You could add swap space, read man mkswap;man swapon to make the system think it has more memory.
To limit ram usage, read man bash and search for ulimit:
ulimit [-HSabcdefiklmnpqrstuvxPT [limit]]
Provides control over the resources available to the shell and to processes started
by it, on systems that allow such control. The -H ...