As far as I know, you can't in a strict way (unless using some kind of virtual machine or using cgroups which is not so easy; you can see this answer from @muru.).
You can limit the memory available with ulimit
, but this will simply have the effect of telling your program that there is no more memory when doing an allocation, or crashing it with a signal if it doesn't handle the out-of-memory condition. Look:
zcat /var/log/syslog.2.gz
it works, lot of output
ulimit -d 100
ulimit -m 100
(This is limiting the memory for data and for core to 100 kB)
[romano:~] 2 % zcat /var/log/syslog.2.gz
/bin/zcat: xmalloc: .././subst.c:3542: cannot allocate 267 bytes (53248 bytes allocated)
But the shell is still able to see all the memory:
[romano:~] 2 % free
total used free shared buffers cached
Mem: 15340736 5148596 10192140 368776 284192 2794848
-/+ buffers/cache: 2069556 13271180
Swap: 31999996 0 31999996
Edit: more info in this unix.se post.