Why buf/cache take so much ram and do not chunk

Question

So I just installed my new Ubuntu 16.04 with all necessary stuff to start programing. But now I have big problem with ram. I can not open more application that I have on my previous notebook with linux 12.04 which has half of my current memory. My 1 problem is with process XORG:

 1040 root      20   0 2403428 1,091g 1,014g S  21,9  7,1 247:35.70 Xorg                                                                                                                   

it take so much ram after one day of using my notebook. Second problem is with buff/cache which always take over 2gb. I read this and they declare that this should chunk when my system need it. So my current state is:

KiB Mem : 16063252 total,  2044612 free, 11033624 used,  2985016 buff/cache
KiB Swap:        0 total,        0 free,        0 used.  2778236 avail Mem 

So I have still like 2GB free.

now I start compile one of my bigger project. When free is close to 0 then compile process crash because it has no free ram:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.16:test (surefire-itest) on project my-project: Execution surefire-itest of goal org.apache.maven.plugins:maven-surefire-plugin:2.16:test failed: The forked VM terminated without saying properly goodbye. VM crash or System.exit called ?
[ERROR] Command was/bin/sh -c cd /my-project && /usr/lib/jvm/java-8-oracle/jre/bin/java -Dfile.encoding=UTF-8 -jar my-project/target/surefire/surefirebooter8879869524337464183.jar my-project/target/surefire/surefire5896025025928619492tmp my-project/target/surefire/surefire_21839336642879397359tmp

this is state couple of second before crash:

KiB Mem : 16063252 total,   178808 free, 13693208 used,  2191236 buff/cache
KiB Swap:        0 total,        0 free,        0 used.   191788 avail Mem 

yes free and avail memory is close to 0 but there is still like 2 GB in buff/cache. Why linux/java is not using this memory ? Is there some way how I can optimized my current memory usage ?

Answer 1

Your compiler runs inside a Java VM, which is by default limited to 1GB, no matter how much free memory you have. Try to supply java command with -Xmx2G to make sure it can reclaim twice as much, and feel free to increase that number further if that's not enough. Note that going over 2GB won't help if you happen to use 32-bit version of Java (which is unlikely, considering the system you have).

Answer 2

Memory used by cache/buffers isn't your problem as it's released on an as needed basis See this.

One possibility:

As Dmitry points out It's likely your java VM that's crashing on you. To expand a bit on his answer, there are a couple switches involved that specify the initial and maximum size of the memory allocation pool available to your Java VM. Below is an excerpt from the java man page

-Xmsn               Specifies the initial size of the memory allocation
                           pool.   This  value  must  be  a  multiple  of 1024
                           greater than 1 MB.  Append the letter  k  or  K  to
                           indicate  kilobytes,  the letter m or M to indicate
                           megabytes, the letter g or G to indicate gigabytes,
                           or  the  letter  t or T to indicate terabytes.  The
                           default value is 2MB. Examples:

                           -Xms6291456
                           -Xms6144k
                           -Xms6m

       -Xmxn               Specifies the maximum size, in bytes, of the memory
                           allocation  pool.  This value must be a multiple of
                           1024 greater than 2 MB.  Append the letter k  or  K
                           to  indicate  kilobytes, the letter m or M to indi-
                           cate megabytes, the letter g or G to indicate giga-
                           bytes,  or the letter t or T to indicate terabytes.
                           The default value is 64MB. Examples:

                           -Xmx83886080
                           -Xmx81920k
                           -Xmx80m

There are some differences depending on the platform you are running on as well.

                       On Mac OS X platforms, the  upper  limit  for  this
                       value when running in 32-bit mode (-d32) is approx-
                       imately 2100m minus overhead amounts, and  approxi-
                       mately  127t minus overhead amounts when running in
                       64-bit mode (-d64).  On Solaris  7  and  Solaris  8
                       SPARC  platforms, the upper limit for this value is
                       approximately  4000m  minus  overhead  amounts.  On
                       Solaris  2.6  and x86 platforms, the upper limit is
                       approximately  2000m  minus  overhead  amounts.  On
                       Linux  platforms,  the upper limit is approximately
                       2000m minus overhead amounts.

Another possibility:

I also note that you have no swap enabled. So if it's not the VM you might want to take a look at this for a method of determining if you need to enable swap. I think you should.

If you are running 14.04 and not 16.04 as it says in your post, you may be affected by this bug or perhaps this one

If you are running 16.04 and Xorg is leaking memory causing your issue perhaps this bug has reappeared due to a regression in which case you should report it on launchpad

Source: http://www.manpagez.com/man/1/java/