I have a 40-core server running Ubuntu 14.04 LTS. I am using an application with multithreading capability. I find that running the application with increasing numbers of threads improves run time until I go over a certain number at which point the run times start increasing again. This thread number is far below the number of cores that I have. Here are some examples (these are "real" times):
8 threads: 1m45.992s 16 threads: 1m7.494s 24 threads: 1m45.174s 32 threads: 3m10.819s 40 threads: 6m12.194s 80 threads: 25m22.937s
I am not running out of memory (only 4 Gb of 128 Gb used) and no swap is being used. No other processes with significant CPU usage are running during these tests.
Interestingly, when I run a version of same application compiled from the same source in OS X with the same data on my PowerMac with 8 cores, I get a steady run time improvement up to 16 threads with only minor (few seconds) slowdown at 32 and 64 threads, so I don't believe it's a problem with the application software. Indeed, when I use another multithread-capable application with similar function as the first on the Ubuntu server, I see similar, though not as dramatic results:
16 threads: 4m4.795s 40 threads: 2m31.430s 60 threads: 3m7.007s 80 threads: 5m6.946s
I usually have to run these analyses sequentially on hundreds of data sets, so any gains in efficiency can make a big difference. My question is whether this may be related to a system software configuration problem vs. an issue with my hardware. Any thoughts on where to start looking to solve this issue and get the full benefit from all my processors would be greatly appreciated.