0

I have a 40-core server running Ubuntu 14.04 LTS. I am using an application with multithreading capability. I find that running the application with increasing numbers of threads improves run time until I go over a certain number at which point the run times start increasing again. This thread number is far below the number of cores that I have. Here are some examples (these are "real" times):

8 threads: 1m45.992s
16 threads: 1m7.494s
24 threads: 1m45.174s
32 threads: 3m10.819s
40 threads: 6m12.194s
80 threads: 25m22.937s

I am not running out of memory (only 4 Gb of 128 Gb used) and no swap is being used. No other processes with significant CPU usage are running during these tests.

Interestingly, when I run a version of same application compiled from the same source in OS X with the same data on my PowerMac with 8 cores, I get a steady run time improvement up to 16 threads with only minor (few seconds) slowdown at 32 and 64 threads, so I don't believe it's a problem with the application software. Indeed, when I use another multithread-capable application with similar function as the first on the Ubuntu server, I see similar, though not as dramatic results:

16 threads: 4m4.795s
40 threads: 2m31.430s
60 threads: 3m7.007s
80 threads: 5m6.946s 

I usually have to run these analyses sequentially on hundreds of data sets, so any gains in efficiency can make a big difference. My question is whether this may be related to a system software configuration problem vs. an issue with my hardware. Any thoughts on where to start looking to solve this issue and get the full benefit from all my processors would be greatly appreciated.

Thank you.

1

You are probably hitting contention points in the code, such as locking (via futexes or the like) where serialization of the code occurs and hence stops the performance scaling.

Also, an x86 CPU may have N cores, each with say 2 threads each, but that does not give you 2 x N performance, since a hyperthread executes when certain execution blocks are available. I believe for a single socketed x86 processor one may get up to 30% extra performance with a hyperthread.

Also, you may be getting contention on memory, be it on cache (L1, L2 or L3) or even on the memory itself. So you may be hitting limitations on the throughput, cache stalls or on the TLB.

With N process > N CPUs, you will end up with more processes than can be runnable, so the scheduler has to perform more work in pre-empting runnable processes and this is another penalty that eats into performance.

You can get low-level performance metrics using tools such as perf. Install it with:

sudo apt-get install linux-tools 

And run you application with perf to get some performance measurements:

perf stat your-program

You can do deeper analysis using perf record and perf report, e.g.

sudo perf record your-program
sudo perf report

Alternatively, run your program and while it is running use perf top to get an interactive real time view of the system activity:

sudo perf top

Hopefully that will give you some idea of where the bottle neck is occurring.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.