26

I'm using python on ubuntu for text analysis. Despite the severe amount of work the program is doing the CPU usage as shown in the system monitor stays consistently at 12%.

I changed the priority of the program from Normal to Very High but that had no effect.

What is limiting the amount of CPU usage which my python program can get and how can I change that, so the program can utilize more cpu power?

73

I assume you have a CPU with 8 virtual cores (quad-core with hyper-threading probably)? That means one fully loaded CPU thread/virtual core equals 12.5% total load.

The Python interpreter is an application which only runs as one single process by default and is therefore not able to take advantage of more than one virtual core. Even if the code you run with it uses multithreading, it will still only use one CPU thread/virtual core, because of the GIL (global interpreter lock).

Only if your Python program uses multiprocessing, which in fact starts up multiple instances of the Python interpreter and lets them perform your tasks truly parallel, you can take advantage of multiple virtual cores/CPU threads. (As @SargeBorsch pointed out in his comment, there are also some advanced ways to achieve this without multiprocessing, but that's normally not something you quickly write yourself.)

  • That actually makes a lot of sense. Yes I'm having a quad-core with 4 cores (8 virtual cores). Ty – Matthias Herrmann Aug 24 '17 at 14:02
  • 9
    @MatthiasHerrmann You could consider a system monitor to show you what percentage each CPU is operating at. That way you could have seen only 1 of 8 CPUs was at 100%. Here is one thread in AU on the subject: Windows “gadgets” equivalent (for wifi and cpu usage)? – WinEunuuchs2Unix Aug 25 '17 at 0:24
  • 7
    Not true, it's possible to use all cores from single python process just fine. One just needs to call C code and release the GIL. And many existing libraries do exactly that (numpy for example). – Sarge Borsch Aug 25 '17 at 4:25
  • 2
    Or use Jython or IronPython, which don't have a GIL. – OrangeDog Aug 25 '17 at 13:42
18

Another possibility, less likely in this case, is that the program is disk-bound, i.e. it is reading and writing to/from the disk which is slow, and the CPU is waiting for the disk.

  • 5
    try iotop to monitor programs bound to iowait – cat Aug 25 '17 at 1:26
  • 1
    Or the code itself is synchronous and blocking. – Zydnar Aug 26 '17 at 13:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.