I'm not very experienced with linux and the OS might not be the problem.
I'm experimenting with tensorflow and hit a wall. I have a bash script that launches python programs consequentially training with different hyperparameters. I use cuda 7.5 and cudnn 4 to compute on the graphics card, the data is in the tensorflow format and I use TFReadRecords as the reader.
Initially everything works fine. The problem is that the python script will slow down 10-20x randomly a few hours after I start the bash script, first usually during one of the first 3 python scripts.
Normally it takes ~50-60% CPU load to read data and transfer it to the GPU, but during slowdown I noticed in the system monitor is that it gets limited to one single cpu core (see screenshots fast, slow).
It can speed up (being distributed on all 4 cores) and slow down again several times. Sometimes it does this on its own, sometimes it gets back to normal when I open and close programs and sometimes it wont speed up again at all. It happens more frequently the longer the bash script runs.
Things I tried and did not help: -setting the nice value to -20 of the python script the bash script or the terminal -changing CPU affinity with taskset -running it from console while stopping lightdm
Any suggestions would be great!