0

I'm not very experienced with linux and the OS might not be the problem.

I'm experimenting with tensorflow and hit a wall. I have a bash script that launches python programs consequentially training with different hyperparameters. I use cuda 7.5 and cudnn 4 to compute on the graphics card, the data is in the tensorflow format and I use TFReadRecords as the reader.

Initially everything works fine. The problem is that the python script will slow down 10-20x randomly a few hours after I start the bash script, first usually during one of the first 3 python scripts.

Normally it takes ~50-60% CPU load to read data and transfer it to the GPU, but during slowdown I noticed in the system monitor is that it gets limited to one single cpu core (see screenshots fast, slow).

It can speed up (being distributed on all 4 cores) and slow down again several times. Sometimes it does this on its own, sometimes it gets back to normal when I open and close programs and sometimes it wont speed up again at all. It happens more frequently the longer the bash script runs.

Things I tried and did not help: -setting the nice value to -20 of the python script the bash script or the terminal -changing CPU affinity with taskset -running it from console while stopping lightdm

Any suggestions would be great!

0

How much memory on your GPU (better yet, its model no.) and do you have an estimate of the problem memory size?

I'm wondering if it's doing some GPU equivalent of thrashing. Still that it could recover from slow back to fast indicates something in the algorithm. Is this a publicly available program (say an example program) or one of your own? Is this the first "large" program that you've tried or run or have you run others, that presumably ran fine.

  • Hi patfla, thanks for the quick answer! its a gtx 1080. it might be the problem, i use 7.5 Gb, I'll shrink down the batch size and run it overnight. The program is my own, but its basically the cifair-10 example modified to work with my data. If less memory usage doesn't help i'll run the example code to see if it persists. Thanks again! – Csaba Konrád Aug 8 '16 at 22:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.