0

I have been succesfully running some TensorFlow code all morning which involves loading relatively large images (1mb) from a folder for use in a training model.

I am not sure if this is relevant but when earlier, I had a power outage when I engaged all 4 of my GPUs. The system rebooted fine. But since then, my whole system is slow doing anything - in particular opening folders. The folder that contains the images (150, 1mb images) is ridiculously slow. The images themselves when the folder is open show the images slowly loading up (very slowly - about 1 min per image)

I have looked at memory usage (top, from terminal; these are the top few)

PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND     
 297 root      20   0       0      0      0 R  35.9  0.0  15:10.55   md126_raid5 
 303 root      20   0       0      0      0 D  29.9  0.0   9:15.02 md126_resy+ 
1387 root      20   0  800764 331476  97912 S   9.3  0.3   4:22.46 Xorg        
4300 simon     20   0  686220  49528  35004 S   5.6  0.0   1:56.81 gnome-syst+ 
1493 root     -51   0       0      0      0 S   4.0  0.0   0:23.52 irq/76-nvi+ 
2583 simon     20   0 1898420 436948  92080 S   3.0  0.3   2:00.12 compiz      
1498 root     -51   0       0      0      0 S   2.0  0.0   0:22.59 irq/78-nvi+ 
1495 root     -51   0       0      0      0 S   1.0  0.0   0:27.90 irq/77-nvi+ 
3065 simon     20   0  663544  41244  28168 S   0.7  0.0   0:06.64 gnome-term+ 

My system has 128 GB RAM, 8T of HD, a 3.2Ghz processor and 4 1080ti GPUs.

I am not sure how to debug this. I downloaded BleachBit and used that to clean out all unused and old files.

How can I get to the bottom of this? - I am fairly new to Ubuntu.

EDIT:

  glxinfo | grep renderer

gives

  OpenGL renderer string: GeForce GTX 1080 Ti/PCIe/SSE2

and

 cat /proc/mdsat

gives

 Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10] 
  md126 : active raid5 sdb[2] sdc[1] sdd[0]
  7814032384 blocks super external:/md127/0 level 5, 128k chunk, algorithm 0 [3/3] [UUU]
  [=================>...]  resync = 89.7% (3507217748/3907016320) finish=59.0min speed=112852K/sec

  md127 : inactive sdb[2](S) sdd[1](S) sdc[0](S)
  6780 blocks super external:imsm

  unused devices: <none>
  • Do you see any errors appearing in journalctf -f for working with the system? What does glxinfo | grep renderer say? – Hi-Angel Nov 7 '17 at 15:12
1

Looks like your RAID is resyncing. Given the size of your hard drive, it may take quite a long time.

  • Thanks Ken - could you give me some insight into why this might the case. How would I know when this process is done? – GhostRider Nov 7 '17 at 19:23
  • Run cat /proc/mdstat to view progress. Update your original post with the output. – vidarlo Nov 7 '17 at 19:27
  • From your post update, it appears that the resync will be finished soon. More than likely this was triggered by the power failure but I'd keep an eye on it if I were you in case a hard drive was damaged somehow. Best of luck to you. – Ken Schumack Nov 8 '17 at 2:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.