7

I have been running a Python script for extended amounts of time that relies on the pyCUDA module.

On two occasions, the desktop has become locked up and totally unresponsive, except, oddly, through SSH. Logging in remotely shows that the Python script is no longer running when the system is (locally) locked up. Because this Python script runs for a long time, I haven't been around to see when exactly it locks up. It has never locked up without the script having run.

Running 12.04 with a GeForce 460 GPU. The Python script doesn't make one CUDA call that lasts for over an hour, but hundreds that last for less than 30 seconds, so I don't think the GPU is locking up the system. Thanks for any ideas.

Improve this question
4
  • 1
    you can check the dmesg system log for any errors
    – Savvas Radevic
    Nov 4, 2012 at 7:43
  • 1
    GUI locked, but kernel and network working correctly... it really does look like a GPU hang.
    – Javier Rivera
    Nov 5, 2012 at 8:26
  • I guess X hangs. Look at the logs of X and syslog and post that information if it happens again.
    – gertvdijk
    Nov 15, 2012 at 17:48
  • You mentioned that when you log in via SSH, the script is no longer running. Is it because it terminated or crashed?
    – Marcin Kaminski
    Nov 23, 2012 at 23:54

2 Answers 2

Reset to default
2

You mentioned that when you log in via SSH, the script is no longer running. Is it because it terminated or crashed?

You can also run your script with strace. This way you can capture every system call it makes, especially what it does when it terminates/crashes.

strace -f /path/script.py -o /tmp/output.log

It will produce a rather large file, so make sure you have enough free space.

Improve this answer
2

A few ideas and debugging hints:

  • When you log in via SSH, is the system idle or is a process hanging with heavy load?
  • What does "totally unresponsive" mean? Can you still get to a virtual terminal hitting Ctrl-Alt-F1? Does hitting CapsLock turn on/off the status LED on the keyboard?
  • Even though your script may not use the GPU for long periods at a time, how much maximum GPU memory does it consume?
  • Does stopping the X server and running the script from a virtual terminal reliably prevent the lockups?
Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .