1

I've been doing CUDA development for a while on Ubuntu 18.04, kernel 4.15.0-46-generic, Nvidia driver 410.48 and a GeForce GTX 1060 card.

Sometimes I cannot do CUDA development any longer on a Clojure REPL using wrappers for JCuda, I'll get only "CUDA_ERROR_UNKNOWN" errors until I reboot the computer. I suspect this wouldn't happen if I'd always remember to release my CUDA resources on the REPL before suspending the computer, but I seem to keep repeating this mistake :(

Oddly the nvidia-smi command works but when I try to run nvidia-docker run --rm nvidia/cuda:9.0-base nvidia-smi it prints out this error message and exits:

docker: Error response from daemon: OCI runtime create failed: container_linux.go:344:
starting container process caused "process_linux.go:424: container init caused \"process_linux.go:407:
running prestart hook 1 caused \\\"error running hook: exit status 1,
stdout: , stderr: exec command: [/usr/bin/nvidia-container-cli --load-kmods configure
--ldconfig=@/sbin/ldconfig.real --device=all --compute --utility --require=cuda>=9.0
--pid=3265 /var/lib/docker/aufs/mnt/a6ddd30bf9b16f4affe5024840625747cf56a7ebee10e5940a90a16770c20190]
\\\\nnvidia-container-cli: initialization error: cuda error: unknown error\\\\n\\\"\"": unknown.

nvidia-smi output:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.48                 Driver Version: 410.48                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 106...  Off  | 00000000:01:00.0  On |                  N/A |
|  0%   36C    P0    27W / 150W |    894MiB /  6075MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      6042      G   /usr/lib/xorg/Xorg                           566MiB |
...

Do you have any tips on how to "reset" the driver state (what ever is causing this) without rebooting? Logging in and out doesn't seem to fix it, and even that is almost as inconvenient than a full restart. I wouldn't like to re-open all my applications.

2 Answers 2

2

On the nvidia forums, someone is suggesting that reloading the driver might do away with the restart.

Here are the steps they mention (I haven't tested this):

  1. Stop all programs using the driver (particularly X11)
  2. lsmod | grep nvidia and rmmod the modules with zero use count. You should do this in dependency order (the column on the right shows the modules that this driver depends on)
  3. Repeat until no Nvidia kernel modules are loaded.
  4. modprobe nvidia to reload the driver.
  5. Restart X11 or whatever was using the GPU

Note that due to step 1, you may lose all open applications (which depend on X11). However, if you setup your X11 to not use NVidia graphics, but the Intel integrated one, then you should be able to reload the driver with minimal impact.

Let us know if this works!

2
  • Thanks for the tip! It seems booting the machine isn't much more itme consuming than following these steps since I am running on a desktop environment.
    – NikoNyrh
    Sep 8, 2019 at 21:12
  • 1
    indeed, rebooting is the faster way. Sometimes you have other processes turning that you may not want to stop. This procedure is for those cases Sep 8, 2019 at 22:46
0

A simplified version of Ciprian's answer consistently works with nvidia-driver-5xx, at least:

  1. Stop all programs using the driver (particularly X11)
  2. sudo rmmod nvidia_uvm
  3. sudo modprobe nvidia_uvm
  4. Restart X11

This has allowed me to again run CUDA applications while circumventing the reboot.

For step 1, dropping into shell with init 3 or ctrl+F[1-6] plus restarting the display manager[0] has worked consistently for me.

[0] with either sudo systemctl restart display-manager or sudo service gdm restart

Related Answers & Questions

My environment

22.04, X11, GNOME V 42.9, nvidia-driver-535, CUDA Toolkit 12.2

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .