I've been doing CUDA development for a while on Ubuntu 18.04, kernel 4.15.0-46-generic, Nvidia driver 410.48 and a GeForce GTX 1060 card.
Sometimes I cannot do CUDA development any longer on a Clojure REPL using wrappers for JCuda, I'll get only "CUDA_ERROR_UNKNOWN" errors until I reboot the computer. I suspect this wouldn't happen if I'd always remember to release my CUDA resources on the REPL before suspending the computer, but I seem to keep repeating this mistake :(
Oddly the nvidia-smi
command works but when I try to run nvidia-docker run --rm nvidia/cuda:9.0-base nvidia-smi
it prints out this error message and exits:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:344:
starting container process caused "process_linux.go:424: container init caused \"process_linux.go:407:
running prestart hook 1 caused \\\"error running hook: exit status 1,
stdout: , stderr: exec command: [/usr/bin/nvidia-container-cli --load-kmods configure
--ldconfig=@/sbin/ldconfig.real --device=all --compute --utility --require=cuda>=9.0
--pid=3265 /var/lib/docker/aufs/mnt/a6ddd30bf9b16f4affe5024840625747cf56a7ebee10e5940a90a16770c20190]
\\\\nnvidia-container-cli: initialization error: cuda error: unknown error\\\\n\\\"\"": unknown.
nvidia-smi output:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.48 Driver Version: 410.48 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 106... Off | 00000000:01:00.0 On | N/A |
| 0% 36C P0 27W / 150W | 894MiB / 6075MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 6042 G /usr/lib/xorg/Xorg 566MiB |
...
Do you have any tips on how to "reset" the driver state (what ever is causing this) without rebooting? Logging in and out doesn't seem to fix it, and even that is almost as inconvenient than a full restart. I wouldn't like to re-open all my applications.