I installed CUDA for my NVIDIA graphics card about a month ago, on a clean install of Ubuntu 16.04. I used the most recent runfile for the installation, and as far as I can tell followed the instructions to the letter, with the exception that the modifications to PATH and LD_LIBRARY_PATH given in the instructions are incorrect. The installation was successful, and even managed to run without interfering with X - my machine has an Intel graphics card as well, which was used for display.
This morning, CUDA programs stopped working, and nvidia-smi reported that it couldn't communicate with the graphics card because it was inactive. I fixed that by putting the display on the NVIDIA graphics card, with the obvious result - the screen can't update while a CUDA kernel is running. Now, when I switch the display back to the Intel graphics card, I get a new error: nvidia-smi reports:
NVIDIA-SMI couldn't find libnvidia-ml.so library in your system. Please
make sure that the NVIDIA Display Driver is properly installed and present in
your system.
Please also try adding directory that contains libnvidia-ml.so to your system PATH.
The directory containing libnvidia-ml.so is on the system PATH. This error, I should emphasize, disappears when the NVIDIA card is set as my main display card.
Whenever I run a CUDA program of any sort with the Intel set as display, I get error 35: "CUDA driver version is insufficient for CUDA runtime version." This seems impossible - "cat /proc/driver/nvidia/version" yields driver version 375.39, and as I understand it this is compatible with CUDA 8.
Using
LD_PRELOAD=/usr/lib/nvidia-375/libnvidia-ml.so nvidia-smi
fixes the first error, but not the second.
I'm reluctant to reinstall CUDA without knowing what went wrong here - I'd rather not have to deal with this once a month. Any suggestions?
Update: When I begin with X running on the NVIDIA card and perform the following sequence of commands (after loading the CUDA .deb package):
sudo apt-get purge nvidia-*
sudo apt-get install nvidia-375
sudo apt-get install cuda
it works. But I think this is because I somehow fooled the NVIDIA card into thinking it's running X, without actually running X; when I reboot, the problem resurfaces (if I run X on the NVIDIA card, everything's fine but I can't run CUDA programs without freezing the screen; if I run X on the Intel card, no CUDA functionality works).