0

A recent kernel update broke my Cuda installation (it works fine if I boot an older kernel) - very similar setups have persisted across kernel updates on other machines, the major difference being that this installation is Ubuntu Server and the others are Ubuntu Desktop. Does this sound like a DKMS issue? Or something else? How can I get my cuda modules to build themselves against new kernels?

I have Ubuntu 16.04, Cuda 10.0 (installed by local .deb), nvidia driver 410.48 (installed automatically during cuda install) and a 2080Ti GPU

$ ls -al /boot
total 111740
drwxr-xr-x  3 root root     4096 Apr  9 12:02 .
drwxr-xr-x 24 root root     4096 Apr  4 16:53 ..
-rw-r--r--  1 root root  1252376 Jan 16 23:29 abi-4.4.0-142-generic
-rw-r--r--  1 root root   190580 Jan 16 23:29 config-4.4.0-142-generic
-rw-r--r--  1 root root   190580 Mar 26 14:02 config-4.4.0-145-generic
drwxr-xr-x  5 root root     4096 Apr  9 12:02 grub
-rw-r--r--  1 root root 50832836 Apr  4 16:54 initrd.img-4.4.0-142-generic
-rw-r--r--  1 root root 39170185 Apr  9 11:15 initrd.img-4.4.0-145-generic
-rw-r--r--  1 root root   182704 Jan 28  2016 memtest86+.bin
-rw-r--r--  1 root root   184380 Jan 28  2016 memtest86+.elf
-rw-r--r--  1 root root   184840 Jan 28  2016 memtest86+_multiboot.bin
-rw-r--r--  1 root root      255 Jan 16 23:29 retpoline-4.4.0-142-generic
-rw-------  1 root root  3904797 Jan 16 23:29 System.map-4.4.0-142-generic
-rw-------  1 root root  3906115 Mar 26 14:02 System.map-4.4.0-145-generic
-rw-------  1 root root  7184032 Jan 16 23:29 vmlinuz-4.4.0-142-generic
-rw-------  1 root root  7188984 Mar 27 10:03 vmlinuz-4.4.0-145-generic

$ dkms status
bbswitch, 0.8, 4.4.0-142-generic, x86_64: installed
bbswitch, 0.8, 4.4.0-145-generic, x86_64: installed
nvidia-410, 410.48, 4.4.0-142-generic, x86_64: installed

$ ls -al /usr/src
total 44
drwxr-xr-x 11 root root 4096 Apr  9 12:02 .
drwxr-xr-x 12 root root 4096 Mar 14 12:56 ..
drwxr-xr-x  2 root root 4096 Mar 14 11:05 bbswitch-0.8
drwxr-xr-x  5 root root 4096 Mar 14 14:55 cudnn_samples_v7
drwxr-xr-x  3 root root 4096 Mar 14 12:56 gmock
drwxr-xr-x  4 root root 4096 Mar 14 12:56 gtest
drwxr-xr-x 27 root root 4096 Feb 27 18:41 linux-headers-4.4.0-142
drwxr-xr-x  7 root root 4096 Feb 27 18:43 linux-headers-4.4.0-142-generic
drwxr-xr-x 27 root root 4096 Apr  4 16:53 linux-headers-4.4.0-145
drwxr-xr-x  7 root root 4096 Apr  4 16:53 linux-headers-4.4.0-145-generic
drwxr-xr-x  8 root root 4096 Mar 14 14:49 nvidia-410-410.48

$ ls -alR /var/lib/dkms
[Very long output] https://pastebin.com/RRMsBT0s
  • Edit your question and show me dkms status and ls -al /boot. Report back to @heynnema – heynnema Apr 10 at 19:28
  • @heynnema have added the diagnostic info you asked for :) – ezekiel Apr 11 at 10:04
  • The nvidia module didn't build for your current kernel. Now show me ls -al /usr/src and ls -alR /var/lib/dkms. What make/model video card do you have? – heynnema Apr 11 at 12:33
  • You accidentally did the first command twice. You didn't tell me what make/model your video card is. – heynnema Apr 11 at 16:51
  • Please see my answer. Please remember to accept it if it solves your problem. Thanks! – heynnema Apr 11 at 17:24
1

$ dkms status

bbswitch, 0.8, 4.4.0-142-generic, x86_64: installed
bbswitch, 0.8, 4.4.0-145-generic, x86_64: installed
nvidia-410, 410.48, 4.4.0-142-generic, x86_64: installed

This shows us that the nvidia-410 dkms driver did not build on kernel 4.4.0-145-generic. There should be a 4th line that looks like:

nvidia-410, 410.48, 4.4.0-145-generic, x86_64: installed

Who knows why it didn't build on the -145 kernel... there IS a dkms.conf file there.

Cuda 10.0 can be downloaded from here. Installation document is here.

Download the Cuda package and reinstall it. Then do a dkms status command and verify that it shows the 4th line, as I show above.

Note: if you'd like a newer version of the Nvidia driver, the latest version is 418.56. I can't say if it's compatible with Cuda 10.0.

Note: if there's a bug in the Cuda/Nvidia software package(s), you may have to do this every time the kernel is updated :-(

Update #1:

Recent updates have caused the Nvidia video driver 410 not to build on the current kernel.

Cuda 10.0 installs Nvidia video driver 410.

Cuda 10.1 gets installed with Nvidia video driver 418.

Cuda 10.1 has problems with tensorflow.

User needs Cuda 10.0 with a working Nvidia video driver.

User is requesting further help from Nvidia.

Update #2:

Try this...

Remove Cuda 10.1 and video driver 418.

Reinstall Cuda 10.0 and video driver 410.

This will put you back to the beginning status.

Let's try and build the 410 driver manually...

sudo dkms build nvidia-410/410.48 # the build may fail, but give us a reason

sudo dkms install nvidia-410/410.48 # run if the build is clean

dkms status # verify 410 installation on current kernel

Update #3:

It turned out that installing either Cuda 10.0/10.1 did not automatically install Nvidia video drivers, and visa-versa.

Final solution was for the user to manually install the required Cuda 10.0, and manually install the latest Nvidia video driver 418.56, and it's all working again.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.