Docker only works with Nvidia drivers upon reinstall

Question

Ubuntu version 20.04 LTS

NVIDIA driver and related package like cuda are all installed properly. Running nvidia-smi and cuda code fine.

Docker related NVIDIA packages are installed too (NVIDIA Container Toolkit). Initial problem is if I try to validate NVIDIA support in docker, I get this error message:

$ sudo docker run --gpus all nvidia/cuda:10.0-base nvidia-smi
docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].

After finding some online discussion, I tried to reinstall docker following instruction here: https://docs.docker.com/engine/install/ubuntu/ It worked for me. NVIDIA works under docker now.

However, after reboot, it will stop working. I will have to do something like:

$ sudo apt-get reinstall docker-ce docker-ce-cli containerd.io

For NVIDIA to work under docker again. Can confirm every reboot will cause this.

How do I make it work so I don't have to reinstall every time after reboot?

Answer 1

In my case, I had docker installed twice, via snap and apt package manager:

After reboot I had:

$ docker images
REPOSITORY              TAG                  IMAGE ID            CREATED             SIZE
ubuntu                  latest               4e2eef94cd6b        3 weeks ago         73.9MB
tensorflow/tensorflow   latest-gpu-jupyter   f0b0261fec71        6 weeks ago         3.3GB
nvidia/cuda             10.0-base            841d44dd4b3c        9 months ago        110MB

If I restart docker service:

$ sudo service docker restart

I have other set of images:

$ docker images
REPOSITORY              TAG                  IMAGE ID            CREATED             SIZE
jupyter/r-notebook      latest               14611e3d9838        2 weeks ago         2.59GB
ubuntu                  latest               4e2eef94cd6b        3 weeks ago         73.9MB
tensorflow/tensorflow   latest-gpu-jupyter   f0b0261fec71        6 weeks ago         3.3GB

$ dpkg -l | grep docker
ii  docker-ce                                  5:19.03.12~3-0~ubuntu-focal           amd64        Docker: the open-source application container engine
ii  docker-ce-cli                              5:19.03.12~3-0~ubuntu-focal           amd64        Docker CLI: the open-source application container engine

$ snap list | grep docker
docker     19.03.11     471    latest/stable  canonical*          -    

I restared OS:

$ sudo init 6

and I removed all images created through snap docker:

$ docker rmi $(docker images -q)

After it I removed snap docker:

$ sudo snap remove docker
$ sudo init 6

Now I have a working docker service:

$ docker run --gpus all -p 8888:8888 -v /tf:/tf -w /tf --name tfgpu --rm tensorflow/tensorflow:latest-gpu-jupyter
[I 07:52:52.707 NotebookApp] Writing notebook server cookie secret to /root/.local/share/jupyter/runtime/notebook_cookie_secret
[I 07:52:52.967 NotebookApp] Serving notebooks from local directory: /tf
[I 07:52:52.967 NotebookApp] The Jupyter Notebook is running at:
[I 07:52:52.967 NotebookApp] http://a1d1932a7004:8888/?token=74b0b061e2a1818b865c1f344be904758f9f0dba73b742d3
[I 07:52:52.967 NotebookApp]  or http://127.0.0.1:8888/?token=74b0b061e2a1818b865c1f344be904758f9f0dba73b742d3
[I 07:52:52.967 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 07:52:52.972 NotebookApp] 

    To access the notebook, open this file in a browser:
        file:///root/.local/share/jupyter/runtime/nbserver-1-open.html
    Or copy and paste one of these URLs:
        http://a1d1932a7004:8888/?token=74b0b061e2a1818b865c1f344be904758f9f0dba73b742d3
     or http://127.0.0.1:8888/?token=74b0b061e2a1818b865c1f344be904758f9f0dba73b742d3