5

Ubuntu version 20.04 LTS

NVIDIA driver and related package like cuda are all installed properly. Running nvidia-smi and cuda code fine.

Docker related NVIDIA packages are installed too (NVIDIA Container Toolkit). Initial problem is if I try to validate NVIDIA support in docker, I get this error message:

$ sudo docker run --gpus all nvidia/cuda:10.0-base nvidia-smi
docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].

After finding some online discussion, I tried to reinstall docker following instruction here: https://docs.docker.com/engine/install/ubuntu/ It worked for me. NVIDIA works under docker now.

However, after reboot, it will stop working. I will have to do something like:

$ sudo apt-get reinstall docker-ce docker-ce-cli containerd.io

For NVIDIA to work under docker again. Can confirm every reboot will cause this.

How do I make it work so I don't have to reinstall every time after reboot?

1 Answer 1

5

In my case, I had docker installed twice, via snap and apt package manager:

After reboot I had:

$ docker images
REPOSITORY              TAG                  IMAGE ID            CREATED             SIZE
ubuntu                  latest               4e2eef94cd6b        3 weeks ago         73.9MB
tensorflow/tensorflow   latest-gpu-jupyter   f0b0261fec71        6 weeks ago         3.3GB
nvidia/cuda             10.0-base            841d44dd4b3c        9 months ago        110MB

If I restart docker service:

$ sudo service docker restart

I have other set of images:

$ docker images
REPOSITORY              TAG                  IMAGE ID            CREATED             SIZE
jupyter/r-notebook      latest               14611e3d9838        2 weeks ago         2.59GB
ubuntu                  latest               4e2eef94cd6b        3 weeks ago         73.9MB
tensorflow/tensorflow   latest-gpu-jupyter   f0b0261fec71        6 weeks ago         3.3GB

$ dpkg -l | grep docker
ii  docker-ce                                  5:19.03.12~3-0~ubuntu-focal           amd64        Docker: the open-source application container engine
ii  docker-ce-cli                              5:19.03.12~3-0~ubuntu-focal           amd64        Docker CLI: the open-source application container engine

$ snap list | grep docker
docker     19.03.11     471    latest/stable  canonical*          -    

I restared OS:

$ sudo init 6

and I removed all images created through snap docker:

$ docker rmi $(docker images -q)

After it I removed snap docker:

$ sudo snap remove docker
$ sudo init 6

Now I have a working docker service:

$ docker run --gpus all -p 8888:8888 -v /tf:/tf -w /tf --name tfgpu --rm tensorflow/tensorflow:latest-gpu-jupyter
[I 07:52:52.707 NotebookApp] Writing notebook server cookie secret to /root/.local/share/jupyter/runtime/notebook_cookie_secret
[I 07:52:52.967 NotebookApp] Serving notebooks from local directory: /tf
[I 07:52:52.967 NotebookApp] The Jupyter Notebook is running at:
[I 07:52:52.967 NotebookApp] http://a1d1932a7004:8888/?token=74b0b061e2a1818b865c1f344be904758f9f0dba73b742d3
[I 07:52:52.967 NotebookApp]  or http://127.0.0.1:8888/?token=74b0b061e2a1818b865c1f344be904758f9f0dba73b742d3
[I 07:52:52.967 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 07:52:52.972 NotebookApp] 

    To access the notebook, open this file in a browser:
        file:///root/.local/share/jupyter/runtime/nbserver-1-open.html
    Or copy and paste one of these URLs:
        http://a1d1932a7004:8888/?token=74b0b061e2a1818b865c1f344be904758f9f0dba73b742d3
     or http://127.0.0.1:8888/?token=74b0b061e2a1818b865c1f344be904758f9f0dba73b742d3
3
  • Sorry for belated confirmation! This is the exact problem I have. Looks like 'snap' is the culprit. Thank you!
    – ccl13
    Oct 9, 2020 at 7:58
  • Cool! this helped me. Although I didnt have two docker containers installed, the snap version just didn't want to work with gpu. regular docker.io works perfectly. btw why would you choose init 6 instead of reboot?
    – Yurkee
    Oct 23, 2020 at 12:01
  • It's just from Solaris, which I've worked with a lot, init is a more graceful way to stop the system than reboot. Now there isn't a difference. Oct 24, 2020 at 13:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.