"Module nvidia is in use" but there are no processes running on the GPU

Question

I am trying to configure VirtualGL, and the configuration gives the following message:

IMPORTANT NOTE: Your system uses modprobe.d to set device permissions. You
must execute rmmod nvidia with the display manager stopped in order for the
new device permission settings to become effective.

When I try running rmmod nvidia (or with sudo), it says that module nvidia is in use:

rmmod: ERROR: Module nvidia is in use by: nvidia_uvm nvidia_modeset

I have already stopped my window manager by running sudo systemctl stop sddm.service, so when I check nvidia-smi it says that there are no processes running on the GPU.

Most of the threads I found on this issue are related to bumblebee, but I don't even have it insalled.

Output of nvidia-smi:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 430.40       Driver Version: 430.40       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 1080    Off  | 00000000:01:00.0 Off |                  N/A |
| 33%   39C    P8    12W / 200W |      9MiB /  8119MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

Ubuntu 18.04

Answer 1

On Ubuntu 20.04, I've been able to fully unload and reload the NVIDIA modules with the following commands (performed via SSH or a text-only tty).

sudo service gdm3 stop

sudo rmmod nvidia_uvm
sudo rmmod nvidia_drm
sudo rmmod nvidia_modeset
sudo rmmod nvidia

# Then, reload them if desired:
sudo modprobe nvidia
sudo modprobe nvidia_modeset
sudo modprobe nvidia_drm
sudo modprobe nvidia_uvm

This came in handy e.g. when upgrading the NVIDIA driver without rebooting (which could happen unexpectedly via unattended apt upgrades).

---

As @Thomas pointed out in a comment, if there's still something preventing one of the modules from being unloaded, it's usually possible to figure out what with:

lsmod | grep nvidia
lsof | grep nvidia