How to use CUDA with NVIDIA Prime

Question

I've found a half a dozen posts on this all over the web, but none of them really answer the question.

I want to set up my nvidia GPU to only do computations, not drive the display. But when I switch to using the Intel GPU in the nvidia-prime configuration, I can no longer load the nvidia module.

modprobe: ERROR: could not insert 'nvidia_352': No such device

Without the module, CUDA doesn't work, obviously.

So what exactly is nvidia-prime doing that makes it impossible to load the module? It's not blacklisted. There's no xorg.conf file, so how does the system know to use the Intel GPU instead the discrete one?

I'm on a Dell 5510 Precision with Ubuntu 14.04 factory installed, and my GPU is Quadro M1000M.

Some suggest using bumblebee, but that shouldn't be necessary for pure compute loads.

Also, apparently bumblebee is able to load the module. So what exactly is it doing?

Update: So why does it always seem that I find the answer when I finally post a question, after hours of trying to figure it out. This actual only a partial answer, but I'm on to something.

So far I've determined that prime does at least two things:

• Switch the GPU off using bbswitch.
• Changes the alternatives for /etc/ld.so.conf.d/x86_64-linux-gnu_GL.conf.

By using bbswitch to turn the GPU back on, I'm now able to load the NVIDIA module.

But the question still remains: What's the best way to configure the system to use the NVIDIA card only for computations?

Should I set nvidia-prime to use the Intel GPU, and try to manually unravel what that did to get CUDA working?

How do I ensure that the system still uses the Intel GPU for the display?

How would I go about simply disabling NVIDIA prime, and configuring it all manually?

Or should I jsut give in and use Bumblebee and optirun? What are the disadvantages of this if any?

Any recommendations?

Answer 1

In my case I found that the NVidia card was not actually turned off, and the only thing I actually needed to do to run CUDA code was:

export LD_LIBRARY_PATH=/usr/lib/nvidia-352

in the shell where I want to run it (I am assuming that globally changing the alternatives setting would break compiz, etc, etc...)

To get to this point (on a Dell Optiplex 7010, with Ubuntu 14.04, CUDA 7.5, and a GTX 980) I believe the steps were:

1. Use the PRIME Profiles tab to select Intel
2. Reboot, and select Intel as the default in the BIOS
3. Shut down the computer
4. Plug the monitors into the onboard video :)

Everything seems to be working fine so far (nvidia-smi sees the card, cuda samples run, theano uses the card, etc...)

Answer 2

I believe I've found at least a cursory solution to this, as described in the update to my original post. Really there's two solutions I found, though I'm sure there are others.

1 - With Prime in Intel mode, re-enable the NVIDIA card via bbswitch, then run modprobe nvidia to load the module and create the device nodes.

2 - Use Bumblebee optirun to launch a bash session from where you can do all your CUDA stuff.

Both these solutions allow you to use the onboard graphics for your display, while using the NVIDIA card for compute loads. The optirun solution seems more versatile, but I prefer the first one for its minimalism.

I'm hoping someone with more understanding will improve on this answer.

Answer 3

I use NVIDIA card only for CUDA executions and find out this approach:

All the time I use intel card and it is confirmed by command lspci | grep -E "VGA|3D":

00:02.0 VGA compatible controller: Intel Corporation Skylake Integrated Graphics (rev 06)
01:00.0 3D controller: NVIDIA Corporation GM107M [GeForce GTX 960M] (rev ff)

In the corresponding row for NVIDIA card you should see (rev ff) means it turned off.

To turn the card on and use it for CUDA computations I use two following commands:

sudo prime-select nvidia
sudo prime-switch

After that command lspci | grep -E "VGA|3D" report:

00:02.0 VGA compatible controller: Intel Corporation Skylake Integrated Graphics (rev 06)
01:00.0 3D controller: NVIDIA Corporation GM107M [GeForce GTX 960M] (rev a2)

Notice about (rev a2), not (rev ff) in the corresponding row. Now card ready to computation.

After computations I use backward actions:

sudo prime-select intel
sudo prime-switch

And lspci | grep -E "VGA|3D" reports:

00:02.0 VGA compatible controller: Intel Corporation Skylake Integrated Graphics (rev 06)
01:00.0 3D controller: NVIDIA Corporation GM107M [GeForce GTX 960M] (rev ff)

Answer 4

In case someone still finds problems after following the steps in the accepted answer, try this:

echo "install bbswitch /bin/true" > /etc/modprobe.d/blacklist-bbswitch.conf
update-initramfs -u

It will disable bbswitch altogether. The drawback is that you won't be able to turn off NVIDIA card for power saving (Xorg still use integrated graphics, as long as prime-select intel).

Answer 5

I'm using a 1070 ti with a thinkpad T420 in an egpu set up to mine crypto-currency while I work. The GPU will theoretically pay for itself after a couple of months this way.

I found that with nvidia 387 export LD_LIBRARY_PATH=/usr/lib/nvidia-387 worked with ethminer using cuda.

However, the only way I could get the system to keep working after a "prime-select intel" was if I had a second monitor plugged into the card when I did the selection and logged off. Otherwise, I would get a "The system is running in low-graphics mode" error, or a continual blank screen. Of course, when I do log-in running intel graphics, the monitor plugged into the GPU displays nothing, so I have to unplug it from the gpu, and plug it back into the system (DVI dock output) to run my duel monitor setup.

I believe this is because gpu-manager detects that bbswitch doesn't work and then deletes the xorg configuration.

I'm posting this to show a workaround for the few who might be in a similar situation, but also to see if anyone has an idea to stop this from happening, as having to move that cable around for every reboot is a bit of an inconvenience.

log_file: /var/log/gpu-manager.log

last_boot_file: /var/lib/ubuntu-drivers-common/last_gfx_boot new_boot_file: /var/lib/ubuntu-drivers-common/last_gfx_boot can't access /run/u-d-c-fglrx-was-loaded file Looking for fglrx modules in /lib/modules/4.4.0-104-generic/updates/dkms Looking for nvidia modules in /lib/modules/4.4.0-104-generic/updates/dkms Found nvidia module: nvidia_387_drm.ko Is nvidia loaded? yes Was nvidia unloaded? no Is nvidia blacklisted? yes Is fglrx loaded? no Was fglrx unloaded? no Is fglrx blacklisted? no Is intel loaded? yes Is radeon loaded? no Is radeon blacklisted? no Is amdgpu loaded? no Is amdgpu blacklisted? no Is nouveau loaded? no Is nouveau blacklisted? yes Is fglrx kernel module available? no Is nvidia kernel module available? yes Vendor/Device Id: 8086:126 BusID "PCI:0@0:2:0" Is boot vga? yes Vendor/Device Id: 10de:1b82 BusID "PCI:5@0:0:0" Is boot vga? no Skipping "/dev/dri/card1", driven by "i915" Skipping "/dev/dri/card0", driven by "nvidia-drm" Skipping "/dev/dri/card1", driven by "i915" Skipping "/dev/dri/card0", driven by "nvidia-drm" Skipping "/dev/dri/card1", driven by "i915" Skipping "/dev/dri/card0", driven by "nvidia-drm" Found "/dev/dri/card1", driven by "i915" output 0: card1-LVDS-1 output 1: card1-HDMI-A-2 Number of connected outputs for /dev/dri/card1: 2 Does it require offloading? yes last cards number = 2 Has amd? no Has intel? yes Has nvidia? yes How many cards? 2 Has the system changed? No main_arch_path x86_64-linux-gnu, other_arch_path i386-linux-gnu Current alternative: /usr/lib/nvidia-387-prime/ld.so.conf Current core alternative: (null) Current egl alternative: /usr/lib/nvidia-387-prime/ld.so.conf Is nvidia enabled? no Is nvidia egl enabled? no Is fglrx enabled? no Is mesa enabled? no Is mesa egl enabled? no Is pxpress enabled? no Is prime enabled? yes Is prime egl enabled? yes Is nvidia available? yes Is nvidia egl available? no Is fglrx available? no Is fglrx-core available? no Is mesa available? yes Is mesa egl available? yes Is pxpress available? no Is prime available? yes Is prime egl available? no Intel IGP detected Intel hybrid system Nvidia driver version 387.34 detected /sys/class/dmi/id/product_version="ThinkPad T420" /sys/class/dmi/id/product_name="4236L23" 1st try: bbswitch without quirks Loading bbswitch with "load_state=-1 unload_state=1" parameters Error: can't open /proc/acpi/bbswitch Removing xorg.conf. Path: /etc/X11/xorg.conf can't access /usr/share/gpu-manager.d/hybrid-power-saving No need to change the current bbswitch status

If I nano xorg.conf after this it's empty. I'm posting this after doing the monitor switch trick, with the mining going on in the background, and my xorg.conf is still empty. So my guess is that for some reason when I keep the monitor plugged in to the GPU on lightdm restart it doesn't matter that my xorg.conf is deleted. Any ideas?