CUDA integration with Theano

Question

I am working with Theano, a deep learning benchmark, on a freshly installed Ubuntu Mate 16.04 machine. Theano can use GPU acceleration for speeding up calculations. I have a NVIDIA K2200M video card which is CUDA-capable and is correctly installed, as the nvidia-smi command shows:

+------------------------------------------------------+                       
| NVIDIA-SMI 361.42     Driver Version: 361.42         |                       
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Quadro K2200M       Off  | 0000:01:00.0     Off |                  N/A |
| N/A   31C    P8    N/A /  N/A |    212MiB /  2047MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0      1090    G   /usr/lib/xorg/Xorg                             200MiB |
|    0      7931    G   /usr/lib/firefox/firefox                         1MiB |
+-----------------------------------------------------------------------------+

A bit of background: Theano needs to be set up so that a few environmental variables have to be defined, such as $CUDA_ROOT, and these refer to /usr/local/cuda but installing nvidia-cuda-* from the official Ubuntu Mate repo does not create those folders. Nevertheless, Theano offers a python code which can help determine if the calculations are being made with the CPU or the GPU and surprisingly Theano finds the CUDA installation.

Here comes the problem: CUDA is recognized by the system but CUDA can't seem to find my GPU, and I get the error WARNING (theano.sandbox.cuda): CUDA is installed, but device gpu is not available (error: cuda unavailable).

I am writing in askubuntu.com and not to the Theano developers because, finding this problem, I uninstalled nvidia-cuda-* and I installed CUDA from the official package provided by NVIDIA following this guide, so that the aforementioned /usr/local/cuda was created and, again, the Theano code recognized the CUDA installation but it still couldn't find my GPU. That is the reason why I think it might be an Ubuntu issue instead of being a faulty implementation on Theano's side.

Answer 1

I kind of figured out it wasn't an Ubuntu issue but a Theano one, as I installed CUDA again from the source I mentioned in the OP and I managed to correctly run sample data from the CUDA package provided by NVIDIA, ruling out (in my unexperienced opinion) an integration problem between NVIDIA-cuda-toolkit and the NVIDIA-drivers.

For those (in the future) having the same issue as me, the problem seems to be in the theano.sandbox.cuda module, when __init__.py tries to compile cuda_ndarray.cu in lines 168-175 (I believe, since I modified the file just slightly), when that file calls the function compiler.compile_str(...), which is a file I believe was created by Theano's development team. NVCC runs correctly but there are problems compiling cuda_ndarray.

So I'll mark this question as answered as soon as askubuntu lets me.

Answer 2

If you are using CUDA 7.5, make sure follow official instruction:

CUDA 7.5 doesn't support the default g++ version. Install an supported version and make it the default.

sudo apt-get install g++-4.9

sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.9 20
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-5 10

sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-4.9 20
sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-5 10

sudo update-alternatives --install /usr/bin/cc cc /usr/bin/gcc 30
sudo update-alternatives --set cc /usr/bin/gcc

sudo update-alternatives --install /usr/bin/c++ c++ /usr/bin/g++ 30
sudo update-alternatives --set c++ /usr/bin/g++

If theano GPU test code has error:

ERROR (theano.sandbox.cuda): Failed to compile cuda_ndarray.cu: libcublas.so.7.5: cannot open shared object file: No such file or directory WARNING (theano.sandbox.cuda): CUDA is installed, but device gpu is not available (error: cuda unavilable)

Just using ldconfig command to link the shared object of CUDA 7.5:

sudo ldconfig /usr/local/cuda-7.5/lib64