1

My friend gave me this laptop a few years ago because he's upgraded since then. It has a GPU and I want to start playing around with TensorFlow or just GPUs in general. I've never been able to get TensorFlow, CUDA, etc. installed on it. It's a GeGorce GTX 460m and an Intel i7. I feel that it has the potential to be the star of my Finux Cluster, which includes antiques such as the ISKCON ppc7450, which is also running Ubuntu 16.04.

Trying to utilize this GPU, I've tried VoidLinux, Linux from Scratch, Ubuntu, Kali, etc., in that order. I'm currently reattempting with a fresh installation of Ubuntu 16.04 server with no graphical interface. It fit on my 1gb flash drive.

What I've figured out is the following. I did it as root, but I probably should have dropped to single-user mode:

  1. Apply any bugfixes, but never do-release-upgrade.
    apt update && apt full-upgrade
    

UPDATE: steps (9-10) can be done here to cut this down to a single reboot.

  1. This doesn't apply in my case, but for graphical installations, blacklist nouveau, and:

    modprobe -r <some variant of nouveau>
    
  2. Download and install video driver 390.138. Other driver versions here, but you need to look in the fine print to find the advanced options a.k.a. legacy and beta versions.

    For newer versions of Ubuntu, there's a compiler version mismatch re: DKMS, NVidia module, GCC 5/7, which can be worked around by update-alternatives. Other work-arounds didn't work for me.

  3. Make the new driver load at boot.

    update-initramfs -u
    
  4. sanity check

    nvidia-smi
    
  5. Download and install CUDA toolkit 9.1 8.0 and its three patches patch. Don't let it overwrite your driver. For other driver versions, there's a compatibility matrix to determine the required SDK version.

    The driver download page helps us select our operating system version, in this case, either 16.04 or 17.04, with the preference being the even version numbers. These install scripts need to be run one at a time, because they don't seem to give a reasonable exit status when they fail.

  6. Reboot and login as non-root. Programs weren't linking correctly, and this is what fixed it for me. I suppose it's got something to do with /etc/environment and /etc/profile, and when and which processes source them.

    UPDATE: maybe just logout. Use /etc/profile.d and /etc/ld.so.conf.d instead of /etc/environment

  7. When compiling the program I want to run, I specify the path to the library with cuInit().

    CUDA_LIB=/usr/local/cuda/lib64/stubs/libcuda.so
    

    I also specify my architecture, since not specifying it yields:

    no kernel image is available for execution on the device.
    

    This is a 460m, so it's got compute capability 2.1.

    CCAP=21
    
    CCAP=20
    CCSM=21
    

    UPDATE: it looks like my system needs "-gencode arch=compute_20,code=[sm_21,compute_20]"

    We might as well specify our CPU. GCC says it's a sandybridge.

    ARCH=sandybridge
    

    Next, passing the args with spaces is probably best done with an array, with the caveat that such variables can't be exported.

    export CFLAGS="-march=$ARCH -mtune=$ARCH"
    CMAKE_CONF=( "-DCUDA_LIB=$CUDA_LIB"
      "-DCUDA_NVCC_FLAGS=-gencode arch=compute_$CCAP,code=[sm_$CCSM,compute_$CCAP]" )
    
    mkdir build && cd build &&
    cmake .. -G Ninja "${CMAKE_CONF[@]}" && cd .. &&
    cmake --build build && cd build
    #sudo cmake --build build --target install
    # ninja: error: unknown target 'install'
    

    UPDATE: CUDA_LIB is a project-specific variable. Instead, do:

    echo /usr/local/cuda/lib64/stubs |
    sudo tee -a /etc/ld.so.conf.d/nvidia.conf &&
    sudo ldconfig
    

    UPDATE: that causes problems with nvidia-smi. fun.

Skipping ahead to TF....

The TF docs say the docker container is supposed to be easier, so I look more deeply into that first.

  1. Install docker.

    sudo apt install \
      apt-transport-https \
      ca-certificates \
      curl \
      gnupg-agent \
      software-properties-common &&
    curl -fsSL https://download.docker.com/linux/ubuntu/gpg | 
    sudo apt-key add - &&
    sudo add-apt-repository \
      "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
      $(lsb_release -cs) \
      stable" &&
    sudo apt update &&
    sudo apt install docker-ce docker-ce-cli containerd.io &&
    sudo adduser $(whoami) docker &&
    sudo reboot # fixes permission problem when connecting to socket
    
  2. Test docker.

    docker run hello-world
    
  3. Install nvidia-docker.

    distribution=$(. /etc/os-release;echo $ID$VERSION_ID) &&
    curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey |
    sudo apt-key add - &&
    curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list |
    sudo tee /etc/apt/sources.list.d/nvidia-docker.list &&
    sudo apt update &&
    sudo apt install -y nvidia-docker2 &&
    sudo systemctl restart docker
    
  4. Test nvidia-docker.

    docker run --rm --gpus all nvidia/cuda:9.1-base nvidia-smi
    
    docker run --rm --gpus all nvidia/cuda:8.0-runtime nvidia-smi
    

UPDATE: as fas as I can tell, everything is successful up to here. Step (13) can't work.

  1. TF docker build.... I'm not sure which image to use. I'm looking through their Dockerfiles, specifically 0.11.0-devel-gpu and 1.12.0-devel-gpu, and I see a COMPUTE_CAPABILITY variable. Same as the crypto miners in step (8), this seems to be a road block is the road block, as these require >2.1 compute compatibility.

    docker pull tensorflow/tensorflow:1.11.0-devel-gpu &&
    docker run --gpus all -it -w /tensorflow -v $PWD:/mnt \
      -e HOST_PERMS="$(id -u):$(id -g)" \
      tensorflow/tensorflow:1.11.0-devel-gpu bash
    

    I ended up switching some versions in their Dockerfile, and the apt install fails:

    E: Unable to locate package libcublas-9-1
    E: Unable to locate package libcufft-9-1                                                                                                                   
    E: Unable to locate package libcurand-9-1                                                                                                                  
    E: Unable to locate package libcusolver-9-1                                                                                                                
    E: Unable to locate package libcusparse-9-1
    

    Searching for those packages just funnels me to the download page for the HPC SDK (libcu++ would be cool), but alas, I don't think there's a version compatible with this old GPU.

UPDATE: 9.1 is the wrong version. Now looking into 8.0. That means tensorflow:0.11.0-devel-gpu was a good lead. It also means that steps (14+) won't do anything productive on my hardware.

Now to try building on the bare metal....

  1. I installed CUPTI.

    echo /usr/local/cuda-9.1/extras/CUPTI/lib64 |
    
    echo /usr/local/cuda-8.0/extras/CUPTI/lib64 |
    sudo tee -a /etc/ld.so.conf.d/nvidia.conf &&
    sudo ldconfig
    
  2. I installed cuDNN 7.1.3 for 9.1 cuDNN v7.1.4 for 8.0.

    sudo dpkg -i libcudnn7_7.1.4.18-1+cuda8.0_amd64.deb \
            libcudnn7-dev_7.1.4.18-1+cuda8.0_amd64.deb
    
  3. find and install libnvinfer7=7.1.3-1+cuda9.1. Looks like Ubuntu 14.04 has one.

  4. Update the host python packages.

    sudo apt install python3-dev python3-pip python3-venv
    
  5. Get the oldest Python version that isn't EOL.

    cd Python-3.6.12 &&
    ./configure --with-univseral-archs=intel-64 --with-hash-algorithm=siphash24 --with-threads --with-ensurepip=upgrade &&
    make -n`nproc` &&
    make DESTDIR=~/python3.6 install
    

    TODO that's a source package, and so I've got to test that command

  6. install GO

    wget https://golang.org/dl/go1.15.6.linux-amd64.tar.gz &&
    tar xf go1.15.6.linux-amd64.tar.gz &&
    sudo chown -R root:root go &&
    sudo mv go /opt/go &&
    sudo tee /etc/profile.d/gopath.sh << "EOF"
    export GOPATH="$HOME/go"
    export GOROOT=/opt/go
    PATH="$PATH:$GOROOT/bin:$GOPATH/bin"
    export PATH=$PATH:$(go env GOPATH)/bin
    EOF
    logout
    
  7. install bazelisk

    go get github.com/bazelbuild/bazelisk &&
    sudo ln -sv $(command -v bazelisk) /usr/local/bin/bazel
    
  8. Compile TF. In the past, I've gotten weird errors suggesting the TF package doesn't exist on PyPI, so I've attempted this before, but the resulting pip packages didn't run. On my system, the documentation's commands give warnings, so I've altered them slightly.

    UPDATE: since my hardware can't support even legacy TF, I switched the build to CPU-only.

    ~/python3.6/usr/local/bin/python3.6 -m venv \
      --system-site-packages tf-env &&
    . tf-env/bin/activate
    pip install -U pip &&
    pip install -U numpy wheel &&
    pip install -U keras_preprocessing --no-deps &&
    
    git clone -b r2.2 --depth=1 --recursive \
      git://github.com/tensorflow/tensorflow.git &&
    cd tensorflow &&
    python configure.py &&
    bazel build  \
      --local_ram_resources=2048 \
      --cxxopt="-D_GLIBCXX_USE_CXX11_ABI=0" \
      //tensorflow/tools/pip_package:build_pip_package &&
    ./bazel-bin/tensorflow/tools/pip_package/build_pip_package \
      /tmp/tensorflow_pkg &&
    pip install /tmp/tensorflow_pkg/tensorflow-version-tags.whl
    

    N.b., the environment changes don't seem to affect anything after the '&&', so don't do:

    . tf-env/bin/activate && whatever
    
  9. TODO Test TF:

    python -c "import tensorflow as tf;print(tf.reduce_sum(tf.random.normal([1000, 1000])))"
    
2
  • With a compute capability of 2.1, you are limited to CUDA 8. Unfortunately, the Deep Neural Networking takes a CC of 3.0, so that makes running the DNN or Tensorflow a problem. I was similarly stuck, until I got an external GPU adapter and hung a GTX 970 off my expresscard slot. See the egpu.io site builds.
    – ubfan1
    Dec 21, 2020 at 16:31
  • Imma install cuda toolkit 8 and then look for software written for older compute compatibilities. Thanks. Dec 21, 2020 at 17:08

1 Answer 1

0
  1. Update system

    apt update && apt full-upgrade
    
  2. Unload driver

    modprobe -r <some variant of nouveau>
    
  3. Download and install video driver. It automatically creates /etc/modprobe.d/nvidia-installer-disable-nouveau.conf. Do a sanity check.

    nvidia-smi
    
  4. Make it load at boot

    update-initramfs -u
    
  5. Download and install CUDA toolkit and patches. Don't overwrite your driver.

  6. Add CUDA toolkit libs to your path

    cat > /etc/ld.so.conf.d/nvidia.conf << "EOF"
    /usr/local/cuda-8.0/lib64
    /usr/local/cuda-8.0/extras/CUPTI/lib64
    EOF
    ldconfig
    
  7. Add CUDA bins to your path

    cat > /etc/profile.d/nvidia.sh << "EOF"
    export PATH="$PATH:/usr/local/cuda-8.0/bin"
    EOF
    . /etc/profile.d/nvidia.sh
    
  8. When compiling, there are usually project-specific ways of specifying the paths to the cuda libraries.

    LEGACY="
      -Wno-deprecated-gpu-targets
      -gencode arch=compute_20,code=[sm_21,compute_20]"
    
    • cmake

      mkdir build && cd build &&
      cmake .. -G Ninja \
        -DCUDA_LIB=/usr/local/cuda/lib64/stubs/libcuda.so \
        -DCUDA_NVCC_FLAGS="$LEGACY" &&
      cd .. &&
      cmake --build build &&
      cmake --build build --target install
      
    • autotools

      ./configure \
        --with-cuda=/usr/local/cuda \
        --with-nvml=libnvidia-ml.so \
        CUDA_NVCC_FLAGS="$LEGACY"
      
      

After modifying some source codes, and getting errors about certain algorithms, I managed to get a couple compiled.

Turns out the hashrate is very low.

The good news is that docker seems to work as expected. TensorFlow isn't happening.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.