1

My question: How do I get Python to use my own compiled OpenBLAS version?

Background

The Theano install guide suggests that one should test that the BLAS library components associated are correctly optimized through these commands:

THEANO_FLAGS=floatX=float32,device=cpu python `python -c "import os, theano; print os.path.dirname(theano.__file__)"`/misc/check_blas.py 
THEANO_FLAGS=floatX=float32,device=gpu python `python -c "import os, theano; print os.path.dirname(theano.__file__)"`/misc/check_blas.py

The results are:

Card: GTX 770
Results on CPU 0.62s and GPU 0.11s

Not sure whether these results were good I decided to compare different libraries as done in this post. As scientific computing requires every possible speed-up this seemed like a good idea.

Compiling and installing OpenBLAS from GitHub

In order to test the latest version I've:

  1. cloned the git git clone https://github.com/xianyi/OpenBLAS.git
  2. entered the directory and made a make TARGET=HASWELL (the TARGET is not really necessary as it detects it automatically)
  3. followed by make PREFIX=/opt/OpenBLAS install.
  4. copied the files under lib (libopenblas.a libopenblas_haswellp-r0.2.14.a libopenblas_haswellp-r0.2.14.so libopenblas.so libopenblas.so.0) to /usr/lib/openblas-haswell_cstm/
  5. added an alternative through sudo update-alternatives --install /usr/lib/libblas.so.3 libblas.so.3 /usr/lib/openblas-haswell_cstm/libopenblas.so.0 30

Unfortunately this gives an error:

$ sudo update-alternatives --config libblas.so.3 
There are 4 choices for the alternative libblas.so.3 (providing /usr/lib/libblas.so.3).

  Selection    Path                                             Priority   Status
------------------------------------------------------------
  0            /usr/lib/openblas-base/libblas.so.3               40        auto mode
* 1            /usr/lib/atlas-base/atlas/libblas.so.3            35        manual mode
  2            /usr/lib/libblas/libblas.so.3                     10        manual mode
  3            /usr/lib/openblas-base/libblas.so.3               40        manual mode
  4            /usr/lib/openblas-haswell_cstm/libopenblas.so.0   30        manual mode

Press enter to keep the current choice[*], or type selection number: 4
update-alternatives: using /usr/lib/openblas-haswell_cstm/libopenblas.so.0 to provide /usr/lib/libblas.so.3 (libblas.so.3) in manual mode
max@MaxLin:~/tutorials$ python
Python 2.7.8 (default, Oct 20 2014, 15:05:19) 
[GCC 4.9.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/dist-packages/numpy/__init__.py", line 153, in <module>
    from . import add_newdocs
  File "/usr/lib/python2.7/dist-packages/numpy/add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "/usr/lib/python2.7/dist-packages/numpy/lib/__init__.py", line 18, in <module>
    from .polynomial import *
  File "/usr/lib/python2.7/dist-packages/numpy/lib/polynomial.py", line 19, in <module>
    from numpy.linalg import eigvals, lstsq, inv
  File "/usr/lib/python2.7/dist-packages/numpy/linalg/__init__.py", line 50, in <module>
    from .linalg import *
  File "/usr/lib/python2.7/dist-packages/numpy/linalg/linalg.py", line 29, in <module>
    from numpy.linalg import lapack_lite, _umath_linalg
ImportError: /usr/lib/liblapack.so.3: undefined symbol: gotoblas

I'm not sure if this has to do with the compiled version not being a .so.3 version. I've tried using the libopenblas_haswellp-r0.2.14.so and the libopenblas.so created by the make but it has no effect on the error.

  • (Re-)installing NumPy shouldn't have any influence on your issue. On a side note, it's a bad idea, to install a Python module both through the system package manager and Pip. Generally, the system packages are preferable, as the package manager can take care of (binary) dependencies. – David Foerster Apr 18 '15 at 22:51
  • @DavidFoerster thanks for the tip on using the system packages. The error is much cleaner now, it seems that installing numpy from pip links to a different library. It should be possible to set the numpy BLAS library using a custom compiled version using pip. Anyway, that's a different problem. Currently I would like to decipher how the packaging system works and how to exchange standard packages with my own versions. – Max Gordon Apr 19 '15 at 7:30
  • Is launchpad.net/ubuntu/+source/openblas unsuitable or too out of date? It appears to have the same code base as the Git repository in your question. This is beyond the scope of this question: if you want to build a custom Debian package, I recommend, that you start with the existing openblas package and How to download, modify, build and install a Debian source package?. You can modify the package name and library file names, if you feel, that your package doesn't fully replace the provided one. – David Foerster Apr 19 '15 at 11:13
  • @DavidFoerster: The current OpenBLAS issue was just because I wanted to test if I can build a standard package from a git repository and hook it into the system. Like many things, it seemed like a good idea at the time but adapting a package sounds like a smoother solution. Especially since there are most likely hints to what kind of dark magic the package system performs to get it to work. Why don't you post that as an answer. Thank you for your time! – Max Gordon Apr 19 '15 at 18:20
  • Because I find it to be “beyond the scope of this question”. If you want to rephrase it to include the way my comment is going, I'll go ahead and make it an answer. – David Foerster Apr 19 '15 at 18:22
0

if you want to build a custom Debian package, I recommend, that you start with the existing openblas package and How to download, modify, build and install a Debian source package?.

You can modify the package name and library file names, if you feel, that your package doesn't fully replace the provided one.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.