CMTH SciPy Stack

Description

This is a full SciPy stack compiled against the Intel compilers and MKL libraries and made available as a module. It contains python 3.6.2 as well as the most recent version at the time of install of the following packages and tools:

• python
• numpy
• scipy
• matplotlib
• cython
• pandas
• ipython
• sympy
• nose
• jupyter
• numba
• bokeh
• tensorflow
• scikit-learn

User instructions

The module containing the latest versions of all packages at the beginning of September 2017 can be loaded as:

module load cmth-scipy-stack/2017.09

Source

• http://www.python.org
• https://www.scipy.org
• https://pypi.python.org

License

Python has the PSF License (Python Software Foundation). From version 2.2 on this is GPL compatible.

Admin notes

Compilation was all done on a workstation. The following additional dependencies were installed: libssl-dev, libsqlite3-dev and tk-dev. First python was compiled. This was done with the Intel compilers as follows

ml purge
ml intel
./configure \
    --prefix=/common/debian/9.1/Compiler/intel/2017.4/cmth-scipy-stack/cmth-scipy-stack-2017.09 \
    --with-icc --without-gcc \
    --with-computed-gotos \
    --with-libm=-limf --with-cxx-main=icpc --with-threads \
    --enable-ipv6 \
    --enable-shared \
    --enable-optimizations \
    CC=icc CXX=icpc LD=xild AR=xiar \
    LIBS="-lpthread -limf -lirc" \
    CFLAGS="-O3 -fp-model strict -fp-model source -xHost -ipo -prec-div -prec-sqrt" \
    LDFLAGS="-ipo" \
    CPPFLAGS="" \
    CPP="icc -E" &> configure.log &
ulimit -s unlimited
make -j4 &> make.log &
make install

This was made into a module before proceeding with the rest.

Then for NumPy the following site.cfg was used to compile it from source:

[mkl]
library_dirs = /common/debian/9.1/Core/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64
include_dirs = /common/debian/9.1/Core/intel/compilers_and_libraries_2017.4.196/linux/mkl/include
mkl_libs = mkl_rt
lapack_libs =

It was then compiled with:

ml purge
ml intel mkl cmth-scipy-stack
python3 setup.py config --compiler=intelem build_clib --compiler=intelem build_ext --compiler=intelem install --prefix=/common/debian/9.1/Compiler/intel/2017.4/cmth-scipy-stack/cmth-scipy-stack-2017.09

SciPy was then compiled from source with:

python3 setup.py config --compiler=intelem --fcompiler=intelem build_clib --compiler=intelem --fcompiler=intelem build_ext --compiler=intelem --fcompiler=intelem install --prefix=/common/debian/9.1/Compiler/intel/2017.4/cmth-scipy-stack/cmth-scipy-stack-2017.09

Everything else was installed from pip3 with the stack module loaded:

package_list="matplotlib cython pandas ipython sympy nose jupyter numba bokeh tensorflow scikit-learn"
for package in $package_list; do
  CC=icc CXX=icpc FC=ifort F77=ifort PYTHONUSERBASE=/common/debian/9.1/Compiler/intel/2017.4/cmth-scipy-stack/cmth-scipy-stack-2017.09 pip3 install --upgrade $package
done