Install and configure python
(Prepared by G. Iadarola)
Introduction
Some operative systems come equipped with a Python installation. Even in this case, it is often convenient to install a user-managed python environment to have full control of the configuration.
In this page we describe two methods:
- Setup of a miniforge installation (easier)
- Compile and configure python from source code (necessary when thw installation needs to be compiled with a specific compiler, for example to be used with MPI).
None of the two methods requires admin rights.
Install and configure miniforge
Step 1: Download and install the most recent version of Miniforge (from here) Something like:
wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh
bash Miniforge3-latest-Linux-x86_64.sh
A miniforge installation is strongly recommended against a miniconda installation as miniforge uses by default the "conda-forge" channel while miniconda uses the "default" channel (https://repo.anaconda.com/pkgs/). While the "default" channel can require a paid license depending on its usage, the "conda-forge" channel is free for all to use (see https://docs.conda.io/projects/conda/en/latest/user-guide/concepts/channels.html).
Nevertheless, if you still prefer a Miniconda installation, you can get it from here and do something like:
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh
Step 2: Activate the miniforge installation
source /home/giadarol/Desktop/PyFRIENDS_python3/miniforge3/bin/activate
Step 2a (optional): Install compilers
To install packages that need to compile code (e.g. PyECLOUD, PyHEADTAIL or Xsuite) you need C and/or FORTRAN compilers. If you don't have compilers available on your system you can install them using conda as discussed in this guide.
For example on Linux or MacOS you can install C, C++ and FORTRAN compilers by:
conda install compilers
Step 3: Install useful packages
pip install numpy scipy matplotlib pandas ipython
Your python installation is ready to be used!
Virtual environments
Virtual environments can be useful to try different package versions and for developments. A detailed guide to conda's virtual environments can be found here:
Compile and configure python from source code
Step 1: We move to the folder where we want to place our installation:
/afs/cern.ch/work/g/giadarol/sim_workspace_mpi_py3
Step 2: We download from the python website, compile and install the latest version of python 3:
mkdir python_src
cd python_src
wget https://www.python.org/ftp/python/3.8.0/Python-3.8.0.tar.xz
tar xvf Python-3.8.0.tar.xz
cd Python-3.8.0
./configure --prefix=/afs/cern.ch/work/g/giadarol/sim_workspace_mpi_py3/python3
make
make install
Possible issues with ssl: In order to be able to use pip at a later stage, python needs to be compiled with ssl support. Please check that this is the case:
/afs/cern.ch/work/g/giadarol/sim_workspace_mpi_py3/python3/bin/python3
import ssl
In case this gives an exception, please install ssl library and recompile python (in Ubuntu this can be done by sudo apt install libssl-dev). Information on how to get the ssl library without admin rights can be font here.
Possible issue with ctypes and libffi: Check that ctypes is working (will be required for several reasons afterwards):
/afs/cern.ch/work/g/giadarol/sim_workspace_mpi_py3/python3/bin/python3
import ctypes
If you get an error, it is because the ffi library is not installed. You can download and install the library as follows:
cd /afs/cern.ch/work/g/giadarol/sim_workspace_mpi_py3/python_src
wget ftp://sourceware.org/pub/libffi/libffi-3.3.tar.gz
tar -xvf libffi-3.3.tar.gz
cd libffi-3.3/
mkdir /afs/cern.ch/work/g/giadarol/sim_workspace_mpi_py3/libffi
./configure --prefix=/afs/cern.ch/work/g/giadarol/sim_workspace_mpi_py3/libffi
make
make install
The library location needs to be added to our library path:
export LD_LIBRARY_PATH=/afs/cern.ch/work/g/giadarol/sim_workspace_mpi_py3/libffi/lib64:$LD_LIBRARY_PATH
The python compilation needs some extra information:
export PKG_CONFIG_PATH=/afs/cern.ch/work/g/giadarol/sim_workspace_mpi_py3/libffi/lib/pkgconfig
./configure --prefix=/afs/cern.ch/work/g/giadarol/sim_workspace_mpi_py3/python3 LDFLAGS="-L/afs/cern.ch/work/g/giadarol/sim_workspace_mpi_py3/libffi/lib/../lib64" CFLAGS="-I/afs/cern.ch/work/g/giadarol/sim_workspace_mpi_py3/libffi/include"
make
make install
Possible issues with sqlite3: In order to be able to use jupyter notebooks at a later stage, python needs to be compiled with sqlite3 support. Please check that this is the case:
/afs/cern.ch/work/g/giadarol/sim_workspace_mpi_py3/python3/bin/python3
import sqlite3
In case this gives an exception, please install sqlite library and recompile python (in Ubuntu this can be done by sudo apt install libsqlite3-dev).
Possible issues with tk: In order to be able to use ipython --pylab at a later stage, python needs to be compiled with tk support. Please check that this is the case:
/afs/cern.ch/work/g/giadarol/sim_workspace_mpi_p3/python3/bin/python3
import _tkinter
In case this gives an exception, please install tk library and recompile python (in Ubuntu this can be done by sudo apt install tklib; sudo apt install tk-dev ).
Step 3: We create a virtual environment
cd /afs/cern.ch/work/g/giadarol/sim_workspace_mpi_py3
mkdir venvs
cd venvs
/afs/cern.ch/work/g/giadarol/sim_workspace_mpi_py3/python3/bin/python3 -m venv py3
Step 4: We activate the virtual environment
source /afs/cern.ch/work/g/giadarol/sim_workspace_mpi_py3/venvs/py3/bin/activate
If we type:
which python
we get:
/afs/cern.ch/work/g/giadarol/sim_workspace_mpi_py3/venvs/py3/bin/python
Step 4b: In case libffi needed to be compiled, you should add libffi path also in activate script
To do this, edit the activate script:
nano /afs/cern.ch/work/g/giadarol/sim_workspace_mpi_py3/venvs/py3/bin/activate
Add after the last line:
export LD_LIBRARY_PATH=/afs/cern.ch/work/g/giadarol/sim_workspace_mpi_py3/libffi/lib64:$LD_LIBRARY_PATH
Step 6: We use pip to install the python modules that we need
pip install numpy scipy cython ipython matplotlib
Steps 5 and 6 are required only to run parallel simulations using MPI.
Step 5 - Check that MPI is available:
You can use the following commands to check that an MPI installation is available
which mpicc
which mpiexec
Step 5.a - Activate MPI:
On certain machines an MPI installation can be activated using module load
(it is the case for CERN lxplus and CERN HPC cluster) or mpi_selector
(it is the case for the INFN-CNAF cluster)
Step 5.b - Install MPI:
If we do not have an MPI installation we need to get one:
cd /afs/cern.ch/work/g/giadarol/sim_workspace_mpi_py3/python_src
wget https://www.open-mpi.org/software/ompi/v1.10/downloads/openmpi-1.10.2.tar.bz2
tar jxf openmpi-1.10.2.tar.bz2
cd openmpi-1.10.2
./configure --prefix=/afs/cern.ch/work/g/giadarol/sim_workspace_mpi_py3/openmpi
make all install
We set the environment variable for the MPI compiler (pip will use the compiler pointed by your MPICC variable to compile mpi4py):
export MPICC=/afs/cern.ch/work/g/giadarol/sim_workspace_mpi_py3/openmpi/bin/mpicc
At this point be careful not to have other MPI paths in your environment (for example set in your .bashrc)
Step 6: Install mpi4py
pip install mpi4py
Important: Python jobs using mpi4py must be run with the mpiexec corresponding to the MPI compiler that has been used. In our case:
/afs/cern.ch/work/g/giadarol/sim_workspace_mpi_py3/openmpi/bin/mpiexec
Of course we can add the folder to our PATH or create a shortcut.
Step 7: Install h5py (if you need it)
If you don't need parallel hdf5 I/O you can just:
pip install h5py
Instead, in case you need parallel hdf5 I/O and you want to compile your own hdf5 library (N.B. this is NOT needed for PyPARIS simulations):
wget https://support.hdfgroup.org/ftp/HDF5/current/src/hdf5-1.10.5.tar.bz2
tar jxf hdf5-1.10.5.tar.bz2
cd hdf5-1.10.5/
./configure --prefix=/afs/cern.ch/work/g/giadarol/sim_workspace_mpi_py3/hdf5lib --enable-parallel --enable-shared
make
make install
CC=/usr/bin/mpicc HDF5_MPI="ON" HDF5_DIR=/afs/cern.ch/work/g/giadarol/sim_workspace_mpi_py3/hdf5lib pip install --no-binary=h5py h5py
In case you don't have libhdf5 installed, but you don't need parallel IO (PyPARIS case), you can:
wget https://support.hdfgroup.org/ftp/HDF5/current/src/hdf5-1.10.5.tar.bz2
tar jxf hdf5-1.10.5.tar.bz2
cd hdf5-1.10.5/
./configure --prefix=/afs/cern.ch/work/g/giadarol/sim_workspace_mpi_py3/hdf5lib --enable-shared
make
make install
HDF5_DIR=/afs/cern.ch/work/g/giadarol/sim_workspace_mpi_py3/hdf5lib pip install --no-binary=h5py h5py
A couple of references that helped with this task:
[1] [[https://www.open-mpi.org/faq/?category=building#easy-build]]