Download and Installation

PERTURBO supports both CPU and GPU configurations. We use the CPU scenario as an example to demonstrate a detailed guide on how to install PERTURBO. Instructions for GPU installation are provided at the end of this page. To ensure compatibility, we suggest compiling both Quantum Espresso (QE) and PERTURBO under the same configuration, either CPU or GPU.

Quantum Espresso and HDF5 Download and Installation

PERTURBO uses a few subroutines from the PWSCF and Phonon packages of Quantum Espresso (QE). Therefore, it needs to be compiled on top of QE. PERTURBO needs the output files from Wannier 90 (W90), which can ba obtained from the QE. In addition, PERTURBO uses the HDF5 format to store data. For the detailed instructions on the installation of these packages, please refer to their official websites: QE, HDF5. If you run the calculations on a cluster or a supercomputer, these packages might be already pre-installed. Here we provide some brief instructions on the installation of these packages. Please note that these instructions can be different for your computing environement.

HDF5

To compile the HDF5 library, please download its source code from the official website. Once the source code is downloaded, please create an empty directory where the HDF5 library will be installed.

For example, the current directory is called ‘mylib’. We download the source code inside the directory called ‘hdf5-1.12.0-source-codes’. Now we are going to install the HDF5 library into a directory called ‘hdf5’.

Generate a Makefile for compiling the HDF5 library using the fortran option: --enable-fortran. Please modify the 'prefix' path to fit your case. Here we compile the serial HDF5 library:

cd hdf5-1.12.0-source-codes
./configure --prefix=mylib/hdf5 --enable-fortran 

One can specify the compilers running the ./configure command with the additional options: CC=<c compiler>, CXX=<c++ compiler>, FC=<fortran compiler>. For more information, run ./configure --help. Compile HDF5:

make
make install

To check whether HDF5 was compiled correctly, one can run the test suite:

make test

Now we have the compiled HDF5 library inside the directory ‘hdf5’. We suggest to use the directory path when compiling QE (--with-hdf5=mylib/hdf5 flag for QE compilation, see below).

Quantum Espresso

To download QE, one can use the wget command:

wget https://github.com/QEF/q-e/archive/qe-7.3.1.tar.gz
tar xvzf qe-7.3.1.tar.gz
cd q-e-qe-7.3.1

or to clone the package from the QE GitHub repository, specifying the version:

git clone https://github.com/QEF/q-e.git
cd q-e
git checkout qe-7.3.1

Once the package is downloaded run the configure command with the hdf5 io option of q-e:

./configure --with-hdf5="/path_to_hdf5"

Other configuration parameters strongly depend on the compilation scenarios (compilers, parallelization, etc.). On the PERTURBO github page you can find several compilation scripts with corresponding parameters.

Make QE also with W90:

make pw ph pp w90

Note that it is possible (but not preferred) to use q-e without hdf5, and then use hdf5 only for PERTURBO calculations, as it is explained below. Fow this, run the configure without --with-hdf5="/path_to_hdf5" option.

PERTURBO Download and Installation

Download

In order to get access to the code, please fill out this form.

To help us keep track of user number, we encourage each individual user to submit a separate request for code download. For example, research groups with multiple users should also have each user submit a request.

If the the form does not work, see here the instructions.

Clone from GitHub (or extract .tar.gz) into the QE directory. There are three subdirectories inside the directory “perturbo”:

• “config” contains the system-dependent configurational files make_XXX.sys.
• “pert-src” contains the source code of perturbo.x to compute electron dynamics
• “qe2pert-src” contains the source code of the interface program qe2pert.x

The source code is supplemented by the tutorial examples input, output files, as well as by the Python postprocessing package Perturbopy.

Installation

PERTURBO uses the configurational file ../make.inc generated by QE and requires another configuration file called make.sys, which specifies additional options required by PERTURBO. The examples of the files make.sys can be found in the config folder.

Copy one of the make.sys files from the config folder:

$ cp config/make_gcc_serial.sys make.sys

and make the necessary changes to it.

For example, you need to specify the path to your HDF5 library, if you haven’t made it on the step of the QE compilation:

IFLAGS += -I/path/to/hdf5/include
HDF5_LIBS = -L/path/to/hdf5/lib -lhdf5 -lhdf5_fortran

Once the file make.sys has been modified, you are ready to compile PERTURBO:

make

After the compilation, a directory called ‘bin’ is generated, which contains two executables, perturbo.x and qe2pert.x.

GPU

The GPU parallelsim is supported in both QE and PERTURBO through OpenACC.

Quantum Espresso

The only different part from CPU is the configuration part, we suggest as below,

FC=nvfortran F90=nvfortran MPIF90=mpif90 CC=cc \
  FFLAGS="-fast -Mlarge_arrays -mcmodel=medium" CFLAGS="-fast -mcmodel=medium" \
  LDFLAGS="-L/opt/nvidia/hpc_sdk/Linux_x86_64/23.1/math_libs/lib64" \
  BLAS_LIBS="-lblas" LAPACK_LIBS="-llapack" \
  ./configure --enable-parallel=yes --enable-openmp=yes \
  --with-cuda=/opt/nvidia/hpc_sdk/Linux_x86_64/23.1/cuda \
  --with-cuda-cc=80 --with-cuda-runtime=12.0 \
  --with-hdf5=/path/to/hdf5

Then you just need to follow the regular workflow of compilation.

PERTURBO

GPU acceleration is supported for both transport and ultrafast dynamics in PERTURBO. The compilation is almost similar as the regular compilation procedures above, so just the different parts will be shown here. Only nvfotran compiler in NVIDIA HPC SDK is well tested on A100 GPUs. Due to a known bug (reported by our group) of the nvfortran compiler, please use version 24.7 and beyond or version 23.1 and below of NVIDIA HPC SDK. Cause PERTURBO uses ../make.inc generated by QE, you are supposed to install QE with OpenACC enabled before starting compilation of GPU version of PERTURBO.

In the PERTURBO folder, copy make_nersc_nvhpc.sys files from the config folder:

cp config/make_nersc_nvhpc.sys make.sys

Besides changing the library path, two lines like below need to be uncommented, which should be the default setting.

FFLAGS += -acc=gpu -Minfo=accel -gpu=cc80,nomanaged,deepcopy,zeroinit
LDFLAGS += -acc=gpu -gpu=cc80,nomanaged,deepcopy,zeroinit

Fine-tuned compilation

To meet different computation demands, we provide multiple compilation flags which can be turn off or on in pert-src/pert_config.h. Currently, these flags only apply to calculations of electron-phonon scattering, transport and dynamics, including calculation modes of 'trans-rta', 'trans-ita', 'trans-mag-rta', 'trans-mag-ita', and 'dynamics-run'. Other calculations, such as spin relaxation times and Cumulant functions, are not supported yet. The explanation of those flags are listed here for reference.

• SCAT_FWD: Defining SCAT_FWD causes PERTURBO to generate a data structure that supports forward traversal, i.e. “source element ordered” calculation.
• SCAT_REV: Defining SCAT_REV causes PERTURBO to generate a data structure that supports reverse traversal, i.e. “target element ordered” calculation.
• STORE_G2DT: Defining STORE_G2DT causes the scatter_channels array to store the two -g2*dt1 and -g2*dt2 terms, instead of just holding g2. This increases memory requirements of PERTURBO, but can result in faster computations.
• CDYN_USE_ARRAY_REDUCE: Defining CDYN_USE_ARRAY_REDUCE causes the OpenMP code to use an array-reduction instead of omp atomic to perform updates to the epcol array. This means that each OpenMP thread has its own copy of epcol, which uses more memory, and combining them at the end will take time as well. However, array-reduction is usually much faster than omp atomic.
• CDYN_SORT_SCAT_TGTS: Defining CDYN_SORT_SCAT_TGTS causes the target-oriented code to order the scatter-target array elements by decreasing sub-array lengths. For the GPU this seems to greatly improve performance, since all threads in a warp will finish working around the same time as each other.

Note that the default values in pert-src/pert_config.h are suggested. Please try other options with care and make sure they align with your intended use.