NWChem¶
NWChem is an open-source quantum chemistry code developed by the Environmental Molecular Sciences Laboratory at Pacific Northwest National Laboratory. The code is hosted on a GitHub repository and official documentation is provided.
Using NWChem at NERSC¶
First Option: Shifter images for release 7.2.0¶
The following scripts use an image created from the 7.2.0 NWChem release using the following Dockerfile to run NWChem on Perlmutter via Shifter.
As far as the execution on GPUs in concerned, the user should read the related documentation on the NWChem website https://nwchemgit.github.io/TCE.html#ccsdt-method-with-cuda
Slurm script for NWChem Shifter image on Perlmutter CPUs¶
#!/bin/bash
#SBATCH -C cpu
#SBATCH -q regular
#SBATCH -t 1:00:00
#SBATCH -N 2
#SBATCH --ntasks-per-node=64
#SBATCH --cpus-per-task=2
#SBATCH -J my_nwchem_job
#SBATCH -o my_nwchem_job.%j.out
#SBATCH -e my_nwchem_job.%j.err
#SBATCH --image=ghcr.io/nwchemgit/nwchem-720.nersc.mpich4.mpi-pr:latest
export OMP_NUM_THREADS=1
export COMEX_MAX_NB_OUTSTANDING=6
export FI_CXI_RX_MATCH_MODE=hybrid
export COMEX_EAGER_THRESHOLD=16384
export FI_CXI_RDZV_THRESHOLD=16384
export FI_CXI_OFLOW_BUF_COUNT=6
export MPICH_SMP_SINGLE_COPY_MODE=CMA
srun -N $SLURM_NNODES --cpu-bind=cores shifter --module=mpich nwchem input.nw
Slurm script for NWChem Shifter image on Perlmutter GPUs¶
#!/bin/bash
#SBATCH -C gpu
#SBATCH -q regular
#SBATCH -t 1:00:00
#SBATCH -N 2
#SBATCH --ntasks-per-node=16
#SBATCH --cpus-per-task=8
#SBATCH --gpus-per-node=4
#SBATCH --gpu-bind=none
#SBATCH -J my_nwchem_job
#SBATCH -o my_nwchem_job.%j.out
#SBATCH -e my_nwchem_job.%j.err
#SBATCH --image=ghcr.io/nwchemgit/nwchem-720.nersc.mpich4.mpi-pr:latest
export OMP_NUM_THREADS=1
export COMEX_MAX_NB_OUTSTANDING=6
export FI_CXI_RX_MATCH_MODE=hybrid
export COMEX_EAGER_THRESHOLD=16384
export FI_CXI_RDZV_THRESHOLD=16384
export FI_CXI_OFLOW_BUF_COUNT=6
export MPICH_SMP_SINGLE_COPY_MODE=CMA
srun -N 1 -n 2 --cpu-bind=cores shifter --module=gpu,mpich nvidia-smi
srun -N $SLURM_NNODES --cpu-bind=cores shifter --module=gpu,mpich nwchem input.nw
Second Option: Shifter images for the master branch¶
The following scripts use an image created from the master branch of the NWChem github repository using the following Dockerfile to run NWChem on Perlmutter via Shifter.
Slurm script for NWChem Shifter image on Perlmutter CPUs¶
#!/bin/bash
#SBATCH -C cpu
#SBATCH -q regular
#SBATCH -t 1:00:00
#SBATCH -N 2
#SBATCH --ntasks-per-node=64
#SBATCH --cpus-per-task=2
#SBATCH -J my_nwchem_job
#SBATCH -o my_nwchem_job.%j.out
#SBATCH -e my_nwchem_job.%j.err
#SBATCH --image=ghcr.io/nwchemgit/nwchem-dev.nersc.mpich4.mpi-pr:latest
export OMP_NUM_THREADS=1
export COMEX_MAX_NB_OUTSTANDING=6
export FI_CXI_RX_MATCH_MODE=hybrid
export COMEX_EAGER_THRESHOLD=16384
export FI_CXI_RDZV_THRESHOLD=16384
export FI_CXI_OFLOW_BUF_COUNT=6
export MPICH_SMP_SINGLE_COPY_MODE=CMA
srun -N $SLURM_NNODES --cpu-bind=cores shifter --module=mpich nwchem input.nw
Slurm script for NWChem Shifter image on Perlmutter GPUs¶
#!/bin/bash
#SBATCH -C gpu
#SBATCH -q regular
#SBATCH -t 1:00:00
#SBATCH -N 2
#SBATCH --ntasks-per-node=16
#SBATCH --cpus-per-task=8
#SBATCH --gpus-per-node=4
#SBATCH --gpu-bind=none
#SBATCH -J my_nwchem_job
#SBATCH -o my_nwchem_job.%j.out
#SBATCH -e my_nwchem_job.%j.err
#SBATCH --image=ghcr.io/nwchemgit/nwchem-dev.nersc.mpich4.mpi-pr:latest
export OMP_NUM_THREADS=1
export COMEX_MAX_NB_OUTSTANDING=6
export FI_CXI_RX_MATCH_MODE=hybrid
export COMEX_EAGER_THRESHOLD=16384
export FI_CXI_RDZV_THRESHOLD=16384
export FI_CXI_OFLOW_BUF_COUNT=6
export MPICH_SMP_SINGLE_COPY_MODE=CMA
srun -N 1 -n 2 --cpu-bind=cores shifter --module=gpu,mpich nvidia-smi
srun -N $SLURM_NNODES --cpu-bind=cores shifter --module=gpu,mpich nwchem input.nw