Skip to content

NWChem

NWChem is an open-source quantum chemistry code developed by the Environmental Molecular Sciences Laboratory at Pacific Northwest National Laboratory. The code is hosted on a GitHub repository and official documentation is provided here.

Using NWChem at NERSC

NWChem is provided on Cori via the nwchem module, which adds the nwchem executable to the user's $PATH. One should load the craype-haswell before loading the nwchem module to use an executable optimized for Cori Haswell compute nodes. Similarly, the craype-mic-knl module should be loaded before the nwchem module to use an executable optimized for Cori KNL nodes.

The general syntax for running NWChem is

srun <Slurm options> nwchem <NWChem inputs file>

For example, if one's inputs file is called input.nw, and one wishes to run NWChem on one KNL node with 1 MPI rank per core, the appropriate commands would be:

#!/bin/bash

#SBATCH -C knl
#SBATCH -t 1:00:00
#SBATCH -q regular
#SBATCH -N 1
#SBATCH -J my_nwchem_job
#SBATCH -o my_nwchem_job.%j.out
#SBATCH -e my_nwchem_job.%j.err

module swap craype-{${CRAY_CPU_TARGET},mic-knl}
module load nwchem

export OMP_NUM_THREADS=1
export OMP_PROC_BIND=spread
export OMP_PLACES=cores
export MPICH_GNI_MAX_EAGER_MSG_SIZE=131026
export MPICH_GNI_NUM_BUFS=80
export MPICH_GNI_NDREG_MAXSIZE=16777216
export MPICH_GNI_MBOX_PLACEMENT=nic
export MPICH_GNI_RDMA_THRESHOLD=65536
export COMEX_MAX_NB_OUTSTANDING=6

srun -N $SLURM_NNODES --ntasks-per-node=64 -c 8 --cpu-bind=cores nwchem input.nw

How NWChem is compiled at NERSC

The nwchem modules at NERSC are compiled using the following script, or with minor variations. Users are welcome to use this script to compile their own versions of NWChem if the versions supplied by NERSC do not satisfy their needs.

Haswell

#!/bin/bash

set -ex

export tarball="nwchem-7.0.2-release.revision-b9985dfa-srconly.2020-10-12.tar.bz2"
export builddir=$(mktemp -d -p /tmp build_nwchem_7.0.2.XXXXXXXXXX)
export builddir_short="/tmp/nwchem"

if [ ! -f ${tarball} ]; then
  wget https://github.com/nwchemgit/nwchem/releases/download/v7.0.2-release/${tarball}
fi
cp ${tarball} ${builddir}

# NWChem forbids the build directory length to contain > 64 chars during
# compilation, which is a constraint we often run into when using a combination
# of $SCRATCH and mktemp with several random chars. So we fool NWChem by making
# the build dir a symlink in /tmp, which has many fewer than 64 chars.
ln -s ${builddir} ${builddir_short}

cd ${builddir_short}
tar xjf ${tarball}
cd nwchem-7.0.2

export NWCHEM_TOP="${PWD}"
export NWCHEM_TARGET="LINUX64"
export USE_MPI="y"
export USE_MPIF="y"
export USE_MPIF4="y"
export LIBMPI=" "
export MPI_LIB=" "
export MPI_INCLUDE=" "
export ARMCI_NETWORK="MPI-PR"
export USE_NOFSCHECK="TRUE"
export USE_NOIO="TRUE"
# We get the correct MKL link flags by using Intel's MKL Link Line Advisor web
# form: https://software.intel.com/en-us/articles/intel-mkl-link-line-advisor
export SCALAPACK_LIB="-L${MKLROOT}/lib/intel64 -lmkl_scalapack_ilp64 -lmkl_intel_ilp64 -lmkl_core -lmkl_sequential -lmkl_blacs_intelmpi_ilp64 -lpthread -lm"
export SCALAPACK_SIZE="8"
export BLAS_SIZE="8"
# Use the same Intel MKL Link Line Advisor flags here.
export BLASOPT="-L${MKLROOT}/lib/intel64 -lmkl_intel_ilp64 -lmkl_core -lmkl_sequential -lmkl_core -liomp5 -lpthread -ldmapp -lm"
export LAPACK_LIB="$BLASOPT"
export NWCHEM_MODULES="all python"
export MRCC_METHODS="TRUE"
module unload cray-libsci
module load python3
export PYTHONVERSION=3.8
export USE_OPENMP="1"
export USE_F90_ALLOCATABLE="T"

cd src
module load gcc # need gfortran in $PATH or else `make nwchem_config` fails
make nwchem_config FC=ftn CC=cc
module unload gcc
make FC=ftn CC=cc 2>&1 | tee $HOME/build_nwchem_7.0.0_intel_hsw_$(date -I"seconds").log
rm ${builddir_short}

KNL

#!/bin/bash

set -ex

module swap craype-{${CRAY_CPU_TARGET},mic-knl}

export tarball="nwchem-7.0.2-release.revision-b9985dfa-srconly.2020-10-12.tar.bz2"
export builddir=$(mktemp -d -p /tmp build_nwchem_7.0.2.XXXXXXXXXX)
export builddir_short="/tmp/nwchem"

if [ ! -f ${tarball} ]; then
  wget https://github.com/nwchemgit/nwchem/releases/download/v7.0.2-release/${tarball}
fi
cp ${tarball} ${builddir}

# NWChem forbids the build directory length to contain > 64 chars during
# compilation, which is a constraint we often run into when using a combination
# of $SCRATCH and mktemp with several random chars. So we fool NWChem by making
# the build dir a symlink in /tmp, which has many fewer than 64 chars.
ln -s ${builddir} ${builddir_short}

cd ${builddir_short}
tar xjf ${tarball}
cd nwchem-7.0.2

export NWCHEM_TOP="${PWD}"
export NWCHEM_TARGET="LINUX64"
export USE_MPI="y"
export USE_MPIF="y"
export USE_MPIF4="y"
export ARMCI_NETWORK="MPI-PR"
export USE_NOFSCHECK="TRUE"
export USE_NOIO="TRUE"
export USE_KNL="y"
# We get the correct MKL link flags by using Intel's MKL Link Line Advisor web
# form: https://software.intel.com/en-us/articles/intel-mkl-link-line-advisor
export SCALAPACK_LIB="-L${MKLROOT}/lib/intel64 -lmkl_scalapack_ilp64 -lmkl_intel_ilp64 -lmkl_core -lmkl_sequential -lmkl_blacs_intelmpi_ilp64 -lpthread -lm"
export SCALAPACK_SIZE="8"
export BLAS_SIZE="8"
# Use the same Intel MKL Link Line Advisor flags here.
export BLASOPT="-L${MKLROOT}/lib/intel64 -lmkl_intel_ilp64 -lmkl_core -lmkl_sequential -lmkl_core -liomp5 -lpthread -ldmapp -lm"
export LAPACK_LIB="$BLASOPT"
export NWCHEM_MODULES="all python"
export MRCC_METHODS="TRUE"
module unload cray-libsci
module load python3
export PYTHONVERSION=3.8
export USE_OPENMP="1"
export USE_F90_ALLOCATABLE="T"
export USE_KNL="1"

cd src
module load gcc # need gfortran in $PATH or else `make nwchem_config` fails
make nwchem_config FC=ftn CC=cc
module unload gcc
make -j16 FC=ftn CC=cc 2>&1 | tee $HOME/build_nwchem_7.0.2_intel_knl_$(date -I"seconds").log
rm ${builddir_short}