Skip to content

NWChem

NWChem is an open-source quantum chemistry code developed by the Environmental Molecular Sciences Laboratory at Pacific Northwest National Laboratory. The code is hosted on a GitHub repository and official documentation is provided here.

Using NWChem at NERSC

NWChem is provided on Cori via the nwchem module, which adds the nwchem executable to the user's $PATH. One should load the craype-haswell before loading the nwchem module to use an executable optimized for Cori Haswell compute nodes. Similarly, the craype-mic-knl module should be loaded before the nwchem module to use an executable optimized for Cori KNL nodes.

The general syntax for running NWChem is

srun <Slurm options> nwchem <NWChem inputs file>

For example, if one's inputs file is called input.nw, and one wishes to run NWChem on one KNL node with 1 MPI rank per core, the appropriate commands would be:

#!/bin/bash

#SBATCH -C knl
#SBATCH -t 1:00:00
#SBATCH -q regular
#SBATCH -N 1
#SBATCH -J my_nwchem_job
#SBATCH -o my_nwchem_job.%j.out
#SBATCH -e my_nwchem_job.%j.err

module swap craype-{${CRAY_CPU_TARGET},mic-knl}
module load nwchem

export OMP_NUM_THREADS=1
export OMP_PROC_BIND=spread
export OMP_PLACES=cores

srun -n 68 -c 4 --cpu-bind=cores nwchem input.nw

How NWChem is compiled at NERSC

The nwchem modules at NERSC are compiled using the following script, or with minor variations. Users are welcome to use this script to compile their own versions of NWChem if the versions supplied by NERSC do not satisfy their needs.

Haswell

#!/bin/bash

set -ex

tarball="nwchem-7.0.0-release.revision-2c9a1c7c-src.2020-02-26.tar.bz2"
builddir=$(mktemp -d -p ${SCRATCH} build_nwchem_7.0.0_intel_19.1.0.166_hsw.XXXXXXXXXX)
builddir_short="/tmp/nwchem"

if [ ! -f ${tarball} ]; then
  wget https://github.com/nwchemgit/nwchem/releases/download/v7.0.0-release/${tarball}
fi
cp ${tarball} ${builddir}

# NWChem forbids the build directory length to contain > 64 chars during
# compilation, which is a constraint we often run into when using a combination
# of $SCRATCH and mktemp with several random chars. So we fool NWChem by making
# the build dir a symlink in /tmp, which has many fewer than 64 chars.
ln -s ${builddir} ${builddir_short}

cd ${builddir_short}
tar xjf nwchem-7.0.0-release.revision-2c9a1c7c-src.2020-02-26.tar.bz2
cd nwchem-7.0.0

NWCHEM_TOP="${PWD}"
NWCHEM_TARGET="LINUX64"
USE_MPI="y"
USE_MPIF="y"
USE_MPIF4="y"
LIBMPI=" "
MPI_LIB=" "
MPI_INCLUDE=" "
ARMCI_NETWORK="MPI-PR"
USE_SCALAPACK="y"
USE_NOFSCHECK="TRUE"
USE_NOIO="TRUE"
# We get the correct MKL link flags by using Intel's MKL Link Line Advisor web
# form: https://software.intel.com/en-us/articles/intel-mkl-link-line-advisor
SCALAPACK="-L${MKLROOT}/lib/intel64 -lmkl_scalapack_ilp64 -lmkl_intel_ilp64 -lmkl_core -lmkl_sequential -lmkl_blacs_intelmpi_ilp64 -lpthread -lm"
SCALAPACK_SIZE="8"
SCALAPACK_LIB=" "
BLAS_SIZE="8"
# Use the same Intel MKL Link Line Advisor flags here.
BLASOPT="-L${MKLROOT}/lib/intel64 -lmkl_intel_ilp64 -lmkl_core -lmkl_sequential -lmkl_core -liomp5 -lpthread -ldmapp -lm"
LAPACK_LIB=" "
NWCHEM_MODULES="all python"
MRCC_METHODS="TRUE"
PYTHONHOME="/usr"
PYTHONVERSION="2.7"
USE_PYTHONCONFIG="Y"
USE_OPENMP="1"
USE_F90_ALLOCATABLE="T"

cd src
module load gcc # need gfortran in $PATH or else `make nwchem_config` fails
make nwchem_config FC=ftn CC=cc
module unload gcc
make FC=ftn CC=cc 2>&1 | tee $HOME/build_nwchem_7.0.0_intel_hsw_$(date -I"seconds").log

KNL

#!/bin/bash

set -ex

module swap craype-{${CRAY_CPU_TARGET},mic-knl}

tarball="nwchem-7.0.0-release.revision-2c9a1c7c-src.2020-02-26.tar.bz2"
builddir=$(mktemp -d -p ${SCRATCH} build_nwchem_7.0.0_intel_19.1.0.166_hsw.XXXXXXXXXX)
builddir_short="/tmp/nwchem"

if [ ! -f ${tarball} ]; then
  wget https://github.com/nwchemgit/nwchem/releases/download/v7.0.0-release/${tarball}
fi
cp ${tarball} ${builddir}

# NWChem forbids the build directory length to contain > 64 chars during
# compilation, which is a constraint we often run into when using a combination
# of $SCRATCH and mktemp with several random chars. So we fool NWChem by making
# the build dir a symlink in /tmp, which has many fewer than 64 chars.
ln -s ${builddir} ${builddir_short}

cd ${builddir_short}
tar xjf nwchem-7.0.0-release.revision-2c9a1c7c-src.2020-02-26.tar.bz2
cd nwchem-7.0.0

NWCHEM_TOP="${PWD}"
NWCHEM_TARGET="LINUX64"
USE_MPI="y"
USE_MPIF="y"
USE_MPIF4="y"
LIBMPI=" "
MPI_LIB=" "
MPI_INCLUDE=" "
ARMCI_NETWORK="MPI-PR"
USE_SCALAPACK="y"
USE_NOFSCHECK="TRUE"
USE_NOIO="TRUE"
# We get the correct MKL link flags by using Intel's MKL Link Line Advisor web
# form: https://software.intel.com/en-us/articles/intel-mkl-link-line-advisor
SCALAPACK="-L${MKLROOT}/lib/intel64 -lmkl_scalapack_ilp64 -lmkl_intel_ilp64 -lmkl_core -lmkl_sequential -lmkl_blacs_intelmpi_ilp64 -lpthread -lm"
SCALAPACK_SIZE="8"
SCALAPACK_LIB=" "
BLAS_SIZE="8"
# Use the same Intel MKL Link Line Advisor flags here.
BLASOPT="-L${MKLROOT}/lib/intel64 -lmkl_intel_ilp64 -lmkl_core -lmkl_sequential -lmkl_core -liomp5 -lpthread -ldmapp -lm"
LAPACK_LIB=" "
NWCHEM_MODULES="all python"
MRCC_METHODS="TRUE"
PYTHONHOME="/usr"
PYTHONVERSION="2.7"
USE_PYTHONCONFIG="Y"
USE_OPENMP="1"
USE_F90_ALLOCATABLE="T"
USE_KNL="1"

cd src
module load gcc # need gfortran in $PATH or else `make nwchem_config` fails
make nwchem_config FC=ftn CC=cc
module unload gcc
make FC=ftn CC=cc 2>&1 | tee $HOME/build_nwchem_7.0.0_intel_knl_$(date -I"seconds").log