Skip to content

Cori Large Memory software

Many software packages available from Cori and other external compute nodes (e.g., Cori GPUs) work on the large memory nodes (aka cmem nodes), too, since they are built with x86 and x64 instruction sets. However, some packages, meant to be used with CUDA on GPUs or to be for a different interconnect fabric, don't work. In that case, we have built separate cmem-specific versions, and differentiate the module names from those for the existing ones, by appending -cmem to the regular names.

If you see module file names for a package, some with the trailing -cmem and others without, you should use one that ends with -cmem. For example, we show below all the llvm versions available, but only the one for llvm-cmem/10.0.1 works on the large memory nodes.

To use modules for the large memory nodes, you can module load cmem.

cmem$ module avail llvm
--------- /global/common/software/nersc/osuse15_cmem/extra_modulefiles ---------

------------- /global/common/software/nersc/cle7/extra_modulefiles -------------


There are several base compilers available.

CCE (Cray Compiler)

cmem$ module load PrgEnv-cray
cmem$ module rm darshan # requires incompatible cray-mpich 
cmem$ export CRAY_CPU_TARGET=x86-64   # set the CPU target to x86-64   


cmem$ module load gcc   

It is suggested to add the processor specific compiler option -march=znver1 for better performance codes.


cmem$ module load intel

For better performance codes, you can make use of the processor specific compiler option for better performances, by adding -xHOST, and make sure that you compile on a cmem node directly.


cmem$ module load cmem
cmem$ module load llvm-cmem


cmem$ module load pgi

It is suggested to use the processor specific compiler option by adding -tp=zen.


Open MPI

Open MPI is provided for the GCC, HPC SDK (formerly PGI), Intel, and CCE compilers, and is provided as the openmpi-cmem/4.0.3 and openmpi-cmem/4.0.5 modules. Users must use this particular version of the openmpi-cmem module.

One must first load a compiler module before loading a openmpi-cmem module, e.g.,

cmem$ module load cmem
cmem$ module load gcc
cmem$ module load openmpi-cmem

After the openmpi-cmem module is loaded, the MPI compiler wrappers will be available as mpicc, mpic++, and mpif90.


Python use on the largemem nodes is largely the same as on Cori. You can find general information about using Python at NERSC at our Python page.

Two main differences Python users should be aware of on the largemem nodes are mpi4py (documented below) and performance issues in libraries that use Intel's MKL like NumPy, SciPy, and scikit-learn. You can find more information about improving MKL performance on AMD CPUs at this Intel MKL on AMD Zen blog post. Alternatively, in NumPy for example, you can use OpenBLAS instead of MKL by installing NumPy from the conda-forge channel (more info at the NumPy installation page). You can also use the nomkl package documented on the Anaconda MKL optimization page.

Using Python mpi4py

Using Python's mpi4py on the Large Memory nodes requires an mpi4py built with Open MPI. This means that the mpi4py in our default Python module will not work on these nodes. It also means that any custom conda environments built with Cray MPICH (following our standard build recipe) will also not work on the Large Memory nodes.

We provide two options for users:

1) Build mpi4py against Open MPI in your own conda environment:

module load cmem
module load python
module swap PrgEnv-intel PrgEnv-gnu
module load openmpi-cmem

conda create -n mylargemem python=3.8 -y
source activate mylargemem

cd $HOME
tar zxvf mpi4py-3.0.3.tar.gz
cd mpi4py-3.0.3
python build
python install


2) Start with our pre-built mpi4py for the Large Memory nodes by cloning an environment:

module load python
conda create -n mylargemem --clone lazy-mpi4py-amd
source activate mylargemem

To run with Slurm:

srun -n 2 python

To run with Open MPI's mpirun:

module load cmem
module load openmpi-cmem
mpirun -n 2 python


cmem$ module load qchem-cmem


cmem$ module load vasp-cmem