Finding and using software on Perlmutter¶

The modules environment¶

Perlmutter's modules environment is similar to Cori's but using Lmod in place of the (earlier) Tcl modules used on Cori.

Lmod works in the same way as the Tcl environment modules familiar to NERSC users, and provides the same user interface and commands - module list, module avail, module load etc. All do the same as they do on Cori, except module avail does not list all hierarchy modules in Lmod.

Lmod also can use the same Tcl-based modulefiles as are used on Cori (though its native and preferred modulefile language is Lua).

Lmod adds some capabilities over Tcl modules, and these lead to some differences in behavior:

• module avail doesn't show everything, module spider shows the rest
• Loading a module can automatically unload a conflicting one
• A new shortcut command, ml

Module hierarchies¶

Lmod supports a module hierarchy, where sets of modules with a common dependency are "under" some required modules. Such module hierarchies exist in both HPE provided programming environments and in NERSC provided E4S software stack.

For example, the HPE provided cray-netcdf module depends on cray-hdf5 as its lower level IO support, and cray-netcdf-hdf5parallel module depends on cray-hdf5-parallel as parallel IO. Also those 200+ software E4S packages (such as gnuplot) installed via Spack by NERSC are only visible after loading the e4s module.

In Lmod, module avail command will show that cray-netcdf is not found, and the output looks the same as checking a truly-non-exist module:

perlmutter$ module avail cray-netcdf
No module(s) or extension(s) found!
Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".

perlmutter$ module avail gnuplot
No module(s) or extension(s) found!
...

perlmutter$ module avail aaa
No module(s) or extension(s) found!
...

Using module load is helpful since it gives the hint that aaa is truly not available and that cray-netcdf exists but can not be loaded directly. It also suggests using module spider to see how to load it:

perlmutter$ module load aaa
Lmod has detected the following error:  The following module(s) are unknown: "aaa"
...
perlmutter$ module load cray-netcdf
Lmod has detected the following error:  These module(s) or extension(s) exist but cannot be loaded as requested: "cray-netcdf"
  Try: "module spider cray-netcdf" to see how to load the module(s).

Using module spider gives the available versions, and suggests to use the spider command on a specific version, which then gives the list of all the dependency modules you need to load first (modules on any one of the lines given depending on your user environment choices such as the compiler version):

perlmutter$ module spider cray-netcdf
-------------------------------------------------------------------------------------
  cray-netcdf:
-------------------------------------------------------------------------------------
     Versions:
        cray-netcdf/4.8.1.1
     Other possible modules matches:
        cray-netcdf-hdf5parallel
-------------------------------------------------------------------------------------
  For detailed information about a specific "cray-netcdf" package (including how to load the modules) use the module's full name.
  Note that names that have a trailing (E) are extensions provided by other modules.
  For example:
     $ module spider cray-netcdf/4.8.1.1

perlmutter$ module spider cray-netcdf/4.8.1,1
------------------------------------------------------------------------------------
 cray-netcdf: cray-netcdf/4.8.1.1
------------------------------------------------------------------------------------
  You will need to load all module(s) on any one of the lines below before the "cray-netcdf/4.8.1.1" module is available to load.

   aocc/3.2.0 cray-hdf5/1.12.1.1
   gcc/10.3.0 cray-hdf5/1.12.1.1
   gcc/11.2.0 cray-hdf5/1.12.1.1
   nvhpc/21.11 cray-hdf5/1.12.1.1
   nvhpc/21.3 cray-hdf5/1.12.1.1
   nvidia/21.11 cray-hdf5/1.12.1.1
   nvidia/21.3 cray-hdf5/1.12.1.1
   nvidia/22.2 cray-hdf5/1.12.1.1

  Help:
   Release info: /opt/cray/pe/netcdf/4.8.1.1/release_info

Module families¶

Lmod has a concept of a "family", which can be thought of as a "niche": only one thing can occupy it at a time. As an example, the craype-hugepages* modules all belong to the "craype_hugepages" family. If you have craype-hugepages2M and run module load craype-hugepages16M, then craype-hugepages2M will be automatically unloaded first.

The ml command¶

Lmod provides an extra command called ml. ml can be used like a short-form of module for most module subcommands, and with no arguments will perform module list.

The Perlmutter software stack¶

GPU-ready applications¶

Please check the Popular applications section for information on the software available or to be available on Perlmutter.

Libraries¶

• MPI / mpi4py (CPU/GPU)
• GASNet (CPU)
• BLAS / LAPACK / ScaLAPACK (CPU/GPU)
• FFTW/cuFFT (CPU/GPU)
• HDF5 / h5py (CPU)
• NetCDF (CPU)
• Spark (CPU/GPU)
• R (CPU/GPU)
• Numpy (CPU/GPU)
• pyCUDA (CPU/GPU)
• Scipy (CPU/GPU)
• Scikit-Learn (CPU/GPU)
• Numba (CPU/GPU)
• TensorFlow (CPU/GPU)
• Keras (CPU/GPU)
• Caffe (CPU/GPU)
• pytorch (CPU/GPU)
• Hypre (CPU/GPU)
• SuperLU-dist (CPU/GPU)
• Sundials (CPU/GPU)
• SLATE (CPU/GPU)
• MAGMA (CPU/GPU)

Programming models¶

• MPI
• OpenMP
• OpenACC
• Kokkos
• Raja
• UPC

Tools¶

Tools supporting GPU development:

• TotalView
• DDT
• ATP
• CUDA-GDB
• CUDA-MEMcheck
• PAPI / cupti
• Nsight

CPU-based tools:

• Cmake
• Spack
• CrayPAT
• Reveal