Skip to content

Finding and using software on Perlmutter

The modules environment

Tip

Always use module spider instead of module avail on Perlmutter to check availability of a specific module.

Perlmutter's uses Lmod as its module environment to manage software.

Lmod works in the same way as the Tcl environment modules familiar to NERSC users, and provides the same user interface and commands - module list, module avail, module load etc. Lmod can also use Tcl-based modulefiles, though its native and preferred modulefile language is Lua.

Lmod adds some capabilities over Tcl modules, and these lead to some differences in behavior:

Module hierarchies

Lmod supports a module hierarchy, where sets of modules with a common dependency are "under" some required modules. Such module hierarchies exist in both HPE provided programming environments and in NERSC provided E4S software stack.

For example, the HPE provided cray-netcdf module depends on cray-hdf5 as its lower level IO support, and cray-netcdf-hdf5parallel module depends on cray-hdf5-parallel as parallel IO. Also those 200+ software E4S packages (such as gnuplot) installed via Spack by NERSC are only visible after loading the e4s module.

In Lmod, module avail command will show that cray-netcdf is not found, and the output looks the same as checking a truly-non-exist module:

perlmutter$ module avail cray-netcdf
No module(s) or extension(s) found!
Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".

perlmutter$ module avail gnuplot
No module(s) or extension(s) found!
...

perlmutter$ module avail aaa
No module(s) or extension(s) found!
...

Using module load is helpful since it gives the hint that aaa is truly not available and that cray-netcdf exists but can not be loaded directly. It also suggests using module spider to see how to load it:

perlmutter$ module load aaa
Lmod has detected the following error:  The following module(s) are unknown: "aaa"
...
perlmutter$ module load cray-netcdf
Lmod has detected the following error:  These module(s) or extension(s) exist but cannot be loaded as requested: "cray-netcdf"
  Try: "module spider cray-netcdf" to see how to load the module(s).

Using module spider gives the available versions, and suggests to use the spider command on a specific version, which then gives the list of all the dependency modules you need to load first (modules on any one of the lines given depending on your user environment choices such as the compiler version):

Tip

Always use module spider instead of module avail to find out how to module load.

perlmutter$ module spider cray-netcdf
-------------------------------------------------------------------------------------
  cray-netcdf:
-------------------------------------------------------------------------------------
     Versions:
        cray-netcdf/4.8.1.1
     Other possible modules matches:
        cray-netcdf-hdf5parallel
-------------------------------------------------------------------------------------
  For detailed information about a specific "cray-netcdf" package (including how to load the modules) use the module's full name.
  Note that names that have a trailing (E) are extensions provided by other modules.
  For example:
     $ module spider cray-netcdf/4.8.1.1

perlmutter$ module spider cray-netcdf/4.8.1,1
------------------------------------------------------------------------------------
 cray-netcdf: cray-netcdf/4.8.1.1
------------------------------------------------------------------------------------
  You will need to load all module(s) on any one of the lines below before the "cray-netcdf/4.8.1.1" module is available to load.

   aocc/3.2.0 cray-hdf5/1.12.1.1
   gcc/10.3.0 cray-hdf5/1.12.1.1
   gcc/11.2.0 cray-hdf5/1.12.1.1
   nvhpc/21.11 cray-hdf5/1.12.1.1
   nvhpc/21.3 cray-hdf5/1.12.1.1
   nvidia/21.11 cray-hdf5/1.12.1.1
   nvidia/21.3 cray-hdf5/1.12.1.1
   nvidia/22.2 cray-hdf5/1.12.1.1

  Help:
   Release info: /opt/cray/pe/netcdf/4.8.1.1/release_info

Module families

Lmod has a concept of a "family", which can be thought of as a "niche": only one thing can occupy it at a time. As an example, the craype-hugepages* modules all belong to the "craype_hugepages" family. If you have craype-hugepages2M and run module load craype-hugepages16M, then craype-hugepages2M will be automatically unloaded first.

The ml command

Lmod provides an extra command called ml. ml can be used like a short-form of module for most module subcommands, and with no arguments will perform module list.

User Contributed Modules

Several projects provide software via their own user-maintained modules for the benefit of NERSC users. To access these modules you must first load the contrib module, as shown:

module load contrib

For example, after running the above command, module avail, will show two additional locations:

------ /global/common/software/m2878/perlmutter/modulefiles ------
   bupc-narrow/2022.10.0           upcxx-extras/master   (D)
   bupc/2022.10.0                  upcxx-extras/2022.3.0
   opencoarrays/2.10.1             upcxx/bleeding-edge
   upcxx-cuda/bleeding-edge        upcxx/nightly
   upcxx-cuda/nightly              upcxx/2022.9.0
   upcxx-cuda/2022.9.0             upcxx/2023.3.0        (D)
   upcxx-cuda/2023.3.0      (D)

-------- /global/common/software/m4232pub/pm/modulefiles ---------
   wrf/4.4.0    wrf/4.5.0 (D)

The first project directory, lists 13 modules provided by the m2878 (UPCXX++ and GASNet-EX) project. The second directory contains two modules provided by m4232 (WRF). Any of these modules can now be loaded as normal.

The Perlmutter software stack

Warning

This is a tentative estimate of software we anticipate will be GPU-ready within the early-access period for Perlmutter. Availability of software may be delayed and our ability to support these packages may be limited.

GPU-ready applications

Please check the Popular applications section for information on the software available or to be available on Perlmutter.

Libraries

Note

Notes for using these libraries with GPUs are still under development.

  • MPI / mpi4py (CPU/GPU)
  • GASNet (CPU)
  • BLAS / LAPACK / ScaLAPACK (CPU/GPU)
  • FFTW/cuFFT (CPU/GPU)
  • HDF5 / h5py (CPU)
  • NetCDF (CPU)
  • Spark (CPU/GPU)
  • R (CPU/GPU)
  • Numpy (CPU/GPU)
  • pyCUDA (CPU/GPU)
  • Scipy (CPU/GPU)
  • Scikit-Learn (CPU/GPU)
  • Numba (CPU/GPU)
  • TensorFlow (CPU/GPU)
  • Keras (CPU/GPU)
  • Caffe (CPU/GPU)
  • pytorch (CPU/GPU)
  • Hypre (CPU/GPU)
  • SuperLU-dist (CPU/GPU)
  • Sundials (CPU/GPU)
  • SLATE (CPU/GPU)
  • MAGMA (CPU/GPU)

Programming models

  • MPI
  • OpenMP
  • OpenACC
  • Kokkos
  • HPX
  • UPC++
  • UPC

Tools

Tools supporting GPU development:

Note

Notes for using these tools with GPUs are still under development.

CPU-based tools: