Finding and using software on Perlmutter¶
The modules environment¶
Tip
Always use module spider
instead of module avail
on Perlmutter to check availability of a specific module.
Perlmutter's uses Lmod as its module environment to manage software.
Lmod works in the same way as the Tcl environment modules familiar to NERSC users, and provides the same user interface and commands - module list
, module avail
, module load
etc. All do the same as they do on Cori, except module avail
does not list all hierarchy modules in Lmod.
Lmod also can use the same Tcl-based modulefiles as are used on Cori (though its native and preferred modulefile language is Lua).
Lmod adds some capabilities over Tcl modules, and these lead to some differences in behavior:
module avail
doesn't show everything,module spider
shows the rest- Loading a module can automatically unload a conflicting one
- A new shortcut command,
ml
Module hierarchies¶
Lmod supports a module hierarchy, where sets of modules with a common dependency are "under" some required modules. Such module hierarchies exist in both HPE provided programming environments and in NERSC provided E4S software stack.
For example, the HPE provided cray-netcdf
module depends on cray-hdf5
as its lower level IO support, and cray-netcdf-hdf5parallel
module depends on cray-hdf5-parallel
as parallel IO. Also those 200+ software E4S packages (such as gnuplot
) installed via Spack by NERSC are only visible after loading the e4s
module.
In Lmod, module avail
command will show that cray-netcdf
is not found, and the output looks the same as checking a truly-non-exist module:
perlmutter$ module avail cray-netcdf
No module(s) or extension(s) found!
Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".
perlmutter$ module avail gnuplot
No module(s) or extension(s) found!
...
perlmutter$ module avail aaa
No module(s) or extension(s) found!
...
Using module load
is helpful since it gives the hint that aaa
is truly not available and that cray-netcdf
exists but can not be loaded directly. It also suggests using module spider
to see how to load it:
perlmutter$ module load aaa
Lmod has detected the following error: The following module(s) are unknown: "aaa"
...
perlmutter$ module load cray-netcdf
Lmod has detected the following error: These module(s) or extension(s) exist but cannot be loaded as requested: "cray-netcdf"
Try: "module spider cray-netcdf" to see how to load the module(s).
Using module spider
gives the available versions, and suggests to use the spider command on a specific version, which then gives the list of all the dependency modules you need to load first (modules on any one of the lines given depending on your user environment choices such as the compiler version):
Tip
Always use module spider
instead of module avail
to find out how to module load
.
perlmutter$ module spider cray-netcdf
-------------------------------------------------------------------------------------
cray-netcdf:
-------------------------------------------------------------------------------------
Versions:
cray-netcdf/4.8.1.1
Other possible modules matches:
cray-netcdf-hdf5parallel
-------------------------------------------------------------------------------------
For detailed information about a specific "cray-netcdf" package (including how to load the modules) use the module's full name.
Note that names that have a trailing (E) are extensions provided by other modules.
For example:
$ module spider cray-netcdf/4.8.1.1
perlmutter$ module spider cray-netcdf/4.8.1,1
------------------------------------------------------------------------------------
cray-netcdf: cray-netcdf/4.8.1.1
------------------------------------------------------------------------------------
You will need to load all module(s) on any one of the lines below before the "cray-netcdf/4.8.1.1" module is available to load.
aocc/3.2.0 cray-hdf5/1.12.1.1
gcc/10.3.0 cray-hdf5/1.12.1.1
gcc/11.2.0 cray-hdf5/1.12.1.1
nvhpc/21.11 cray-hdf5/1.12.1.1
nvhpc/21.3 cray-hdf5/1.12.1.1
nvidia/21.11 cray-hdf5/1.12.1.1
nvidia/21.3 cray-hdf5/1.12.1.1
nvidia/22.2 cray-hdf5/1.12.1.1
Help:
Release info: /opt/cray/pe/netcdf/4.8.1.1/release_info
Module families¶
Lmod has a concept of a "family", which can be thought of as a "niche": only one thing can occupy it at a time. As an example, the craype-hugepages*
modules all belong to the "craype_hugepages" family. If you have craype-hugepages2M
and run module load craype-hugepages16M
, then craype-hugepages2M
will be automatically unloaded first.
The ml
command¶
Lmod provides an extra command called ml. ml
can be used like a short-form of module
for most module subcommands, and with no arguments will perform module list
.
The Perlmutter software stack¶
Warning
This is a tentative estimate of software we anticipate will be GPU-ready within the early-access period for Perlmutter. Availability of software may be delayed and our ability to support these packages may be limited.
GPU-ready applications¶
Please check the Popular applications section for information on the software available or to be available on Perlmutter.
Libraries¶
Note
Notes for using these libraries with GPUs are still under development.
- MPI / mpi4py (CPU/GPU)
- GASNet (CPU)
- BLAS / LAPACK / ScaLAPACK (CPU/GPU)
- FFTW/cuFFT (CPU/GPU)
- HDF5 / h5py (CPU)
- NetCDF (CPU)
- Spark (CPU/GPU)
- R (CPU/GPU)
- Numpy (CPU/GPU)
- pyCUDA (CPU/GPU)
- Scipy (CPU/GPU)
- Scikit-Learn (CPU/GPU)
- Numba (CPU/GPU)
- TensorFlow (CPU/GPU)
- Keras (CPU/GPU)
- Caffe (CPU/GPU)
- pytorch (CPU/GPU)
- Hypre (CPU/GPU)
- SuperLU-dist (CPU/GPU)
- Sundials (CPU/GPU)
- SLATE (CPU/GPU)
- MAGMA (CPU/GPU)
Programming models¶
Tools¶
Tools supporting GPU development:
Note
Notes for using these tools with GPUs are still under development.
CPU-based tools: