Skip to content

Lmod

Perlmutter uses Lmod as its module environment to manage software.

Lmod is a Lua-based module system that helps manage the user environment (PATH, LD_LIBRARY_PATH, etc.) through modulefiles. Lmod is an extension of environment-modules that supports Tcl modules along with hierarchical MODULEPATH.

About module

module is a bash function read by $LMOD_CMD which points to the Lmod command that reads a modulefile and evaluates each modulefile using the eval command:

elvis@perlmutter> type module
module is a function
module () 
{ 
    eval $($LMOD_CMD bash "$@") && eval $(${LMOD_SETTARG_CMD:-:} -s sh)
}

Command Summary

Command Description
module list List active modules in the user environment
module av [module] List available modules in MODULEPATH
module spider [module] Query all modules in MODULEPATH and any module hierarchy
module overview [module] List all modules with a count of each module
module load [module] Load a modulefile in the user environment
module unload [module] Remove a loaded module from the user environment
module purge Remove all modules from the user environment
module swap [module1] [module2] Replace module1 with module2
module show [module] Show content of commands performed by loading module
module --raw show [module] Show raw content of modulefile
module help [module] Show help for a given module
module whatis [module] A brief description of the module, generally single line
module savelist List all user collections
module save [collection] Save active modules in a user collection
module describe [collection] Show content of user collection
module restore [collection] Load modules from a collection
module disable [collection] Disable a user collection
module --config Show Lmod configuration
module use [-a] [path] Prepend or append path to MODULEPATH
module unuse [path] Remove path from MODULEPATH
module --show_hidden av Show all available modules in MODULEPATH including hidden modules
module --show_hidden spider Show all possible modules in MODULEPATH and module hierarchy including hidden modules

ml

Lmod provides another convenient shortcut command for the module command called ml for user convenience, which mimics the module command.

Invoking the ml command without arguments is equivalent to module list; any sub-commands to module are usable with the ml command. For example, ml avail is equivalent to module avail; ml spider is equivalent to module spider, etc.

The syntax for loading and unloading modules with the ml command differs from the module command. You can type ml <module> to load a module, and unload by prepending a minus sign (-): ml -<module>.

For instance if we want to load the gcc module we can run ml gcc. To remove this module from the set of loaded modules we would run ml -gcc. 

elvis@perlmutter> ml gcc
elvis@perlmutter> ml

Currently Loaded Modules:
  1) gcc/10.2.0 (c)

  Where:
   c:  Compiler

One can load and unload module in a single ml command: for instance if you want to unload gcc and load cuda you could run: ml -gcc cuda or ml cuda -gcc.

Finding Modules

Lmod provides several commands to help you find modules including module avail, module spider and even module overview. We will discuss a few of these commands here.

To see list of available modules, you can run 

module avail

The output of this will be quite long depending on the number of modulefiles, for instance if you want to see all gcc modules you can run 

module avail gcc

The module spider command reports all modules in your system in MODULEPATH along with all module trees in the hierarchical system. Note that module avail doesn't show modules from all trees in the hierarchical system. If you want to know all available software on the system, please use module spider.

The output will be a list of software entries with corresponding versions:

elvis@perlmutter> module spider

------------------------------------------------------------------------------------------------------------------------------------------------------------------------
The following is a list of the modules and extensions currently available:
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  aocc: aocc/2.2.0.1

  atp: atp/3.13.1

  cce: cce/11.0.4

  clingo: clingo/git_20210514

  cmake: cmake/3.18.4

  cpe: cpe/21.04

  cray-ccdb: cray-ccdb/4.11.1

  cray-cti: cray-cti/2.13.6

  cray-dsmml: cray-dsmml/0.1.4

  cray-fftw: cray-fftw/3.3.8.9

  cray-hdf5: cray-hdf5/1.12.0.3

  ...

The module spider can report all versions of the software. For instance if we want to see all gcc compilers we can run the following. In this example we have three versions of gcc (8.1.0, 9.3.0, and 10.2.0):

elvis@perlmutter> module spider gcc

------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  gcc:
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
     Versions:
        gcc/8.1.0
        gcc/9.3.0
        gcc/10.2.0

------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  For detailed information about a specific "gcc" package (including how to load the modules) use the module's full name.
  Note that names that have a trailing (E) are extensions provided by other modules.
  For example:

     $ module spider gcc/10.2.0
------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Lmod recently introduced a new command called module overview which displays each module in short form with a number of module versions. Shown below is a preview of module overview

elvis@perlmutter> module overview

------------------------------------------------------------------------------------------------------------------ /opt/cray/pe/lmod/lmod/modulefiles/Core -------------------------------------------------------------------------------------------------------------------
lmod (1)   settarg (1)

--------------------------------------------------------------------------------------------------------------------- /opt/cray/pe/lmod/modulefiles/core ---------------------------------------------------------------------------------------------------------------------
aocc (1)   cpe-cuda  (2)   cray-cti     (4)   cray-libpals (2)   cray-pals   (2)   cray-R    (1)   craypkg-gen (2)   gdb4hpc (2)   nvidia         (5)   PrgEnv-aocc (1)   PrgEnv-nvhpc   (1)   valgrind4hpc (2)
atp  (2)   cpe       (2)   cray-dsmml   (1)   cray-libsci  (2)   cray-pmi    (2)   cray-stat (1)   cudatoolkit (3)   iobuf   (1)   papi           (2)   PrgEnv-cray (1)   PrgEnv-nvidia  (1)
cce  (2)   cray-ccdb (1)   cray-dyninst (1)   cray-mrnet   (1)   cray-python (1)   craype    (1)   gcc         (4)   nvhpc   (5)   perftools-base (2)   PrgEnv-gnu  (1)   sanitizers4hpc (2)

------------------------------------------------------------------------------------------------------------ /opt/cray/pe/lmod/modulefiles/craype-targets/default ------------------------------------------------------------------------------------------------------------
craype-accel-amd-gfx908 (1)   craype-accel-nvidia70 (1)   craype-hugepages128M (1)   craype-hugepages256M (1)   craype-hugepages32M  (1)   craype-hugepages64M (1)   craype-network-ofi (1)   craype-x86-milan-x (1)   craype-x86-spr-hbm (1)
craype-accel-amd-gfx90a (1)   craype-accel-nvidia80 (1)   craype-hugepages16M  (1)   craype-hugepages2G   (1)   craype-hugepages4M   (1)   craype-hugepages8M  (1)   craype-network-ucx (1)   craype-x86-milan   (1)   craype-x86-spr     (1)
craype-accel-host       (1)   craype-arm-grace      (1)   craype-hugepages1G   (1)   craype-hugepages2M   (1)   craype-hugepages512M (1)   craype-network-none (1)   craype-x86-genoa   (1)   craype-x86-rome    (1)   craype-x86-trento  (1)

-------------------------------------------------------------------------------------------------------- /global/common/software/nersc/pm-2022.12.0/extra_modulefiles --------------------------------------------------------------------------------------------------------
allinea-forge (3)   cmake   (2)   cudnn    (3)   evp-patch    (1)   gpu     (1)   idl   (1)    lammps      (1)   mongodb (2)   nccl           (3)   nvidia     (1)   pytorch (4)   taskfarmer (1)   valgrind (2)
amber         (2)   codee   (3)   darshan  (4)   fast-mkl-amd (1)   gromacs (3)   jamo  (2)    llvm        (2)   mumps   (1)   Nsight-Compute (1)   parallel   (1)   qchem   (2)   tensorflow (2)   vasp-tpc (4)
arm-forge     (3)   contrib (1)   e4s      (2)   globus-tools (1)   gsl     (1)   jgi   (1)    mathematica (1)   mvasp   (1)   Nsight-Systems (1)   pwanalyzer (2)   spack   (4)   totalview  (3)   vasp     (5)
berkeleygw    (2)   cpu     (1)   espresso (3)   gpu-test     (1)   hip     (1)   julia (27)   matlab      (2)   namd    (3)   nvidia-mixed   (1)   python     (1)   spin    (1)   training   (1)   xalt     (1)

------------------------------------------------------------------------------------------------------------------- /usr/share/lmod/lmod/modulefiles/Core --------------------------------------------------------------------------------------------------------------------
lmod (1)   settarg (1)

--------------------------------------------------------------------------------------------------------------------------- /opt/cray/modulefiles ----------------------------------------------------------------------------------------------------------------------------
chapel (1)   cray-lustre-client-ofed (1)   dvs (1)   libfabric (1)   xpmem (1)

If you want to filter this output by the name of module you can pass the name as an argument. For example, if you want to know the number of modules named PrgEnv-gnu and PrgEnv-cray you can run:

module overview PrgEnv-cray PrgEnv-gnu

Module hierarchies

Lmod supports a module hierarchy, where sets of modules with a common dependency are "under" some required modules. Such module hierarchies exist in both HPE-provided programming environments and in the NERSC-provided E4S software stack.

For example, the HPE-provided cray-netcdf module depends on cray-hdf5 as its lower-level I/O support, and the cray-netcdf-hdf5parallel module depends on cray-hdf5-parallel as parallel I/O. Additionally, the 200+ E4S packages (such as gnuplot) installed via Spack by NERSC are only visible after loading the e4s module.

In Lmod, the module avail command will show that cray-netcdf is not found, and the output looks the same as checking a module which truly does not exist:

perlmutter$ module avail cray-netcdf
No module(s) or extension(s) found!
Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".

perlmutter$ module avail gnuplot
No module(s) or extension(s) found!
...

perlmutter$ module avail aaa
No module(s) or extension(s) found!
...

Using module load is helpful here since it gives the hint that aaa is truly not available but that cray-netcdf exists but can not be loaded directly. It also suggests using module spider to see how to load it:

perlmutter$ module load aaa
Lmod has detected the following error:  The following module(s) are unknown: "aaa"
...
perlmutter$ module load cray-netcdf
Lmod has detected the following error:  These module(s) or extension(s) exist but cannot be loaded as requested: "cray-netcdf"
  Try: "module spider cray-netcdf" to see how to load the module(s).

Using module spider gives the available versions, and suggests to use the spider command on a specific version, which then gives the list of all the dependency modules you need to load first (modules on any one of the lines given depending on your user environment choices such as the compiler version):

Tip

Always use module spider instead of module avail to find out how to module load.

User-Contributed Modules

Several projects provide software via their own user-maintained modules for the benefit of NERSC users. To access these modules you must first load the contrib module, as shown:

module load contrib

For example, after running the above command, module avail, will show two additional locations:

------ /global/common/software/m2878/perlmutter/modulefiles ------
   bupc-narrow/2022.10.0           upcxx-extras/master   (D)
   bupc/2022.10.0                  upcxx-extras/2022.3.0
   opencoarrays/2.10.1             upcxx/bleeding-edge
   upcxx-cuda/bleeding-edge        upcxx/nightly
   upcxx-cuda/nightly              upcxx/2022.9.0
   upcxx-cuda/2022.9.0             upcxx/2023.3.0        (D)
   upcxx-cuda/2023.3.0      (D)

-------- /global/common/software/m4232pub/pm/modulefiles ---------
   wrf/4.4.0    wrf/4.5.0 (D)

The first project directory lists 13 modules provided by the m2878 (UPCXX++ and GASNet-EX) project. The second directory contains two modules provided by m4232 (WRF). Any of these modules can now be loaded as usual.

User Collections

Lmod introduced the concept of user collections, allowing a user to reference a group of modules with a unique name. This is particularly useful if there is a set of modules you load regularly to do a particular task. You can save the modules into a collection to save the trouble of retyping them every time, loading them with the module restore command. Note that Lmod can load only one user collection at a time.

To save modules in a collection, load the relevant modules and run module save. In this example, we save our active modules into the default collection. 

elvis@perlmutter> module purge
elvis@perlmutter> module load gcc
elvis@perlmutter> module list

Currently Loaded Modules:
  1) gcc/10.2.0 (c)

  Where:
   c:  Compiler



elvis@perlmutter> module save
Saved current collection of modules to: "default", for system: "perlmutter"

Lmod will store user collection in $HOME/.lmod.d. You can view all collections using module savelist.

elvis@perlmutter> module savelist
Named collection list (For LMOD_SYSTEM_NAME = "perlmutter"):
  1) default

You can see contents of a collection using module describe, which shows the modules that will be loaded when restoring from the collection. In this next example, we will purge and restore from the collection, which will simply load the gcc module:

elvis@perlmutter> module describe
Collection "default" contains: 
   1) gcc
elvis@perlmutter> module purge
elvis@perlmutter> module list
No modules loaded
elvis@perlmutter> module restore 
Restoring modules from user's default, for system: "perlmutter"
elvis@perlmutter> module list

Currently Loaded Modules:
  1) gcc/10.2.0 (c)

  Where:
   c:  Compiler

Showing the Contents of Modulefiles

There are several commands to reveal the contents of modulefiles: modulewhatis, module help, module show, and module spider.

The module whatis is a single line summary of modulefile and module help is a multi-line description of the modulefile.

The module show command displays the commands executed when loading the module (module load). Shown below are commands run in your user shell when loading PrgEnv-nvidia module.

Note

The output of module show is not the content of the modulefile. If you want to see the content of the modulefile, add the --raw option: module --raw show PrgEnv-nvidia.

elvis@perlmutter> module show PrgEnv-nvidia
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
   /opt/cray/pe/lmod/modulefiles/core/PrgEnv-nvidia/8.0.0.lua:
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
family("PrgEnv")
help([[
Use 'module restore PrgEnv-nvidia' to load the default nvidia programming environment.

]])
whatis("Enables the Programming Environment using the nvidia compilers.")
setenv("PE_ENV","NVIDIA")
load("nvidia")
load("craype")
load("craype-x86-rome")
load("craype-network-ofi")
load("cray-dsmml")
load("perftools-base")
load("xpmem")
load("cray-mpich")
load("cray-libsci")

Lmod Configuration for Perlmutter

We have made a few changes to Lmod configuration which are described below.

Note

Lmod configuration files can be found at $LMOD_CONFIG_DIR. If you want to view Lmod configuration you can run module --config. Please see https://lmod.readthedocs.io/en/latest/090_configuring_lmod.html for an overview of Lmod configuration values and their description.

NERSC Defaults

NERSC provides gpu and cpu modules that load recommended defaults for compiling on GPUs and CPUs, respectively. We may update these modules as new information or recommendations are made. You can use module show <gpu/cpu> to see what each module does. You can only have one of these modules loaded at a time and loading one unloads the other. Currently, the gpu module is loaded by default.

Format Titles for Module Trees

We have set the environment LMOD_AVAIL_STYLE="grouped:system" which alters the output of module avail. Under this setting, Lmod displays titles chosen by NERSC for each module tree, rather than the directory path as its default behavior.

When you invoke module avail you will see a title for each module tree.

module avail with grouped style 
elvis@perlmutter> module av

----------------------------------------------------------------- Cray Compiler/Network Dependent Packages -----------------------------------------------------------------
   cray-mpich-abi/8.1.4    cray-mpich/8.1.4

--------------------------------------------------------------------- Cray Compiler Dependent Modules ----------------------------------------------------------------------
   cray-hdf5/1.12.0.3

---------------------------------------------------------------------- Additional available software -----------------------------------------------------------------------
   clingo/git_20210514    darshan/3.2.1       mpich/3.4.1        npe-llvm/0.1              py-cffi/1.14.3                 spack/0.16.1
   cmake/3.18.4           fast-mkl-amd/1.0    nccl/2.9.6         nvidia-nersc/20.11        python/3.8-anaconda-2020.11    xalt/2.10.2
   cudnn/8.2.0            llvm/11.0.1         nccl/2.9.8  (D)    nvidia-nersc/21.5  (D)    pytorch/1.8.0

----------------------------------------------------------------------------- Lmod Modulefiles -----------------------------------------------------------------------------
   lmod    settarg

------------------------------------------------------------------------ Cray MPI Dependent Modules ------------------------------------------------------------------------
   cray-hdf5-parallel/1.12.0.3    cray-hdf5-parallel/1.12.0.3 (D)    cray-parallel-netcdf/1.12.1.3 (D)    cray-parallel-netcdf/1.12.1.3

---------------------------------------------------------------------------- Cray Core Modules -----------------------------------------------------------------------------
   aocc/2.2.0.1     (D)    cray-dsmml/0.1.4         cray-R/4.0.3.0        (D)    gcc/10.2.0              (L,D)    nvidia/20.9            (D)    PrgEnv-nvidia/8.0.0
   atp/3.13.1              cray-jemalloc/5.1.0.4    cray-stat/4.10.1             gdb4hpc/4.12.5                   papi/6.0.0.6                  valgrind4hpc/2.11.1
   cce/11.0.4              cray-libsci/21.04.1.1    craype/2.7.6                 iobuf/2.0.10                     perftools-base/21.02.0
   cpe/21.04               cray-pmi-lib/6.0.10      craypkg-gen/1.3.14           nvhpc-byo-compiler/20.9 (D)      PrgEnv-aocc/8.0.0
   cray-ccdb/4.11.1        cray-pmi/6.0.10          cudatoolkit/20.9_11.0 (D)    nvhpc-nompi/20.9        (D)      PrgEnv-cray/8.0.0
   cray-cti/2.13.6         cray-python/3.8.5.0      gcc/9.3.0                    nvhpc/20.9              (D)      PrgEnv-gnu/8.0.0

----------------------------------------------------------- /opt/cray/pe/lmod/modulefiles/craype-targets/default -----------------------------------------------------------
   craype-accel-amd-gfx908    craype-accel-nvidia80    craype-hugepages1G      craype-hugepages2M     craype-hugepages512M    craype-network-none    craype-x86-milan
   craype-accel-host          craype-hugepages128M     craype-hugepages256M    craype-hugepages32M    craype-hugepages64M     craype-network-ofi     craype-x86-rome
   craype-accel-nvidia70      craype-hugepages16M      craype-hugepages2G      craype-hugepages4M     craype-hugepages8M      craype-network-ucx

------------------------------------------------------------------------------- Cray Modules -------------------------------------------------------------------------------
   cray-ucx/2.7.0-1    cudatoolkit/20.9_11.0    libfabric/1.11.0.3.66    nvhpc-byo-compiler/20.9    nvhpc-nompi/20.9    nvhpc/20.9

------------------------------------------------------------------------------ Cray Compilers ------------------------------------------------------------------------------
   aocc/2.2.0.1    cray-R/4.0.3.0    gcc/8.1.0    gcc/9.3.0    gcc/10.2.0    nvidia/20.9

------------------------------------------------------------------------- NERSC-provided Software --------------------------------------------------------------------------
   Default

  Where:
   L:  Module is loaded
   D:  Default Module

Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".

If you want to see the full path to module tree you can use the -s <system> option, which changes the Lmod avail style to prefer system instead of grouped. To make this change persistent, set the environment variable LMOD_AVAIL_STYLE="system:grouped" Shown below is an output using the system avail format.

module avail with system style 
elvis@perlmutter> module -s system av

----------------------------------------------------------- /opt/cray/pe/lmod/modulefiles/comnet/gnu/8.0/ofi/1.0 -----------------------------------------------------------
   cray-mpich-abi/8.1.4    cray-mpich/8.1.4

-------------------------------------------------------------- /opt/cray/pe/lmod/modulefiles/compiler/gnu/8.0 --------------------------------------------------------------
   cray-hdf5/1.12.0.3

-------------------------------------------------------- /global/common/software/nersc/shasta2105/extra_modulefiles --------------------------------------------------------
   clingo/git_20210514    darshan/3.2.1       mpich/3.4.1        npe-llvm/0.1              py-cffi/1.14.3                 spack/0.16.1
   cmake/3.18.4           fast-mkl-amd/1.0    nccl/2.9.6         nvidia-nersc/20.11        python/3.8-anaconda-2020.11    xalt/2.10.2
   cudnn/8.2.0            llvm/11.0.1         nccl/2.9.8  (D)    nvidia-nersc/21.5  (D)    pytorch/1.8.0

------------------------------------------------------------------ /usr/share/lmod/lmod/modulefiles/Core -------------------------------------------------------------------
   lmod    settarg

---------------------------------------------------- /opt/cray/pe/lmod/modulefiles/mpi/nvidia/20/ofi/1.0/cray-mpich/8.0 ----------------------------------------------------
   cray-hdf5-parallel/1.12.0.3    cray-parallel-netcdf/1.12.1.3 (D)

------------------------------------------------- /opt/cray/pe/lmod/modulefiles/mpi/crayclang/10.0/ofi/1.0/cray-mpich/8.0 --------------------------------------------------
   cray-hdf5-parallel/1.12.0.3 (D)    cray-parallel-netcdf/1.12.1.3

-------------------------------------------------------------------- /opt/cray/pe/lmod/modulefiles/core --------------------------------------------------------------------
   aocc/2.2.0.1     (D)    cray-dsmml/0.1.4         cray-R/4.0.3.0        (D)    gcc/10.2.0              (L,D)    nvidia/20.9            (D)    PrgEnv-nvidia/8.0.0
   atp/3.13.1              cray-jemalloc/5.1.0.4    cray-stat/4.10.1             gdb4hpc/4.12.5                   papi/6.0.0.6                  valgrind4hpc/2.11.1
   cce/11.0.4              cray-libsci/21.04.1.1    craype/2.7.6                 iobuf/2.0.10                     perftools-base/21.02.0
   cpe/21.04               cray-pmi-lib/6.0.10      craypkg-gen/1.3.14           nvhpc-byo-compiler/20.9 (D)      PrgEnv-aocc/8.0.0
   cray-ccdb/4.11.1        cray-pmi/6.0.10          cudatoolkit/20.9_11.0 (D)    nvhpc-nompi/20.9        (D)      PrgEnv-cray/8.0.0
   cray-cti/2.13.6         cray-python/3.8.5.0      gcc/9.3.0                    nvhpc/20.9              (D)      PrgEnv-gnu/8.0.0

----------------------------------------------------------- /opt/cray/pe/lmod/modulefiles/craype-targets/default -----------------------------------------------------------
   craype-accel-amd-gfx908    craype-accel-nvidia80    craype-hugepages1G      craype-hugepages2M     craype-hugepages512M    craype-network-none    craype-x86-milan
   craype-accel-host          craype-hugepages128M     craype-hugepages256M    craype-hugepages32M    craype-hugepages64M     craype-network-ofi     craype-x86-rome
   craype-accel-nvidia70      craype-hugepages16M      craype-hugepages2G      craype-hugepages4M     craype-hugepages8M      craype-network-ucx

-------------------------------------------------------------------------- /opt/cray/modulefiles ---------------------------------------------------------------------------
   cray-ucx/2.7.0-1    cudatoolkit/20.9_11.0    libfabric/1.11.0.3.66    nvhpc-byo-compiler/20.9    nvhpc-nompi/20.9    nvhpc/20.9

----------------------------------------------------------------------------- /opt/modulefiles -----------------------------------------------------------------------------
   aocc/2.2.0.1    cray-R/4.0.3.0    gcc/8.1.0    gcc/9.3.0    gcc/10.2.0    nvidia/20.9

----------------------------------------------------------- /global/common/software/nersc/shasta2105/modulefiles -----------------------------------------------------------
   Default

  Where:
   L:  Module is loaded
   D:  Default Module

Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".

Lmod Families

The Lmod family("name") Lua function enables the concept of families to be enforced.

The basic idea is that modules belong to a family group, and only one module from the family group can be loaded at a given time. The purpose is to protect users from loading conflicting modules, e.g., two MPI libraries such as openmpi and mpich that provide the same binaries mpicc, mpifort, mpic++.

HPE/Cray has defined several families in their software stack. The compiler family contains the gcc, cce, aocc and nvidia modules. Lmod will automatically swap modules when you load modules from same family group. Shown below, we see that Lmod swaps nvidia with aocc and aocc with gcc since Lmod permits only one module from the compiler family to be loaded at a time.

elvis@perlmutter> module load aocc

Lmod is automatically replacing "nvidia/20.9" with "aocc/2.2.0.1".

elvis@perlmutter> module load gcc

Lmod is automatically replacing "aocc/2.2.0.1" with "gcc/10.2.0".

A quick way to check the active families is to run module --mt, which shows the module tables state along with family names. In the below output we see there are five families defined: PrgEnv, compiler, craype, craype_cpu, and craype_network.

elvis@perlmutter> module --mt
_ModuleTable_ = {
  ["MTversion"] = 3,
  ["c_rebuildTime"] = 7200.0,
  ["c_shortTime"] = 0.53126287460327,
  depthT = {},
  family = {
    ["PrgEnv"] = "PrgEnv-nvidia",
    ["compiler"] = "gcc",
    ["craype"] = "craype",
    ["craype_cpu"] = "craype-x86-rome",
    ["craype_network"] = "craype-network-ofi",
  },
...

Lmod will set the environment variables LMOD_FAMILIY_<NAME> and LMOD_FAMILY_<NAME>_VERSION, where <name> is the name of family that can be used to reference family.

Module Properties

We have configured module properties for a subset of modules to help classify the software provided. If you run module list, you will see properties such as cpe, math, io, c, dev to help you identify what type of software the module provides.

elvis@perlmutter> module list

Currently Loaded Modules:
  1) nvidia/20.9     (g,c)   4) libfabric/1.11.0.3.66   7) perftools-base/21.02.0                    (dev)   10) PrgEnv-nvidia/8.0.0 (cpe)  13) cray-pmi/6.0.10
  2) craype/2.7.6    (c)     5) craype-network-ofi      8) xpmem/2.2.40-7.0.1.0_1.9__g1d7a24d.shasta (H)     11) xalt/2.10.2                14) cray-pmi-lib/6.0.10
  3) craype-x86-rome         6) cray-dsmml/0.1.4        9) cray-libsci/21.04.1.1                     (math)  12) darshan/3.2.1       (io)   15) Default

  Where:
   g:     built for GPU
   cpe:   Cray Programming Environment Modules
   math:  Mathematical libraries
   io:    Input/output software
   c:     Compiler
   dev:   Development Tools and Programming Languages
   H:                Hidden Module

Startup module

We have configured Lmod to provide a module that is loaded by default for all users, called Default. This module can be loaded using module load Default. If you are unsure how to get to the startup configuration you can do one of the following:

  1. Run module restore
  2. Run module purge && module load Default

Note

If you have a default collection (a collection entitled default), then module restore will restore your user collection instead of the system default. You can check your collections by running module savelist or examining the content of the files in $HOME/.lmod.d

If you don't have a default user collection, then Lmod will restore from the startup modulefile Default. This is done by Lmod by reading the environment variable LMOD_SYSTEM_DEFAULT_MODULES which should not be changed. 

elvis@perlmutter> module purge
elvis@perlmutter> module restore
Resetting modules to system default. Reseting $MODULEPATH back to system default. All extra directories will be removed from $MODULEPATH.
elvis@perlmutter> ml

Currently Loaded Modules:
  1) cray-mpich/8.1.4      (mpi)   7) cray-dsmml/0.1.4                                  13) darshan/3.2.1       (io)
  2) nvidia/20.9           (g,c)   8) perftools-base/21.02.0                    (dev)   14) cray-pmi/6.0.10
  3) craype/2.7.6          (c)     9) xpmem/2.2.40-7.0.1.0_1.9__g1d7a24d.shasta (H)     15) cray-pmi-lib/6.0.10
  4) craype-x86-rome              10) cray-libsci/21.04.1.1                     (math)  16) Default
  5) libfabric/1.11.0.3.66        11) PrgEnv-nvidia/8.0.0                       (cpe)
  6) craype-network-ofi           12) xalt/2.10.2

  Where:
   g:     built for GPU
   mpi:   MPI Providers
   cpe:   Cray Programming Environment Modules
   math:  Mathematical libraries
   io:    Input/output software
   c:     Compiler
   dev:   Development Tools and Programming Languages
   H:                Hidden Module

Please refer to see https://lmod.readthedocs.io/en/latest/070_standard_modules.html for details related to startup modules

Useful Tips

Redirecting Module Output

Lmod will redirect output to stderr by default, which means this won't work as you expected, since its output will not be stored in a file.

elvis@perlmutter> module list > active.txt

Currently Loaded Modules:
  1) craype/2.7.6          (c)   5) cray-dsmml/0.1.4                                   9) PrgEnv-nvidia/8.0.0 (cpe)  13) cray-pmi-lib/6.0.10
  2) craype-x86-rome             6) perftools-base/21.02.0                    (dev)   10) xalt/2.10.2                14) Default
  3) libfabric/1.11.0.3.66       7) xpmem/2.2.40-7.0.1.0_1.9__g1d7a24d.shasta (H)     11) darshan/3.2.1       (io)   15) gcc/10.2.0          (c)
  4) craype-network-ofi          8) cray-libsci/21.04.1.1                     (math)  12) cray-pmi/6.0.10

  Where:
   cpe:   Cray Programming Environment Modules
   math:  Mathematical libraries
   io:    Input/output software
   c:     Compiler
   dev:   Development Tools and Programming Languages
   H:                Hidden Module

The --redirect option to any module command will redirect stderr to stdout so that shell commands can capture module commands. 

elvis@perlmutter> module --redirect list > active.txt
elvis@perlmutter> cat active.txt 

Currently Loaded Modules:
  1) craype/2.7.6          (c)   5) cray-dsmml/0.1.4                                   9) PrgEnv-nvidia/8.0.0 (cpe)  13) cray-pmi-lib/6.0.10
  2) craype-x86-rome             6) perftools-base/21.02.0                    (dev)   10) xalt/2.10.2                14) Default
  3) libfabric/1.11.0.3.66       7) xpmem/2.2.40-7.0.1.0_1.9__g1d7a24d.shasta (H)     11) darshan/3.2.1       (io)   15) gcc/10.2.0          (c)
  4) craype-network-ofi          8) cray-libsci/21.04.1.1                     (math)  12) cray-pmi/6.0.10

  Where:
   cpe:   Cray Programming Environment Modules
   math:  Mathematical libraries
   io:    Input/output software
   c:     Compiler
   dev:   Development Tools and Programming Languages
   H:                Hidden Module

Autoswap modules of the same name

Lmod will automatically swap modules of the same name. For instance, if you load gcc/9.3.0, Lmod will remove gcc/10.2.0 from your user environment:

elvis@perlmutter> module load gcc/9.3.0 

The following have been reloaded with a version change:
  1) gcc/10.2.0 => gcc/9.3.0

Debugging Modules

If you want to debug the state of modules, any of these command options can help:

  • Tracing modules: module -T
  • Print Module Table: module --mt
  • Debug Level: module --debug=[1|2|3]

To debug modulefiles themselves, please see https://lmod.readthedocs.io/en/latest/160_debugging_modulefiles.html

Parsing output

Use the -t option with module avail, module spider, module list, module spider and module savelist commands to produce parsable output.

To see loaded modules in a parsable format, the environment variable LOADEDMODULES is a colon separated list of active modules loaded in your shell.

elvis@perlmutter> echo $LOADEDMODULES
gcc/10.2.0:dvs/2.12_4.0.102-7.0.1.0_8.1__g30d29e7a

Similarly, the full path to the modulefiles of active modules can be retrieved using the _LMFILES_ environment variable. The output is a colon-separated list of modulefiles.

elvis@perlmutter> echo $_LMFILES_
/opt/cray/pe/lmod/modulefiles/core/gcc/10.2.0.lua:/opt/cray/modulefiles/dvs/2.12_4.0.102-7.0.1.0_8.1__g30d29e7a

Seeing Defaults for Modulefiles

The module -d avail command will report the default for every module. Lmod will load the default module if you don't specify the full version (i.e., module load gcc) since there is only one default for every module name.

For example, to list the default for gcc:

elvis@perlmutter> module -d avail gcc

---------------------------------------------------------------------------- Cray Core Modules -----------------------------------------------------------------------------
   gcc/10.2.0 (L)

  Where:
   L:  Module is loaded

Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".

Hidden Modules

Hidden modules are modulefiles with a leading dot (.) in front of the version. These modules are not shown when running module avail or module spider. In order to see these modules you need to use the --show_hidden option. This option is also required if you want to show the content of a hidden modulefile: module --show_hidden show <name>, module --show_hidden help <name>, or module --show_hidden whatis <name>.

A hidden module will have a symbol (H) next to the modulefile.

In example below we have dvs as a hidden module which is not reported in the output of module avail 

elvis@perlmutter> module avail dvs
No module(s) or extension(s) found!
Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".



elvis@perlmutter> module --show_hidden avail dvs

------------------------------------------------------------------------------- Cray Modules -------------------------------------------------------------------------------
   dvs/2.12_4.0.102-7.0.1.0_8.1__g30d29e7a (H)

  Where:
   H:  Hidden Module

For Perlmutter, we have configured our own system defaults and hidden modules in the modulerc.lua file, which can be retrieved by the environment variable $LMOD_MODULERCFILE.

Managing User Collection

Lmod will store user collections in $HOME/.lmod.d, so if you want to delete a user collection, you can simply delete the file in the $HOME/.lmod.d directory.

Shown below we have a default collection stored this in the file $HOME/.lmod.d/default.perlmutter. 

elvis@perlmutter> ls ~/.lmod.d/
default.perlmutter
elvis@perlmutter> module -t savelist
default

If you want to delete the user collection default, you can remove the file

elvis@perlmutter> rm ~/.lmod.d/default.perlmutter
elvis@perlmutter>  module describe default
Lmod Warning:  No collection named "default" found.

Alternatively you can use module disable [collection] to disable a collection name. Lmod will rename the collection name by appending a ~ to the collection name so that Lmod won't read the collection. In the below example we create a user collection named gcc and running module disable gcc will rename the collection to gcc.perlmutter~; however Lmod doesn't recognize the gcc collection when running module describe gcc. To re-enable the user collection, rename the file by removing the ~.

elvis@perlmutter> module purge
elvis@perlmutter> module load gcc
elvis@perlmutter> module save gcc
Saved current collection of modules to: "gcc", for system: "perlmutter"

elvis@perlmutter> ls ~/.lmod.d/
gcc.perlmutter
elvis@perlmutter> module disable gcc
Disabling gcc collection by renaming with a "~"
elvis@perlmutter> ls ~/.lmod.d/
gcc.perlmutter~
elvis@perlmutter> module describe gcc
Lmod Warning:  No collection named "gcc" found.

Customize Module Defaults at User Level

Lmod has support for customizing module defaults at the user level. This is done by creating a file $HOME/.modulerc.lua.

Let's assume you want to use the latest PrgEnv-nvidia/8.4.0 as the default module however the system default is PrgEnv-nvidia/8.3.3 so you can simply type module load PrgEnv-nvidia without having to type the version. Unfortunately, you can't make this change at the system level since it will affect all users, however you can make this change at the user level.

elvis@perlmutter> ml av PrgEnv-nvidia

------------------------------------------------------------------------------------------- Cray Core Modules --------------------------------------------------------------------------------------------
   PrgEnv-nvidia/8.3.3 (D)    PrgEnv-nvidia/8.4.0

  Where:
   D:  Default Module

If the avail list is too long consider trying:

"module --default avail" or "ml -d av" to just list the default modules.
"module overview" or "ml ov" to display the number of modules for each name.

Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".

To make this change you can add the following line in the ~/.modulerc.lua file.

module_version("PrgEnv-nvidia/8.4.0","default")

Next, let's rerun the module av command and take note of the (D) next to the module name.

elvis@perlmutter> module av PrgEnv-nvidia

------------------------------------------------------------------------------------------- Cray Core Modules --------------------------------------------------------------------------------------------
   PrgEnv-nvidia/8.3.3    PrgEnv-nvidia/8.4.0 (D)

  Where:
   D:  Default Module

If the avail list is too long consider trying:

"module --default avail" or "ml -d av" to just list the default modules.
"module overview" or "ml ov" to display the number of modules for each name.

Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".

The .modulerc.lua file can be used for further customization such as hiding modules or setting aliases. For more details, we encourage you to check out https://lmod.readthedocs.io/en/latest/093_modulerc.html

Troubleshooting User Environment Issues

If you have issues with your user environment, please review your startup configuration files for any module commands. Look at the files sourced as a result of your configuration.

  • bash/sh users: $HOME/.bashrc, $HOME/.bashrc_profile, $HOME/.profile
  • csh/tcsh shell: $HOME/.cshrc

In particular, look for changes to the environment variable MODULEPATH or use of module use command in your startup configuration.

'module: Command not found.' with csh

Users of csh/tcsh may encounter the error module: Command not found. when trying to load modules on Perlmutter. To mitigate this issue, we suggest that csh users execute source $LMOD_PKG/init/csh prior to running any module commands on Perlmutter.

References

Please see the references below for additional help. NERSC's Lmod Training (the final link below) is a self-paced exercise covering the basics of Lmod and Lua modulefiles in a Docker container.