Skip to content

Environment

NERSC User Environment

Home Directories, Shells and Dotfiles

All NERSC systems use global home directories. NERSC supports bash, csh, and tcsh as login shells. Other shells (ksh, sh, and zsh) are also available. The default login shell at NERSC is bash. NERSC does not populate shell initialization files (also known as dotfiles) on users' home directories. You can create dotfiles (e.g., ~/.bashrc, ~/.bash_profile, etc.) as needed to put your personal shell modifications.

No more .ext dotfiles at NERSC since February 21, 2020.

NERSC used to reserve the standard dotfiles (~/.bashrc, ~/.bash_profile, ~/.cshrc, ~/.login, etc.) for system use so that users had to use the corresponding .ext files (e.g., ~/.bshrc.ext, ~/.bash_profile.ext, etc.) for their shell modifications. This is not the case anymore! You can modify those standard dotfiles for your personal use now.

The actual dotfile transition occurred during the center maintenance on February 21-25, 2020. To mitigate the interruptions to existing workloads, we have preserved shell environments by replacing dotfiles with template dotfiles that source .ext files. For example, if you are an existing user at NERSC, here is how your ~/.bashrc file would look like,

# begin .bashrc
if [ -z "$SHIFTER_RUNTIME" ]
then
    . $HOME/.bashrc.ext
fi
# end .bashrc

You are recommended to move the contents of your ~/.bashrc.ext file into your ~/.bashrc file after the transition (and remove the .ext files afterwards).

Changing Default Login Shell

Use Iris to change your default login shell. Log in, then under the "Details" tab look for the "Server Logins" section. Click on "Edit" under the "Actions" column.

Customizing Shell Environment

You can create dotfiles (e.g., .bashrc, .bash_profile, or .profile, etc) in your $HOME directory to put your personal shell modifications.

Note

On Cori ~/.bash_profile and ~/.profile are sourced by login shells, while ~/.bashrc is sourced by most of the shell invocations including the login shells. In general you can put the environment variables, such as PATH, which are inheritable to subshells in ~/.bash_profile or ~/.profile and functions and aliases in the ~/.bashrc file in order to make them available in subshells.

System specific customizations

All NERSC systems share the Global HOME; the same $HOME is available regardless of the platform. To make system specific customizations use the pre-defined environment variable NERSC_HOST.

Example

case $NERSC_HOST in
    "cori")
        : # settings for Cori
        export MYVARIABLE="value-for-cori"
        ;;
    "datatran")
        : # settings for DTN nodes
        export MYVARIABLE="value-for-dtn"
        ;;
    *)
        : # default value for other nodes
        export MYVARIABLE="default-value"
        ;;
esac

darshan and altd

NERSC loads a light I/O profiling tool, darshan, and altd (a library tracking tool) on Cori by default. If you encounter any problems with them, you can unload them in your ~/.bash_profile, or ~/.login file:

module unload darshan
module unload altd

shifter

If you run shifter applications, you may want to skip the dotfiles. You can use the following if block in your dotfiles:

if [ -z "$SHIFTER_RUNTIME" ]; then
    : # Settings for when *not* in shifter
fi

missing NERSC variables

If any of the NERSC defined environment variables such as $SCRATCH, are missing in your shell invocations, you can add them in your ~/.bashrc file as follows:

if [ -z "$SCRATCH" ]; then
    export SCRATCH=/global/cscratch1/sd/$USER
fi

crontabs

If you run bash scripts in crontabs, you may want to invoke a login shell (#!/bin/bash -l) in order to get the NERSC defined environment variables, such as NERSC_HOST, SCRATCH, CSCRATCH, and to get the module command defined.

NERSC Modules Environment

NERSC uses the module utility to manage nearly all software. There are two advantages of the module approach:

  1. NERSC can provide many different versions and/or installations of a single software package on a given machine, including a default version as well as several older and newer version.
  2. Users can easily switch to different versions or installations without having to explicitly specify different paths. With modules, the MANPATH and related environment variables are automatically managed.

What is module

module is a shell function that modifies user shell upon load of a modulefile. The module function is defined as follows

$ type module
module is a function
module () 
{ 
    eval `/opt/cray/pe/modules/3.2.11.4/bin/modulecmd bash $*`
}

Note

module is not a program

Module Commands

General usage:

nersc$ module [ switches ] [ subcommand ] [subcommand-args ]

Further reading:

  • module help
  • man module
  • man modulefile
  • Online manual (note: some features may only be available in later versions than what is installed on NERSC systems)

Common commands

List currently loaded modules:

module list

List all available modules:

module avail 
module av

Show availability of specific module:

module avail <module-name>

Show availability of all modules containing a substring:

module avail -S <substring>

Display what changes are made when a module is loaded:

module display <module-name>
module show <module-name>

Add a module to your current environment:

module load <module-name>
module add <module-name>

Note

This command is silent unless there are problems with the module.

Tip

If you load then generic name of a module, you will get the default version.

module load gcc

To load a specific version use the full name

module load gcc/8.1.0

Remove module from the current environment:

module unload <module-name>
module rm <module-name>

Note

This command will fail silently if the specified module is not loaded.

Switch currently loaded module with a new module:

module swap <old-module> <new-module>
module switch <old-module> <new-module>

To purge all modules::

modle purge

To view help for a particular module::

module help <module-name>

To see a condensed list of module you can use module -t and use this with list or avail

$ module -t list
Currently Loaded Modulefiles:
modules/3.2.11.4
nsg/1.2.0
altd/2.0
darshan/3.1.7
intel/19.0.3.199
craype-network-aries
craype/2.6.2
cray-libsci/19.06.1
udreg/2.3.2-7.0.1.1_3.29__g8175d3d.ari
ugni/6.0.14.0-7.0.1.1_7.32__ge78e5b0.ari
pmi/5.0.14
dmapp/7.1.1-7.0.1.1_4.43__g38cf134.ari
gni-headers/5.0.12.0-7.0.1.1_6.27__g3b1768f.ari
xpmem/2.2.20-7.0.1.1_4.8__g0475745.ari
job/2.2.4-7.0.1.1_3.34__g36b56f4.ari
dvs/2.12_2.2.156-7.0.1.1_8.6__g5aab709e
alps/6.6.57-7.0.1.1_5.10__g1b735148.ari
rca/2.2.20-7.0.1.1_4.42__g8e3fb5b.ari
atp/2.1.3
PrgEnv-intel/6.0.5
craype-haswell
cray-mpich/7.7.10
craype-hugepages2M

Creating a Custom Module Environment

You can modify your environment so that certain modules are loaded whenever you log in.

The first option is to use shell commands.

bash

In ~/.bash_profile

module swap PrgEnv-${PE_ENV,,} PrgEnv-gnu

csh

In ~/.login

set pe = ` echo $PE_ENV | tr "[:upper:]" "[:lower:]" `
module swap PrgEnv-${pe} PrgEnv-gnu

snapshots

The second option is to use the "snapshot" feature of modules.

  1. swap and load modules to your desired configuration
  2. save a "snapshot" with module snapshot <snapshot-filename>

Then at any time later restore the environment with module restore <snapshot-filename>.

Install Your Own Customized Modules

You can create and install your own modules for your convenience or for sharing software among collaborators. See the man modulefile or the modulefile documentation for details of the required format and available commands. These custom modulefiles can be made visible to the module command by module use /path/to/the/custom/modulefiles.

Tip

Global Common is the recommended location to install software.

Note

Make sure the UNIX file permissions grant access to all users who want to use the software.

Warning

Do not give write permissions to your home directory to anyone else.

Note

The module use command adds new directories before other module search paths (defined as $MODULEPATH), so modules defined in a custom directory will have precedence if there are other modules with the same name in the module search paths. If you prefer to have the new directory added at the end of $MODULEPATH, use module use -a instead of module use.

Module FAQ

  1. Is there an environment variable that captures loaded modules?

Yes, active modules can be retrieved via $LOADEDMODULES, this environment variable is automatically changed to reflect active loaded modules that is reflected via module list. If you want to access modulefile path for loaded modules you can retrieve via $_LM_FILES

  1. How to restore MODULEPATH in user session?

If you run into an error such as following::

$ module avail
ModuleCmd_Avail.c(217):ERROR:107: 'MODULEPATH' not set

You should try a new login shell and see if it fixes the issue. Check to see if your startup scripts (~/.bashrc, ~/.bash_profile) or ~/.cshrc for tcsh/csh