Environment¶

NERSC User Environment¶

Home Directories, Shells and Dotfiles¶

All NERSC systems use global home directories, which are are pre-populated with shell initialization files (also known as dotfiles) for all available shells. NERSC supports bash, csh, and tcsh as login shells. Other shells (ksh, sh, and zsh) are also available. The default shell at NERSC is bash.

The standard dotfiles are symbolic links to read-only files that NERSC controls. For each standard dotfile, there is a user-writeable ".ext" file.

Fixing Dotfiles¶

Occasionally, a user will accidentally delete the symbolic links to the standard dotfiles, or otherwise damage the dotfiles to the point that it becomes difficult to do anything. In this case, the user should run the command /usr/common/software/bin/fixdots. This will recreate the original dotfile configuration, after first saving the current configuration in the directory $HOME/KeepDots.timestamp is a string that includes the current date and time. After running fixdots, the user should carefully incorporate the saved customizations into the newly-created .ext files.

Dotfile changes planned in February 2020¶

To reduce the shell startup overhead, and to bring NERSC in line with other HPC centers, NERSC will NOT reserve your home dotfiles for system use anymore starting from February 2020 (The exact transition date is still TBA). The NERSC settings in the current home dotifles will be migrated to the /etc/profile.d directory on each system. After this transition, you can use your home .bash_profile, .bashrc, etc., for your personal shell modifications (You don't have to work with those .ext files anymore!).

Notes for existing user accounts¶

To mitigate the interruptions to your workloads, we will do the following to preserve your current shell environment on the transition day.

• We will replace your dotfiles, that are currenly symbolic links to the files under NERSC control with template dotfiles that source their .ext files. For example, here is how your ~/.bashrc file would look like,

  # begin .bashrc
  if [[ -z "$SHIFTER_RUNTIME" ]]
  then
      . $HOME/.bashrc.ext
  fi
  # end .bashrc
  

  You are recommended to move what's in your ~/.bashrc.ext file into your .bashrc file after the transition (and remove the .ext files after the move). If you do not use shifter applications, then you can remove the if block for testing shifter runtime.

• If you have already been using local dotfiles (i.e., not using the dotfiles reserved by the system), we will not make any changes to your dotifles. You may find the altd and darshan modules are loaded in your environment, but you can unload them in your dotfiles if you prefer.

While we expect that this transition is transparent to most users, some users may experience changes in their shell environment. We have configured cori12 (one of the Cori login nodes) and dtn12 (DTN test node), so that you can test your shell environment now before the actual changes take place. You can type the following commands to get the dotfiles that will be populated on your account on the transition day (type dotmgr -h for more info):

dotmgr -s # save your current dotfiles, and print the location
dotmgr -e # get the dotfiles that will be populated on your account on the transition day

You can then login to cori12 and/or dtn12 to check whether this affected your environment.

ssh cori12

You can then return to your current dotfiles after testing with the following command:

dotmgr -r <directory-that-the-save-step-returned>

You'll need to log out of cori12/dtn12, and log back into a Cori/DTN login node, to get your restored dotfiles take effect.

Notice that now you can test your new dotfiles on the login nodes only. Since the environment on the login nodes are passed to the compute nodes by the batch system, you will see similar shell environment on the compute nodes. We will arrange compute nodes access in next January for you to test as well. Please let us know of any problems you encounter, by filing a ticket at help.nersc.gov

Notes for new user accounts created after February 2020¶

If your NERSC account is created after the dotfile migration (February 2020), NERSC will not populate any dotfiles on your home directory. You need to create your own dotfiles (e.g., .bashrc, .bash_profile, etc) as needed to put your personal shell modifications.

• NERSC loads darshan (a lightweight I/O profiling tool) and altd (a library tracking tool) by default. If you encounter any problems with them, you can unload them in your .bash_proifle, or .login file:

  module unload darshan
  module unload altd
  

• If you run shifter applications, you many want to skip the dotfiles on your home. You can use the following if block in your dotfiles (for bash)

  if [[ -z "$SHIFTER_RUNTIME" ]]; then
  #put your Cori setting here
  ... 
  
  fi  
  

• If any of the NERSC defined environment variables, e.g., SCRATCH, are missing in your shell invocations, you can add them in your ~/.bashrc file as follows:

  if [ -n "$SCRATCH" ]; then
      export SCRATCH=/global/cscratch1/sd/$USER
  fi
  

Changing Default Login Shell¶

Use NIM to change your default login shell. Login, then select Change Shell from the Actions pull-down menu.

NERSC Modules Environment¶

NERSC uses the module utility to manage nearly all software. There are two huge advantages of the module approach:

1. NERSC can provide many different versions and/or installations of a single software package on a given machine, including a default version as well as several older and newer version.
2. Users can easily switch to different versions or installations without having to explicitly specify different paths. With modules, the MANPATH and related environment variables are automatically managed.

Module Command¶

The following is a list of commands available in the Modules Environment tool available on Cori.

module help¶

To get a usage list of module options type the following (listing is abbreviated):

module help

  Available Commands and Usage:

     + add|unload      modulefile [modulefile …]
     + rm|unload       modulefile [modulefile …]
     + switch          modulefile1 modulefile2
     + display         modulefile [modulefile …]
     + avail           path [path]
     + list
     + help            modulefile [modulefile …]

module list¶

module list

module avail¶

   # To get all available packages
   module avail

   # To know the availability of a specific software
   module avail netcdf

   # To know all packages that contain a substring, use -S flag.
   module avail -S netcdf

module display¶

To see what changes are made to your environment when a module is loaded:

module display [modulefile]

module show [modulefile]

module load¶

This command will add one or more modulefiles to your current environment. It does so silently, but will throw errors if there are any problems with the modulefile. If you load the generic name of a module, you will get the default version. To load a specific version, load the modulefile using the full specification.

module load [modulefile1][modulefile2]

# Load visit
module load visit

# Load visit version 2.1.2
module load visit/2.1.2

module unload¶

Unloads the specified modulefile from the user's environment. This command will fail silently if the modulefile you specify is not already loaded.

module unload [modulefile]

module swap¶

The modules environment allows you to swap between versions of packages

module swap [old modulefile] [new modulefile]

Creating a Custom Environment¶

You can modify your environment so that certain modules are loaded whenever you log in. Put changes in one of the following files, depending on your shell:

• .cshrc.ext or .tcshrc.ext
• .bashrc.ext

Users may have certain customizations that are appropriate for one NERSC platform, but not for others. This can be accomplished by testing the value of the environment variable $NERSC_HOST. For example, on Cori the default programming environment is Intel (PrgEnv-Intel). A C-shell user who wants to use the GNU programming environment should include the following module command in their .cshrc.ext file:

Cori¶

if ($NERSC_HOST == "cori") then
  module swap PrgEnv-intel PrgEnv-gnu
endif

Install Your Own Customized Modules¶

You can create and install your own modules for your convenience or for sharing software among collaborators. The module definition files can be placed in the following locations:

• project directory
• your home directory
• available file system.

Make sure the UNIX file permissions grant access to all users who want to use the software.

As an example, we have modulefile named myzlib located in

/global/project/projectdirs/mpccc/usg/modulefiles/cori

To register this modulefile with our modules environment we run the following commands:

nersc$ module use /global/project/projectdirs/mpccc/usg/modulefiles/cori
nersc$ module load myzlib/1.2.7