Perlmutter Software¶

Environment¶

Lmod - modules¶

Perlmutter uses Lmod, a Lua based module system, to manage and dynamically change user's environment settings such as which directories are included in the PATH environment variable.

Detailed Usage Guide for Lmod at NERSC

NERSC Defaults¶

NERSC provides gpu and cpu modules that load recommended defaults for compiling on GPUs and CPUs, respectively. We may update these modules as new information or recommendations are made. You can use module show <gpu/cpu> to see what each module does. You can only have one of these modules loaded at a time and loading one unloads the other. Currently, the gpu module is loaded by default.

Older Environments¶

Generally we recommend that you use the most recent programming environment installed on Perlmutter. However, sometimes it is convenient to have access to previous programming environments to check things like compile options and libraries, etc. You can use module load cpe/YY.XX to load the previous programming environment from year YY and month XX. We will remove cpe modules for environments that no longer work on our system due to changes in underlying dependencies like network libraries.

Please keep in mind that these cpe modules are offered for convenience sake. If you require reproducibility across environments we encourage you to investigate container-based options like Shifter.

Spack¶

NERSC provides the Spack package manager via a modulefile. This spack instance is preconfigured by NERSC to integrate with the Perlmutter software environment.

In order to use the default Spack instance

module load spack

In order to use the E4S instance

module load e4s

The Spack documentation provides on overview of how to use spack and Spack at NERSC provides details of the NERSC provided instance along with additional recommendations for how to effectively use Spack at NERSC.

Finding Software¶

NERSC provides a wide array of software: compilers, libraries, frameworks, profilers, debuggers, utilities and applications.

Typically these are available via the module system.

See finding software for further details about finding pre-installed software.

Packaging¶

Containers¶

Containers provide a stable fixed environment for your software that greatly reduces the dependency on any details of NERSC's software stack while retaining and in some cases improving performance!

If your application depends on MPI then NERSC recommends installing MPICH inside the container from source (i.e. not from apt-get) with shared libraries. This allows your software to use the ABI compatible Cray MPICH which has been optimized for the Perlmutter network.

Best Practice¶

When installing software on NERSC's filesystems it is recommended to use the /global/common/software/<your NERSC project> directories. This filesystem is optimized for sharing software.

Compilers¶

Cray, AMD, NVIDIA, LLVM and GNU compilers are available through modules on Perlmutter.

Cray provides PrgEnv-<compiler vendor> modules that load components of specific toolchain including MPI and LibSci (man intro_libsci). Included with these modules are compilers wrappers (similar to mpicc) that include additional link options and compiler flags:

• cc for C
• CC for C++
• ftn for Fortran

Linking¶

Languages¶

Recommended compilers and programming models to target GPUs. Further details can be found in the NERSC Perlmutter Readiness Guide.

C¶

C++¶

Fortran¶

Python¶

NERSC provides a python installation based on Anaconda python. See the NERSC page on using python on Perlmutter for details about how to manage environments and target the Perlmutter GPUs.

MPI¶

Cray MPICH is a CUDA-aware MPI implementation - allowing programmers to pass pointers to buffers in GPU memory to MPI API routines. Example CUDA Aware MPI program.

To utilize CUDA-Aware MPI the feature must be enabled at both compile and run time:

At compile time the HPE GPU Transport Layer (GTL) libraries must be linked. In order to enable this take at least one the following actions:

1. Keep the default gpu module loaded
2. Load the cudatoolkit and craype-accel-nvidia80 modules
3. Set the environment variable CRAY_ACCEL_TARGET=nvidia80
4. Pass the compiler flag -target-accel=nvidia80

At run time MPICH_GPU_SUPPORT_ENABLED=1 must be set. If it is not set there will be Errors similar to

MPIDI_CRAY_init: GPU_SUPPORT_ENABLED is requested, but GTL library is not linked

For further details about Cray MPICH see the manpage on Perlmutter man intro_mpi.

Programming Models¶

OpenMP¶

See the overview of OpenMP at NERSC for a summary of key features, links to NERSC training and other useful references.

GPU Offload¶

PrgEnv-nvidia¶

Use the -mp=gpu option.

Details of OpenMP support and programming recommendation are available at in the OpenMP section of the Perlmutter Readiness Guide.

PrgEnv-cray¶

warning: linking module '@': linking module flags 'SDK Version': IDs have conflicting values ('[2 x i32] [i32 11, i32 1]' from /opt/cray/pe/cce/14.0.1/cce-clang/x86_64/lib/libcrayomptarget-nvptx-sm_80.bc with '[2 x i32] [i32 11, i32 5]' from ../benchmarks/babelStream/main.cpp) [-Wlinker-warnings]

PrgEnv-gnu¶

PrgEnv-gnu is not recommended for OpenMP offloading.

To access it:

ml use /global/cfs/cdirs/m1759/wwei/Modules/perlmutter/modulefiles/
ml gcc/13.1.0

An example to compile your code:

gcc -Ofast -fopenmp -foffload=nvptx-none="-Ofast -lm -latomic -misa=sm_80"  main.c -o main

OpenACC¶

GPU Offload¶

PrgEnv-nvidia¶

Use the -acc=gpu option.

CUDA¶

On Perlmutter CUDA is available via the cudatoolkit modules. The toolkit modules contain GPU-accelerated libraries, profiling tools (nsight compute & systems), debugger tools (cuda-gdb & cuda-memcheck) a runtime library and nvcc CUDA compiler.

NVIDIA maintains extensive documentation for CUDA toolkits.

PrgEnv-nvidia¶

The host compilers nvc / nvc++ (accessible through the cc/ CC wrapper) in NVIDIA SDK has CUDA opt-in support. To compile a single source C / C++ code (host & device code in the same source file) with the Cray wrappers you must add the -cuda flag to their compilation step which notifies the nvc/ nvc++ compiler to accept CUDA runtime APIs. Omitting the -cuda flag will result in your application compiling without any of the CUDA API calls, and will generate an executable with undefined behavior.

PrgEnv-gnu¶

When using the PrgEnv-gnu environment in conjunction with the cudatoolkit module (i.e., if compiling any application for both host and device side), you must note that not every version of gcc is compatible with every version of nvcc - supported host compilers for each nvcc installation.

Versions¶

NERSC generally aims to make the latest versions of cudatoolkit available. In some cases a specific version other than what is installed is needed.

In this situation one should first check if the version needed is compatible. Generally CUDA is forward compatible. For example code written for 11.3 should work with 11.7.

See the CUDA Compatibility Document which describes the details.

If this is not an option next one should consider using containers through Shifter for the specific desired CUDA version.

UPC++¶

On Perlmutter the UPC++ PGAS library is available as user-managed software, contributed by the UPC++ library maintainers at LBNL.

Also see UPC++ documentation for Perlmutter for site-specific details and recommendations regarding use of UPC++ at NERSC.