Skip to content

NPE and PrgEnv-llvm

NPE or NERSC Programming Environment complements and extends the Cray Programming Environment provided by HPE.

NPE and PrgEnv-LLVM are currently Experimental

These are new efforts under active development. Long term support options are being planned and evaluated, but for now significant breaking changes may occur at any time.

Your feedback is welcome!


NPE is not yet part of the default set of modules and must be added via:

module use /global/cfs/cdirs/nstaff/cookbg/pe/modulefiles

after which NPE can be loaded

ml npe


Additional versions of the npe stack will be made available as additional npe modules.


LLVM compilers with support for OpenMP offload and SYCL with support for MPI via MPICH.

ml PrgEnv-llvm

If GPU support is desired then

ml cudatoolkit craype-accel-nvidia80
ml PrgEnv-llvm


Use mpic++, mpicc to compile, not cc or CC.


Fortran (flang) is not supported at this time.

intel-llvm is the default version and supports OpenMP offload and SYCL for A100. Nightly versions are also available (see ml avail intel-llvm).

llvm is based on released versions and supports OpenMP offload for A100.


The mpich module is available for:

  • PrgEnv-gnu
  • PrgEnv-nvidia
  • PrgEnv-llvm


mpic++, mpicc and mpif90 should be used instead of cc, CC, and ftn in this release.


The cudatoolkit and craype-accel-nvidia80 modules must be loaded before mpich to enable CUDA aware MPI.


Flux is a flexible framework for resource management consisting of a suite of projects, tools, and libraries which may be used to build site-custom resource managers for HPC centers.


Reminder that everything in NPE is considered "experimental" NERSC's current workflow tool recommendations.

Start multi-node interactive job

salloc -C gpu -A <account> -N 2 --ntasks-per-node=1 --gpus-per-node 4 -q debug -t 30

Load the flux-sched module

module load flux-sched

Start flux

srun --mpi=pmi2 --pty flux start

Verify that flux can see gpu resources

$ flux resource list
      free      2      128        8 nid[001704-001705]
 allocated      0        0        0
      down      0        0        0


Please report any issues, comments or feedback via the NERSC Help Desk.

Release notes


  • flux support added (flux-sched/0.22.0)
  • Reframe test for flux
  • also build llvm/14.0.0


Initial "alpha" release includes

  • mpich/4.0 for PrgEnv-gnu, PrgEnv-nvidia and PrgEnv-llvm mpich source
  • PrgEnv-llvm/0.1 a new LLVM based PrgEnv
  • llvm/14.0.3 from source with OpenMP offload support for A100.
  • intel-llvm/2021-12 from Intel's fork of LLVM with OpenMP offload and SYCL support for A100.
  • intel-llvm/sycl-nightly/<date> Access to nightly builds of intel-llvm.
  • ninja/1.10.2 Ninja build tool
  • cmake/3.22.1 CMake build system
  • reframe/3.10.1 HPC testing framework