Skip to content

Perlmutter Spack Environments

In this guide, we will demonstrate how one can leverage Spack to build software on Perlmutter using your own spack instance.

Setup

Before you begin, you will need to clone an instance of Spack if you do not already have one. For more details please refer to Installation of Spack

git clone https://github.com/spack/spack

Next you will need to clone the NERSC Spack Infrastructure repository, which contains some relevant configuration files

git clone https://github.com/NERSC/spack-infrastructure.git

Before you get started, we recommend you source the setup-env.sh script found in the root of the spack-infrastructure repo. This will create a Python environment in order for you to do Spack builds. Spack requires clingo in order to bootstrap clingo; however, we observed issues where Spack was unable to bootstrap clingo - see Spack issue 28315 for details. We found that installing clingo as a Python package addressed the issue.

elvis@perlmutter> cd spack-infrastructure/
elvis@perlmutter> source setup-env.sh
Collecting clingo
  Using cached clingo-5.5.2-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB)
Collecting cffi
  Using cached cffi-1.15.1-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (402 kB)
Collecting pycparser
  Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Installing collected packages: pycparser, cffi, clingo
Successfully installed cffi-1.15.1 clingo-5.5.2 pycparser-2.21
WARNING: You are using pip version 20.2.3; however, version 21.3.1 is available.
You should consider upgrading via the '/global/homes/e/elvis/spack-infrastructure/spack-pyenv/bin/python3 -m pip install --upgrade pip' command.
/global/homes/e/elvis/spack-infrastructure/spack-pyenv/bin/python
Package    Version
---------- -------
cffi       1.15.1
clingo     5.5.2
pip        20.2.3
pycparser  2.21
setuptools 44.1.1
WARNING: You are using pip version 20.2.3; however, version 21.3.1 is available.
You should consider upgrading via the '/global/homes/e/elvis/spack-infrastructure/spack-pyenv/bin/python3 -m pip install --upgrade pip' command.

Please source the Spack setup-env.sh script to install Spack. This can be done by running the following command, assuming Spack is cloned in your HOME directory:

elvis@perlmutter> cd $HOME
elvis@perlmutter> source spack/share/spack/setup-env.sh

Spack builds are sensitive to user environment

User modules can cause spack builds to behave differently, see spack/27124. We recommend you reset your environment and load the following modules on Perlmutter:

module reset
module load cpu

We provide a template Spack configuration that you can use to create a Spack environment with pre-configured settings.

cd spack-infrastructure/
spack env create demo spack-configs/perlmutter-user-spack/spack.yaml
spack env activate demo

Spack Configuration

Changes to spack.yaml between Spack releases

Spack has changed the YAML structure of spack.yaml between releases, therefore you may need to update the spack.yaml to work with your Spack instance. We will keep the Spack configuration file up to date with the most recent Spack deployment.

NERSC provides a template Spack configuration on Github Shown below is the template Spack configuration for Perlmutter:

# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
  view: false
  config:
    concretization: separately
    build_stage: $spack/var/spack/stage
    misc_cache: $spack/var/spack/misc_cache
    concretizer: clingo
    install_tree: $spack/opt/spack
  concretizer:
    reuse: true
  # Perlmutter compiler and package preferences
  include:
  - /global/common/software/spackecp/perlmutter/spack_settings/compilers.yaml
  - /global/common/software/spackecp/perlmutter/spack_settings/packages.yaml

  mirrors:
    perlmutter-e4s-23.08: file:///global/common/software/spackecp/mirrors/perlmutter-e4s-23.08

  # Spack Chaining, if you want to use existing software stack
  upstreams:
    perlmutter-e4s-23.08:
      install_tree: /global/common/software/spackecp/perlmutter/e4s-23.08/default/spack/opt/spack

  # add package specs to the `specs` list
  specs: [ ]

Compiler and Package Preferences

In system-provided Spack instances, we configured the settings to use NERSC's recommended compilers and package preferences. This is defined via the include keyword.

  include:
  - /global/common/software/spackecp/perlmutter/spack_settings/compilers.yaml
  - /global/common/software/spackecp/perlmutter/spack_settings/packages.yaml

We included these configurations in the spack.yaml so you don't have to define them, and we encourage you to use these settings and override any preferences by defining them in your own Spack configuration. You are welcome to add any additional compilers or package preferences.

Configuring your Spack Instance

By default, your Spack instance may not be configured with the Perlmutter compiler and package preferences. This can be addressed by copying the files into the $SPACK_ROOT/etc/spack directory which applies configuration changes to this instance of Spack.

You can copy the configuration files by running the following commands: 

elvis@perlmutter> cp /global/common/software/spackecp/perlmutter/spack_settings/compilers.yaml $SPACK_ROOT/etc/spack/
elvis@perlmutter> cp /global/common/software/spackecp/perlmutter/spack_settings/packages.yaml $SPACK_ROOT/etc/spack/

By running these commands you instruct Spack to use the NERSC provided compilers and package preferences. Furthermore, these changes will affect each Spack environment in the Spack instance.

Likewise, if you want to set the Spack upstreams in your Spack instance, you can run the following:

elvis@perlmutter> spack config get upstreams > $SPACK_ROOT/etc/upstreams.yaml

Please make sure you are in the spack environment, and the expected output should be similar to the one found in spack.yaml as shown below

elvis@perlmutter> spack config get upstreams
upstreams:
  perlmutter-e4s-23.08:
    install_tree: /global/common/software/spackecp/perlmutter/e4s-23.08/default/spack/opt/spack

This would be good time to test the changes you made by deactivating your Spack environment.

elvis@perlmutter> despacktivate
elvis@perlmutter>  spack env st
==> No active environment
elvis@perlmutter> spack compiler list
==> Available compilers
-- cce sles15-any -----------------------------------------------
cce@17.0.0  cce@16.0.0  cce@15.0.1

-- gcc sles15-any -----------------------------------------------
gcc@12.3.0  gcc@11.2.0  gcc@10.3.0

-- nvhpc sles15-any ---------------------------------------------
nvhpc@23.9  nvhpc@23.1  nvhpc@22.7

You can run the following command to see if package preference is picked up from $SPACK_ROOT/etc/spack/packages.yaml

elvis@perlmutter> spack config blame packages

The output may be long, but you want to pay attention to section where it defines package preference such as cray-mpich, since that is the preferred MPI provider on Perlmutter. You can extract this information via 

elvis@perlmutter> spack config blame packages | grep -A11 'cray-mpich:'
/global/u1/e/elvis/gitrepos/spack/etc/spack/packages.yaml:28             cray-mpich:
/global/u1/e/elvis/gitrepos/spack/etc/spack/packages.yaml:29               buildable: False
/global/u1/e/elvis/gitrepos/spack/etc/spack/packages.yaml:30               externals:
/global/u1/e/elvis/gitrepos/spack/etc/spack/packages.yaml:31               - spec: cray-mpich@os %gcc@12.3.0
/global/u1/e/elvis/gitrepos/spack/etc/spack/packages.yaml:32                 prefix: /opt/cray/pe/mpich/8.1.28/ofi/gnu/12.3
/global/u1/e/elvis/gitrepos/spack/etc/spack/packages.yaml:33                 modules: [cray-mpich/8.1.27, cudatoolkit/12.2]
/global/u1/e/elvis/gitrepos/spack/etc/spack/packages.yaml:34               - spec: cray-mpich@os %nvhpc@23.9
/global/u1/e/elvis/gitrepos/spack/etc/spack/packages.yaml:35                 prefix: /opt/cray/pe/mpich/8.1.28/ofi/nvidia/23.3
/global/u1/e/elvis/gitrepos/spack/etc/spack/packages.yaml:36                 modules: [cray-mpich/8.1.27, cudatoolkit/12.2]
/global/u1/e/elvis/gitrepos/spack/etc/spack/packages.yaml:37               - spec: cray-mpich@os %cce@17.0.0
/global/u1/e/elvis/gitrepos/spack/etc/spack/packages.yaml:38                 prefix: /opt/cray/pe/mpich/8.1.28/ofi/cray/17.0
/global/u1/e/elvis/gitrepos/spack/etc/spack/packages.yaml:39                 modules: [cray-mpich/8.1.27, cudatoolkit/12.2]

Once you are satisfied with the changes, please activate your Spack environment that you created earlier by running:

elvis@perlmutter> spack env activate demo

Spack Upstreams

Spack upstreams are directories containing other Spack instances that Spack can search for pre-installed Spack packages. They make installing additional packages quicker by avoiding installing Spack packages and dependencies already available in another Spack instance. We define Spack upstreams install locations in the spack.yaml file. For example, we defined the perlmutter-e4s-23.08 upstream install location with the upstreams keyword as follows,

  upstreams:
    perlmutter-e4s-23.08:
      install_tree: /global/common/software/spackecp/perlmutter/e4s-23.08/default/spack/opt/spack

Let's say you want to install zlib. Take note, we install the packages in the Spack upstream location since they were previously installed:

elvis@perlmutter> spack install --add zlib
==> Installing gmake-4.4.1-6b72ibntxvoxfjc3vs32fsm3wlxccjgu [1/2]
==> No binary for gmake-4.4.1-6b72ibntxvoxfjc3vs32fsm3wlxccjgu found: installing from source
==> Using cached archive: /global/u1/e/elvis/spack/var/spack/cache/_source-cache/archive/dd/dd16fb1d67bfab79a72f5e8390735c49e3e8e70b4945a15ab1f81ddb78658fb3.tar.gz
==> No patches needed for gmake
==> gmake: Executing phase: 'autoreconf'
==> gmake: Executing phase: 'configure'
==> gmake: Executing phase: 'build'
==> gmake: Executing phase: 'install'
==> gmake: Successfully installed gmake-4.4.1-6b72ibntxvoxfjc3vs32fsm3wlxccjgu
  Stage: 0.17s.  Autoreconf: 0.00s.  Configure: 14.96s.  Build: 1.29s.  Install: 22.98s.  Post-install: 0.09s.  Total: 39.56s
[+] /global/u1/e/elvis/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/gmake-4.4.1-6b72ibntxvoxfjc3vs32fsm3wlxccjgu
==> Installing zlib-1.3-psgimln4rrkgwy3auz2vmfitwybxvpoo [2/2]
==> No binary for zlib-1.3-psgimln4rrkgwy3auz2vmfitwybxvpoo found: installing from source
==> Using cached archive: /global/u1/e/elvis/spack/var/spack/cache/_source-cache/archive/ff/ff0ba4c292013dbc27530b3a81e1f9a813cd39de01ca5e0f8bf355702efa593e.tar.gz
==> No patches needed for zlib
==> zlib: Executing phase: 'edit'
==> zlib: Executing phase: 'build'
==> zlib: Executing phase: 'install'
==> zlib: Successfully installed zlib-1.3-psgimln4rrkgwy3auz2vmfitwybxvpoo
  Stage: 0.14s.  Edit: 1.01s.  Build: 0.78s.  Install: 0.08s.  Post-install: 0.11s.  Total: 2.17s
[+] /global/u1/e/elvis/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/zlib-1.3-psgimln4rrkgwy3auz2vmfitwybxvpoo

Buildcache Mirrors

We configured buildcache mirrors so you can install packages from the buildcache instead of building from source. Please note that mirror precedence is top-down as Spack attempts to search for specs from each mirror. We defined the following mirrors:

  mirrors:
    perlmutter-e4s-23.08: file:///global/common/software/spackecp/mirrors/perlmutter-e4s-23.08

You will need to install the GPG key to use the buildcache since packages are signed and Spack needs the GPG key for verification. Set the environment variable SPACK_GNUPGHOME to point to your $HOME/.gnupg. That way Spack will install the GPG key in your user space.

elvis@perlmutter> export SPACK_GNUPGHOME=$HOME/.gnupg
elvis@perlmutter> spack buildcache keys -it
==> Fetching file:///global/common/software/spackecp/mirrors/perlmutter-spack-develop/build_cache/_pgp/B5FDE18F615783AF078ED29C3BD6B0E9935AEB8F.pub
gpg: key 3BD6B0E9935AEB8F: "GPG Key - e4s <shahzebsiddiqui@lbl.gov>" not changed
gpg: Total number processed: 1
gpg:              unchanged: 1
gpg: key 3BD6B0E9935AEB8F: "GPG Key - e4s <shahzebsiddiqui@lbl.gov>" not changed
gpg: Total number processed: 1
gpg:              unchanged: 1

You can see a list of mirrors by running the following:

elvis@perlmutter> spack mirror list
perlmutter-e4s-23.08 [sb] file:///global/common/software/spackecp/mirrors/perlmutter-e4s-23.08
spack-public         [s ] https://mirror.spack.io

You can see all packages in the buildcache by running spack buildcache list, which will show for all mirrors. If you want to see specs from a particular mirror, we recommend you remove one of the mirrors and rerun the same command.

Use Cases

Use Cases 1: Overriding Package Preferences

Often times, you will run into a situation where you need to override a package preference for a particular spec. We provide default package preferences that are provided in the packages.yaml file that is included in the Spack configuration.

Let's try running spack spec hdf5 and take note that Spack will use an existing installation of HDF5 which is denoted by the [^] symbol.

(spack-pyenv) elvis@perlmutter> spack spec -IL hdf5
Input spec
--------------------------------
 -   hdf5

Concretized
--------------------------------
[^]  sbbcwluvwjlzxtb5rloktld3ul7mrkby  hdf5@1.14.2%gcc@11.2.0~cxx+fortran+hl~ipo~java~map+mpi+shared~szip~threadsafe+tools api=v18 build_system=cmake build_type=Release generator=make arch=linux-sles15-zen3
[^]  xzmiunmizltzk5duclcgwd74jf5ljelh      ^cmake@3.26.3%gcc@11.2.0~doc+ncurses+ownlibs build_system=generic build_type=Release arch=linux-sles15-zen3
[e]  g2bxzmnqb7z2qouuzmyx2j7l3ofhfkrr          ^curl@7.66.0%gcc@11.2.0~gssapi~ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools libs=shared,static tls=openssl arch=linux-sles15-zen3
[e]  3whqqtcmv6y4dwmbq7gnww5i6ajis4c5          ^ncurses@6.1%gcc@11.2.0~symlinks+termlib abi=none build_system=autotools arch=linux-sles15-zen3
[e]  hxsudafifjkox7ydibdrcecq2to2iofx      ^cray-mpich@os%gcc@11.2.0 build_system=generic arch=linux-sles15-zen3
[^]  v2n34otbjhmndbudp3giswqphlii7bsu      ^gmake@4.4.1%gcc@11.2.0~guile build_system=autotools arch=linux-sles15-zen3
[e]  widvp4yazpnoc2u6aqgvjoyth6uwbzs4      ^pkg-config@0.29.2%gcc@11.2.0+internal_glib build_system=autotools arch=linux-sles15-zen3
[^]  tx35gb7xta67ja4l2e44jlz5wxy3ax6n      ^zlib@1.2.13%gcc@11.2.0+optimize+pic+shared build_system=makefile arch=linux-sles15-zen3

This is set because we have set reuse: True in the Spack configuration which can be shown by running

(spack-pyenv) elvis@perlmutter> spack config blame concretizer | grep -A1 'reuse:'
/global/u1/e/elvis/spack/var/spack/environments/demo/spack.yaml:14    reuse: True
/global/u1/e/elvis/spack/etc/spack/defaults/concretizer.yaml:23       targets:

Spack will reuse existing installations of packages if available since we set reuse: True in the Spack configuration. In order to build a new installation of HDF5, we must disable reuse by setting reuse: False in the Spack configuration. Let's update the spack.yaml by changing reuse: true to reuse: false and rerun the same command. Take note that the hash has changed from the previous output and the [^] symbol is no longer present.

(spack-pyenv) elvis@perlmutter> spack spec -IL hdf5
Input spec
--------------------------------
 -   hdf5

Concretized
--------------------------------
 -   so4umdd2hleawdgqezwawfhfyi2zqjip  hdf5@1.14.3%gcc@11.2.0~cxx+fortran+hl~ipo~java~map+mpi+shared~subfiling~szip~threadsafe+tools api=v18 build_system=cmake build_type=Release generator=make arch=linux-sles15-zen3
 -   t5dmmijqy2ofnlqzi3oeiiqf37ezfg6a      ^cmake@3.27.7%gcc@11.2.0~doc+ncurses+ownlibs build_system=generic build_type=Release arch=linux-sles15-zen3
[e]  g2bxzmnqb7z2qouuzmyx2j7l3ofhfkrr          ^curl@7.66.0%gcc@11.2.0~gssapi~ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools libs=shared,static tls=openssl arch=linux-sles15-zen3
[e]  3whqqtcmv6y4dwmbq7gnww5i6ajis4c5          ^ncurses@6.1%gcc@11.2.0~symlinks+termlib abi=none build_system=autotools arch=linux-sles15-zen3
[e]  hxsudafifjkox7ydibdrcecq2to2iofx      ^cray-mpich@os%gcc@11.2.0 build_system=generic arch=linux-sles15-zen3
 -   34q2rrsht3es5twmcb2yjcrgwihyv3xk      ^gmake@4.4.1%gcc@11.2.0~guile build_system=generic arch=linux-sles15-zen3
[e]  widvp4yazpnoc2u6aqgvjoyth6uwbzs4      ^pkg-config@0.29.2%gcc@11.2.0+internal_glib build_system=autotools arch=linux-sles15-zen3
 -   g5jnaexsu2foavtounyfifq5zu6kh6rz      ^zlib-ng@2.1.4%gcc@11.2.0+compat+opt build_system=autotools arch=linux-sles15-zen3

If you analyze the output of spack config blame packages you will see an entry for hdf5 as follows, which will be used as default preference whenever you build HDF5:

elvis@perlmutter> spack config blame packages | grep -A1 'hdf5:'
/global/common/software/spackecp/perlmutter/spack_settings/packages.yaml:93     hdf5:
/global/common/software/spackecp/perlmutter/spack_settings/packages.yaml:94       variants: +fortran +hl +shared api=v18

Let's say you want to disable the fortran variant (~fortran) for hdf5. You could run spack spec hdf5 ~fortran but we want to make these changes in spack.yaml so they are permanent. To accomplish this task, use a double colon :: next to the package section that needs to be overridden.

Shown below is an example of overriding the fortran variant for hdf5, take note of the double colon:

# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
  view: false
  config:
    concretization: separately
    build_stage: $spack/var/spack/stage
    misc_cache: $spack/var/spack/misc_cache
    concretizer: clingo
    install_tree: $spack/opt/spack
  concretizer:
    reuse: false
  packages:
    hdf5::
      variants: ~fortran
  include:
  - /global/common/software/spackecp/perlmutter/spack_settings/compilers.yaml
  - /global/common/software/spackecp/perlmutter/spack_settings/packages.yaml

  mirrors:
    perlmutter-e4s-23.08: file:///global/common/software/spackecp/mirrors/perlmutter-e4s-23.08

  upstreams:
    perlmutter-e4s-23.08:
      install_tree: /global/common/software/spackecp/perlmutter/e4s-23.08/default/spack/opt/spack
  specs:
  - hdf5

Let's rerun spack spec hdf5 and take note the ~fortran variant is set for the HDF5 package.

elvis@perlmutter> spack spec hdf5
Input spec
--------------------------------
 -   hdf5

Concretized
--------------------------------
 -   ad6kcu33p6bo7lcwzculsaols4yewxez  hdf5@1.14.3%gcc@11.2.0~cxx~fortran~hl~ipo~java~map+mpi+shared~subfiling~szip~threadsafe+tools api=default build_system=cmake build_type=Release generator=make arch=linux-sles15-zen3
 -   t5dmmijqy2ofnlqzi3oeiiqf37ezfg6a      ^cmake@3.27.7%gcc@11.2.0~doc+ncurses+ownlibs build_system=generic build_type=Release arch=linux-sles15-zen3
[e]  g2bxzmnqb7z2qouuzmyx2j7l3ofhfkrr          ^curl@7.66.0%gcc@11.2.0~gssapi~ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools libs=shared,static tls=openssl arch=linux-sles15-zen3
[e]  3whqqtcmv6y4dwmbq7gnww5i6ajis4c5          ^ncurses@6.1%gcc@11.2.0~symlinks+termlib abi=none build_system=autotools arch=linux-sles15-zen3
[e]  hxsudafifjkox7ydibdrcecq2to2iofx      ^cray-mpich@os%gcc@11.2.0 build_system=generic arch=linux-sles15-zen3
 -   34q2rrsht3es5twmcb2yjcrgwihyv3xk      ^gmake@4.4.1%gcc@11.2.0~guile build_system=generic arch=linux-sles15-zen3
[e]  widvp4yazpnoc2u6aqgvjoyth6uwbzs4      ^pkg-config@0.29.2%gcc@11.2.0+internal_glib build_system=autotools arch=linux-sles15-zen3
 -   g5jnaexsu2foavtounyfifq5zu6kh6rz      ^zlib-ng@2.1.4%gcc@11.2.0+compat+opt build_system=autotools arch=linux-sles15-zen3

If we run spack config blame packages you should see the the HDF5 preference will be overridden by the spack.yaml and not the packages.yaml:

elvis@perlmutter> spack config blame packages | grep -A1 'hdf5:'
---                                                                        packages:
/global/u1/e/elvis/spack/environments/demo/spack.yaml:14                                  hdf5:
/global/u1/e/elvis/spack/environments/demo/spack.yaml:15                                    variants:  ~fortran

Use Case 2: Changing compiler preferences

By default, we pick a gcc compiler as the default compiler. This can be shown in the output of spack config blame packages; under the all section you will see a list of compilers defined by keyword compiler:

elvis@perlmutter> spack config blame packages | grep -A1 'all:'
/global/common/software/spackecp/perlmutter/spack_settings/packages.yaml:2      all:
/global/u1/e/elvis/spack/etc/spack/defaults/packages.yaml:18                   compiler: [gcc@11.2.0, nvhpc@22.7, cce@15.0.0, gcc, intel, pgi, clang, xl, nag, fj, aocc]

Let's change the default compiler to cce@16.0.0. To do this, we simply add the following two lines to the Spack configuration under the packages: heading:

    all:
      compiler:: [cce@16.0.0]

For this exercise, use this Spack configuration:

# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
  view: false
  config:
    concretization: separately
    build_stage: $spack/var/spack/stage
    misc_cache: $spack/var/spack/misc_cache
    concretizer: clingo
    install_tree: $spack/opt/spack
  concretizer:
    reuse: false
  packages:
    all:
      compiler:: [cce@16.0.0]
  include:
  - /global/common/software/spackecp/perlmutter/spack_settings/compilers.yaml
  - /global/common/software/spackecp/perlmutter/spack_settings/packages.yaml

  mirrors:
    perlmutter-e4s-23.08: file:///global/common/software/spackecp/mirrors/perlmutter-e4s-23.08

  upstreams:
    perlmutter-e4s-23.08:
      install_tree: /global/common/software/spackecp/perlmutter/e4s-23.08/default/spack/opt/spack
  specs: [ ]

Let's try running spack spec gmake zlib. Take note of the compiler preference, and notice that all packages will be built with the cce compiler.

elvis@perlmutter> spack spec gmake zlib
Input spec
--------------------------------
 -   gmake

Concretized
--------------------------------
 -   2rhamsokocf57xm4ao43rqcngjr22i7y  gmake@4.4.1%cce@16.0.0~guile build_system=generic arch=linux-sles15-zen3

Input spec
--------------------------------
 -   zlib

Concretized
--------------------------------
 -   t2vpomnjk4i4mtwdsi6fs66umtlljy74  zlib@1.3%cce@16.0.0+optimize+pic+shared build_system=makefile arch=linux-sles15-zen3
 -   2rhamsokocf57xm4ao43rqcngjr22i7y      ^gmake@4.4.1%cce@16.0.0~guile build_system=generic arch=linux-sles15-zen3