Perlmutter Spack Environments¶
In this guide, we will demonstrate how one can leverage Spack to build software on Perlmutter and install arbitrary software. We recommend you also look at Spack Training for Perlmutter.
Setup¶
In order to get started you will need to clone an instance of Spack and the spack-infrastructure repository in your user space.
git clone https://github.com/spack/spack
git clone https://github.com/NERSC/spack-infrastructure.git
Before you get started, we recommend you source the setup-env.sh script found in the root of the spack-infrastructure repo. This will create a Python environment in order for you to do Spack builds. Spack requires clingo in-order to bootstrap clingo however we observed issues where Spack was unable to bootstrap clingo see spack/28315. We found that installing clingo as a Python package addressed the issue.
elvis@login34> cd spack-infrastructure/
elvis@login34> source setup-env.sh
Collecting clingo
Using cached clingo-5.5.2-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB)
Collecting cffi
Using cached cffi-1.15.1-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (402 kB)
Collecting pycparser
Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Installing collected packages: pycparser, cffi, clingo
Successfully installed cffi-1.15.1 clingo-5.5.2 pycparser-2.21
WARNING: You are using pip version 20.2.3; however, version 21.3.1 is available.
You should consider upgrading via the '/global/homes/e/elvis/spack-infrastructure/spack-pyenv/bin/python3 -m pip install --upgrade pip' command.
/global/homes/e/elvis/spack-infrastructure/spack-pyenv/bin/python
Package Version
---------- -------
cffi 1.15.1
clingo 5.5.2
pip 20.2.3
pycparser 2.21
setuptools 44.1.1
WARNING: You are using pip version 20.2.3; however, version 21.3.1 is available.
You should consider upgrading via the '/global/homes/e/elvis/spack-infrastructure/spack-pyenv/bin/python3 -m pip install --upgrade pip' command.
We provide a template Spack configuration that you can use to create a Spack environment with pre-configured settings.
cd spack-infrastructure/
spack env create demo spack-configs/perlmutter-user-spack/spack.yaml
spack env activate demo
Spack Configuration¶
Changes to spack.yaml between Spack releases
Spack has changed the YAML structure of spack.yaml between releases, therefore you may need to update the spack.yaml to work with your Spack instance. We will keep the Spack configuration file up to date with the most recent E4S deployment.
You can find the template spack configuration by clicking here Shown below is the template Spack configuration for Perlmutter:
# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
view: false
config:
concretization: separately
build_stage: $spack/var/spack/stage
misc_cache: $spack/var/spack/misc_cache
concretizer: clingo
install_tree: $spack/opt/spack
# Perlmutter compiler and package preferences
include:
- /global/common/software/spackecp/perlmutter/spack_settings/compilers.yaml
- /global/common/software/spackecp/perlmutter/spack_settings/packages.yaml
mirrors:
perlmutter-spack-develop: file:///global/common/software/spackecp/mirrors/perlmutter-spack-develop
perlmutter-e4s-22.11: file:///global/common/software/spackecp/mirrors/perlmutter-e4s-22.11
perlmutter-e4s-22.05: file:///global/common/software/spackecp/mirrors/perlmutter-e4s-22.05
# Spack Chaining, if you want to use existing software stack
upstreams:
perlmutter-e4s-22.11:
install_tree: /global/common/software/spackecp/perlmutter/e4s-22.11/default/spack/opt/spack
perlmutter-e4s-22.05:
install_tree: /global/common/software/spackecp/perlmutter/e4s-22.05/default/spack/opt/spack
# add package specs to the `specs` list
specs: [ ]
Compiler and Package Preferences¶
In system provided Spack instances, We configured the settings to use NERSC's recommended compilers and package preferences. This is defined via the include keyword.
include:
- /global/common/software/spackecp/perlmutter/spack_settings/compilers.yaml
- /global/common/software/spackecp/perlmutter/spack_settings/packages.yaml
We included these configurations in the spack.yaml so you don't have to define them, and we encourage you use these settings and override any preferences by defining them in your own Spack configuration. You are welcome to add any additional compilers or package preferences.
Spack Upstreams¶
Spack upstreams are directories containing other Spack instances that Spack can search for pre-installed Spack packages. They make installing additional packages quicker by avoiding installing Spack packages and dependencies already available in another Spack instance. We define Spack upstreams install locations in the spack.yaml file. For example, we defined the perlmutter-e4s-22.05 upstream install location with the upstreams keyword as follows,
upstreams:
perlmutter-e4s-22.05:
install_tree: /global/common/software/spackecp/perlmutter/e4s-22.05/default/spack/opt/spack
Let's say you want to install cmake and papi. Take note, we install the packages in the Spack upstream location since they were previously installed:
elvis@perlmutter> spack install cmake papi
==> Warning: included configuration files should be updated manually [files=/global/common/software/spackecp/perlmutter/spack_settings/compilers.yaml, /global/common/software/spackecp/perlmutter/spack_settings/packages.yaml]
==> All of the packages are already installed
==> Updating view at /global/u1/s/elvis/spack-infrastructure/spack/var/spack/environments/demo/.spack-env/view
==> Warning: Skipping external package: curl@7.66.0%gcc@11.2.0~gssapi~ldap~libidn2~librtmp~libssh~libssh2~nghttp2 libs=shared,static tls=openssl arch=cray-sles15-zen3/zkrv7nh
==> Warning: Skipping external package: bzip2@1.0.6%gcc@11.2.0~debug~pic+shared arch=cray-sles15-zen3/4g7s6qp
==> Warning: Skipping external package: ncurses@6.1%gcc@11.2.0~symlinks+termlib abi=none arch=cray-sles15-zen3/i6ri5ef
Examine the directory paths. We see cmake was pulled from the Spack upstream while, PAPI was installed into the directory of our Spack instance.
elvis@perlmutter> spack find -Lvp cmake papi
==> In environment demo
==> Root specs
-------------------------------- cmake -------------------------------- papi
==> 2 installed packages
-- cray-sles15-zen3 / gcc@11.2.0 --------------------------------
p23fzuowp4yuitemelic7f65nwybthxd cmake@3.23.1~doc+ncurses~ownlibs~qt build_type=Release /global/common/software/spackecp/perlmutter/e4s-22.05/73973/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/cmake-3.23.1-p23fzuowp4yuitemelic7f65nwybthxd
s2y4nrvu6whr6hhgi63aa3nqwz2d35af papi@6.0.0.1~cuda+example~infiniband~lmsensors~nvml~powercap~rapl~rocm~rocm_smi~sde+shared~static_tools /global/u1/s/elvis/spack-infrastructure/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/papi-6.0.0.1-s2y4nrvu6whr6hhgi63aa3nqwz2d35af
Buildcache Mirrors¶
We configured buildcache mirrors so you can install packages from the buildcache instead of building from source. Please note that mirror precedence is top-down as Spack attempts to search for specs from each mirror. We defined the following mirrors,
mirrors:
perlmutter-spack-develop: file:///global/common/software/spackecp/mirrors/perlmutter-spack-develop
perlmutter-e4s-22.05: file:///global/common/software/spackecp/mirrors/perlmutter-e4s-22.05
You will need to install the GPG key to use the buildcache since packages are signed and Spack needs the GPG key for verification. Set the environment variable SPACK_GNUPGHOME to point to your $HOME/.gnupg. That way Spack will install the GPG key in your user space.
elvis@perlmutter> export SPACK_GNUPGHOME=$HOME/.gnupg
elvis@perlmutter> spack buildcache keys -it
==> Fetching file:///global/common/software/spackecp/mirrors/perlmutter-spack-develop/build_cache/_pgp/B5FDE18F615783AF078ED29C3BD6B0E9935AEB8F.pub
gpg: key 3BD6B0E9935AEB8F: "GPG Key - e4s <shahzebsiddiqui@lbl.gov>" not changed
gpg: Total number processed: 1
gpg: unchanged: 1
gpg: key 3BD6B0E9935AEB8F: "GPG Key - e4s <shahzebsiddiqui@lbl.gov>" not changed
gpg: Total number processed: 1
gpg: unchanged: 1
You can see a list of mirrors by running the following,
elvis@perlmutter> spack mirror list
perlmutter-spack-develop file:///global/common/software/spackecp/mirrors/perlmutter-spack-develop
perlmutter-e4s-22.05 file:///global/common/software/spackecp/mirrors/perlmutter-e4s-22.05
spack-public https://mirror.spack.io
You can see all packages in the buildcache by running spack buildcache list, which will show for all mirrors. If you want to see specs from a particular mirror, we recommend you remove one of the mirrors and rerun the same command.
Use Cases¶
Use Cases 1: Overriding Package Preferences¶
Often times, you will run into a situation where you need to override a package preference for a particular spec. We provide default package preferences that are provided in the packages.yaml file that is included in the spack configuration.
For instance, let's say you run the following command spack spec hdf5. You may notice the following output
elvis@perlmutter> spack spec hdf5
Input spec
--------------------------------
hdf5
Concretized
--------------------------------
hdf5@1.12.2%gcc@11.2.0~cxx+fortran+hl~ipo~java+mpi+shared~szip~threadsafe+tools api=v18 build_type=RelWithDebInfo arch=cray-sles15-zen3
...
If you analyze the output of spack config blame packages you will see an entry for hdf5 as follows, which will be used as default preference whenever you build hdf5
/global/common/software/spackecp/perlmutter/spack_settings/packages.yaml:80 hdf5:
/global/common/software/spackecp/perlmutter/spack_settings/packages.yaml:81 variants: +fortran +hl +shared api=v18
Let's say you want to disable the fortran variant for hdf5. You could do run spack spec hdf5 ~fortran but we want to make these changes in spack.yaml so they are permanent. To accomplish this task, use double colon :: next to the package section that needs to be overridden.
Shown below is an example of overriding the fortran variant for hdf5, take note of the double colon:
# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
view: false
config:
concretization: separately
build_stage: $spack/var/spack/stage
misc_cache: $spack/var/spack/misc_cache
concretizer: clingo
install_tree: $spack/opt/spack
packages:
hdf5::
variants: ~fortran +hl +shared api=v18
include:
- /global/common/software/spackecp/perlmutter/spack_settings/compilers.yaml
- /global/common/software/spackecp/perlmutter/spack_settings/packages.yaml
mirrors:
perlmutter-spack-develop: file:///global/common/software/spackecp/mirrors/perlmutter-spack-develop
perlmutter-e4s-22.11: file:///global/common/software/spackecp/mirrors/perlmutter-e4s-22.11
perlmutter-e4s-22.05: file:///global/common/software/spackecp/mirrors/perlmutter-e4s-22.05
specs:
- hdf5
Let's concretize this environment and take note of the variant ~fortran in the output for hdf5
elvis@perlmutter> spack concretize -f
==> Starting concretization
- rne4jm2 hdf5@1.12.2%gcc@11.2.0~cxx~fortran+hl~ipo~java+mpi+shared~szip~threadsafe+tools api=v18 build_type=RelWithDebInfo arch=cray-sles15-zen3
- xqy4j4r ^cmake@3.23.3%gcc@11.2.0~doc+ncurses~ownlibs~qt build_type=Release arch=cray-sles15-zen3
- 4tvm5an ^curl@7.66.0%gcc@11.2.0~gssapi~ldap~libidn2~librtmp~libssh~libssh2~nghttp2 libs=shared,static tls=openssl arch=cray-sles15-zen3
- kb64bk6 ^expat@2.4.8%gcc@11.2.0+libbsd arch=cray-sles15-zen3
- 6ii42bx ^libbsd@0.11.5%gcc@11.2.0 arch=cray-sles15-zen3
- nzueemp ^libmd@1.0.4%gcc@11.2.0 arch=cray-sles15-zen3
- 2slycpx ^libarchive@3.5.2%gcc@11.2.0+iconv compression=bz2lib,lz4,lzma,lzo2,zlib,zstd crypto=mbedtls libs=shared,static programs=none xar=expat arch=cray-sles15-zen3
- huv6fd6 ^bzip2@1.0.6%gcc@11.2.0~debug~pic+shared arch=cray-sles15-zen3
- po4cjxv ^libiconv@1.16%gcc@11.2.0 libs=shared,static arch=cray-sles15-zen3
- xxbkaph ^lz4@1.9.3%gcc@11.2.0 libs=shared,static arch=cray-sles15-zen3
- rlpwurc ^lzo@2.10%gcc@11.2.0 libs=shared,static arch=cray-sles15-zen3
- ucarxpp ^mbedtls@2.28.0%gcc@11.2.0+pic build_type=Release libs=static arch=cray-sles15-zen3
- 53dgd2y ^xz@5.2.5%gcc@11.2.0~pic libs=shared,static arch=cray-sles15-zen3
- ymk6b64 ^zlib@1.2.12%gcc@11.2.0+optimize+pic+shared patches=0d38234 arch=cray-sles15-zen3
- rtaozj7 ^zstd@1.5.2%gcc@11.2.0~programs libs=shared,static arch=cray-sles15-zen3
- chuwtnb ^libuv@1.44.1%gcc@11.2.0 arch=cray-sles15-zen3
- 7m3pjmb ^ncurses@6.1%gcc@11.2.0~symlinks+termlib abi=none arch=cray-sles15-zen3
- xjxbsef ^rhash@1.4.2%gcc@11.2.0 patches=093518c,3fbfe46 arch=cray-sles15-zen3
- puisy4b ^cray-mpich@8.1.22%gcc@11.2.0+wrappers arch=cray-sles15-zen3
- pysfq3q ^pkgconf@1.8.0%gcc@11.2.0 arch=cray-sles15-zen3
==> Warning: included configuration files should be updated manually [files=/global/common/software/spackecp/perlmutter/spack_settings/compilers.yaml, /global/common/software/spackecp/perlmutter/spack_settings/packages.yaml]
If we run spack config blame packages you should see hdf5 preference will be overridden by the spack.yaml and not the packages.yaml
elvis@perlmutter> spack config blame packages
--- packages:
/global/u1/e/elvis/spack-demo/spack.yaml:14 hdf5:
/global/u1/e/elvis/spack-demo/spack.yaml:15 variants: ~fortran +hl +shared api=v18
Use Case 2: Changing compiler preferences¶
By default, we pick a gcc compiler as the default compiler. This can be shown in the output of spack config blame packages under the all section you will see a list of compilers defined by keyword compiler
/global/common/software/spackecp/perlmutter/spack_settings/packages.yaml:2 all:
/global/u1/e/elvis/spack/etc/spack/defaults/packages.yaml:18 compiler: [gcc@11.2.0, nvhpc@22.5, cce@15.0.0, gcc, intel, pgi, clang, xl, nag, fj, aocc]
Let's change the default compiler to cce@15.0.0, To do this, we simply add the following two lines to the Spack configuration under the packages: heading.
all:
compiler:: [cce@15.0.0]
Afterwards, the Spack configuration will appear as follows:
# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
view: false
config:
concretization: separately
build_stage: $spack/var/spack/stage
misc_cache: $spack/var/spack/misc_cache
concretizer: clingo
install_tree: $spack/opt/spack
packages:
all:
compiler:: [cce@15.0.0]
hdf5::
variants: ~fortran +hl +shared api=v18
include:
- /global/common/software/spackecp/perlmutter/spack_settings/compilers.yaml
- /global/common/software/spackecp/perlmutter/spack_settings/packages.yaml
mirrors:
perlmutter-spack-develop: file:///global/common/software/spackecp/mirrors/perlmutter-spack-develop
perlmutter-e4s-22.11: file:///global/common/software/spackecp/mirrors/perlmutter-e4s-22.11
perlmutter-e4s-22.05: file:///global/common/software/spackecp/mirrors/perlmutter-e4s-22.05
specs:
- hdf5
Let's concretize this environment. Take note of the compiler preference, and notice that all packages will be built with cce compiler.
elvis@perlmutter> spack concretize -f
==> Starting concretization
- yhmflwx hdf5@1.12.2%cce@15.0.0~cxx~fortran+hl~ipo~java+mpi+shared~szip~threadsafe+tools api=v18 build_type=RelWithDebInfo arch=cray-sles15-zen3
- mnftbts ^cmake@3.23.3%cce@15.0.0~doc+ncurses~ownlibs~qt build_type=Release arch=cray-sles15-zen3
- anoeglb ^curl@7.66.0%cce@15.0.0~gssapi~ldap~libidn2~librtmp~libssh~libssh2~nghttp2 libs=shared,static tls=openssl arch=cray-sles15-zen3
- ehz6aeu ^expat@2.4.8%cce@15.0.0+libbsd arch=cray-sles15-zen3
- lu52rr5 ^libbsd@0.11.5%cce@15.0.0 arch=cray-sles15-zen3
- p4t4462 ^libmd@1.0.4%cce@15.0.0 arch=cray-sles15-zen3
- prl5lb5 ^libarchive@3.5.2%cce@15.0.0+iconv compression=bz2lib,lz4,lzma,lzo2,zlib,zstd crypto=mbedtls libs=shared,static programs=none xar=expat arch=cray-sles15-zen3
- fgynajv ^bzip2@1.0.6%cce@15.0.0~debug~pic+shared arch=cray-sles15-zen3
- dw3j4po ^libiconv@1.16%cce@15.0.0 libs=shared,static arch=cray-sles15-zen3
- zb4uvsn ^lz4@1.9.3%cce@15.0.0 libs=shared,static arch=cray-sles15-zen3
- pyuisfk ^lzo@2.10%cce@15.0.0 libs=shared,static arch=cray-sles15-zen3
- 533hwpb ^mbedtls@2.28.0%cce@15.0.0+pic build_type=Release libs=static arch=cray-sles15-zen3
- cbnaud7 ^xz@5.2.5%cce@15.0.0~pic libs=shared,static arch=cray-sles15-zen3
- hytkr3d ^zlib@1.2.12%cce@15.0.0+optimize+pic+shared patches=0d38234 arch=cray-sles15-zen3
- 3skpogk ^zstd@1.5.2%cce@15.0.0~programs libs=shared,static arch=cray-sles15-zen3
- kk2j24a ^libuv@1.44.1%cce@15.0.0 arch=cray-sles15-zen3
- evngor4 ^ncurses@6.1%cce@15.0.0~symlinks+termlib abi=none arch=cray-sles15-zen3
- 3ji5563 ^rhash@1.4.2%cce@15.0.0 patches=093518c,3fbfe46 arch=cray-sles15-zen3
- k3ga2ni ^cray-mpich@8.1.22%cce@15.0.0+wrappers arch=cray-sles15-zen3
- u5yavug ^pkgconf@1.8.0%cce@15.0.0 arch=cray-sles15-zen3