Skip to content

Spack

Spack is a package manager for supercomputers, Linux, and macOS. Spack is designed to support multiple versions and configurations of software on a wide variety of platforms and environments. Spack is non-destructive: installing a new version of a package does not break existing installations, so many configurations of the same package can coexist.

Spack Commands

Shown below are some of the most common Spack commands you will use. For a complete list of commands, see the Spack Command Reference.

Command Description
spack install <spec> Install a package by spec.
spack find List installed packages.
spack list List available packages.
spack load <spec> Load a package into the current environment.
spack unload <spec> Unload a package from the current environment.
spack load --list List packages loaded into the current environment.
spack env create <name> Create a new environment.
spack env activate <name> Activate an environment.
spack env deactivate Deactivate the current environment.
spack env status Show the status of the current environment.
spack env create -d <dir> Create an environment in a directory.
spack env activate -d <dir> Activate an environment in a directory.
spack concretize -f Force concretize an environment.
spack spec <spec> Show what would be installed given a spec.
spack spec -Il <spec> Show the install status and dependencies with hashes.
spack buildcache list List available binary packages.
spack info <spec> Show information about a package.
spack edit <spec> Open a package file in an editor.
spack config get <section> Show the YAML configuration for a section.
spack config list List all configuration scopes.
spack config blame <section> Show the YAML configuration file and line numbers for a section.

Loading spack packages into your user environment

Unlike module load that is used for loading modules into your user environment, in Spack you must use spack load to load a Spack package into your user environment.

Use Case Example: Loading HDF5

First, let's activate the gcc environment by running the following:

spack env activate -V gcc

Let's assume you want to use HDF5, first you need to determine which Spack package you want to load. We can run the following to show the installed versions of HDF5.

spack find hdf5

This will output a list of installed packages as shown:

==> 5 installed packages
-- cray-sles15-zen3 / gcc@11.2.0 --------------------------------
hdf5@1.8.22  hdf5@1.10.7  hdf5@1.10.8  hdf5@1.12.2  hdf5@1.12.2

Let's load HDF5 version 1.10.7 that was compiled with GCC version 11.2.0 into our user environment. To specify the version use @ and to specify the compiler use %. These symbols can also be combined as is the case with loading GCC version 11.2.0. Take note that Spack will load the dependencies in your user environment which is the default behavior.

elvis@perlmutter> spack load hdf5@1.10.7%gcc@11.2.0
elvis@perlmutter> spack load --list
==> 20 loaded packages
-- cray-sles15-zen3 / gcc@11.2.0 --------------------------------
bzip2@1.0.6   cray-mpich@8.1.22  expat@2.4.8  libarchive@3.5.2  libiconv@1.16  libuv@1.44.1  lzo@2.10        ncurses@6.1    rhash@1.4.2  zlib@1.2.12
cmake@3.23.1  curl@7.66.0        hdf5@1.10.7  libbsd@0.11.5     libmd@1.0.4    lz4@1.9.3     mbedtls@2.28.0  pkgconf@1.8.0  xz@5.2.5     zstd@1.5.2

Your user environment will be updated. That includes PATH, LD_LIBRARY_PATH, and other variables relevant for loading the package. The HDF5 package provides the h5cc program which will appear in your environment as shown below.

elvis@perlmutter> which h5cc
/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.10.7-6opvtfcoaxox7oskb7dzisnb32fnk5ne/h5cc
dir is /global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.10.7-6opvtfcoaxox7oskb7dzisnb32fnk5ne
gcc (GCC) 11.2.0 20210728 (Cray Inc.)
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

This confirms that HDF5 was successfully loaded.

View Shell Commands to Load the Package

It can be informative to view the commands that Spack uses when a package is loaded. For example, to see the shell commands used when zlib is loaded, use spack load --sh zlib%gcc@11.2.0.

elvis@perlmutter> spack load --sh zlib%gcc@11.2.0
export CMAKE_PREFIX_PATH=/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.12-ozmcyfjfv7i5gjjgklfsh43h67vzsuc5:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.10.7-6opvtfcoaxox7oskb7dzisnb32fnk5ne:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/pkgconf-1.8.0-nugfov2wsvpllczfy6ff7bytzt4oi34q:/opt/cray/pe/mpich/8.1.22/ofi/gnu/9.1:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/cmake-3.23.1-p23fzuowp4yuitemelic7f65nwybthxd:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/rhash-1.4.2-xjxbsefahiue4isqlzwcobgc7sgfhmqm:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/libuv-1.44.1-gitz4v6soymrlpymzvygqlqibktfsp7a:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/libarchive-3.5.2-d4gpdx2rxj4calconu7cd6kgq2ev5roe:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zstd-1.5.2-rtaozj7xdfhxengzppbqbth4mya3edpq:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.12-ozmcyfjfv7i5gjjgklfsh43h67vzsuc5:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/xz-5.2.5-tyswlp4epqidsxgtgxyefciqoz2ju6oe:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/mbedtls-2.28.0-ucarxpp47pj7yo7abt5povg3swygev3q:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/lzo-2.10-rlpwurcqberlfjlpk62go3arnneonv6u:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/lz4-1.9.3-xxbkaphnofa6gxugxszqr5kzew7chrqa:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/libiconv-1.16-3xwztgyaaco4thqqhacdst65mmnr5is3:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/expat-2.4.8-yvchkv23jw7po7meyuxckpre5uroipmd:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/libbsd-0.11.5-uomjj7io3hhggbfyzofrw5jpw5chlv74:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/libmd-1.0.4-iynqjjkz4jgtxagqy2o2tvspqfn3yvog:/global/common/software/nersc/pm-2021q4/easybuild/software/Nsight-Systems/2022.2.1:/global/common/software/nersc/pm-2021q4/easybuild/software/Nsight-Compute/2022.1.1;
export LD_LIBRARY_PATH=/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.12-ozmcyfjfv7i5gjjgklfsh43h67vzsuc5/lib:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.10.7-6opvtfcoaxox7oskb7dzisnb32fnk5ne/lib:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/pkgconf-1.8.0-nugfov2wsvpllczfy6ff7bytzt4oi34q/lib:/opt/cray/pe/mpich/8.1.22/ofi/gnu/9.1/lib:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/rhash-1.4.2-xjxbsefahiue4isqlzwcobgc7sgfhmqm/lib:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/libuv-1.44.1-gitz4v6soymrlpymzvygqlqibktfsp7a/lib:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/libarchive-3.5.2-d4gpdx2rxj4calconu7cd6kgq2ev5roe/lib:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zstd-1.5.2-rtaozj7xdfhxengzppbqbth4mya3edpq/lib:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.12-ozmcyfjfv7i5gjjgklfsh43h67vzsuc5/lib:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/xz-5.2.5-tyswlp4epqidsxgtgxyefciqoz2ju6oe/lib:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/mbedtls-2.28.0-ucarxpp47pj7yo7abt5povg3swygev3q/lib:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/lzo-2.10-rlpwurcqberlfjlpk62go3arnneonv6u/lib:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/lz4-1.9.3-xxbkaphnofa6gxugxszqr5kzew7chrqa/lib:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/libiconv-1.16-3xwztgyaaco4thqqhacdst65mmnr5is3/lib:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/expat-2.4.8-yvchkv23jw7po7meyuxckpre5uroipmd/lib:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/libbsd-0.11.5-uomjj7io3hhggbfyzofrw5jpw5chlv74/lib:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/libmd-1.0.4-iynqjjkz4jgtxagqy2o2tvspqfn3yvog/lib:/opt/nvidia/hpc_sdk/Linux_x86_64/22.5/math_libs/11.7/lib64:/opt/nvidia/hpc_sdk/Linux_x86_64/22.5/cuda/11.7/extras/CUPTI/lib64:/opt/nvidia/hpc_sdk/Linux_x86_64/22.5/cuda/11.7/extras/Debugger/lib64:/opt/nvidia/hpc_sdk/Linux_x86_64/22.5/cuda/11.7/nvvm/lib64:/opt/nvidia/hpc_sdk/Linux_x86_64/22.5/cuda/11.7/lib64:/global/common/software/nersc/pm-2022q3/sw/darshan/3.4.0/lib:/opt/cray/pe/papi/6.0.0.16/lib64:/opt/cray/pe/gcc/11.2.0/snos/lib64:/opt/cray/libfabric/1.15.2.0/lib64;
export MANPATH=/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.12-ozmcyfjfv7i5gjjgklfsh43h67vzsuc5/share/man:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/pkgconf-1.8.0-nugfov2wsvpllczfy6ff7bytzt4oi34q/share/man:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/rhash-1.4.2-xjxbsefahiue4isqlzwcobgc7sgfhmqm/share/man:/usr/share/man:/usr/man:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/libarchive-3.5.2-d4gpdx2rxj4calconu7cd6kgq2ev5roe/share/man:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.12-ozmcyfjfv7i5gjjgklfsh43h67vzsuc5/share/man:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/xz-5.2.5-tyswlp4epqidsxgtgxyefciqoz2ju6oe/share/man:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/lz4-1.9.3-xxbkaphnofa6gxugxszqr5kzew7chrqa/share/man:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/libiconv-1.16-3xwztgyaaco4thqqhacdst65mmnr5is3/share/man:/usr/share/man:/usr/man:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/libbsd-0.11.5-uomjj7io3hhggbfyzofrw5jpw5chlv74/share/man:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/libmd-1.0.4-iynqjjkz4jgtxagqy2o2tvspqfn3yvog/share/man:/usr/share/man:/usr/man:/opt/nvidia/hpc_sdk/Linux_x86_64/22.5/cuda/11.7/doc/man:/opt/cray/pe/perftools/22.09.0/man:/opt/cray/pe/papi/6.0.0.16/share/pdoc/man:/opt/cray/pe/gcc/11.2.0/snos/share/man:/opt/cray/pe/craype/2.7.19/man:/opt/cray/pe/mpich/8.1.22/ofi/man:/opt/cray/pe/mpich/8.1.22/man/mpich:/opt/cray/pe/libsci/22.11.1.2/man:/opt/cray/pe/man/csmlversion:/opt/cray/pe/dsmml/0.2.2/dsmml/man:/opt/cray/libfabric/1.15.2.0/share/man:/usr/share/lmod/lmod/share/man:/usr/local/man:/usr/share/man:/usr/man:;
export PKG_CONFIG_PATH=/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.12-ozmcyfjfv7i5gjjgklfsh43h67vzsuc5/lib/pkgconfig:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.10.7-6opvtfcoaxox7oskb7dzisnb32fnk5ne/lib/pkgconfig:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/pkgconf-1.8.0-nugfov2wsvpllczfy6ff7bytzt4oi34q/lib/pkgconfig:/opt/cray/pe/mpich/8.1.22/ofi/gnu/9.1/lib/pkgconfig:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/rhash-1.4.2-xjxbsefahiue4isqlzwcobgc7sgfhmqm/lib/pkgconfig:/usr/share/pkgconfig:/usr/lib64/pkgconfig:/usr/lib/pkgconfig:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/libuv-1.44.1-gitz4v6soymrlpymzvygqlqibktfsp7a/lib/pkgconfig:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/libarchive-3.5.2-d4gpdx2rxj4calconu7cd6kgq2ev5roe/lib/pkgconfig:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zstd-1.5.2-rtaozj7xdfhxengzppbqbth4mya3edpq/lib/pkgconfig:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.12-ozmcyfjfv7i5gjjgklfsh43h67vzsuc5/lib/pkgconfig:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/xz-5.2.5-tyswlp4epqidsxgtgxyefciqoz2ju6oe/lib/pkgconfig:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/lzo-2.10-rlpwurcqberlfjlpk62go3arnneonv6u/lib/pkgconfig:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/lz4-1.9.3-xxbkaphnofa6gxugxszqr5kzew7chrqa/lib/pkgconfig:/usr/share/pkgconfig:/usr/lib64/pkgconfig:/usr/lib/pkgconfig:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/expat-2.4.8-yvchkv23jw7po7meyuxckpre5uroipmd/lib/pkgconfig:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/libbsd-0.11.5-uomjj7io3hhggbfyzofrw5jpw5chlv74/lib/pkgconfig:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/libmd-1.0.4-iynqjjkz4jgtxagqy2o2tvspqfn3yvog/lib/pkgconfig:/usr/share/pkgconfig:/usr/lib64/pkgconfig:/usr/lib/pkgconfig:/opt/modulefiles/cudatoolkit:/global/common/software/nersc/pm-2022q3/sw/darshan/3.4.0/lib/pkgconfig:/opt/cray/pe/craype/2.7.19/pkg-config:/opt/cray/pe/dsmml/0.2.2/dsmml/lib/pkgconfig:/opt/cray/libfabric/1.15.2.0/lib64/pkgconfig;
export SPACK_LOADED_HASHES=ozmcyfjfv7i5gjjgklfsh43h67vzsuc5:6opvtfcoaxox7oskb7dzisnb32fnk5ne:nugfov2wsvpllczfy6ff7bytzt4oi34q:6yvclv72nfvhlx2fj72n5npigrgouojd:p23fzuowp4yuitemelic7f65nwybthxd:xjxbsefahiue4isqlzwcobgc7sgfhmqm:i6ri5efvxr243dakip6bk7n6b3qfruyb:gitz4v6soymrlpymzvygqlqibktfsp7a:d4gpdx2rxj4calconu7cd6kgq2ev5roe:rtaozj7xdfhxengzppbqbth4mya3edpq:ozmcyfjfv7i5gjjgklfsh43h67vzsuc5:tyswlp4epqidsxgtgxyefciqoz2ju6oe:ucarxpp47pj7yo7abt5povg3swygev3q:rlpwurcqberlfjlpk62go3arnneonv6u:xxbkaphnofa6gxugxszqr5kzew7chrqa:3xwztgyaaco4thqqhacdst65mmnr5is3:4g7s6qpmt2nsclyi2d2ndfek2ptey4m6:yvchkv23jw7po7meyuxckpre5uroipmd:uomjj7io3hhggbfyzofrw5jpw5chlv74:iynqjjkz4jgtxagqy2o2tvspqfn3yvog:zkrv7nh3jglxsimtfzld7vo4mhdqlh3u;

Find the Install Prefix for a Package

For your application you may want to specify the location of the installed package as an environment variable. In this case, let's set HDF5_DIR to the parent directory of libs and include --the install prefix for our HDF5 package. A convenient way to do this is as follows,

elvis@perlmutter> export HDF5_DIR=$(spack location -i hdf5@1.10.7%gcc@11.2.0)

Unloading Spack Packages From Your Environment

You can use spack unload to remove packages from your environment which functions similarly to module unload. spack unload will only work on installed specs that were loaded in your environment. In the example below, we show how to unload the cmake Spack package from the user environment. Take note as we unload cmake via spack unload cmake the user environment was updated and it now resorts to the system cmake /usr/bin/cmake

elvis@perlmutter> spack load --list
==> 20 loaded packages
-- cray-sles15-zen3 / gcc@11.2.0 --------------------------------
bzip2@1.0.6   cray-mpich@8.1.22  expat@2.4.8  libarchive@3.5.2  libiconv@1.16  libuv@1.44.1  lzo@2.10        ncurses@6.1    rhash@1.4.2  zlib@1.2.12
cmake@3.23.1  curl@7.66.0        hdf5@1.10.7  libbsd@0.11.5     libmd@1.0.4    lz4@1.9.3     mbedtls@2.28.0  pkgconf@1.8.0  xz@5.2.5     zstd@1.5.2

elvis@perlmutter> which cmake
/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/cmake-3.23.1-p23fzuowp4yuitemelic7f65nwybthxd/bin/cmake

elvis@perlmutter> spack unload xz@5.2.5

elvis@perlmutter>  which cmake
/usr/bin/cmake

elvis@perlmutter> spack load --list
==> 19 loaded packages
-- cray-sles15-zen3 / gcc@11.2.0 --------------------------------
bzip2@1.0.6  cray-mpich@8.1.22  curl@7.66.0  expat@2.4.8  hdf5@1.10.7  libarchive@3.5.2  libbsd@0.11.5  libiconv@1.16  libmd@1.0.4  libuv@1.44.1  lz4@1.9.3  lzo@2.10  mbedtls@2.28.0  ncurses@6.1  pkgconf@1.8.0  rhash@1.4.2  xz@5.2.5  zlib@1.2.12  zstd@1.5.2

If you want to remove all packages from your user environment you can run the following:

elvis@perlmutter> spack unload --all
elvis@perlmutter> spack load --list
==> 0 loaded packages

Using Hashes for Packages with Multiple Versions

Sometimes, you will need to use hashes instead of specs to load a specific package variant. Let's assume we want to load CMake and we get the error, "Use a more specific spec". Note that in this example we have three specs for same version but each has a different hash (The 7 character code proceeding the spec).

Error

elvis@perlmutter> spack load cmake@3.23.1%gcc@11.2.0
==> Error: cmake@3.23.1%gcc@11.2.0 matches multiple packages.
  Matching packages:
    p23fzuo cmake@3.23.1%gcc@11.2.0 arch=cray-sles15-zen3
    6v4d2hu cmake@3.23.1%gcc@11.2.0 arch=cray-sles15-zen3
  Use a more specific spec.

In this case, we can use the hash to uniquely identify the desired variant of CMake. For example, if we want to load the spec with hash p23fzuo, we would run the following:

elvis@perlmutter> spack load /p23fzuo

Loading a Spack Package Without Dependencies

The command spack load will load a package and its dependencies which is recommended when using software, however sometimes you don't want to load all the extra dependencies in your environment. Spack provides a way to load only the package via --only package which will load the package and not its dependencies. In the example shown below, we load one package. Take note of this compared to its normal behavior which loads all dependencies.

elvis@perlmutter> spack load --list
==> 0 loaded packages
elvis@perlmutter> spack load --only package /p23fzuo
elvis@perlmutter> spack load --list
==> 1 loaded package
-- cray-sles15-zen3 / gcc@11.2.0 --------------------------------
cmake@3.23.1

elvis@perlmutter> spack unload -a
elvis@perlmutter> spack load --list
==> 0 loaded packages
elvis@perlmutter> spack load /p23fzuo
elvis@perlmutter> spack load --list
==> 17 loaded packages
-- cray-sles15-zen3 / gcc@11.2.0 --------------------------------
bzip2@1.0.6  cmake@3.23.1  curl@7.66.0  expat@2.4.8  libarchive@3.5.2  libbsd@0.11.5  libiconv@1.16  libmd@1.0.4  libuv@1.44.1  lz4@1.9.3  lzo@2.10  mbedtls@2.28.0  ncurses@6.1  rhash@1.4.2  xz@5.2.5  zlib@1.2.12  zstd@1.5.2

The e4s/<version> modules comes with several environments such as: gcc, nvhpc, cce, cuda. The cuda environment provides packages built with CUDA support with the gcc@11.2.0 compiler. Depending on the environment you are using, please consider loading the appropriate modules

  1. Load the PrgEnv-gnu, cpu module when using gcc environment
  2. Load the PrgEnv-cce, cpu module when using cce environment
  3. Load the PrgEnv-nvhpc, cpu module when using nvhpc environment
  4. Load the PrgEnv-gnu, gpu module when using cuda environment

When in doubt, please refer to the compiler settings to determine the list of modules used. To retrieve a list of compilers you can do spack compiler list. To query a compiler instance like gcc@11.2.0 you can run spack compiler info gcc@11.2.0.

In the cuda environment, you will see packages are built with variants +cuda cuda_arch=80 that are optimized for NVIDIA A100 GPUs (+cuda cuda_arch=80). You should make sure the craype-accel-nvidia modulefile is loaded. Usually this is loaded with the gpu modulefile at login. You can check the accelerator target by running the following:

elvis@perlmutter> echo $CRAY_ACCEL_TARGET
nvidia80

Developer Workflow

Shown below is a guide to get you started building packages with our Spack instance. First, we recommend you source the following file which will activate a python environment and install clingo.

# bash, sh, zsh users
source $SPACK_ROOT/bin/spack-setup.sh

# csh or tcsh users
source $SPACK_ROOT/bin/spack-setup.csh

We recommend you run the following to see if Spack will concretize a spec and not try to bootstrap clingo. If you are in a Spack environment, please deactivate the environment using despacktivate command. You can confirm your active environment by running spack env status. Next try running spack spec zlib to check if Spack will concretize zlib which will use clingo installed in your Python environment.

(.spack-pyenv) elvis@perlmutter> despacktivate
(.spack-pyenv) elvis@perlmutter> spack env status
==> No active environment

(.spack-pyenv) elvis@perlmutter> spack spec zlib
Input spec
--------------------------------
zlib

Concretized
--------------------------------
zlib@1.2.12%gcc@11.2.0+optimize+pic+shared patches=0d38234 arch=cray-sles15-zen3

We recommend you create a environment in your user space to install your Spack stack. This can be done using the spack env create -d <dir> command. This will create a directory with an empty spack.yaml. Now you can activate your newly created environment using the -d option to specify a directory. This will create a directory with an empty spack.yaml.

In example below, we will create a environment in our home directory called e4s-22.05-demo.

(.spack-pyenv) elvis@perlmutter> spack env create -d $HOME/e4s-22.05-demo
==> Created environment in /global/homes/s/elvis/e4s-22.05-demo
==> You can activate this environment with:
==>   spack env activate /global/homes/s/elvis/e4s-22.05-demo

(.spack-pyenv) elvis@perlmutter> spack env activate $HOME/e4s-22.05-demo

(.spack-pyenv) elvis@perlmutter> ls -l $HOME/e4s-22.05-demo
total 1
-rw-rw---- 1 elvis elvis 199 Jul 19 12:10 spack.yaml

Querying Spack Configuration

The spack config command can be used to query Spack configuration. Spack has several configuration scopes where configuration can be pulled. The configuration scope that is most relevant to you are site, user and defaults.

You can use spack config get [section] to query the configuration section. If you are unsure of all the configuration sections, you can run 

spack config list

If you want to query the package configuration, you can run spack config get packages. The spack config --scope option can be useful to query by configuration scope. We have set environment $SPACK_SYSTEM_CONFIG_PATH in our e4s modules that provides our global spack settings for all e4s stacks. The environment should be set to the following which should contain the following configuration files:

elvis@perlmutter> echo  $SPACK_SYSTEM_CONFIG_PATH
/global/common/software/spackecp/perlmutter/spack_settings

elvis@perlmutter> ls -l $SPACK_SYSTEM_CONFIG_PATH
total 10
-rw-rw-r-- 1 e4s spackecp 3609 Mar  7 07:27 compilers.yaml
-rw-rw-r-- 1 e4s spackecp 4494 Mar  7 07:27 packages.yaml

This Spack instance is configured to install packages in your HOME directory, the Spack site configuration can be found in this directory $SPACK_ROOT/etc/spack/ which applies configuration to all users but this can be overridden in your spack.yaml. Take note of the following settings below for build stage, and installation tree. Note that HOME directory is shared between Cori and Perlmutter, therefore we organize user installation by system. We recommend you adopt a similar naming scheme if you plan on building Spack packages on both systems.

(.spack-pyenv) elvis@perlmutter> cat $SPACK_ROOT/etc/spack/config.yaml
config:
  build_stage: $HOME/spack-workspace/perlmutter/build_stage
  misc_cache: $HOME/spack-workspace/perlmutter/misc_cache
  install_tree:
    root: $HOME/spack-workspace/perlmutter/software
  source_cache: $HOME/.spack/cache

Installing packages

Now let's add the following package in your spack.yaml and try building it via Spack. We will build papi. Shown below is the content of spack.yaml, please copy this content to your spack.yaml.

# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
  # add package specs to the `specs` list
  specs:
  - papi
  view: true

Note we don't provide a buildcache so papi will be built from source. Now let's install all packages in your environment by running spack install.

(.spack-pyenv) elvis@perlmutter:~/e4s-22.05-demo> spack install
==> Starting concretization
==> Environment concretized in 13.19 seconds.
==> Concretized papi
 -   s2y4nrv  papi@6.0.0.1%gcc@11.2.0~cuda+example~infiniband~lmsensors~nvml~powercap~rapl~rocm~rocm_smi~sde+shared~static_tools arch=cray-sles15-zen3

==> Installing environment /global/homes/s/elvis/e4s-22.05-demo
==> Installing papi-6.0.0.1-s2y4nrvu6whr6hhgi63aa3nqwz2d35af
==> No binary for papi-6.0.0.1-s2y4nrvu6whr6hhgi63aa3nqwz2d35af found: installing from source
==> Using cached archive: /global/homes/s/elvis/.spack/cache/_source-cache/archive/3c/3cd7ed50c65b0d21d66e46d0ba34cd171178af4bbf9d94e693915c1aca1e287f.tar.gz
==> No patches needed for papi
==> papi: Executing phase: 'autoreconf'
==> papi: Executing phase: 'configure'
==> papi: Executing phase: 'build'
==> papi: Executing phase: 'install'
==> papi: Successfully installed papi-6.0.0.1-s2y4nrvu6whr6hhgi63aa3nqwz2d35af
  Fetch: 0.01s.  Build: 25.80s.  Total: 25.81s.
[+] /global/homes/s/elvis/spack-workspace/perlmutter/software/cray-sles15-zen3/gcc-11.2.0/papi-6.0.0.1-s2y4nrvu6whr6hhgi63aa3nqwz2d35af
==> Updating view at /global/homes/s/elvis/e4s-22.05-demo/.spack-env/view

We can find the root directory for papi by running the following. We see the package was installed in our home directory.

(.spack-pyenv) elvis@perlmutter:~/e4s-22.05-demo> spack location -i papi
/global/homes/s/elvis/spack-workspace/perlmutter/software/cray-sles15-zen3/gcc-11.2.0/papi-6.0.0.1-s2y4nrvu6whr6hhgi63aa3nqwz2d35af

Creating Modulefiles via spack

In this section, we will create modules via spack. We recommend you review the modules documentation in spack. The spack configuration for modules is defined in modules.yaml.

Shown below is the site configuration for modules.yaml, we have configured the module root to be $HOME/spack-workspace/perlmutter/modules where TCL modules will be generated. Furthermore we have customized the TCL module setting when generating modules. You are welcome to override these configurations in your spack.yaml.

(.spack-pyenv) elvis@perlmutter> cat $SPACK_ROOT/etc/spack/modules.yaml
modules:
  default:
    roots:
      tcl: $HOME/spack-workspace/perlmutter/modules
    tcl:
      naming_scheme: '{name}/{version}-{compiler.name}-{compiler.version}'
      all:
        conflict:
        - '{name}'
        environment:
          set:
            '{name}_ROOT': '{prefix}'

We can create the module tree and retrieve the location of our generated module as shown below.

(.spack-pyenv) elvis@perlmutter:~/e4s-22.05-demo> spack module tcl refresh --delete-tree -y
==> Regenerating tcl module files
(.spack-pyenv) elvis@perlmutter:~/e4s-22.05-demo> spack module tcl find --full-path papi
/global/homes/s/elvis/spack-workspace/perlmutter/modules/cray-sles15-zen3/papi/6.0.0.1-gcc-11.2.0-s2y4nrv

If you want to add this to MODULEPATH you can simply run the following:

(.spack-pyenv) elvis@perlmutter:~/e4s-22.05-demo> module use $HOME/spack-workspace/perlmutter/modules/$(spack arch)

Now we see the modulefile papi/6.0.0.1-gcc-11.2.0-s2y4nrv is visible and can be loaded as any other software package.

(.spack-pyenv) elvis@perlmutter:~/e4s-22.05-demo> ml -t av papi
/global/homes/e/elvis/spack-workspace/perlmutter/modules/cray-sles15-zen3:
papi/6.0.0.1-gcc-11.2.0-s2y4nrv
/opt/cray/pe/lmod/modulefiles/core:
papi/6.0.0.16
papi/7.0.0.1

Spack Views

Spack environments can define filesystem views to directly access the software without having to load software.

If you are in an existing environment, you will want to deactivate and reactivate environment with view.

elvis@perlmutter> despacktivate

In this example, let's enable Spack view for nvhpc spack environment

Note

Activating an environment with view may take some time depending on size of environment. If you run spack env activate <env>, it will activate the environment with a view. We recommend you don't enable view since it may end up corrupting your user environment with the entire stack.

Spack View Enabled

elvis@perlmutter> spack env activate --with-view nvhpc

or 

elvis@perlmutter> spack env activate nvhpc

Spack View Disabled

elvis@perlmutter> spack env activate -V nvhpc

Spack will automatically load everything in the view, Take note, we have 0 loaded packages in the output of spack load --list, however we can access software from the view since $PATH, $LD_LIBRARY_PATH, etc... has been updated in your user environment. Spack will present software in a view similar to how one access software in Linux in /usr/local or /usr/bin.

elvis@perlmutter> spack load --list
==> 0 loaded packages

elvis@perlmutter> which h5cc
/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/var/spack/environments/nvhpc/views/default/hdf5/1.10.7-nvhpc-22.5/bin/h5cc

elvis@perlmutter> which cmake
/global/common/software/spackecp/perlmutter/e4s-22.05/75197/spack/views/default/cmake/3.23.1-gcc-11.2.0/bin/cmake

Now you may be wondering which packages you can access with Spack views. We have selected packages in view to avoid conflicts during view generation. For instance you can't have two instances of cmake in the same view.

Binary Cache Mirror

We have setup a binary cache mirror named perlmutter-e4s-22.05 that can be added into your spack.yaml.

You can check this by running spack mirror list to see the list of mirrors.

elvis@perlmutter>  spack mirror list
perlmutter-e4s-22.05    file:///global/common/software/spackecp/mirrors/perlmutter-e4s-22.05
source_mirror           file:///global/cfs/cdirs/m3503/mirrors/source_mirror
spack-public            https://mirror.spack.io

Spack has support signing and verifying packages via GPG Key which is required when installing packages from the binary cache. To learn more about the binary cache, please see https://spack-tutorial.readthedocs.io/en/latest/tutorial_binary_cache.html#setting-up-a-binary-cache-mirror

First, you will need to install the GPG key locally, note this is only required once. Please run the following command.

elvis@perlmutter> spack buildcache keys -it
==> Fetching file:///global/common/software/spackecp/mirrors/perlmutter-e4s-22.05/build_cache/_pgp/B5FDE18F615783AF078ED29C3BD6B0E9935AEB8F.pub
gpg: key 3BD6B0E9935AEB8F: public key "GPG Key - e4s <shahzebsiddiqui@lbl.gov>" imported
gpg: Total number processed: 1
gpg:               imported: 1
gpg: inserting ownertrust of 6

You can see list of gpg keys by running gpg -k make sure you see the following entry in the output.

pub   rsa2048 2022-02-23 [SC]
      B5FDE18F615783AF078ED29C3BD6B0E9935AEB8F
uid           [ultimate] GPG Key - e4s <shahzebsiddiqui@lbl.gov>
sub   rsa2048 2022-02-23 [E]

If you have successfully imported the gpg key you should be able to run spack buildcache list to see all the specs in the binary cache.

We have set the environment variable SPACK_GNUPGHOME which points to $HOME/.gnupg in case you run any commands like spack gpg which will typically fail since Spack attempts to write GPG keys into the Spack instance at $SPACK_ROOT/spack/opt/gpg which you won't have access to.

If you are still in your environment that you created in section Developer Workflow, let's install a package from the binary cache, you will notice that Spack will install packages from the binary cache instead of building from source.

elvis@perlmutter> spack install raja@0.14.0~cuda+examples+exercises~ipo+openmp~rocm+shared~tests build_type=RelWithDebInfo
==> Installing pkgconf-1.8.0-nugfov2wsvpllczfy6ff7bytzt4oi34q
==> Fetching file:///global/common/software/spackecp/mirrors/perlmutter-e4s-22.05/build_cache/cray-sles15-zen3/gcc-11.2.0/pkgconf-1.8.0/cray-sles15-zen3-gcc-11.2.0-pkgconf-1.8.0-nugfov2wsvpllczfy6ff7bytzt4oi34q.spack
==> Extracting pkgconf-1.8.0-nugfov2wsvpllczfy6ff7bytzt4oi34q from binary cache
gpg: Signature made Fri 14 Oct 2022 08:12:17 AM PDT
gpg:                using RSA key B5FDE18F615783AF078ED29C3BD6B0E9935AEB8F
gpg: Good signature from "GPG Key - e4s <shahzebsiddiqui@lbl.gov>" [ultimate]
[+] /global/homes/s/elvis/spack-workspace/perlmutter/software/cray-sles15-zen3/gcc-11.2.0/pkgconf-1.8.0-nugfov2wsvpllczfy6ff7bytzt4oi34q
==> Installing berkeley-db-18.1.40-hafeanvncfcwgstenlf5cc7ttyobmgt3
==> Fetching file:///global/common/software/spackecp/mirrors/perlmutter-e4s-22.05/build_cache/cray-sles15-zen3/gcc-11.2.0/berkeley-db-18.1.40/cray-sles15-zen3-gcc-11.2.0-berkeley-db-18.1.40-hafeanvncfcwgstenlf5cc7ttyobmgt3.spack
==> Extracting berkeley-db-18.1.40-hafeanvncfcwgstenlf5cc7ttyobmgt3 from binary cache
gpg: Signature made Thu 13 Oct 2022 11:52:15 AM PDT
gpg:                using RSA key B5FDE18F615783AF078ED29C3BD6B0E9935AEB8F
gpg: Good signature from "GPG Key - e4s <shahzebsiddiqui@lbl.gov>" [ultimate]
[+] /global/homes/s/elvis/spack-workspace/perlmutter/software/cray-sles15-zen3/gcc-11.2.0/berkeley-db-18.1.40-hafeanvncfcwgstenlf5cc7ttyobmgt3

...

Need Help with Spack

If you need help with Spack, please join the Spack Slack Channel, once you have registered you can go to slack workspace at https://spackpm.slack.com/.