Skip to content

Spack - A package manager

Spack automates the download-build-install process for software - including dependencies - and provides convenient management of versions and build configurations.

NERSC provides Spack via module load spack. For most uses, the module for the newest available version is recommended.

The default Spack instance is configured for installing software to a location in your $HOME, and to be able to use software installed by NERSC and also the E4S Stack.

Using Spack at NERSC

See the Spack documentation for detailed information, but a summary of the procedure is as follows, and we have a glossary of Spack terms further below to help you get started.

To install software with Spack, perform the following steps. In this example we will install GNU Octave into $HOME/sw:

  1. module load spack

  2. Check that Spack has a package for the software you want to install, and read about the versions and variants available. spack list <word1> <word2> searches where <word1> or <word2> appear in the name, for example:

    cori$ spack list gnu octave
==> 12 packages.
dejagnu  gnuradio  octave-optim    perl-term-readline-gnu
gnupg    gnutls    octave-splines  py-gnuplot
gnuplot  octave    octave-struct   ruby-gnuplot

    In this case, octave looks like what we want, and we can find out about it with 

    spack info octave

    Tip

    You can also check whether there is an already-installed version of what you want with spack find. For example, to check for installed instances of the Spack package octave:

    spack find -vpl octave

    The -v here shows the variants used in each install, the -p shows the path to where it is installed and the -l shows the hash.

    Tip

    There is a full list of the packages Spack knows about at https://spack.readthedocs.io/en/latest/package_list.html.

  3. Check the list of dependencies Spack will install:

    cori$ spack spec -Il octave
...
 -   xsuyhgx  octave@5.2.0%intel@19.1.2.254~arpack~curl~fftw~fltk~fontc...
 -   t7ue4px      ^cray-libsci@19.06.1%intel@19.1.2.254~mpi~openmp+shar...
[+]  rbctzhm      ^pcre@8.44%intel@19.1.2.254~jit+multibyte+shared+utf ...
[^]  xh3ldnf      ^pkg-config@0.29.2%intel@19.1.2.254+internal_glib arc...
[+]  kvywvaf      ^readline@8.0%intel@19.1.2.254 arch=cray-cnl7-haswell
[^]  3huqp2j          ^ncurses@6.2%intel@19.1.2.254+shared~symlinks+ter...

    Things Spack sees already installed in your installation tree will have a [+] in the first column, and things Spack found installed upstream will have [^]. A - means Spack did not find it, although for some packages (such as cray-libsci) Spack will be able to use the external (ie, not installed via Spack) instance.

    If you have already-installed software to build on top of, but that Spack doesn't see, you may need to create an environment and continue your build there.

    You can specify on the command-line the version, compiler, variants, etc that you want for the package, for example spack spec -Il octave%gcc means "Octave built with the gcc compiler". Run spack help --spec to see syntax and examples.

    In this example we will build Octave with gcc, because there is a known bug in Octave when using the Intel compiler.

  4. When you're satisfied with what Spack plans to do, install it:

    spack install octave%gcc

Using software installed with Spack

If the install worked, Spack will create a modulefile that you could add to your $MODULEPATH. By default, in the NERSC Spack setup, these will be written to $HOME/sw/share/spack/modules/ 

cori$ module use $HOME/sw/share/spack/modules/cray-*
cori$ module avail octave

--- $HOME/sw/share/spack/modules/cray-cnl7-haswell ----
octave/5.2.0-gcc-10.1.0-sqnlr7q

And optionally copy the file to a location where you install your own custom modulefiles.

Alternatively, you can use Spack to manage your environment directly. For example, you can check for installed instances of octave with:

spack find octave

Tip

To limit the search to only instances targeting the current architecture, try: 

spack find octave arch=`spack arch`

You can then make the instance available in your environment with spack load, for example:

spack load octave

Spack will then update environment variables including PATH, INCLUDE and LIBRARY_PATH to include the instance of octave that you used Spack to install.

What to do when spack install fails

  1. Check if the error output suggests an easy solution. You can increase the verbosity of spack install with spack -dv install, which might help in identifying the cause of a failure.

    Some failures can be avoided by:

    • building a different (eg earlier) version of the application
    • building with a different compiler (try %gcc or %cray)
    • disabling a variant
    • modifying the spec of a dependency (see Specs and Dependencies in the official Spack documentation)

    Other errors might be fixed by modifying the package - see Debugging, developing or just reading Spack packages for tips on this.

  2. Seek help: you can check the official Spack documentation, open a ticket at https://help.nersc.gov or ask the Spack community via the Spack Slack. (The Spack Slack community is very active and helpful)

Debugging, developing or just reading Spack packages

spack edit octave will open the package.py file in your default editor. In most cases this file is read-only, but you can copy it to your local repo with:

cp -vr $SPACK_NERSC_REPO/packages/octave $SPACK_MYREPO/

And after this, spack edit octave will edit your local copy of the package, and spack spec and spack install will also preferentially use your copy of octave .

Spack Locations

In this document we use the following environment variables (which are set in the modulefile for spack) to refer to specific locations:

  • $SPACK_ROOT is the location where Spack itself is installed. This is significant because the builtin repo is stored within it.

  • $SPACK_BASE is the location where Spack will look first for packages and modulefiles, and where it will install software to. By default this is at $HOME/sw, but you can change this by setting $SPACK_USER_PREFERRED_BASE to an alternative location.

    For example, if you are mostly building software for your team, you might want to add this line to your .bashrc (and encourage your team to do the same):

    # I'm in project m1234 and want my team to share software:
export SPACK_USER_PREFERRED_BASE /global/common/software/m1234/sw

  • $SPACK_MYREPO is a directory for Spack packages you have modified. When searching for a package, Spack will search in the following locations, in order:

    1. $SPACK_MYREPO
    2. $SPACK_NERSC_REPO
    3. The built-in repo for the Spack instance

  • $SPACK_NERSC_REPO is a Spack repo which can include packages updated by NERSC. For some Spack modules, $SPACK_NERSC_REPO will simply refer to the builtin repo for that instance.

  • $SPACK_BUILTIN is the location of package directories in the Spack instance. $SPACK_NERSC_REPO and $SPACK_MYREPO are for convenience in the event that you need to modify a package: since the builtin repos are
    read-only, you can make an editable copy of a package (for example the octave package we used in Using Spack at NERSC with:

    cp -r $SPACK_NERSC_REPO/packages/octave $SPACK_MYREPO

Spack Glossary

The diagram below illustrates the words describing some key Spack concepts:

diagram of relationships between Spack words

Package

"Source code" describing a piece of software and how to build it (actually a Python class, defined in package.py), in a directory with any patches that might need to be applied first.

elvis@cori$ ls -l $SPACK_ROOT/var/spack/repos/builtin/packages/zlib
total 7
drwxrwxr-x+ 2 swowner swowner  512 May 26 13:00 __pycache__
-rw-rw-r--+ 1 swowner swowner 2074 Apr  3 13:40 package.py
-rw-rw-r--+ 1 swowner swowner  564 Mar 30 19:16 w_patch.patch

The Spack Packaging Guide has detail about creating and maintaining packages.

Repo

A collection ("repository") of packages. Almost everything is in the "builtin" repo, but Spack has a "repos" config section where you can specify locations and order of repos to search.

See NERSC Spack repo locations for repo locations used at NERSC, and the Spack package repository documentation for more about Spack repos.

Spec

Spack defines a Domain-Specific Language (DSL) for describing the build parameters and dependencies with which a package was, or should be, built. The Spack Specs and Dependencies documentation describes this in detail, and the example below shows the main elements of a spec:

elements of a Spack Spec

Variant

A variant is a selectable build option for a package. Available variants for a package are defined in the Spack package. You can, in the spec, enable (+), disable (- or ~) or set (name=value) a variant. Variants usually correspond to a ./configure or cmake setting.

Hash

Often directory or modulefile names created by Spack include a string of arbitrary characters, such as x2icaez below:

elvis@cori:~> module av cmake

----------------------- /global/common/sw/modulefiles/cray-cnl7-haswell -----------------------
cmake/3.18.4-gcc-8.3.0-cg2udpv  cmake/3.20.5-gcc-10.1.0-x2icaez

This is a "hash": Spack identifies a unique build spec by computing a hash of the package build settings and dependencies. This is a convenient way to refer to a unique instance of software without typing the full specification.

One common use for a hash is to instruct Spack to build software on a specific, already installed library, e.g. here we can see 2 instances of METIS:

elvis@cori$ spack find -vl metis arch=$(spack arch)
==> 2 installed packages
-- cray-cnl7-haswell / intel@19.0.3.199 ------------------------..
kmwfurn metis@5.1.0~gdb~int64~real64+shared build_type=Release p..

-- cray-cnl7-haswell / intel@19.1.2.254 ------------------------..
lzrewvi metis@5.1.0~gdb~int64~real64+shared build_type=Release p..

Spack will prefer the one with the hash lzrewvi:

elvis@cori$ spack spec -Il mumps+metis
[snip]
Concretized
--------------------------------
 -   my5bvud  mumps@5.3.3%intel@19.1.2.254+complex+double+float~..
 -   t7ue4px      ^cray-libsci@19.06.1%intel@19.1.2.254~mpi~open..
[^]  lzrewvi      ^metis@5.1.0%intel@19.1.2.254~gdb~int64~real64..
 -   yezois3          ^cmake@3.20.5%intel@19.1.2.254~doc+ncurses..
[^]  wcezuim      ^mpich@3.1%intel@19.1.2.254~argobots+fortran+h..

But we can specify a preference to use the other one:

elvis@cori$ spack spec -Il mumps+metis ^metis/kmwfurn
[snip]
Concretized
--------------------------------
==> Warning: Intel's compilers may or may not optimize to the sa..
 -   d5q6rxk  mumps@5.3.3%intel@19.1.2.254+complex+double+float~..
 -   t7ue4px      ^cray-libsci@19.06.1%intel@19.1.2.254~mpi~open..
[^]  kmwfurn      ^metis@5.1.0%intel@19.0.3.199~gdb~int64~real64..
 -   2t6titn          ^cmake@3.20.5%intel@19.0.3.199~doc+ncurses..
[^]  wcezuim      ^mpich@3.1%intel@19.1.2.254~argobots+fortran+h..

Installation tree

A directory for Spack-installed software. Spack allows specification of a directory structure and naming convention for installed software. At NERSC this defaults to a location in your $HOME and the directory structure starts with the target architecture, e.g. cray-cnl7-haswell for Cori Haswell software. Spack uses the keyword install_tree to refer to the installation tree.

Upstream

Spack can only install software to a place where you have write permission, but it can find software (to satisfy dependencies) in other locations, called "upstream" locations. At NERSC, Spack modules are each configured to use a location in /global/common/software/nersc as an upstream - NERSC consultants install some software there as a convenience.

Another important upstream is the E4S Stack.

To see the list of upstreams Spack will use, run spack config get upstreams:

$ spack config get upstreams
upstreams:
  nersc-installs:
    install_tree: /global/common/software/nersc/cori-2021q4/spack
    modules:
      tcl: /global/common/software/nersc/cori-2021q4/spack-modulefiles
  e4s-2105:
    install_tree: /global/common/software/spackecp/e4s-21.05/software
    modules:
      tcl: /global/common/software/spackecp/e4s-21.05/modules

Environment

The NERSC Spack configuration aims to make installing software simple, but there are some situations in which it may not be the configuration you want, for example:

  • you want to install different software to different locations (for example, you build software for two different projects, and want to install some software to /global/common/software/project1 and some to /global/common/software/project2.
  • you have software built outside of Spack, and need Spack to build software on your external build, rather than its default approach of building each dependency package.
  • you want to use different conventions for directory structure or modulefile names than the defaults set by NERSC.

A Spack environment is a directory with a spack.yaml file that outlines software to build, and configuration modifications to use while building it.

To create and activate an environment:

nersc$ mkdir my_spack_env
nersc$ cd my_spack_env
nersc$ spack env create -d $PWD
[snip]
==> You can activate this environment with:
==>   spack env activate /path/to/my_spack_env
nersc$ spack env activate -d $PWD

By default the environment will install things to your regular installation tree, but you could change this in spack.yaml, for example:

spack:
  config:
    install_tree:
      # install things into the `install` subdirectory of this environment:
      root: ${SPACK_ENV}/install
      # alternatively: install things into my project's /global/common/software:
      # root: /global/common/software/my_project
    # put modulefiles generated by Spack in this environment directory:
    module_roots:
      tcl:    ${SPACK_ENV}/modules
      lmod:   ${SPACK_ENV}/lmod
  view: true
  specs: []

If your software stack needs something that was installed external to Spack you can add a packages section, e.g.:

spack:
  # [snip]
  packages:
    # use the ncl from "module load ncl"
    ncl:
      externals:
      - spec: ncl@6.5.0 
      modules: [ncl/6.5.0] 
    # use my local install of fftw 3.3.8, as the preferred fftw version:
    fftw:
      version: [3.3.8]
      externals:
      - spec: fftw@3.3.8
        prefix: /path/to/my/fftw/install

For further details about using Spack environments see https://spack.readthedocs.io/en/latest/environments.html

Spack Resources