Spack - A package manager¶
Spack automates the download-build-install process for software - including dependencies - and provides convenient management of versions and build configurations.
NERSC provides Spack via module load spack
. For most uses, the module for the newest available version is recommended.
The default Spack instance is configured for installing software to a location in your $HOME
, and to be able to use software installed by NERSC and also the E4S Stack.
Using Spack at NERSC¶
See the Spack documentation for detailed information, but a summary of the procedure is as follows, and we have a glossary of Spack terms further below to help you get started.
To install software with Spack, perform the following steps. In this example we will install GNU Octave into $HOME/sw
:
-
module load spack
-
Check that Spack has a package for the software you want to install, and read about the versions and variants available.
spack list <word1> <word2>
searches where<word1>
or<word2>
appear in the name, for example:cori$ spack list gnu octave ==> 12 packages. dejagnu gnuradio octave-optim perl-term-readline-gnu gnupg gnutls octave-splines py-gnuplot gnuplot octave octave-struct ruby-gnuplot
In this case,
octave
looks like what we want, and we can find out about it withspack info octave
Tip
You can also check whether there is an already-installed version of what you want with
spack find
. For example, to check for installed instances of the Spack packageoctave
:spack find -vpl octave
The
-v
here shows the variants used in each install, the-p
shows the path to where it is installed and the-l
shows the hash.Tip
There is a full list of the packages Spack knows about at https://spack.readthedocs.io/en/latest/package_list.html.
-
Check the list of dependencies Spack will install:
cori$ spack spec -Il octave ... - xsuyhgx octave@5.2.0%intel@19.1.2.254~arpack~curl~fftw~fltk~fontc... - t7ue4px ^cray-libsci@19.06.1%intel@19.1.2.254~mpi~openmp+shar... [+] rbctzhm ^pcre@8.44%intel@19.1.2.254~jit+multibyte+shared+utf ... [^] xh3ldnf ^pkg-config@0.29.2%intel@19.1.2.254+internal_glib arc... [+] kvywvaf ^readline@8.0%intel@19.1.2.254 arch=cray-cnl7-haswell [^] 3huqp2j ^ncurses@6.2%intel@19.1.2.254+shared~symlinks+ter...
Things Spack sees already installed in your installation tree will have a
[+]
in the first column, and things Spack found installed upstream will have[^]
. A-
means Spack did not find it, although for some packages (such as cray-libsci) Spack will be able to use the external (ie, not installed via Spack) instance.If you have already-installed software to build on top of, but that Spack doesn't see, you may need to create an environment and continue your build there.
You can specify on the command-line the version, compiler, variants, etc that you want for the package, for example
spack spec -Il octave%gcc
means "Octave built with the gcc compiler". Runspack help --spec
to see syntax and examples.In this example we will build Octave with gcc, because there is a known bug in Octave when using the Intel compiler.
-
When you're satisfied with what Spack plans to do, install it:
spack install octave%gcc
Using software installed with Spack¶
If the install worked, Spack will create a modulefile that you could add to your $MODULEPATH
. By default, in the NERSC Spack setup, these will be written to $HOME/sw/share/spack/modules/
cori$ module use $HOME/sw/share/spack/modules/cray-*
cori$ module avail octave
--- $HOME/sw/share/spack/modules/cray-cnl7-haswell ----
octave/5.2.0-gcc-10.1.0-sqnlr7q
And optionally copy the file to a location where you install your own custom modulefiles.
Alternatively, you can use Spack to manage your environment directly. For example, you can check for installed instances of octave
with:
spack find octave
Tip
To limit the search to only instances targeting the current architecture, try:
spack find octave arch=`spack arch`
You can then make the instance available in your environment with spack load
, for example:
spack load octave
Spack will then update environment variables including PATH
, INCLUDE
and LIBRARY_PATH
to include the instance of octave
that you used Spack to install.
What to do when spack install
fails¶
-
Check if the error output suggests an easy solution. You can increase the verbosity of
spack install
withspack -dv install
, which might help in identifying the cause of a failure.Some failures can be avoided by:
- building a different (eg earlier) version of the application
- building with a different compiler (try
%gcc
or%cray
) - disabling a variant
- modifying the spec of a dependency (see Specs and Dependencies in the official Spack documentation)
Other errors might be fixed by modifying the package - see Debugging, developing or just reading Spack packages for tips on this.
-
Seek help: you can check the official Spack documentation, open a ticket at https://help.nersc.gov or ask the Spack community via the Spack Slack. (The Spack Slack community is very active and helpful)
Debugging, developing or just reading Spack packages¶
spack edit octave
will open the package.py
file in your default editor. In most cases this file is read-only, but you can copy it to your local repo with:
cp -vr $SPACK_NERSC_REPO/packages/octave $SPACK_MYREPO/
And after this, spack edit octave
will edit your local copy of the package, and spack spec
and spack install
will also preferentially use your copy of octave
.
Spack Locations¶
In this document we use the following environment variables (which are set in the modulefile for spack
) to refer to specific locations:
-
$SPACK_ROOT
is the location where Spack itself is installed. This is significant because thebuiltin
repo is stored within it. -
$SPACK_BASE
is the location where Spack will look first for packages and modulefiles, and where it will install software to. By default this is at$HOME/sw
, but you can change this by setting$SPACK_USER_PREFERRED_BASE
to an alternative location.For example, if you are mostly building software for your team, you might want to add this line to your
.bashrc
(and encourage your team to do the same):# I'm in project m1234 and want my team to share software: export SPACK_USER_PREFERRED_BASE /global/common/software/m1234/sw
-
$SPACK_MYREPO
is a directory for Spack packages you have modified. When searching for a package, Spack will search in the following locations, in order:$SPACK_MYREPO
$SPACK_NERSC_REPO
- The built-in repo for the Spack instance
-
$SPACK_NERSC_REPO
is a Spack repo which can include packages updated by NERSC. For some Spack modules,$SPACK_NERSC_REPO
will simply refer to the builtin repo for that instance. -
$SPACK_BUILTIN
is the location of package directories in the Spack instance.$SPACK_NERSC_REPO
and$SPACK_MYREPO
are for convenience in the event that you need to modify a package: since the builtin repos are
read-only, you can make an editable copy of a package (for example theoctave
package we used in Using Spack at NERSC with:cp -r $SPACK_NERSC_REPO/packages/octave $SPACK_MYREPO
Spack Glossary¶
The diagram below illustrates the words describing some key Spack concepts:
Package¶
"Source code" describing a piece of software and how to build it (actually a Python class, defined in package.py
), in a directory with any patches that might need to be applied first.
elvis@cori$ ls -l $SPACK_ROOT/var/spack/repos/builtin/packages/zlib
total 7
drwxrwxr-x+ 2 swowner swowner 512 May 26 13:00 __pycache__
-rw-rw-r--+ 1 swowner swowner 2074 Apr 3 13:40 package.py
-rw-rw-r--+ 1 swowner swowner 564 Mar 30 19:16 w_patch.patch
The Spack Packaging Guide has detail about creating and maintaining packages.
Repo¶
A collection ("repository") of packages. Almost everything is in the "builtin" repo, but Spack has a "repos" config section where you can specify locations and order of repos to search.
See NERSC Spack repo locations for repo locations used at NERSC, and the Spack package repository documentation for more about Spack repos.
Spec¶
Spack defines a Domain-Specific Language (DSL) for describing the build parameters and dependencies with which a package was, or should be, built. The Spack Specs and Dependencies documentation describes this in detail, and the example below shows the main elements of a spec:
Variant¶
A variant is a selectable build option for a package. Available variants for a package are defined in the Spack package. You can, in the spec, enable (+
), disable (-
or ~
) or set (name=value
) a variant. Variants usually correspond to a ./configure
or cmake
setting.
Hash¶
Often directory or modulefile names created by Spack include a string of arbitrary characters, such as x2icaez
below:
elvis@cori:~> module av cmake
----------------------- /global/common/sw/modulefiles/cray-cnl7-haswell -----------------------
cmake/3.18.4-gcc-8.3.0-cg2udpv cmake/3.20.5-gcc-10.1.0-x2icaez
This is a "hash": Spack identifies a unique build spec by computing a hash of the package build settings and dependencies. This is a convenient way to refer to a unique instance of software without typing the full specification.
One common use for a hash is to instruct Spack to build software on a specific, already installed library, e.g. here we can see 2 instances of METIS:
elvis@cori$ spack find -vl metis arch=$(spack arch)
==> 2 installed packages
-- cray-cnl7-haswell / intel@19.0.3.199 ------------------------..
kmwfurn metis@5.1.0~gdb~int64~real64+shared build_type=Release p..
-- cray-cnl7-haswell / intel@19.1.2.254 ------------------------..
lzrewvi metis@5.1.0~gdb~int64~real64+shared build_type=Release p..
Spack will prefer the one with the hash lzrewvi
:
elvis@cori$ spack spec -Il mumps+metis
[snip]
Concretized
--------------------------------
- my5bvud mumps@5.3.3%intel@19.1.2.254+complex+double+float~..
- t7ue4px ^cray-libsci@19.06.1%intel@19.1.2.254~mpi~open..
[^] lzrewvi ^metis@5.1.0%intel@19.1.2.254~gdb~int64~real64..
- yezois3 ^cmake@3.20.5%intel@19.1.2.254~doc+ncurses..
[^] wcezuim ^mpich@3.1%intel@19.1.2.254~argobots+fortran+h..
But we can specify a preference to use the other one:
elvis@cori$ spack spec -Il mumps+metis ^metis/kmwfurn
[snip]
Concretized
--------------------------------
==> Warning: Intel's compilers may or may not optimize to the sa..
- d5q6rxk mumps@5.3.3%intel@19.1.2.254+complex+double+float~..
- t7ue4px ^cray-libsci@19.06.1%intel@19.1.2.254~mpi~open..
[^] kmwfurn ^metis@5.1.0%intel@19.0.3.199~gdb~int64~real64..
- 2t6titn ^cmake@3.20.5%intel@19.0.3.199~doc+ncurses..
[^] wcezuim ^mpich@3.1%intel@19.1.2.254~argobots+fortran+h..
Installation tree¶
A directory for Spack-installed software. Spack allows specification of a directory structure and naming convention for installed software. At NERSC this defaults to a location in your $HOME
and the directory structure starts with the target architecture, e.g. cray-cnl7-haswell
for Cori Haswell software. Spack uses the keyword install_tree
to refer to the installation tree.
Upstream¶
Spack can only install software to a place where you have write permission, but it can find software (to satisfy dependencies) in other locations, called "upstream" locations. At NERSC, Spack modules are each configured to use a location in /global/common/software/nersc
as an upstream - NERSC consultants install some software there as a convenience.
Another important upstream is the E4S Stack.
To see the list of upstreams Spack will use, run spack config get upstreams
:
$ spack config get upstreams
upstreams:
nersc-installs:
install_tree: /global/common/software/nersc/cori-2021q4/spack
modules:
tcl: /global/common/software/nersc/cori-2021q4/spack-modulefiles
e4s-2105:
install_tree: /global/common/software/spackecp/e4s-21.05/software
modules:
tcl: /global/common/software/spackecp/e4s-21.05/modules
Environment¶
The NERSC Spack configuration aims to make installing software simple, but there are some situations in which it may not be the configuration you want, for example:
- you want to install different software to different locations (for example, you build software for two different projects, and want to install some software to
/global/common/software/project1
and some to/global/common/software/project2
. - you have software built outside of Spack, and need Spack to build software on your external build, rather than its default approach of building each dependency package.
- you want to use different conventions for directory structure or modulefile names than the defaults set by NERSC.
A Spack environment is a directory with a spack.yaml
file that outlines software to build, and configuration modifications to use while building it.
To create and activate an environment:
nersc$ mkdir my_spack_env
nersc$ cd my_spack_env
nersc$ spack env create -d $PWD
[snip]
==> You can activate this environment with:
==> spack env activate /path/to/my_spack_env
nersc$ spack env activate -d $PWD
By default the environment will install things to your regular installation tree, but you could change this in spack.yaml
, for example:
spack:
config:
install_tree:
# install things into the `install` subdirectory of this environment:
root: ${SPACK_ENV}/install
# alternatively: install things into my project's /global/common/software:
# root: /global/common/software/my_project
# put modulefiles generated by Spack in this environment directory:
module_roots:
tcl: ${SPACK_ENV}/modules
lmod: ${SPACK_ENV}/lmod
view: true
specs: []
If your software stack needs something that was installed external to Spack you can add a packages
section, e.g.:
spack:
# [snip]
packages:
# use the ncl from "module load ncl"
ncl:
externals:
- spec: ncl@6.5.0
modules: [ncl/6.5.0]
# use my local install of fftw 3.3.8, as the preferred fftw version:
fftw:
version: [3.3.8]
externals:
- spec: fftw@3.3.8
prefix: /path/to/my/fftw/install
For further details about using Spack environments see https://spack.readthedocs.io/en/latest/environments.html
Spack Resources¶
- Documentation: https://spack.readthedocs.io/en/latest/index.html
- Tutorial: https://spack.readthedocs.io/en/latest/tutorial.html
- Source code: https://github.com/spack/spack
- Spack Slack: http://spackpm.slack.com/