Skip to content

Spack Gitlab Pipeline

This guide will discuss how we can leverage spack and gitlab to automate spack builds using continuous integration. If you are a developer using spack, or member of Application Development or Software Technology that wants to install your software product at NERSC, you should read this guide. We will discuss how one can push specs to buildcache and later install from the buildcache. We will use a spack manifest (spack.yaml) to define our spack configuration.

For this example we picked HDF5 package and build from source and push to buildcache on a nightly basis using gitlab scheduled pipeline. To get started you will need access to our gitlab server https://software.nersc.gov and refer to the project https://software.nersc.gov/ci-resources/hdf5-nightly-build for implementation of spack gitlab pipeline.

HDF5 nightly buildcache

This project is responsible for rebuilding hdf5%gcc@9.3.0 using a fixed release of spack on a nightly basis. We have setup a scheduled pipeline to run daily builds from source. This can be achieved by removing the buildcache prior to build. As a user, you can access hdf5 assuming it is pushed into buildcache on a nightly basis. If you are unable to see see buildcache, please check the pipeline status to see if there is an active job.

If you want to run pipeline, you can press play button the scheduled pipeline at https://software.nersc.gov/siddiq90/hdf5-nightly-build/-/pipeline_schedules. For more details on running pipelines manually click here.

Please note this workflow is tied to spack tag e4s-21.02, you should use same tag if you want consistent behavior.

Relevant Files

  • spack.yaml - spack.yaml used to push to buildcache at /global/common/software/spackecp/mirrors/hdf5-nightly on nightly basis
  • .gitlab-ci.yml - gitlab CI file
  • examples/spack.yaml - example spack.yaml used to install hdf5 from buildcache.
  • setclonepath.sh - used to create a random spack directory name in $SCRATCH this was done to avoid a shorter path name because spack was cloned in $CI_PROJECT_DIR which was too long path for spack to apply padding. We picked shared file system to troubleshoot build since jobs go through compute nodes and we don't have access to logs if there is a build failure.

User Development Workflow

This guide will help users install from a buildcache.

To get started clone spack at branch e4s-21.02 as follows:

git clone https://github.com/spack/spack.git -b e4s-21.02

Next, we need to source spack in our user environment by sourcing the setup script as follows:

cd spack
source share/spack/setup-env.sh

We need to set SPACK_GNUPGHOME in order to access buildcache. First you can set SPACK_GNUPGHOME to $HOME/.gnupg which will instruct spack to store gpg keys in your $HOME directory:

export SPACK_GNUPGHOME=$HOME/.gnupg

Now let's trust the gpg key via spack gpg trust:

spack gpg trust /global/common/software/spackecp/gpgkeys/e4s2010.pub

You can check if gpg key is imported by running:

spack gpg list

Once gpg key is added, you can copy the examples/spack.yaml locally and create an environment as follows:

spack env create hdf5 spack.yaml
spack env activate hdf5

If you have got this far, you should see an entry for hdf5 mirror as follows:

$ spack mirror list
hdf5            file:///global/common/software/spackecp/mirrors/hdf5-nightly
spack-public    https://spack-llnl-mirror.s3-us-west-2.amazonaws.com/

You should be able to see specs in this buildcache by running spack buildcache list with an output as follows:

$ spack buildcache list   
==> 3 cached builds.
-- cray-cnl7-haswell / gcc@9.3.0 --------------------------------
hdf5@1.10.7  mpich@3.1  zlib@1.2.11

Now we let's install hdf5 from buildcache.

spack install
elvis@cori11:> spack install
==> Installing environment hdf5
==> mpich@3.1 : has external module in ['cray-mpich/7.7.10']
[+] /opt/cray/pe/mpt/7.7.10/gni/mpich-gnu/8.2 (external mpich-3.1-2ymwqzfqnyrsyp7adgzmiylwgeto3zb4)
==> Installing zlib-1.2.11-uslkm4qg6dwtgqdjctoz6hyahizidlvz
==> Fetching file:///global/common/software/spackecp/mirrors/hdf5-nightly/build_cache/cray-cnl7-haswell/gcc-9.3.0/zlib-1.2.11/cray-cnl7-haswell-gcc-9.3.0-zlib-1.2.11-uslkm4qg6dwtgqdjctoz6hyahizidlvz.spack
#################################################################################################################################################### 100.0%
==> Extracting zlib-1.2.11-uslkm4qg6dwtgqdjctoz6hyahizidlvz from binary cache
gpgconf: socketdir is '/global/homes/y/elvis/.gnupg'
gpgconf:    no /run/user dir
gpgconf:    using homedir as fallback
gpg: Signature made Wed Mar  3 11:47:21 2021 PST
gpg:                using RSA key 1C0641AF18F4EA5A2628CBDDAC248997479D4AB0
gpg: Good signature from "Spack GPG Key (Spack E4S GPG Key) <shahzebsiddiqui@lbl.gov>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: EA17 2EB6 343D 3075 0A56  522F 0140 A256 659E 0CBD
    Subkey fingerprint: 1C06 41AF 18F4 EA5A 2628  CBDD AC24 8997 479D 4AB0
==> Installing patchelf-0.10-b4iqpjdvgkvc5yuw6lsdoyg3jnkedo3s
==> No binary for patchelf-0.10-b4iqpjdvgkvc5yuw6lsdoyg3jnkedo3s found: installing from source
==> Fetching https://spack-llnl-mirror.s3-us-west-2.amazonaws.com/_source-cache/archive/b2/b2deabce05c34ce98558c0efb965f209de592197b2c88e930298d740ead09019.tar.gz
#################################################################################################################################################### 100.0%
==> No patches needed for patchelf
==> patchelf: Executing phase: 'autoreconf'
==> patchelf: Executing phase: 'configure'
==> patchelf: Executing phase: 'build'
==> patchelf: Executing phase: 'install'
==> patchelf: Successfully installed patchelf-0.10-b4iqpjdvgkvc5yuw6lsdoyg3jnkedo3s
Fetch: 0.46s.  Build: 10.09s.  Total: 10.55s.
[+] /global/cscratch1/sd/elvis/spack-envs/hdf5/spack/opt/spack/cray-cnl7-haswell/gcc-9.3.0/patchelf-0.10-b4iqpjdvgkvc5yuw6lsdoyg3jnkedo3s
[+] /global/cscratch1/sd/elvis/spack-envs/hdf5/spack/opt/spack/cray-cnl7-haswell/gcc-9.3.0/zlib-1.2.11-uslkm4qg6dwtgqdjctoz6hyahizidlvz
==> Installing hdf5-1.10.7-4iumqkfbojam6unwkb2y5t45deds7hb3
==> Fetching file:///global/common/software/spackecp/mirrors/hdf5-nightly/build_cache/cray-cnl7-haswell/gcc-9.3.0/hdf5-1.10.7/cray-cnl7-haswell-gcc-9.3.0-hdf5-1.10.7-4iumqkfbojam6unwkb2y5t45deds7hb3.spack
#################################################################################################################################################### 100.0%
==> Extracting hdf5-1.10.7-4iumqkfbojam6unwkb2y5t45deds7hb3 from binary cache
gpg: Signature made Wed Mar  3 11:47:44 2021 PST
gpg:                using RSA key 1C0641AF18F4EA5A2628CBDDAC248997479D4AB0
gpg: Good signature from "Spack GPG Key (Spack E4S GPG Key) <shahzebsiddiqui@lbl.gov>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: EA17 2EB6 343D 3075 0A56  522F 0140 A256 659E 0CBD
    Subkey fingerprint: 1C06 41AF 18F4 EA5A 2628  CBDD AC24 8997 479D 4AB0
[+] /global/cscratch1/sd/elvis/spack-envs/hdf5/spack/opt/spack/cray-cnl7-haswell/gcc-9.3.0/hdf5-1.10.7-4iumqkfbojam6unwkb2y5t45deds7hb3
==> Updating view at /global/cscratch1/sd/elvis/spack-envs/hdf5/spack/var/spack/environments/hdf5/.spack-env/view
==> Warning: [/global/cscratch1/sd/elvis/spack-envs/hdf5/spack/var/spack/environments/hdf5/.spack-env/view] Skipping external package: mpich@3.1%gcc@9.3.0~argobots+fortran+hwloc+hydra+libxml2+pci+romio~slurm~verbs+wrapperrpath device=ch4 netmod=ofi pmi=pmi arch=cray-cnl7-haswell/2ymwqzf
==> Updating view at /global/cscratch1/sd/elvis/spack-envs/hdf5/spack/var/spack/environments/hdf5/.spack-env/view
==> Warning: [/global/cscratch1/sd/elvis/spack-envs/hdf5/spack/var/spack/environments/hdf5/.spack-env/view] Skipping external package: mpich@3.1%gcc@9.3.0~argobots+fortran+hwloc+hydra+libxml2+pci+romio~slurm~verbs+wrapperrpath device=ch4 netmod=ofi pmi=pmi arch=cray-cnl7-haswell/2ymwqzf

Once installation is complete, we can see our installed specs via spack find:

elvis@cori11:> spack find
==> In environment hdf5
==> Root specs
-- no arch / gcc@9.3.0 ------------------------------------------
hdf5%gcc@9.3.0

==> 3 installed packages
-- cray-cnl7-haswell / gcc@9.3.0 --------------------------------
hdf5@1.10.7  mpich@3.1  zlib@1.2.11

Congratulations, you have now installed spack packages from a buildcache!

Things to Consider

  1. If you plan to host a buildcache and mirror, you need world readable (o+rx) permission to read and install from buildcache.
  2. Please consider a short path when building to buildcache to some shared filesystem ($CFS, $SCRATCH). If you plan to use gitlab pipeline to build spack pipeline, the CI_PROJECT_DIR is a long path on Cori so a short a padding length such as 127 is sufficient. If you plan to clone spack outside of CI_PROJECT_DIR such as $CFS or $SCRATCH then please consider a unique spack path to avoid name conflicts between pipeline. Please refer to setclonepath.sh script.
  3. The public gpg key must be stored in a world-readable location in order for other users to trust gpg key spack gpg trust.