Skip to content

E4S 21.11 - Perlmutter

Overview

This is the first deployment of E4S on Perlmutter based on E4S 21.11 release provided by E4S team. We provide a containerized deployment and standard source build deployment accessible via modules.

Containerized Deployment with Shifter

You can access the E4S container using Shifter on Perlmutter.

To launch an E4S 21.11 container on a single node with 4 gpus you can do the following:

$> salloc --nodes 1 --qos interactive --time 01:00:00 --constraint gpu --gpus 4 --account=<account>_g

$> shifterimg images | grep ecpe4s 
perlmutter docker     READY    e2da787b6a   2021-11-15T07:58:10 ecpe4s/ubuntu20.04-gpu-x86_64:21.11

$> shifter -E --image=ecpe4s/ubuntu20.04-gpu-x86_64:21.11 /bin/bash --rcfile /etc/bashrc
(base) $ which spack
/spack/bin/spack

Once you are in the container you can access the software stack by running spack find.

Output of spack find inside the container
adiak@0.2.1%gcc@9.3.0: ~ipo+mpi+shared build_type=RelWithDebInfo
adios@1.13.1%gcc@9.3.0: +blosc~bzip2~fortran~hdf5~infiniband+lz4+mpi~netcdf+shared+sz~szip+zfp+zlib patches=01113e9efb929d71c28bf33cc8b7f215d85195ec700e99cb41164e2f8f830640,8ae17f655248e87cbab1d1ed794e15364a38d2f5f8d971b1086702f72d79bd42,d24b79b795f66e40ddcd331ea4be896ac9c393d6f68f4318616d23928b0694e9 staging=none
adios2@2.7.1%gcc@9.3.0: +blosc+bzip2~dataman~dataspaces~endian_reverse+fortran~hdf5~ipo+mpi+pic+png~python+shared+ssc+sst+sz+zfp build_type=Release patches=8d301e8232baf4049b547f22bd73774309662017a62dac36360d2965907062bf
adlbx@1.0.0%gcc@9.3.0:
adol-c@2.7.2%gcc@9.3.0: ~advanced_branching+atrig_erf~boost+doc+examples~openmp~sparse
alquimia@1.0.9%gcc@9.3.0: ~ipo+shared build_type=RelWithDebInfo
aml@0.1.0%gcc@9.3.0:
amrex@21.11%gcc@9.3.0: ~amrdata~cuda~eb~fortran~hdf5~hypre~ipo+linear_solvers+mpi~openmp~particles~petsc~pic~plotfile_tools+rocm~shared~sundials amdgpu_target=gfx908 build_type=RelWithDebInfo cuda_arch=none dimensions=3 precision=double
amrex@21.11%gcc@9.3.0: ~amrdata~cuda~eb~fortran~hdf5~hypre~ipo+linear_solvers+mpi~openmp~particles~petsc~pic~plotfile_tools~rocm~shared~sundials amdgpu_target=none build_type=RelWithDebInfo cuda_arch=none dimensions=3 precision=double
amrex@21.11%gcc@9.3.0: ~amrdata+cuda~eb~fortran~hdf5~hypre~ipo+linear_solvers+mpi~openmp~particles~petsc~pic~plotfile_tools~rocm~shared~sundials amdgpu_target=none build_type=RelWithDebInfo cuda_arch=70 dimensions=3 precision=double
amrex@21.11%gcc@9.3.0: ~amrdata+cuda~eb~fortran~hdf5~hypre~ipo+linear_solvers+mpi~openmp~particles~petsc~pic~plotfile_tools~rocm~shared~sundials amdgpu_target=none build_type=RelWithDebInfo cuda_arch=80 dimensions=3 precision=double
ant@1.10.7%gcc@9.3.0:
arborx@1.1%gcc@9.3.0: ~cuda~ipo+mpi~openmp~rocm+serial~sycl~trilinos build_type=RelWithDebInfo
argobots@1.1%gcc@9.3.0: ~affinity~debug+perf~stackunwind~tool~valgrind stackguard=none
arpack-ng@3.8.0%gcc@9.3.0: +mpi+shared
ascent@0.7.1%gcc@9.3.0: ~adios~adios2~babelflow~cuda~doc~dray~fides+fortran~ipo~mfem+mpi+openmp~python+serial+shared+test+vtkh build_type=RelWithDebInfo cuda_arch=none
asio@1.20.0%gcc@9.3.0: ~boost_coroutine~boost_regex~separate_compilation cxxstd=17
assimp@5.0.1%gcc@9.3.0: ~ipo+shared build_type=RelWithDebInfo
autoconf@2.69%gcc@9.3.0:  patches=35c449281546376449766f92d49fc121ca50e330e60fefcfc9be2af3253082c2,7793209b33013dc0f81208718c68440c5aae80e7a1c4b8d336e382525af791a7,a49dd5bac3b62daa0ff688ab4d508d71dbd2f4f8d7e2a02321926346161bf3ee
autoconf-archive@2019.01.06%gcc@9.3.0:
automake@1.16.3%gcc@9.3.0:
axl@0.3.0%gcc@9.3.0: +bbapi~bbapi_fallback~dw~ipo+shared async_api=daemon build_type=RelWithDebInfo
axl@0.5.0%gcc@9.3.0: +bbapi~bbapi_fallback~dw~ipo+shared async_api=daemon build_type=RelWithDebInfo
axom@0.5.0%gcc@9.3.0: +cpp14~cuda~debug~devtools+examples+fortran+hdf5~ipo+lua~mfem+mpi+openmp~python+raja~scr+shared+tools+umpire build_type=RelWithDebInfo cuda_arch=none
berkeley-db@18.1.40%gcc@9.3.0: +cxx~docs+stl patches=b231fcc4d5cff05e5c3a4814f6a5af0e9a966428dc2176540d2c05aff41de522
berkeley-db@18.1.40%gcc@11.1.0: +cxx~docs+stl patches=b231fcc4d5cff05e5c3a4814f6a5af0e9a966428dc2176540d2c05aff41de522
binutils@2.33.1%gcc@9.3.0: ~gas+gold+headers~interwork+ld+libiberty~lto~nls+plugins libs=shared,static
binutils@2.36.1%gcc@9.3.0: ~gas+gold+headers~interwork+ld+libiberty~lto~nls+plugins libs=shared,static patches=a51b7bf4607645235be01683a3e2deb3a87d767cc124ea4ef6febf9f32abf4a7
binutils@2.36.1%gcc@11.1.0: ~gas+gold+headers~interwork+ld+libiberty~lto~nls+plugins libs=shared,static patches=a51b7bf4607645235be01683a3e2deb3a87d767cc124ea4ef6febf9f32abf4a7
bison@3.8.2%gcc@9.3.0:
blaspp@2021.04.01%gcc@9.3.0: ~cuda~ipo+openmp+rocm+shared amdgpu_target=gfx908 build_type=RelWithDebInfo cuda_arch=none
blaspp@2021.04.01%gcc@9.3.0: ~cuda~ipo+openmp~rocm+shared amdgpu_target=none build_type=RelWithDebInfo cuda_arch=none
blaspp@2021.04.01%gcc@9.3.0: +cuda~ipo+openmp~rocm+shared amdgpu_target=none build_type=RelWithDebInfo cuda_arch=none
blt@0.4.1%gcc@9.3.0:
bmi@main%gcc@9.3.0:
bolt@2.0%gcc@9.3.0: ~ipo build_type=RelWithDebInfo
boost@1.77.0%gcc@9.3.0: +atomic+chrono~clanglibcpp~container~context~coroutine+date_time~debug+exception~fiber+filesystem+graph~icu+iostreams+locale+log+math~mpi+multithreaded~numpy~pic+program_options~python+random+regex+serialization+shared+signals~singlethreaded+system~taggedlayout+test+thread+timer~versionedlayout+wave cxxstd=17 patches=93f4aad8f88d1437e50d95a2d066390ef3753b99ef5de24f7a46bc083bd6df06 visibility=global
boost@1.77.0%gcc@9.3.0: +atomic+chrono~clanglibcpp~container~context~coroutine+date_time~debug+exception~fiber+filesystem+graph~icu+iostreams+locale+log+math~mpi+multithreaded~numpy~pic+program_options~python+random+regex+serialization+shared+signals~singlethreaded+system~taggedlayout+test+thread+timer~versionedlayout+wave cxxstd=98 patches=93f4aad8f88d1437e50d95a2d066390ef3753b99ef5de24f7a46bc083bd6df06 visibility=global
butterflypack@2.0.0%gcc@9.3.0: ~ipo+shared build_type=RelWithDebInfo
bzip2@1.0.8%gcc@9.3.0: ~debug~pic+shared
bzip2@1.0.8%gcc@11.1.0: ~debug~pic+shared
c-blosc@1.21.0%gcc@9.3.0: +avx2~ipo build_type=RelWithDebInfo
cabana@0.4.0%gcc@9.3.0: ~arborx~cuda~heffte~hypre~ipo+mpi~openmp~pthread~rocm+serial+shared~sycl build_type=RelWithDebInfo
caliper@2.7.0%gcc@9.3.0: +adiak~cuda~fortran+gotcha~ipo+libdw~libpfm+libunwind+mpi+papi+sampler+shared~sosflow build_type=RelWithDebInfo cuda_arch=none
caliper@2.7.0%gcc@9.3.0: +adiak+cuda~fortran+gotcha~ipo+libdw~libpfm+libunwind+mpi+papi+sampler+shared~sosflow build_type=RelWithDebInfo cuda_arch=70
caliper@2.7.0%gcc@9.3.0: +adiak+cuda~fortran+gotcha~ipo+libdw~libpfm+libunwind+mpi+papi+sampler+shared~sosflow build_type=RelWithDebInfo cuda_arch=80
camp@0.1.0%gcc@9.3.0: ~cuda~ipo~rocm~tests amdgpu_target=none build_type=RelWithDebInfo cuda_arch=none
camp@0.2.2%gcc@9.3.0: ~cuda~ipo~rocm~tests amdgpu_target=none build_type=RelWithDebInfo cuda_arch=none
camp@0.2.2%gcc@9.3.0: +cuda~ipo~rocm~tests amdgpu_target=none build_type=RelWithDebInfo cuda_arch=70
camp@0.2.2%gcc@9.3.0: +cuda~ipo~rocm~tests amdgpu_target=none build_type=RelWithDebInfo cuda_arch=80
catalyst@5.6.0%gcc@9.3.0: ~essentials~extras~ipo+osmesa~python~python3+rendering build_type=RelWithDebInfo
chai@2.4.0%gcc@9.3.0: ~benchmarks~cuda~enable_pick+examples~ipo~openmp~raja~rocm+shared~tests amdgpu_target=none build_type=RelWithDebInfo cuda_arch=none
chai@2.4.0%gcc@9.3.0: ~benchmarks+cuda~enable_pick+examples~ipo~openmp~raja~rocm+shared~tests amdgpu_target=none build_type=RelWithDebInfo cuda_arch=70
chai@2.4.0%gcc@9.3.0: ~benchmarks+cuda~enable_pick+examples~ipo~openmp~raja~rocm+shared~tests amdgpu_target=none build_type=RelWithDebInfo cuda_arch=80
charliecloud@0.25%gcc@9.3.0: ~docs
cinch@master%gcc@9.3.0:
cmake@3.21.4%gcc@9.3.0: ~doc+ncurses+openssl+ownlibs~qt build_type=Release
cmake@3.21.4%gcc@11.1.0: ~doc+ncurses+openssl+ownlibs~qt build_type=Release
conduit@0.7.2%gcc@9.3.0: ~adios~doc~doxygen+fortran+hdf5+hdf5_compat~ipo+mpi~parmetis~python+shared~silo+test~zfp build_type=RelWithDebInfo patches=89d1829ad52f503f6179e43efddf998c239a95c14ca1f248463a3f61ad7d5cf7
conduit@0.7.2%gcc@9.3.0: ~adios~doc~doxygen+fortran+hdf5+hdf5_compat~ipo+mpi~parmetis~python+shared~silo+test~zfp build_type=RelWithDebInfo patches=89d1829ad52f503f6179e43efddf998c239a95c14ca1f248463a3f61ad7d5cf7
cub@1.12.0-rc0%gcc@9.3.0:
cuda@11.4.2%gcc@9.3.0: ~dev
cuda@11.4.2%gcc@11.1.0: ~dev
curl@7.79.0%gcc@9.3.0: ~gssapi~ldap~libidn2~librtmp~libssh~libssh2~nghttp2 tls=openssl
czmq@4.1.1%gcc@9.3.0:
darshan-runtime@3.3.1%gcc@9.3.0: ~apmpi~apmpi_sync~apxc~hdf5+mpi scheduler=NONE
darshan-util@3.3.1%gcc@9.3.0: ~apmpi~apxc~bzip2
datatransferkit@3.1-rc3%gcc@9.3.0: ~external-arborx~ipo~openmp+serial+shared build_type=RelWithDebInfo
diffutils@3.8%gcc@9.3.0:
diffutils@3.8%gcc@11.1.0:
double-conversion@3.1.5%gcc@9.3.0: ~ipo build_type=RelWithDebInfo
doxygen@1.8.20%gcc@9.3.0: ~graphviz~ipo~mscgen build_type=RelWithDebInfo
dtcmp@1.1.3%gcc@9.3.0: +shared
dyninst@11.0.1%gcc@9.3.0: ~ipo+openmp~stat_dysect~static build_type=RelWithDebInfo
eigen@3.4.0%gcc@9.3.0: ~ipo build_type=RelWithDebInfo
elfutils@0.185%gcc@9.3.0: +bzip2~debuginfod~nls+xz patches=d786d49c28d7f0c8fc27bab39ca8714e5f4d128c7f09bb18533a8ec99b38dbf8
er@0.1.0%gcc@9.3.0: ~ipo+shared build_type=RelWithDebInfo
exaworks@0.1.0%gcc@9.3.0:
exmcutils@0.6.0%gcc@9.3.0:
expat@2.4.1%gcc@9.3.0: +libbsd
expat@2.4.1%gcc@11.1.0: +libbsd
faodel@1.2108.1%gcc@9.3.0: ~hdf5~ipo+mpi+shared+tcmalloc build_type=RelWithDebInfo logging=stdout network=nnti serializer=xdr
fftw@3.3.10%gcc@9.3.0: +mpi+openmp~pfft_patches precision=double,float
findutils@4.8.0%gcc@9.3.0:
flecsi@1.4.2%gcc@9.3.0: ~coverage~cuda~debug_backend~disable_metis~doc~doxygen+external_cinch~flecstan~flog~graphviz+hdf5~ipo~kokkos~openmp+shared~tutorial~unit_tests backend=mpi build_type=RelWithDebInfo caliper_detail=none cuda_arch=none
flecsi@2.1.0%gcc@9.3.0: ~coverage+cuda~debug_backend~disable_metis~doc~doxygen~external_cinch~flecstan~flog~graphviz+hdf5~ipo~kokkos~openmp+shared~tutorial~unit_tests backend=mpi build_type=RelWithDebInfo caliper_detail=none cuda_arch=70
flecsi@2.1.0%gcc@9.3.0: ~coverage+cuda~debug_backend~disable_metis~doc~doxygen~external_cinch~flecstan~flog~graphviz+hdf5~ipo~kokkos~openmp+shared~tutorial~unit_tests backend=mpi build_type=RelWithDebInfo caliper_detail=none cuda_arch=80
flex@2.6.3%gcc@9.3.0: +lex~nls
flit@2.1.0%gcc@9.3.0:
flux-core@0.30.0%gcc@9.3.0: ~cuda~docs patches=0329f4b38b5ce4e1a3dee212cb93ce40388d1bde47c82a7a8e73bc7c38274c81
flux-sched@0.19.0%gcc@9.3.0: ~cuda
fortrilinos@2.0.0%gcc@9.3.0: +hl~ipo+shared build_type=RelWithDebInfo
freetype@2.11.0%gcc@9.3.0:
gasnet@2021.9.0%gcc@9.3.0: ~debug conduits=smp
gawk@5.1.0%gcc@9.3.0:
gdbm@1.19%gcc@9.3.0:
gdbm@1.19%gcc@11.1.0:
gettext@0.21%gcc@9.3.0: +bzip2+curses+git~libunistring+libxml2+tar+xz
gettext@0.21%gcc@11.1.0: +bzip2+curses+git~libunistring+libxml2+tar+xz
ginkgo@1.4.0%gcc@9.3.0: ~cuda~develtools~full_optimizations~hwloc~ipo+openmp+rocm+shared amdgpu_target=gfx908 build_type=Release cuda_arch=none patches=43e571a49ea774793d89d145571c95ae7cdf991f5220d8fe84db217a3c613a32
ginkgo@1.4.0%gcc@9.3.0: ~cuda~develtools~full_optimizations~hwloc~ipo+openmp~rocm+shared amdgpu_target=none build_type=Release cuda_arch=none patches=43e571a49ea774793d89d145571c95ae7cdf991f5220d8fe84db217a3c613a32
ginkgo@1.4.0%gcc@9.3.0: +cuda~develtools~full_optimizations~hwloc~ipo+openmp~rocm+shared amdgpu_target=none build_type=Release cuda_arch=70 patches=43e571a49ea774793d89d145571c95ae7cdf991f5220d8fe84db217a3c613a32
ginkgo@1.4.0%gcc@9.3.0: +cuda~develtools~full_optimizations~hwloc~ipo+openmp~rocm+shared amdgpu_target=none build_type=Release cuda_arch=80 patches=43e571a49ea774793d89d145571c95ae7cdf991f5220d8fe84db217a3c613a32
git@2.31.1%gcc@9.3.0: +man+nls+perl~svn~tcltk
glib@2.70.0%gcc@9.3.0: ~libmount patches=8547954e50d0a720fa9794b84c31569df6a7419319b95ac9de0ed861d8969ffd tracing=none
globalarrays@5.8%gcc@9.3.0: ~scalapack armci=mpi-ts
glproto@1.4.17%gcc@9.3.0:
gmake@4.3%gcc@9.3.0: ~guile+nls
gmp@6.2.1%gcc@9.3.0:
googletest@1.8.1%gcc@9.3.0: +gmock~ipo+pthreads+shared build_type=RelWithDebInfo
gotcha@1.0.3%gcc@9.3.0: ~ipo~test build_type=RelWithDebInfo
gperftools@2.9.1%gcc@9.3.0: +debugalloc~dynamic_sized_delete_support+libunwind~sized_delete
gsl@2.7%gcc@9.3.0: ~external-cblas
hdf5@1.8.22%gcc@9.3.0: ~cxx~fortran+hl~ipo~java+mpi+shared~szip~threadsafe+tools api=v18 build_type=RelWithDebInfo
hdf5@1.8.22%gcc@9.3.0: ~cxx+fortran+hl~ipo~java+mpi+shared~szip~threadsafe+tools api=v18 build_type=RelWithDebInfo
hdf5@1.12.0%gcc@9.3.0: ~cxx+fortran+hl~ipo~java+mpi+shared~szip~threadsafe+tools api=default build_type=RelWithDebInfo
hdf5@1.12.1%gcc@9.3.0: ~cxx+fortran+hl~ipo~java+mpi+shared~szip~threadsafe+tools api=v18 build_type=RelWithDebInfo
heffte@2.2.0%gcc@9.3.0: ~cuda~fftw~fortran~ipo~magma~mkl~python+rocm+shared amdgpu_target=gfx908 build_type=RelWithDebInfo cuda_arch=none
heffte@2.2.0%gcc@9.3.0: ~cuda+fftw~fortran~ipo~magma~mkl~python~rocm+shared amdgpu_target=none build_type=RelWithDebInfo cuda_arch=none
heffte@2.2.0%gcc@9.3.0: +cuda~fftw~fortran~ipo~magma~mkl~python~rocm+shared amdgpu_target=none build_type=RelWithDebInfo cuda_arch=70
heffte@2.2.0%gcc@9.3.0: +cuda~fftw~fortran~ipo~magma~mkl~python~rocm+shared amdgpu_target=none build_type=RelWithDebInfo cuda_arch=80
help2man@1.47.16%gcc@9.3.0:
hip@4.3.1%gcc@9.3.0: ~ipo build_type=Release patches=2a4190477b7d9206b9cd8d70770ba0bc007273cbe54772efb12f9ca2e37c0392,99190b4616edb362d48f9b265c3018a3c6339481b0729d9fe46185fca25bc54b,e276c4acf3d37712b6bea306fea34f539d3c4f743471e9da208b5eb17b16ae67
hipblas@4.3.1%gcc@9.3.0: ~ipo build_type=Release
hipsparse@4.3.1%gcc@9.3.0: ~ipo build_type=Release
hpctoolkit@2021.10.15%gcc@9.3.0: ~all-static~cray~cuda~debug+mpi+papi~rocm+viewer
hpcviewer@2021.10%gcc@9.3.0:
hpx@1.7.1%gcc@9.3.0: ~async_cuda~async_mpi~cuda~examples~generic_coroutines~ipo~rocm~tools amdgpu_target=none build_type=RelWithDebInfo cuda_arch=none cxxstd=17 instrumentation=none malloc=tcmalloc max_cpu_count=64 networking=tcp
hpx@1.7.1%gcc@9.3.0: ~async_cuda~async_mpi+cuda~examples~generic_coroutines~ipo~rocm~tools amdgpu_target=none build_type=RelWithDebInfo cuda_arch=70 cxxstd=17 instrumentation=none malloc=tcmalloc max_cpu_count=64 networking=tcp
hpx@1.7.1%gcc@9.3.0: ~async_cuda~async_mpi+cuda~examples~generic_coroutines~ipo~rocm~tools amdgpu_target=none build_type=RelWithDebInfo cuda_arch=80 cxxstd=17 instrumentation=none malloc=tcmalloc max_cpu_count=64 networking=tcp
hsa-rocr-dev@4.3.1%gcc@9.3.0: +image~ipo+shared build_type=Release patches=71e6851d9be8113bfb8d71b66a3171e0d0401bae5e6f161c9e7fe32558261a46
hwloc@2.6.0%gcc@9.3.0: ~cairo~cuda~gl~libudev+libxml2~netloc~nvml~opencl+pci~rocm+shared
hwloc@2.6.0%gcc@9.3.0: ~cairo+cuda~gl~libudev+libxml2~netloc~nvml~opencl+pci~rocm+shared
hwloc@2.6.0%gcc@11.1.0: ~cairo+cuda~gl~libudev+libxml2~netloc~nvml~opencl+pci~rocm+shared
hypre@2.23.0%gcc@9.3.0: ~complex~cuda~debug+fortran~int64~internal-superlu~mixedint+mpi~openmp+shared~superlu-dist~unified-memory cuda_arch=none
hypre@2.23.0%gcc@9.3.0: ~complex+cuda~debug+fortran~int64~internal-superlu~mixedint+mpi~openmp+shared~superlu-dist~unified-memory cuda_arch=70
hypre@2.23.0%gcc@9.3.0: ~complex+cuda~debug+fortran~int64~internal-superlu~mixedint+mpi~openmp+shared~superlu-dist~unified-memory cuda_arch=80
inputproto@2.3.2%gcc@9.3.0:
intel-tbb@2020.3%gcc@9.3.0: ~ipo+shared+tm build_type=RelWithDebInfo cxxstd=default patches=62ba015ebd1819c45bef47411540b789b493e31ca668c4ff4cb2afcbc306b476,ce1fb16fb932ce86a82ca87cf0431d1a8c83652af9f552b264213b2ff2945d73,d62cb666de4010998c339cde6f41c7623a07e9fc69e498f2e149821c0c2c6dd0
intel-xed@12.0.1%gcc@9.3.0: ~debug~pic patches=acffa07ee4b8e898df18f1439547e45ec9c609328dfb6e1ad2dce416587334d4
jansson@2.13.1%gcc@9.3.0: ~ipo+shared build_type=RelWithDebInfo
json-c@0.15%gcc@9.3.0: ~ipo build_type=RelWithDebInfo
jsoncpp@1.9.4%gcc@9.3.0: ~ipo build_type=RelWithDebInfo cxxstd=default
kbproto@1.0.7%gcc@9.3.0:
kokkos@3.4.01%gcc@9.3.0: ~aggressive_vectorization~compiler_warnings~cuda~cuda_constexpr~cuda_lambda~cuda_ldg_intrinsic~cuda_relocatable_device_code~cuda_uvm~debug~debug_bounds_check~debug_dualview_modify_check~deprecated_code~examples~explicit_instantiation~hpx~hpx_async_dispatch~hwloc~ipo~memkind~numactl~openmp~pic+profiling~profiling_load_print~pthread~qthread+rocm+serial+shared~sycl~tests~tuning~wrapper amdgpu_target=gfx908 build_type=RelWithDebInfo cuda_arch=none std=14
kokkos@3.4.01%gcc@9.3.0: ~aggressive_vectorization~compiler_warnings~cuda~cuda_constexpr~cuda_lambda~cuda_ldg_intrinsic~cuda_relocatable_device_code~cuda_uvm~debug~debug_bounds_check~debug_dualview_modify_check~deprecated_code~examples~explicit_instantiation~hpx~hpx_async_dispatch~hwloc~ipo~memkind~numactl~openmp~pic+profiling~profiling_load_print~pthread~qthread~rocm+serial+shared~sycl~tests~tuning~wrapper amdgpu_target=none build_type=RelWithDebInfo cuda_arch=none std=14
kokkos@3.4.01%gcc@9.3.0: ~aggressive_vectorization~compiler_warnings~cuda~cuda_constexpr~cuda_lambda~cuda_ldg_intrinsic~cuda_relocatable_device_code~cuda_uvm~debug~debug_bounds_check~debug_dualview_modify_check~deprecated_code~examples~explicit_instantiation~hpx~hpx_async_dispatch~hwloc~ipo~memkind~numactl+openmp~pic+profiling~profiling_load_print~pthread~qthread~rocm+serial+shared~sycl~tests~tuning~wrapper amdgpu_target=none build_type=RelWithDebInfo cuda_arch=none std=14
kokkos@3.4.01%gcc@9.3.0: ~aggressive_vectorization~compiler_warnings+cuda~cuda_constexpr~cuda_lambda~cuda_ldg_intrinsic~cuda_relocatable_device_code~cuda_uvm~debug~debug_bounds_check~debug_dualview_modify_check~deprecated_code~examples~explicit_instantiation~hpx~hpx_async_dispatch~hwloc~ipo~memkind~numactl~openmp~pic+profiling~profiling_load_print~pthread~qthread~rocm+serial+shared~sycl~tests~tuning+wrapper amdgpu_target=none build_type=RelWithDebInfo cuda_arch=70 std=14
kokkos@3.4.01%gcc@9.3.0: ~aggressive_vectorization~compiler_warnings+cuda~cuda_constexpr~cuda_lambda~cuda_ldg_intrinsic~cuda_relocatable_device_code~cuda_uvm~debug~debug_bounds_check~debug_dualview_modify_check~deprecated_code~examples~explicit_instantiation~hpx~hpx_async_dispatch~hwloc~ipo~memkind~numactl~openmp~pic+profiling~profiling_load_print~pthread~qthread~rocm+serial+shared~sycl~tests~tuning+wrapper amdgpu_target=none build_type=RelWithDebInfo cuda_arch=80 std=14
kokkos-kernels@3.4.01%gcc@9.3.0: ~blas~cblas~cublas~cuda~cusparse~ipo~lapack~lapacke~mkl+openmp~pthread~serial+shared~superlu build_type=RelWithDebInfo cuda_arch=none execspace_cuda=auto execspace_openmp=auto execspace_serial=auto execspace_threads=auto layouts=left memspace_cudaspace=auto memspace_cudauvmspace=auto offsets=int,size_t ordinals=int scalars=double
kokkos-kernels@3.4.01%gcc@9.3.0: ~blas~cblas~cublas+cuda~cusparse~ipo~lapack~lapacke~mkl~openmp~pthread~serial+shared~superlu build_type=RelWithDebInfo cuda_arch=70 execspace_cuda=auto execspace_openmp=auto execspace_serial=auto execspace_threads=auto layouts=left memspace_cudaspace=auto memspace_cudauvmspace=auto offsets=int,size_t ordinals=int scalars=double
kokkos-kernels@3.4.01%gcc@9.3.0: ~blas~cblas~cublas+cuda~cusparse~ipo~lapack~lapacke~mkl~openmp~pthread~serial+shared~superlu build_type=RelWithDebInfo cuda_arch=80 execspace_cuda=auto execspace_openmp=auto execspace_serial=auto execspace_threads=auto layouts=left memspace_cudaspace=auto memspace_cudauvmspace=auto offsets=int,size_t ordinals=int scalars=double
kokkos-nvcc-wrapper@3.2.00%gcc@9.3.0: +mpi
kokkos-nvcc-wrapper@3.2.00%gcc@9.3.0: +mpi
kvtree@1.2.0%gcc@9.3.0: ~ipo+mpi+shared build_type=RelWithDebInfo file_lock=FLOCK
lapackpp@2021.04.00%gcc@9.3.0: ~ipo+shared build_type=RelWithDebInfo
lapackpp@2021.04.00%gcc@9.3.0: ~ipo+shared build_type=RelWithDebInfo
lapackpp@2021.04.00%gcc@9.3.0: ~ipo+shared build_type=RelWithDebInfo
legion@21.03.0%gcc@9.3.0: ~bindings~bounds_checks~cuda~cuda_hijack~cuda_unsupported_compiler~enable_tls~fortran~gasnet_debug~hdf5~hwloc~ipo~kokkos+libdl~native~openmp~papi~privilege_checks~python~redop_complex~shared~spy+zlib build_type=RelWithDebInfo conduit=none cuda_arch=70 gasnet_root=none max_dims=3 max_fields=512 network=none output_level=warning
libarchive@3.5.2%gcc@9.3.0: +iconv compression=bz2lib,lz4,lzma,lzo2,zlib,zstd crypto=mbedtls libs=shared,static programs=none xar=libxml2
libbsd@0.11.3%gcc@9.3.0:
libbsd@0.11.3%gcc@11.1.0:
libcap@2.25%gcc@9.3.0:  patches=3db844dd771b320b8117b72f4e00b560051311fbbd4cba1ebcbd7c19116c1d66
libcircle@0.3.0%gcc@9.3.0:
libdwarf@20180129%gcc@9.3.0:
libedit@3.1-20210216%gcc@9.3.0:
libedit@3.1-20210216%gcc@11.1.0:
libevent@2.1.12%gcc@9.3.0: +openssl
libfabric@1.13.2%gcc@9.3.0: ~debug~kdreg fabrics=rxm,sockets,tcp,udp
libffi@3.3%gcc@9.3.0:  patches=26f26c6f29a7ce9bf370ad3ab2610f99365b4bdd7b82e7c31df41a3370d685c0
libffi@3.3%gcc@11.1.0:  patches=26f26c6f29a7ce9bf370ad3ab2610f99365b4bdd7b82e7c31df41a3370d685c0
libgcrypt@1.9.3%gcc@9.3.0:
libgit2@1.1.1%gcc@9.3.0: ~curl~ipo+ssh build_type=RelWithDebInfo https=system
libgpg-error@1.42%gcc@9.3.0:
libiberty@2.33.1%gcc@9.3.0: +pic
libice@1.0.9%gcc@9.3.0:
libiconv@1.16%gcc@9.3.0:  libs=shared,static
libiconv@1.16%gcc@11.1.0:  libs=shared,static
libidn2@2.3.0%gcc@9.3.0:
libjpeg-turbo@2.1.0%gcc@9.3.0:
libmd@1.0.3%gcc@9.3.0:
libmd@1.0.3%gcc@11.1.0:
libmonitor@2021.04.27%gcc@9.3.0: ~commrank~dlopen+hpctoolkit
libnrm@0.1.0%gcc@9.3.0:
libogg@1.3.4%gcc@9.3.0:
libpciaccess@0.16%gcc@9.3.0:
libpciaccess@0.16%gcc@11.1.0:
libpng@1.6.37%gcc@9.3.0:
libpthread-stubs@0.4%gcc@9.3.0:
libquo@1.3.1%gcc@9.3.0:
libsigsegv@2.13%gcc@9.3.0:
libsigsegv@2.13%gcc@11.1.0:
libsm@1.2.3%gcc@9.3.0:
libsodium@1.0.18%gcc@9.3.0:
libssh2@1.8.0%gcc@9.3.0: ~ipo+shared build_type=RelWithDebInfo
libtheora@1.1.1%gcc@9.3.0:  patches=057836ee5a5e28cae435c20d31173e20a3f96cfe680505836f19a77e985f7100,2e4f891f6880386d9391f3e4eaf4a23493de4eea532f9b5cb8a04b5f7cd09301,ff8478dbe3f19dd24f01d220e3262c1819d101b013d8ab3d7a972528252f1a7e
libtiff@4.3.0%gcc@9.3.0: ~jbig~jpeg~jpeg12~lerc~libdeflate~lzma~old-jpeg~pixarlog~webp~zlib~zstd
libtool@2.4.6%gcc@9.3.0:
libtool@2.4.6%gcc@11.1.0:
libunistring@0.9.10%gcc@9.3.0:
libunwind@1.5.0%gcc@9.3.0: ~block_signals~conservative_checks~cxx_exceptions~debug~debug_frame+docs+pic+tests+weak_backtrace+xz~zlib components=none libs=shared,static
libx11@1.7.0%gcc@9.3.0:
libxau@1.0.8%gcc@9.3.0:
libxcb@1.14%gcc@9.3.0:
libxdmcp@1.1.2%gcc@9.3.0:
libxext@1.3.3%gcc@9.3.0:
libxml2@2.9.12%gcc@9.3.0: ~python
libxml2@2.9.12%gcc@11.1.0: ~python
libxt@1.1.5%gcc@9.3.0:
libyaml@0.2.5%gcc@9.3.0:
libyogrt@1.24%gcc@9.3.0: ~static scheduler=slurm
libzmq@4.3.3%gcc@9.3.0: ~drafts+libsodium
llvm-amdgpu@4.3.1%gcc@9.3.0: ~ipo+openmp+rocm-device-libs build_type=Release patches=d999f3b235e655ee07f6dd2590302082feaa06d32c5c6b53aae9c5cf1e45b644
loki@0.1.7%gcc@9.3.0: +shared
lua@5.3.5%gcc@9.3.0: ~pcfile+shared
lua-luaposix@35.0%gcc@9.3.0:
lwgrp@1.0.4%gcc@9.3.0: +shared
lz4@1.9.3%gcc@9.3.0:  libs=shared,static
lzo@2.10%gcc@9.3.0:  libs=shared,static
m4@1.4.19%gcc@9.3.0: +sigsegv patches=9dc5fbd0d5cb1037ab1e6d0ecc74a30df218d0a94bdd5a02759a97f62daca573,bfdffa7c2eb01021d5849b36972c069693654ad826c1a20b53534009a4ec7a89
m4@1.4.19%gcc@11.1.0: +sigsegv patches=9dc5fbd0d5cb1037ab1e6d0ecc74a30df218d0a94bdd5a02759a97f62daca573,bfdffa7c2eb01021d5849b36972c069693654ad826c1a20b53534009a4ec7a89
magma@2.6.1%gcc@9.3.0: ~cuda+fortran~ipo+rocm+shared amdgpu_target=gfx908 build_type=RelWithDebInfo cuda_arch=none
magma@2.6.1%gcc@9.3.0: +cuda+fortran~ipo~rocm+shared amdgpu_target=none build_type=RelWithDebInfo cuda_arch=70
magma@2.6.1%gcc@9.3.0: +cuda+fortran~ipo~rocm+shared amdgpu_target=none build_type=RelWithDebInfo cuda_arch=80
mbedtls@2.27.0%gcc@9.3.0: +pic build_type=Release libs=static
mbedtls@3.0.0%gcc@9.3.0: +pic build_type=Release libs=static
memkind@1.10.1%gcc@9.3.0:
mercury@1.0.1%gcc@9.3.0: +bmi+boostsys~cci+checksum~debug~ipo+mpi+ofi+shared+sm~udreg build_type=RelWithDebInfo patches=34fc95b3599c74a8cece6e873cfdc8bc0afe2dc0deabb6e2d11ea2a93f0cebf5
mercury@2.0.1%gcc@9.3.0: ~bmi+boostsys~cci+checksum~debug~ipo+mpi+ofi+shared+sm~udreg build_type=RelWithDebInfo
mesa18@18.3.6%gcc@9.3.0: +glx~llvm~opengles+osmesa patches=55a5611ca9fcbe8324c4d68a07b338134954ff12c5b122dc78ff376f012a1414 swr=none
meson@0.60.0%gcc@9.3.0:  patches=aa6c50d5a2aeb1a487d16f6712be4357fefb923aae37ab830699b07338388287
metall@0.17%gcc@9.3.0: ~ipo build_type=RelWithDebInfo
metis@5.1.0%gcc@9.3.0: ~gdb~int64~real64+shared build_type=Release patches=4991da938c1d3a1d3dea78e49bbebecba00273f98df2a656e38b83d55b281da1,b1225da886605ea558db7ac08dd8054742ea5afe5ed61ad4d0fe7a495b1270d2
mfem@4.3.0%gcc@9.3.0: ~amgx~conduit~cuda~debug~examples~gnutls~gslib~lapack~libceed~libunwind+metis~miniapps~mpfr+mpi~netcdf~occa~openmp~petsc~pumi~raja+rocm~shared~slepc+static~strumpack~suite-sparse~sundials~superlu-dist~threadsafe~umpire+zlib amdgpu_target=gfx908 cuda_arch=none patches=ccc98d6d874a760218c9f6e4bcf3a97ea79746b0f28997f4312fcf02a0beae6e timer=auto
mfem@4.3.0%gcc@9.3.0: ~amgx~conduit~cuda~debug~examples~gnutls~gslib~lapack~libceed~libunwind+metis~miniapps~mpfr+mpi~netcdf~occa~openmp~petsc~pumi~raja~rocm~shared~slepc+static~strumpack~suite-sparse~sundials~superlu-dist~threadsafe~umpire+zlib amdgpu_target=none cuda_arch=none patches=ccc98d6d874a760218c9f6e4bcf3a97ea79746b0f28997f4312fcf02a0beae6e timer=auto
mfem@4.3.0%gcc@9.3.0: ~amgx~conduit+cuda~debug~examples~gnutls~gslib~lapack~libceed~libunwind+metis~miniapps~mpfr+mpi~netcdf~occa~openmp~petsc~pumi~raja~rocm~shared~slepc+static~strumpack~suite-sparse~sundials~superlu-dist~threadsafe~umpire+zlib amdgpu_target=none cuda_arch=70 patches=ccc98d6d874a760218c9f6e4bcf3a97ea79746b0f28997f4312fcf02a0beae6e timer=auto
mfem@4.3.0%gcc@9.3.0: ~amgx~conduit+cuda~debug~examples~gnutls~gslib~lapack~libceed~libunwind+metis~miniapps~mpfr+mpi~netcdf~occa~openmp~petsc~pumi~raja~rocm~shared~slepc+static~strumpack~suite-sparse~sundials~superlu-dist~threadsafe~umpire+zlib amdgpu_target=none cuda_arch=80 patches=ccc98d6d874a760218c9f6e4bcf3a97ea79746b0f28997f4312fcf02a0beae6e timer=auto
mochi-margo@0.4.3%gcc@9.3.0:
mpark-variant@1.4.0%gcc@9.3.0: ~ipo build_type=RelWithDebInfo patches=21a4f8de3525204ee6db2e53758a3e3fd9c13817df29d2926d24376858a369e7,4e173fe8c853eb92956a40371688b4a19498189fe65b7ceac30f6b9d6663a985,b3501f726fd40129b4aaa11453a5891c8953a34af8ac84f5ab10a22afa5e7b9b
mpc@1.1.0%gcc@9.3.0:
mpfr@4.1.0%gcc@9.3.0:
mpich@3.4.2%gcc@9.3.0: ~argobots+fortran+hwloc+hydra+libxml2+pci+romio~slurm~verbs~wrapperrpath device=ch4 netmod=ofi pmi=pmi
mpich@3.4.2%gcc@9.3.0: ~argobots+fortran+hwloc+hydra+libxml2+pci+romio~slurm~verbs~wrapperrpath device=ch4 netmod=ofi pmi=pmi
mpifileutils@0.11%gcc@9.3.0: ~experimental~gpfs~lustre~xattr
munge@0.5.14%gcc@9.3.0:  localstatedir=PREFIX/var
muparser@2.2.6.1%gcc@9.3.0:
nasm@2.15.05%gcc@9.3.0:
ncurses@6.2%gcc@9.3.0: ~symlinks+termlib abi=none
ncurses@6.2%gcc@11.1.0: ~symlinks+termlib abi=none
netcdf-c@4.8.1%gcc@9.3.0: ~dap~fsync~hdf4~jna+mpi~parallel-netcdf+pic+shared
netlib-scalapack@2.1.0%gcc@9.3.0: ~ipo~pic+shared build_type=Release patches=1c9ce5fee1451a08c2de3cc87f446aeda0b818ebbce4ad0d980ddf2f2a0b2dc4,f2baedde688ffe4c20943c334f580eb298e04d6f35c86b90a1f4e8cb7ae344a2
ninja@1.10.2%gcc@9.3.0:
nlohmann-json@3.10.4%gcc@9.3.0: ~ipo~multiple_headers build_type=RelWithDebInfo
nrm@0.1.0%gcc@9.3.0:
numactl@2.0.14%gcc@9.3.0:  patches=4e1d78cbbb85de625bad28705e748856033eaafab92a66dffd383a3d7e00cc94,62fc8a8bf7665a60e8f4c93ebbd535647cebf74198f7afafec4c085a8825c006,ff37630df599cfabf0740518b91ec8daaf18e8f288b19adaae5364dc1f6b2296
nvhpc@21.9%gcc@9.3.0: +blas+lapack+mpi install_type=single
oce@0.18.3%gcc@9.3.0: ~X11+tbb
omega-h@9.34.1%gcc@9.3.0: ~examples~ipo+mpi+optimize+shared+symbols~throw+trilinos~warnings+zlib build_type=RelWithDebInfo
openblas@0.3.18%gcc@9.3.0: ~bignuma~consistent_fpcsr~ilp64+locking+pic+shared threads=openmp
openjdk@11.0.12_7%gcc@9.3.0:
openmpi@4.1.1%gcc@9.3.0: ~atomics~cuda~cxx~cxx_exceptions+gpfs~internal-hwloc~java~legacylaunchers~lustre~memchecker~pmi~pmix~singularity~sqlite3+static~thread_multiple+vt+wrapper-rpath fabrics=none schedulers=none
openpmd-api@0.14.3%gcc@9.3.0: ~adios1+adios2+hdf5~ipo+mpi~python+shared build_type=RelWithDebInfo
openssh@8.7p1%gcc@9.3.0:
openssl@1.1.1l%gcc@9.3.0: ~docs certs=system
openssl@1.1.1l%gcc@11.1.0: ~docs certs=system
otf2@2.3%gcc@9.3.0:
papi@6.0.0.1%gcc@9.3.0: ~cuda+example~infiniband~lmsensors~nvml~powercap~rapl~rocm~rocm_smi~sde+shared~static_tools amdgpu_target=none
papyrus@1.0.1%gcc@9.3.0: ~ipo build_type=RelWithDebInfo
parallel-netcdf@1.12.2%gcc@9.3.0: ~burstbuffer+cxx+fortran+pic+shared
paraview@5.9.1%gcc@9.3.0: ~adios2~advanced_debug~cuda+development_files~examples~hdf5~ipo+kits+mpi+opengl2~osmesa~python~python3~qt+shared build_edition=canonical build_type=RelWithDebInfo cuda_arch=none
parmetis@4.0.3%gcc@9.3.0: ~gdb~int64~ipo+shared build_type=RelWithDebInfo patches=4f892531eb0a807eb1b82e683a416d3e35154a455274cf9b162fb02054d11a5b,50ed2081bc939269689789942067c58b3e522c269269a430d5d34c00edbc5870,704b84f7c7444d4372cb59cca6e1209df4ef3b033bc4ee3cf50f369bce972a9d
parmetis@4.0.3%gcc@9.3.0: ~gdb~int64~ipo+shared build_type=RelWithDebInfo patches=4f892531eb0a807eb1b82e683a416d3e35154a455274cf9b162fb02054d11a5b,50ed2081bc939269689789942067c58b3e522c269269a430d5d34c00edbc5870,704b84f7c7444d4372cb59cca6e1209df4ef3b033bc4ee3cf50f369bce972a9d
parsec@3.0.2012%gcc@9.3.0: ~cuda~debug_verbose~ipo~profile+shared build_type=RelWithDebInfo cuda_arch=none
parsec@3.0.2012%gcc@9.3.0: +cuda~debug_verbose~ipo~profile+shared build_type=RelWithDebInfo cuda_arch=70
parsec@3.0.2012%gcc@9.3.0: +cuda~debug_verbose~ipo~profile+shared build_type=RelWithDebInfo cuda_arch=80
pcre@8.44%gcc@9.3.0: ~jit+multibyte+utf
pcre@8.44%gcc@11.1.0: ~jit+multibyte+utf
pcre2@10.36%gcc@9.3.0: ~jit+multibyte
pdsh@2.31%gcc@9.3.0: +ssh+static_modules
pdt@3.25.1%gcc@9.3.0: ~pic
pegtl@3.2.0%gcc@9.3.0: ~ipo build_type=RelWithDebInfo
perl@5.34.0%gcc@9.3.0: +cpanm+shared+threads
perl@5.34.0%gcc@11.1.0: +cpanm+shared+threads
perl-data-dumper@2.173%gcc@11.1.0:
petsc@3.16.1%gcc@9.3.0: ~X~batch~cgns~complex~cuda~debug+double~exodusii~fftw~giflib+hdf5~hpddm~hwloc+hypre~int64~jpeg~knl~libpng~libyaml~memkind+metis~mkl-pardiso~mmg~moab~mpfr+mpi~mumps~openmp~p4est~parmmg~ptscotch~random123~rocm~saws~scalapack+shared~strumpack~suite-sparse+superlu-dist~tetgen~trilinos~valgrind amdgpu_target=none clanguage=C cuda_arch=none
petsc@3.16.1%gcc@9.3.0: ~X~batch~cgns~complex+cuda~debug+double~exodusii~fftw~giflib+hdf5~hpddm~hwloc+hypre~int64~jpeg~knl~libpng~libyaml~memkind+metis~mkl-pardiso~mmg~moab~mpfr+mpi~mumps~openmp~p4est~parmmg~ptscotch~random123~rocm~saws~scalapack+shared~strumpack~suite-sparse+superlu-dist~tetgen~trilinos~valgrind amdgpu_target=none clanguage=C cuda_arch=70
petsc@3.16.1%gcc@9.3.0: ~X~batch~cgns~complex+cuda~debug+double~exodusii~fftw~giflib+hdf5~hpddm~hwloc+hypre~int64~jpeg~knl~libpng~libyaml~memkind+metis~mkl-pardiso~mmg~moab~mpfr+mpi~mumps~openmp~p4est~parmmg~ptscotch~random123~rocm~saws~scalapack+shared~strumpack~suite-sparse+superlu-dist~tetgen~trilinos~valgrind amdgpu_target=none clanguage=C cuda_arch=80
petsc@3.16.1%gcc@9.3.0: ~X~batch~cgns~complex+cuda~debug+double~exodusii~fftw~giflib+hdf5~hpddm~hwloc+hypre~int64~jpeg~knl~libpng~libyaml~memkind+metis~mkl-pardiso~mmg~moab~mpfr+mpi~mumps~openmp~p4est~parmmg~ptscotch~random123~rocm~saws~scalapack+shared~strumpack~suite-sparse+superlu-dist~tetgen~trilinos~valgrind amdgpu_target=none clanguage=C cuda_arch=none
pflotran@3.0.2%gcc@9.3.0:
pfunit@3.3.3%gcc@9.3.0: ~docs~ipo+mpi~openmp+shared~use_comm_world build_type=RelWithDebInfo max_array_rank=5
phist@1.9.5%gcc@9.3.0: +fortran+host+int64~ipo+mpi+openmp~parmetis+scamac+shared~trilinos build_type=RelWithDebInfo kernel_lib=builtin outlev=2 patches=1fc9ac6c1cce2804d171dd12470e709cd1e3839be6d51f7e5c33427ba4b379ca
pkgconf@1.8.0%gcc@9.3.0:
pkgconf@1.8.0%gcc@11.1.0:
plasma@21.8.29%gcc@9.3.0: ~ipo~lua+shared build_type=RelWithDebInfo
precice@2.3.0%gcc@9.3.0: ~ipo+mpi+petsc~python+shared build_type=RelWithDebInfo
protobuf@3.18.0%gcc@9.3.0: +shared build_type=Release
pugixml@1.11.4%gcc@9.3.0: ~ipo+pic+shared build_type=RelWithDebInfo
pumi@2.2.6%gcc@9.3.0: ~fortran~int64~ipo~shared+simmodsuite_version_check~testing~zoltan build_type=RelWithDebInfo simmodsuite=none
py-alabaster@0.7.12%gcc@9.3.0:
py-alembic@1.5.5%gcc@9.3.0:
py-anyio@3.3.4%gcc@9.3.0:
py-apache-libcloud@1.2.1%gcc@9.3.0:
py-argon2-cffi@21.1.0%gcc@9.3.0:
py-async-generator@1.10%gcc@9.3.0:
py-attrs@21.2.0%gcc@9.3.0:
py-babel@2.7.0%gcc@9.3.0:
py-backcall@0.2.0%gcc@9.3.0:
py-bcrypt@3.2.0%gcc@9.3.0:
py-beniget@0.4.1%gcc@9.3.0:
py-bleach@4.1.0%gcc@9.3.0:
py-blinker@1.4%gcc@9.3.0:
py-bottleneck@1.3.2%gcc@9.3.0:
py-certifi@2021.10.8%gcc@9.3.0:
py-certipy@0.1.3%gcc@9.3.0:
py-cffi@1.15.0%gcc@9.3.0:
py-chardet@4.0.0%gcc@9.3.0:
py-cinemasci@1.3%gcc@9.3.0: +mpi
py-colorama@0.4.4%gcc@9.3.0:
py-cryptography@3.4.7%gcc@9.3.0: ~idna
py-cython@0.29.24%gcc@9.3.0:
py-cython@0.29.24%gcc@9.3.0:
py-debugpy@1.4.1%gcc@9.3.0:
py-decorator@5.1.0%gcc@9.3.0:
py-defusedxml@0.7.1%gcc@9.3.0:
py-dill@0.3.4%gcc@9.3.0:
py-docutils@0.17.1%gcc@9.3.0:
py-entrypoints@0.3%gcc@9.3.0:
py-fuzzywuzzy@0.18.0%gcc@9.3.0: ~speedup
py-gast@0.5.2%gcc@9.3.0:
py-gevent@1.5.0%gcc@9.3.0:
py-globus-sdk@3.0.2%gcc@9.3.0:
py-greenlet@1.1.0%gcc@9.3.0:
py-h5py@3.4.0%gcc@9.3.0: +mpi
py-idna@2.9%gcc@9.3.0:
py-imagesize@1.1.0%gcc@9.3.0:
py-iniconfig@1.1.1%gcc@9.3.0:
py-ipykernel@6.4.1%gcc@9.3.0:
py-ipython@7.28.0%gcc@9.3.0:
py-ipython-genutils@0.2.0%gcc@9.3.0:
py-ipywidgets@7.6.5%gcc@9.3.0:
py-jedi@0.18.0%gcc@9.3.0:
py-jinja2@3.0.1%gcc@9.3.0: ~i18n
py-json5@0.9.4%gcc@9.3.0:
py-jsonpatch@1.23%gcc@9.3.0:
py-jsonpointer@1.9%gcc@9.3.0:
py-jsonschema@2.6.0%gcc@9.3.0:
py-jsonschema@3.2.0%gcc@9.3.0:
py-jupyter-client@6.1.12%gcc@9.3.0:
py-jupyter-core@4.7.1%gcc@9.3.0:
py-jupyter-packaging@0.7.12%gcc@9.3.0:
py-jupyter-server@1.9.0%gcc@9.3.0:
py-jupyter-telemetry@0.1.0%gcc@9.3.0:
py-jupyterhub@1.4.1%gcc@9.3.0:
py-jupyterlab@3.0.14%gcc@9.3.0:
py-jupyterlab-pygments@0.1.2%gcc@9.3.0:
py-jupyterlab-server@2.6.0%gcc@9.3.0:
py-jupyterlab-widgets@1.0.2%gcc@9.3.0:
py-libensemble@0.8.0%gcc@9.3.0: ~deap+mpi~mpmath~nlopt~petsc4py~pyyaml~scipy~tasmanian
py-mako@1.1.4%gcc@9.3.0:
py-markupsafe@2.0.1%gcc@9.3.0:
py-matplotlib-inline@0.1.2%gcc@9.3.0:
py-mistune@0.8.4%gcc@9.3.0:
py-mpi4py@3.1.2%gcc@9.3.0:
py-msgpack@1.0.2%gcc@9.3.0:
py-mypy@0.910%gcc@9.3.0: ~python2
py-mypy-extensions@0.4.3%gcc@9.3.0:
py-nbclassic@0.3.1%gcc@9.3.0:
py-nbclient@0.5.0%gcc@9.3.0:
py-nbconvert@6.0.1%gcc@9.3.0: +serve
py-nbformat@5.1.3%gcc@9.3.0:
py-nest-asyncio@1.4.0%gcc@9.3.0:
py-netifaces@0.10.5%gcc@9.3.0:
py-notebook@6.1.4%gcc@9.3.0:
py-ntplib@0.4.0%gcc@9.3.0:
py-numexpr@2.7.3%gcc@9.3.0:
py-numpy@1.21.3%gcc@9.3.0: +blas+lapack patches=873745d7b547857fcfec9cae90b09c133b42a4f0c23b6c2d84cf37e2dd816604
py-numpy@1.21.3%gcc@9.3.0: +blas+lapack patches=873745d7b547857fcfec9cae90b09c133b42a4f0c23b6c2d84cf37e2dd816604
py-oauthlib@3.1.0%gcc@9.3.0: +extras
py-packaging@21.0%gcc@9.3.0:
py-pamela@1.0.0%gcc@9.3.0:
py-pandas@1.3.4%gcc@9.3.0:
py-pandocfilters@1.4.2%gcc@9.3.0:
py-paramiko@2.7.1%gcc@9.3.0: ~invoke
py-parse@1.18.0%gcc@9.3.0:
py-parsl@1.1.0%gcc@9.3.0:
py-parso@0.8.1%gcc@9.3.0:
py-pbr@5.4.3%gcc@9.3.0:
py-periodictable@1.5.0%gcc@9.3.0:
py-petsc4py@3.16.1%gcc@9.3.0: +mpi patches=d344e0e8b4c5ebb96961e1905bdd749162c52b151538238609f4dec77e2a6360
py-pexpect@4.7.0%gcc@9.3.0:
py-pickleshare@0.7.5%gcc@9.3.0:
py-picmistandard@0.0.14%gcc@9.3.0:
py-pika@0.13.0%gcc@9.3.0:
py-pillow@8.0.0%gcc@9.3.0: ~freetype~imagequant+jpeg~jpeg2000~lcms~tiff~webp~webpmux~xcb+zlib
py-pip@21.1.2%gcc@9.3.0:
py-pip@21.1.2%gcc@9.3.0:
py-pkgconfig@1.5.1%gcc@9.3.0:
py-pluggy@0.13.0%gcc@9.3.0:
py-ply@3.11%gcc@9.3.0:
py-prometheus-client@0.7.1%gcc@9.3.0: ~twisted
py-prompt-toolkit@3.0.17%gcc@9.3.0:
py-psutil@5.8.0%gcc@9.3.0:
py-ptyprocess@0.7.0%gcc@9.3.0:
py-py@1.9.0%gcc@9.3.0:
py-pybind11@2.6.2%gcc@9.3.0: ~ipo build_type=RelWithDebInfo
py-pycparser@2.20%gcc@9.3.0:
py-pyelftools@0.26%gcc@9.3.0:
py-pygments@2.10.0%gcc@9.3.0:
py-pyjwt@2.1.0%gcc@9.3.0: ~crypto
py-pyjwt@2.1.0%gcc@9.3.0: +crypto
py-pymongo@3.9.0%gcc@9.3.0:
py-pynacl@1.4.0%gcc@9.3.0:
py-pyopenssl@19.0.0%gcc@9.3.0:
py-pyparsing@2.4.7%gcc@9.3.0:
py-pyrsistent@0.15.7%gcc@9.3.0:
py-pytest@6.2.4%gcc@9.3.0:
py-pytest-runner@5.1%gcc@9.3.0:
py-python-dateutil@2.8.2%gcc@9.3.0:
py-python-editor@1.0.4%gcc@9.3.0:
py-python-json-logger@0.1.11%gcc@9.3.0:
py-pythran@0.9.12%gcc@9.3.0:  patches=b427dcaaa74ed4dfa39c58819e446e86055b65648826a101c08634d3c0be1f95
py-pytz@2021.1%gcc@9.3.0:
py-pyyaml@5.3.1%gcc@9.3.0: +libyaml
py-pyzmq@17.1.2%gcc@9.3.0:
py-pyzmq@18.1.0%gcc@9.3.0:
py-radical-entk@1.8.0%gcc@9.3.0:
py-radical-gtod@1.6.7%gcc@9.3.0:
py-radical-pilot@1.8.0%gcc@9.3.0:
py-radical-saga@1.8.0%gcc@9.3.0:
py-radical-utils@1.8.0%gcc@9.3.0:
py-regex@2020.11.13%gcc@9.3.0:
py-requests@2.25.1%gcc@9.3.0: ~socks
py-requests-unixsocket@0.2.0%gcc@9.3.0:
py-ruamel-yaml@0.17.16%gcc@9.3.0:
py-ruamel-yaml-clib@0.2.0%gcc@9.3.0:
py-scipy@1.7.1%gcc@9.3.0:
py-semantic-version@2.8.2%gcc@9.3.0:
py-send2trash@1.5.0%gcc@9.3.0:
py-setproctitle@1.1.10%gcc@9.3.0:
py-setuptools@57.4.0%gcc@9.3.0:
py-setuptools@58.2.0%gcc@9.3.0:
py-setuptools-rust@0.12.1%gcc@9.3.0:
py-setuptools-scm@6.3.2%gcc@9.3.0: +toml
py-setuptools-scm@6.3.2%gcc@9.3.0: +toml
py-six@1.16.0%gcc@9.3.0:
py-sniffio@1.2.0%gcc@9.3.0:
py-snowballstemmer@2.0.0%gcc@9.3.0:
py-sphinx@4.1.2%gcc@9.3.0:
py-sphinx-autodoc-typehints@1.12.0%gcc@9.3.0:
py-sphinx-rtd-theme@0.5.1%gcc@9.3.0:
py-sphinxcontrib-applehelp@1.0.1%gcc@9.3.0:
py-sphinxcontrib-devhelp@1.0.1%gcc@9.3.0:
py-sphinxcontrib-htmlhelp@2.0.0%gcc@9.3.0:
py-sphinxcontrib-jsmath@1.0.1%gcc@9.3.0:
py-sphinxcontrib-qthelp@1.0.2%gcc@9.3.0:
py-sphinxcontrib-serializinghtml@1.1.5%gcc@9.3.0:
py-sphobjinv@2.1%gcc@9.3.0:
py-sqlalchemy@1.4.20%gcc@9.3.0:  backend=none
py-tblib@1.6.0%gcc@9.3.0:
py-terminado@0.8.3%gcc@9.3.0:
py-testpath@0.4.2%gcc@9.3.0:
py-toml@0.10.2%gcc@9.3.0:
py-tomli@1.2.1%gcc@9.3.0:
py-tornado@5.1.1%gcc@9.3.0:
py-tornado@6.1%gcc@9.3.0:
py-traitlets@5.0.4%gcc@9.3.0:
py-typed-ast@1.4.3%gcc@9.3.0:
py-typeguard@2.12.1%gcc@9.3.0:
py-typing-extensions@3.10.0.2%gcc@9.3.0:
py-urllib3@1.26.6%gcc@9.3.0: ~brotli~secure~socks
py-vcversioner@2.16.0.0%gcc@9.3.0:
py-warlock@1.3.3%gcc@9.3.0:
py-warpx@21.11%gcc@9.3.0: +mpi
py-warpx@21.11%gcc@9.3.0: +mpi
py-warpx@21.11%gcc@9.3.0: +mpi
py-wcwidth@0.1.7%gcc@9.3.0:
py-webencodings@0.5.1%gcc@9.3.0:
py-websocket-client@0.57.0%gcc@9.3.0:
py-wheel@0.36.2%gcc@9.3.0:
py-widgetsnbextension@3.5.1%gcc@9.3.0:
python@3.8.12%gcc@9.3.0: +bz2+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4+uuid+zlib patches=0d98e93189bc278fbc37a50ed7f183bd8aaf249a8e1670a465f0db6bb4f8cf87,4c2457325f2b608b1b6a2c63087df8c26e07db3e3d493caf36a56f0ecf6fb768,f2fd060afc4b4618fe8104c4c5d771f36dc55b1db5a4623785a4ea707ec72fb4
python@3.8.12%gcc@11.1.0: +bz2+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4+uuid+zlib patches=0d98e93189bc278fbc37a50ed7f183bd8aaf249a8e1670a465f0db6bb4f8cf87,4c2457325f2b608b1b6a2c63087df8c26e07db3e3d493caf36a56f0ecf6fb768,f2fd060afc4b4618fe8104c4c5d771f36dc55b1db5a4623785a4ea707ec72fb4
qthreads@1.16%gcc@9.3.0: +hwloc~spawn_cache+static scheduler=distrib stack_size=4096
raja@0.13.0%gcc@9.3.0: ~cuda+examples+exercises~ipo+openmp~rocm+shared~tests amdgpu_target=none build_type=RelWithDebInfo cuda_arch=none patches=57dd531a50ac791b4bb214d34a4bf3fca1349354927c72915b7ccd20524701a9
raja@0.14.0%gcc@9.3.0: ~cuda+examples+exercises~ipo+openmp~rocm+shared~tests amdgpu_target=none build_type=RelWithDebInfo cuda_arch=none
raja@0.14.0%gcc@9.3.0: +cuda+examples+exercises~ipo+openmp~rocm+shared~tests amdgpu_target=none build_type=RelWithDebInfo cuda_arch=70
raja@0.14.0%gcc@9.3.0: +cuda+examples+exercises~ipo+openmp~rocm+shared~tests amdgpu_target=none build_type=RelWithDebInfo cuda_arch=80
rankstr@0.1.0%gcc@9.3.0: ~ipo+shared build_type=RelWithDebInfo
readline@8.1%gcc@9.3.0:
readline@8.1%gcc@11.1.0:
redset@0.1.0%gcc@9.3.0: ~ipo+shared build_type=RelWithDebInfo
rempi@1.1.0%gcc@9.3.0:
rocblas@4.3.1%gcc@9.3.0: ~ipo build_type=Release tensile_architecture=all
rocfft@4.3.1%gcc@9.3.0: ~ipo amdgpu_target=gfx701 amdgpu_target_sram_ecc=none build_type=Release
rocrand@4.3.1%gcc@9.3.0: ~ipo build_type=Release
rocsolver@4.3.1%gcc@9.3.0: ~ipo+optimal amdgpu_target=gfx906:xnack- build_type=Release
rust@1.51.0%gcc@9.3.0: +analysis+clippy~rls+rustfmt+src extra_targets=none
scr@3.0rc2%gcc@9.3.0: +bbapi~bbapi_fallback~dw+examples+fortran~ipo+libyogrt+pdsh+shared+tests build_type=RelWithDebInfo cache_base=/dev/shm cntl_base=/dev/shm copy_config=none file_lock=FLOCK resource_manager=SLURM scr_config=scr.conf
shuffile@0.1.0%gcc@9.3.0: ~ipo+shared build_type=RelWithDebInfo
slate@2021.05.02%gcc@9.3.0: ~cuda~ipo+mpi+openmp+rocm+shared amdgpu_target=gfx908 build_type=RelWithDebInfo cuda_arch=none
slate@2021.05.02%gcc@9.3.0: ~cuda~ipo+mpi+openmp~rocm+shared amdgpu_target=none build_type=RelWithDebInfo cuda_arch=none
slate@2021.05.02%gcc@9.3.0: +cuda~ipo+mpi+openmp~rocm+shared amdgpu_target=none build_type=RelWithDebInfo cuda_arch=70
slate@2021.05.02%gcc@9.3.0: +cuda~ipo+mpi+openmp~rocm+shared amdgpu_target=none build_type=RelWithDebInfo cuda_arch=80
slepc@3.16.0%gcc@9.3.0: +arpack~blopex~cuda~rocm amdgpu_target=none cuda_arch=none
slepc@3.16.0%gcc@9.3.0: +arpack~blopex+cuda~rocm amdgpu_target=none cuda_arch=70
slepc@3.16.0%gcc@9.3.0: +arpack~blopex+cuda~rocm amdgpu_target=none cuda_arch=80
slurm@21-08-1-1%gcc@9.3.0: ~gtk~hdf5~hwloc~mariadb~pmix+readline~restd sysconfdir=PREFIX/etc
snappy@1.1.8%gcc@9.3.0: ~ipo+pic+shared build_type=RelWithDebInfo patches=c9cfecb1f7a623418590cf4e00ae7d308d1c3faeb15046c2e5090e38221da7cd
spath@0.1.0%gcc@9.3.0: ~ipo~mpi+shared build_type=RelWithDebInfo
spath@0.1.0%gcc@9.3.0: ~ipo+mpi+shared build_type=RelWithDebInfo
sqlite@3.36.0%gcc@9.3.0: +column_metadata+fts~functions~rtree
sqlite@3.36.0%gcc@11.1.0: +column_metadata+fts~functions~rtree
stc@0.9.0%gcc@9.3.0:
strumpack@6.1.0%gcc@9.3.0: +butterflypack+c_interface~count_flops~cuda~ipo+mpi+openmp+parmetis+rocm~scotch+shared~slate~task_timers+zfp amdgpu_target=gfx908 build_type=RelWithDebInfo cuda_arch=none
strumpack@6.1.0%gcc@9.3.0: +butterflypack+c_interface~count_flops~cuda~ipo+mpi+openmp+parmetis~rocm~scotch+shared~slate~task_timers+zfp amdgpu_target=none build_type=RelWithDebInfo cuda_arch=none
strumpack@6.1.0%gcc@9.3.0: +butterflypack+c_interface~count_flops+cuda~ipo+mpi+openmp+parmetis~rocm~scotch+shared~slate~task_timers+zfp amdgpu_target=none build_type=RelWithDebInfo cuda_arch=70
strumpack@6.1.0%gcc@9.3.0: +butterflypack+c_interface~count_flops+cuda~ipo+mpi+openmp+parmetis~rocm~scotch+shared~slate~task_timers+zfp amdgpu_target=none build_type=RelWithDebInfo cuda_arch=80
suite-sparse@5.10.1%gcc@9.3.0: ~cuda~openmp+pic~tbb
sundials@5.8.0%gcc@9.3.0: +ARKODE+CVODE+CVODES+IDA+IDAS+KINSOL~cuda+examples+examples-install~f2003~fcmix+generic-math~hypre~int64~ipo~klu~lapack~monitoring+mpi~openmp~petsc~pthread~raja~rocm+shared+static~superlu-dist~superlu-mt~sycl~trilinos amdgpu_target=none build_type=RelWithDebInfo cuda_arch=none precision=double
sundials@5.8.0%gcc@9.3.0: +ARKODE+CVODE+CVODES+IDA+IDAS+KINSOL+cuda+examples+examples-install~f2003~fcmix+generic-math~hypre~int64~ipo~klu~lapack~monitoring+mpi~openmp~petsc~pthread~raja~rocm+shared+static~superlu-dist~superlu-mt~sycl~trilinos amdgpu_target=none build_type=RelWithDebInfo cuda_arch=70 precision=double
sundials@5.8.0%gcc@9.3.0: +ARKODE+CVODE+CVODES+IDA+IDAS+KINSOL+cuda+examples+examples-install~f2003~fcmix+generic-math~hypre~int64~ipo~klu~lapack~monitoring+mpi~openmp~petsc~pthread~raja~rocm+shared+static~superlu-dist~superlu-mt~sycl~trilinos amdgpu_target=none build_type=RelWithDebInfo cuda_arch=80 precision=double
superlu@5.3.0%gcc@9.3.0: ~ipo+pic build_type=RelWithDebInfo
superlu-dist@7.1.1%gcc@9.3.0: ~cuda~int64~ipo~openmp+shared build_type=RelWithDebInfo cuda_arch=none
superlu-dist@7.1.1%gcc@9.3.0: ~cuda~int64~ipo~openmp+shared build_type=RelWithDebInfo cuda_arch=none
superlu-dist@7.1.1%gcc@9.3.0: +cuda~int64~ipo~openmp+shared build_type=RelWithDebInfo cuda_arch=70
superlu-dist@7.1.1%gcc@9.3.0: +cuda~int64~ipo~openmp+shared build_type=RelWithDebInfo cuda_arch=80
swig@4.0.2%gcc@9.3.0:
swig@4.0.2%gcc@11.1.0:
swig@4.0.2-fortran%gcc@9.3.0:
symengine@0.7.0%gcc@9.3.0: ~boostmp~flint~ipo~llvm+mpc+mpfr~openmp~piranha+shared+thread_safe build_type=Release
sz@1.4.12.3%gcc@9.3.0: ~fortran~hdf5~ipo~netcdf~pastri~python~random_access+shared~stats~time_compression build_type=RelWithDebInfo
sz@2.1.12%gcc@9.3.0: ~fortran~hdf5~ipo~netcdf~pastri~python~random_access+shared~stats~time_compression build_type=RelWithDebInfo
tar@1.34%gcc@9.3.0:
tar@1.34%gcc@11.1.0:
tasmanian@7.7%gcc@9.3.0: ~blas~cuda~fortran~ipo~magma+mpi+openmp~python~rocm~xsdkflags amdgpu_target=none build_type=Release cuda_arch=none
tasmanian@7.7%gcc@9.3.0: ~blas+cuda~fortran~ipo~magma+mpi+openmp~python~rocm~xsdkflags amdgpu_target=none build_type=Release cuda_arch=70
tasmanian@7.7%gcc@9.3.0: ~blas+cuda~fortran~ipo~magma+mpi+openmp~python~rocm~xsdkflags amdgpu_target=none build_type=Release cuda_arch=80
tau@2.31%gcc@9.3.0: ~adios2+binutils~comm~craycnl~cuda+elf+fortran~gasnet+io~level_zero+libdwarf+libunwind~likwid+mpi~ompt~opari~opencl~openmp+otf2+papi+pdt~phase~ppc64le~profileparam+pthreads+python~rocm~rocprofiler~roctracer~scorep~shmem~sqlite~x86_64
tcl@8.6.11%gcc@9.3.0:
texinfo@6.5%gcc@9.3.0:  patches=12f6edb0c6b270b8c8dba2ce17998c580db01182d871ee32b7b6e4129bd1d23a,1732115f651cff98989cb0215d8f64da5e0f7911ebf0c13b064920f088f2ffe1
trilinos@13.0.1%gcc@9.3.0: ~adios2+amesos+amesos2+anasazi+aztec~basker+belos+boost~chaco~complex~cuda~cuda_rdc~debug~dtk+epetra+epetraext~epetraextbtf~epetraextexperimental~epetraextgraphreorderings~exodus+explicit_template_instantiation~float+fortran~gtest~hdf5~hypre+ifpack+ifpack2+intrepid+intrepid2~ipo+isorropia+kokkos~mesquite+minitensor+ml+mpi+muelu~mumps+nox~openmp+phalanx+piro~python+rol+rythmos+sacado~scorec+shards+shared+shylu+stk+stokhos+stratimikos~strumpack~suite-sparse~superlu+superlu-dist+teko+tempus+tpetra+trilinoscouplings~wrapper~x11+zoltan+zoltan2 build_type=RelWithDebInfo cuda_arch=none cxxstd=14 gotype=long_long
trilinos@13.2.0%gcc@9.3.0: ~adios2+amesos+amesos2+anasazi+aztec~basker+belos+boost~chaco~complex+cuda~cuda_rdc~debug~dtk+epetra+epetraext~epetraextbtf~epetraextexperimental~epetraextgraphreorderings~exodus+explicit_template_instantiation~float+fortran~gtest~hdf5~hypre+ifpack+ifpack2+intrepid+intrepid2~ipo+isorropia+kokkos~mesquite+minitensor+ml+mpi+muelu~mumps+nox~openmp+phalanx+piro~python+rol+rythmos+sacado~scorec+shards+shared+shylu+stk+stokhos+stratimikos~strumpack~suite-sparse~superlu+superlu-dist+teko+tempus+tpetra+trilinoscouplings+wrapper~x11+zoltan+zoltan2 build_type=RelWithDebInfo cuda_arch=70 cxxstd=14 gotype=long_long
trilinos@13.2.0%gcc@9.3.0: ~adios2+amesos+amesos2+anasazi+aztec~basker+belos+boost~chaco~complex+cuda~cuda_rdc~debug~dtk+epetra+epetraext~epetraextbtf~epetraextexperimental~epetraextgraphreorderings~exodus+explicit_template_instantiation~float+fortran~gtest~hdf5~hypre+ifpack+ifpack2+intrepid+intrepid2~ipo+isorropia+kokkos~mesquite+minitensor+ml+mpi+muelu~mumps+nox~openmp+phalanx+piro~python+rol+rythmos+sacado~scorec+shards+shared+shylu+stk+stokhos+stratimikos~strumpack~suite-sparse~superlu+superlu-dist+teko+tempus+tpetra+trilinoscouplings+wrapper~x11+zoltan+zoltan2 build_type=RelWithDebInfo cuda_arch=80 cxxstd=14 gotype=long_long
turbine@1.3.0%gcc@9.3.0: ~hdf5~python~r
umap@2.1.0%gcc@9.3.0: ~ipo~logging~tests build_type=RelWithDebInfo
umpire@5.0.1%gcc@9.3.0: +c~cuda~deviceconst+examples~fortran~ipo~numa+openmp~rocm+shared amdgpu_target=none build_type=RelWithDebInfo cuda_arch=none patches=135bbc7d2f371531f432672b115ac0a407968aabfffc5b8a941db9b493dbf81f,f5c691752e4833a936bce224bbe0fe884d3afa84c5e5a4a481f59a12840159c9 tests=none
umpire@6.0.0%gcc@9.3.0: +c~cuda~deviceconst+examples~fortran~ipo~numa~openmp~rocm+shared amdgpu_target=none build_type=RelWithDebInfo cuda_arch=none tests=none
umpire@6.0.0%gcc@9.3.0: +c+cuda~deviceconst+examples~fortran~ipo~numa~openmp~rocm~shared amdgpu_target=none build_type=RelWithDebInfo cuda_arch=70 tests=none
umpire@6.0.0%gcc@9.3.0: +c+cuda~deviceconst+examples~fortran~ipo~numa~openmp~rocm~shared amdgpu_target=none build_type=RelWithDebInfo cuda_arch=80 tests=none
unifyfs@0.9.1%gcc@9.3.0: +auto-mount~fortran~pmi~pmix+spath patches=8a9c20c857c728637d994c097505cdce780f4b8e61535d221117864f75795313,f6d61ac7cee0d690f50a9a99ff9d2242f494c3baf7374baaf6779340728566b1
unzip@6.0%gcc@9.3.0:
upcxx@2021.9.0%gcc@9.3.0: ~cuda~gasnet+mpi cross=none
upcxx@2021.9.0%gcc@9.3.0: +cuda~gasnet+mpi cross=none
util-linux-uuid@2.36.2%gcc@9.3.0:
util-linux-uuid@2.36.2%gcc@11.1.0:
util-macros@1.19.3%gcc@9.3.0:
util-macros@1.19.3%gcc@11.1.0:
veloc@1.5%gcc@9.3.0: ~ipo build_type=RelWithDebInfo
vtk-h@0.7.1%gcc@9.3.0: ~contourtree~cuda~logging+mpi+openmp+serial+shared cuda_arch=none
vtk-m@1.6.0%gcc@9.3.0: ~64bitids+ascent_types~cuda+doubleprecision~hip~ipo~logging+mpi+openmp+rendering~shared~tbb~virtuals amdgpu_target=none build_type=Release cuda_arch=none
vtk-m@1.6.0%gcc@9.3.0: ~64bitids+ascent_types+cuda+doubleprecision~hip~ipo~logging+mpi+openmp+rendering~shared~tbb~virtuals amdgpu_target=none build_type=Release cuda_arch=70
vtk-m@1.6.0%gcc@9.3.0: ~64bitids+ascent_types+cuda+doubleprecision~hip~ipo~logging+mpi+openmp+rendering~shared~tbb~virtuals amdgpu_target=none build_type=Release cuda_arch=80
warpx@21.11%gcc@9.3.0: +app~ascent~eb~ipo+lib+mpi+mpithreadmultiple+openpmd+psatd+qed~qedtablegen+shared+tprof build_type=RelWithDebInfo compute=omp dims=2 precision=double
warpx@21.11%gcc@9.3.0: +app~ascent~eb~ipo+lib+mpi+mpithreadmultiple+openpmd+psatd+qed~qedtablegen+shared+tprof build_type=RelWithDebInfo compute=omp dims=3 precision=double
warpx@21.11%gcc@9.3.0: +app~ascent~eb~ipo+lib+mpi+mpithreadmultiple+openpmd+psatd+qed~qedtablegen+shared+tprof build_type=RelWithDebInfo compute=omp dims=rz precision=double
xcb-proto@1.14.1%gcc@9.3.0:
xerces-c@3.2.3%gcc@9.3.0:  cxxstd=default netaccessor=curl transcoder=iconv
xextproto@7.3.0%gcc@9.3.0:
xproto@7.0.31%gcc@9.3.0:
xtrans@1.3.5%gcc@9.3.0:
xz@5.2.5%gcc@9.3.0: +pic libs=shared,static
xz@5.2.5%gcc@11.1.0: +pic libs=shared,static
yaml-cpp@0.7.0%gcc@9.3.0: ~ipo+pic+shared~tests build_type=RelWithDebInfo
z3@4.8.9%gcc@11.1.0: ~gmp~ipo~python build_type=RelWithDebInfo
zfp@0.5.5%gcc@9.3.0: ~aligned~c~cuda~fasthash~fortran~ipo~openmp~profile~python+shared~strided~twoway bsws=64 build_type=RelWithDebInfo cuda_arch=none
zfp@0.5.5%gcc@9.3.0: ~aligned~c+cuda~fasthash~fortran~ipo~openmp~profile~python+shared~strided~twoway bsws=64 build_type=RelWithDebInfo cuda_arch=70
zfp@0.5.5%gcc@9.3.0: ~aligned~c+cuda~fasthash~fortran~ipo~openmp~profile~python+shared~strided~twoway bsws=64 build_type=RelWithDebInfo cuda_arch=80
zlib@1.2.11%gcc@9.3.0: +optimize+pic+shared
zlib@1.2.11%gcc@11.1.0: +optimize+pic+shared
zsh@5.8%gcc@9.3.0: +skip-tcsetpgrp-test
zstd@1.5.0%gcc@9.3.0: ~programs

Standard Deployment

Note

This stack was built using the Cray Programming Environment 21.12 and is subject to change upon with CPE upgrade.

This stack can be accessible from one of two modules (e4s/21.11-lmod and e4s/21.11-tcl). The e4s/21.11-lmod will expose the e4s stack using the Lmod hierarchy while e4s/21.11-tcl will show the stack using the flat hierarchy in tcl format. Both modulefile point to the same spack configuration however the only difference is the MODULEPATH set for each stack. Shown below are the modulefiles available for e4s/21.11

elvis@perlmutter> ml -t av e4s/21.11
/global/common/software/nersc/shasta-21.11/extra_modulefiles:
e4s/21.11-lmod
e4s/21.11-tcl

In order to get started you need to load one of the following modules and you will be in a spack environment named e4s

elvis@perlmutter> ml e4s/21.11-tcl
elvis@perlmutter> spack env st
==> In environment e4s

Spack Configuration

Our deployment spack configuration is as follows

Production spack configuration - spack.yaml
# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
  view: false
  config:
    build_stage: $CFS/m3503/spack-stage
    misc_cache: $CFS/m3503/misc_cache
    install_tree:
      concretizer: clingo
      root: /global/common/software/spackecp/perlmutter/e4s-21.11/software
    module_roots:
      tcl: /global/common/software/spackecp/perlmutter/e4s-21.11/modules/tcl
      lmod: /global/common/software/spackecp/perlmutter/e4s-21.11/modules/lmod
  modules:
    default:
      enable:
      - lmod
      lmod:
        blacklist_implicits: true
        hash_length: 0
        core_compilers:
        - gcc@7.5.0
        hierarchy:
        - mpi
        all:
          autoload: direct
          environment:
            set:
              '{name}_ROOT': '{prefix}'
          suffixes:
            ^cuda: cuda
        projections:
          all: '{name}/{version}'
    tcl:
      hash_length: 0
      naming_scheme: '{name}/{version}-{compiler.name}-{compiler.version}'
      blacklist_implicits: true
      all:
        autoload: direct
        conflict:
        - '{name}'
        environment:
          set:
            '{name}_ROOT': '{prefix}'
        suffixes:
          ^cray-mpich: mpi
          ^cuda: cuda
      blacklist:
      - gcc
      - nvhpc
      - cray-mpich
  compilers:
  - compiler:
      spec: gcc@7.5.0
      paths:
        cc: /usr/bin/gcc
        cxx: /usr/bin/g++
        f77: /usr/bin/gfortran
        fc: /usr/bin/gfortran
      flags: {}
      operating_system: sles15
      target: any
      modules: []
  - compiler:
      spec: gcc@9.3.0
      paths:
        cc: /opt/cray/pe/craype/default/bin/cc
        cxx: /opt/cray/pe/craype/default/bin/CC
        f77: /opt/cray/pe/craype/default/bin/ftn
        fc: /opt/cray/pe/craype/default/bin/ftn
      flags: {}
      operating_system: sles15
      target: any
      modules:
      - PrgEnv-gnu
      - gcc/9.3.0
      - craype-x86-milan
      - libfabric
      #environment:
      #  append_path:
      #    LIBRARY_PATH: /opt/nvidia/hpc_sdk/Linux_x86_64/21.9/math_libs/lib64
      #    LD_LIBRARY_PATH: /opt/nvidia/hpc_sdk/Linux_x86_64/21.9/math_libs/lib64
      #    CPATH: /opt/nvidia/hpc_sdk/Linux_x86_64/21.9/math_libs/include
  - compiler:
      spec: nvhpc@21.9
      paths:
        cc: /opt/cray/pe/craype/default/bin/cc
        cxx: /opt/cray/pe/craype/default/bin/CC
        f77: /opt/cray/pe/craype/default/bin/ftn
        fc: /opt/cray/pe/craype/default/bin/ftn
      flags: {}
      operating_system: sles15
      target: any
      modules:
      - PrgEnv-nvidia
      - nvidia/21.9
      - craype-x86-milan
      - libfabric
  definitions:
  - gcc_compilers: ['%gcc@9.3.0']
  - nvhpc_compilers: ['%nvhpc@21.9']
  - compilers: [gcc@9.3.0 %gcc@7.5.0, nvhpc@21.9 %gcc@7.5.0]
  - mpis: [cray-mpich@8.1.12 %gcc@9.3.0, cray-mpich@8.1.12 %nvhpc@21.9]
  - gcc_specs:
    - adios2@2.7.1
    - amrex@21.11 +fortran +hypre +openmp +petsc +shared
    - gasnet@2021.9.0
    - hdf5@1.12.1
    - hypre@2.23.0 +openmp +superlu-dist
    - kokkos@3.4.01 +openmp
    - kokkos-kernels@3.4.01 +openmp
    - papi@6.0.0.1
    - petsc@3.16.1 +openmp +strumpack
    - raja@0.14.0
    - slepc@3.16.0
    - strumpack@6.1.0 ~slate
    - sundials@5.8.0 +openmp +hypre
    - superlu@5.3.0
    - superlu-dist@7.1.1 +openmp
    - trilinos@13.0.1 +amesos +amesos2 +anasazi +aztec +belos +boost +epetra +epetraext
      +ifpack +ifpack2 +intrepid +intrepid2 +isorropia +kokkos +ml +minitensor +muelu
      +nox +piro +phalanx +rol +rythmos +sacado +stk +shards +shylu +stokhos +stratimikos
      +teko +tempus +tpetra +trilinoscouplings +zoltan +zoltan2 +superlu-dist gotype=long_long
    - upcxx@2021.9.0 +gasnet +mpi
    #- dyninst@11.0.1 failed to install elfutils
  - cuda_specs:
    - hypre@2.23.0+cuda cuda_arch=80
    - petsc@3.16.1 +cuda cuda_arch=80
    - kokkos-kernels@3.4.01 +openmp +cuda cuda_arch=80 ^kokkos +openmp +wrapper +cuda
      cuda_arch=80
    - kokkos@3.4.01 +openmp +wrapper +cuda cuda_arch=80
    - raja@0.14.0+cuda cuda_arch=80
    - slepc@3.16.0 +cuda cuda_arch=80 ^petsc@3.16.1 +cuda cuda_arch=80
    - umpire@6.0.0 ~shared +cuda cuda_arch=80
    - upcxx@2021.9.0 +cuda
    - zfp@0.5.5 +cuda cuda_arch=80
    # following specs failed to find cublas 
    #- amrex@21.11 +cuda cuda_arch=80
    #- magma@2.6.1+cuda cuda_arch=80
    #- mfem@4.3.0+cuda cuda_arch=80
    #- strumpack@6.1.0 ~slate +cuda cuda_arch=80
    #- slate@2021.05.02 +cuda cuda_arch=80 
    #- superlu-dist@7.1.1 +openmp +cuda cuda_arch=80
    #- sundials@5.8.0 +openmp +cuda cuda_arch=80
  - nvhpc_specs:
    - hypre@2.23.0 +cuda cuda_arch=80
    - umpire@6.0.0 ~shared +cuda cuda_arch=80
    - upcxx@2021.9.0 +cuda
    - zfp@0.5.5 +cuda cuda_arch=80
    # unable to install for nvhpc compiler  
    #- amrex@21.11 +cuda cuda_arch=80
    #- petsc@3.16.1 +cuda cuda_arch=80
    #- kokkos-kernels@3.4.01 +openmp +wrapper +cuda cuda_arch=80 ^kokkos +openmp +wrapper +cuda cuda_arch=80
    #- kokkos@3.4.01 +openmp +wrapper +cuda cuda_arch=80
    #- raja@0.14.0+cuda cuda_arch=80
  - nersc_specs:
    - plumed@2.6.3
    - wannier90@3.1.0
  specs:
  - matrix:
    - [$gcc_specs]
    - [$gcc_compilers]
  - matrix:
    - [$cuda_specs]
    - [$gcc_compilers]
  - matrix:
    - [$nvhpc_specs]
    - [$nvhpc_compilers]
  - $compilers
  - $mpis
  - $nersc_specs
  packages:
    all:
      compiler: [gcc@9.3.0, nvhpc@21.9]
      providers:
        blas: [cray-libsci]
        mpi: [cray-mpich]
    amrex:
      variants: +fortran +hypre +openmp +petsc +shared
    bzip2:
      version: [1.0.6]
      externals:
      - spec: bzip2@1.0.6
        prefix: /usr
    cray-libsci:
      buildable: false
      externals:
      - spec: cray-libsci@21.08.1.2
        modules:
        - cray-libsci/21.08.1.2
    cray-mpich:
      buildable: false
      externals:
      - spec: cray-mpich@8.1.12 %gcc@9.3.0
        prefix: /opt/cray/pe/mpich/8.1.12/ofi/gnu/9.1
        modules:
        - cray-mpich/8.1.12
        - cudatoolkit/21.9_11.4
      - spec: cray-mpich@8.1.12 %nvhpc@21.9
        prefix: /opt/cray/pe/mpich/8.1.12/ofi/nvidia/20.7
        modules:
        - cray-mpich/8.1.12
        - cudatoolkit/21.9_11.4
    cuda:
      buildable: false
      version: [11.4.0]
      externals:
      - spec: cuda@11.4.0
        prefix: /opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4
        modules:
        - cudatoolkit/21.9_11.4

    diffutils:
      version: [3.6]
      externals:
      - spec: diffutils@3.6
        prefix: /usr
    findutils:
      version: [4.6.0]
      externals:
      - spec: findutils@4.6.0
        prefix: /usr
    hdf5:
      variants: +fortran +hl +shared api=v18
      version: [1.12.1]
    hypre:
      variants: +openmp +superlu-dist
    gcc:
      buildable: false
      externals:
      - spec: gcc@9.3.0 %gcc@7.5.0
        modules:
        - PrgEnv-gnu
        - gcc/9.3.0
        - craype-x86-milan
        - libfabric
    git:
      version: [2.26.2]
      buildable: false
      externals:
      - spec: git@2.26.2
        prefix: /usr
    gettext:
      version: [0.19.8.1]
      buildable: false
      externals:
      - spec: gettext@0.19.8.1
        prefix: /usr
    libfabric:
      buildable: false
      variants: fabrics=sockets,tcp,udp,rxm
      externals:
      - spec: libfabric@1.11.0.4.75
        prefix: /opt/cray/libfabric/1.11.0.4.75
        modules:
        - libfabric/1.11.0.4.75
    libunwind:
      variants: +pic +xz
    mercury:
      variants: ~bmi
    mesa:
      variants: ~llvm
    mesa18:
      variants: ~llvm
    m4:
      buildable: false
      externals:
      - spec: m4@1.4.18
        prefix: /usr
    mpich:
      variants: ~wrapperrpath
    ncurses:
      variants: +termlib
      externals:
      - spec: ncurses@6.1
        prefix: /usr
    nvhpc:
      buildable: false
      externals:
      - spec: nvhpc@21.9 %gcc@7.5.0
        modules:
        - PrgEnv-nvidia
        - nvidia/21.9
        - craype-x86-milan
        - libfabric
    openssl:
      version: [1.1.0i]
      buildable: false
      externals:
      - spec: openssl@1.1.0i
        prefix: /usr
    openssh:
      version: [7.9p1]
      buildable: false
      externals:
      - spec: openssh@7.9p1
        prefix: /usr
    petsc:
      variants: +openmp +strumpack
    pdsh:
      buildable: false
      externals:
      - spec: pdsh@2.34
        prefix: /usr
    readline:
      version: [7.0]
      buildable: false
      externals:
      - spec: readline@7.0
        prefix: /usr
    slurm:
      buildable: false
      version: [20-11-8-1]
      externals:
      - spec: slurm@20-11-8-1
        prefix: /usr
    superlu-dist:
      variants: +openmp
    strumpack:
      variants: ~slate
    tar:
      version: [1.3]
      buildable: false
      externals:
      - spec: tar@1.30
        prefix: /usr
    trilinos:
      variants: +amesos +amesos2 +anasazi +aztec +belos +boost +epetra +epetraext
        +ifpack +ifpack2 +intrepid +intrepid2 +isorropia +kokkos +ml +minitensor +muelu
        +nox +piro +phalanx +rol +rythmos +sacado +stk +shards +shylu +stokhos +stratimikos
        +teko +tempus +tpetra +trilinoscouplings +zoltan +zoltan2 +superlu-dist gotype=long_long
    unzip:
      version: [6.0]
      buildable: false
      externals:
      - spec: unzip@6.0
        prefix: /usr
    util-linux-uuid:
      version: [2.33.1]
      buildable: false
      externals:
      - spec: util-linux-uuid@2.33.1
        prefix: /usr
    xz:
      version: [5.2.3]
      buildable: false
      externals:
      - spec: xz@5.2.3
        prefix: /usr
    zsh:
      version: [5.6]
      buildable: false
      externals:
      - spec: zsh@5.6
        prefix: /usr

Overview

Shown below is an output of spack find showing only explicit specs that were installed along with hash and variants for each spec. This deployment utilize gcc@9.3.0 and nvhpc@21.9 compiler for building for building the software stack.

Output of spack find
elvis@perlmutter> spack find -x --format "{name}@{version}%{compiler.name}%{compiler.version} /{hash:7}:{variants}"
adios2@2.7.1%gcc%9.3.0 /id67zxy:+blosc+bzip2~dataman~dataspaces~endian_reverse+fortran~hdf5~ipo+mpi+pic+png~python+shared+ssc+sst+sz+zfp build_type=Release patches=8d301e8232baf4049b547f22bd73774309662017a62dac36360d2965907062bf
amrex@21.11%gcc%9.3.0 /im7eqou:~amrdata~cuda~eb+fortran~hdf5+hypre~ipo+linear_solvers+mpi+openmp~particles+petsc~pic~plotfile_tools~rocm+shared~sundials amdgpu_target=none build_type=RelWithDebInfo cuda_arch=none dimensions=3 precision=double
cray-mpich@8.1.12%gcc%9.3.0 /mvyqt7u:
cray-mpich@8.1.12%nvhpc%21.9 /e7ifg65:
gasnet@2021.9.0%gcc%9.3.0 /owzisjf:~debug conduits=smp
gcc@9.3.0%gcc%7.5.0 /suufi5c:~binutils+bootstrap~graphite~nvptx~piclibs~strip languages=c,c++,fortran patches=b48e48736062e64a6da7cbe7e21a6c1c89422d1f49ef547c73b479a3f3f4935f
hypre@2.23.0%gcc%9.3.0 /vzrzuch:~complex+cuda~debug+fortran~int64~internal-superlu~mixedint+mpi+openmp+shared+superlu-dist~unified-memory cuda_arch=80
hypre@2.23.0%nvhpc%21.9 /5gcmt4f:~complex+cuda~debug+fortran~int64~internal-superlu~mixedint+mpi+openmp+shared+superlu-dist~unified-memory cuda_arch=80
kokkos@3.4.01%gcc%9.3.0 /43pn73s:~aggressive_vectorization~compiler_warnings~cuda~cuda_constexpr~cuda_lambda~cuda_ldg_intrinsic~cuda_relocatable_device_code~cuda_uvm~debug~debug_bounds_check~debug_dualview_modify_check~deprecated_code~examples~explicit_instantiation~hpx~hpx_async_dispatch~hwloc~ipo~memkind~numactl+openmp~pic+profiling~profiling_load_print~pthread~qthread~rocm+serial+shared~sycl~tests~tuning~wrapper amdgpu_target=none build_type=RelWithDebInfo cuda_arch=none std=14
kokkos-kernels@3.4.01%gcc%9.3.0 /yxp2agn:~blas~cblas~cublas~cuda~cusparse~ipo~lapack~lapacke~mkl+openmp~pthread~serial+shared~superlu build_type=RelWithDebInfo cuda_arch=none execspace_cuda=auto execspace_openmp=auto execspace_serial=auto execspace_threads=auto layouts=left memspace_cudaspace=auto memspace_cudauvmspace=auto offsets=int,size_t ordinals=int scalars=double
kokkos-kernels@3.4.01%gcc%9.3.0 /ax5a6kg:~blas~cblas~cublas+cuda~cusparse~ipo~lapack~lapacke~mkl+openmp~pthread~serial+shared~superlu build_type=RelWithDebInfo cuda_arch=80 execspace_cuda=auto execspace_openmp=auto execspace_serial=auto execspace_threads=auto layouts=left memspace_cudaspace=auto memspace_cudauvmspace=auto offsets=int,size_t ordinals=int scalars=double
nvhpc@21.9%gcc%7.5.0 /7drpywv:+blas+lapack~mpi install_type=single
papi@6.0.0.1%gcc%9.3.0 /my65i47:~cuda+example~infiniband~lmsensors~nvml~powercap~rapl~rocm~rocm_smi~sde+shared~static_tools amdgpu_target=none
petsc@3.16.1%gcc%9.3.0 /4dajvqv:~X~batch~cgns~complex+cuda~debug+double~exodusii~fftw~giflib+hdf5~hpddm~hwloc+hypre~int64~jpeg~knl~libpng~libyaml~memkind+metis~mkl-pardiso~mmg~moab~mpfr+mpi~mumps+openmp~p4est~parmmg~ptscotch~random123~rocm~saws~scalapack+shared+strumpack~suite-sparse+superlu-dist~tetgen~trilinos~valgrind amdgpu_target=none clanguage=C cuda_arch=80
plumed@2.6.3%gcc%9.3.0 /emtm3wd:+gsl+mpi+shared arrayfire=none optional_modules=all
raja@0.14.0%gcc%9.3.0 /uqv24cu:~cuda+examples+exercises~ipo+openmp~rocm+shared~tests amdgpu_target=none build_type=RelWithDebInfo cuda_arch=none
raja@0.14.0%gcc%9.3.0 /w7shro2:+cuda+examples+exercises~ipo+openmp~rocm+shared~tests amdgpu_target=none build_type=RelWithDebInfo cuda_arch=80
slepc@3.16.0%gcc%9.3.0 /hftdevq:+arpack~blopex~cuda~rocm amdgpu_target=none cuda_arch=none
slepc@3.16.0%gcc%9.3.0 /voqdvvo:+arpack~blopex+cuda~rocm amdgpu_target=none cuda_arch=80
sundials@5.8.0%gcc%9.3.0 /n4rh37o:+ARKODE+CVODE+CVODES+IDA+IDAS+KINSOL~cuda+examples+examples-install~f2003~fcmix+generic-math+hypre~int64~ipo~klu~lapack~monitoring+mpi+openmp~petsc~pthread~raja~rocm+shared+static~superlu-dist~superlu-mt~sycl~trilinos amdgpu_target=none build_type=RelWithDebInfo cuda_arch=none precision=double
superlu@5.3.0%gcc%9.3.0 /g7kbe4f:~ipo+pic build_type=RelWithDebInfo
trilinos@13.0.1%gcc%9.3.0 /duxpdgk:~adios2+amesos+amesos2+anasazi+aztec~basker+belos+boost~chaco~complex~cuda~cuda_rdc~debug~dtk+epetra+epetraext~epetraextbtf~epetraextexperimental~epetraextgraphreorderings~exodus+explicit_template_instantiation~float+fortran~gtest~hdf5~hypre+ifpack+ifpack2+intrepid+intrepid2~ipo+isorropia+kokkos~mesquite+minitensor+ml+mpi+muelu~mumps+nox~openmp+phalanx+piro~python+rol+rythmos+sacado~scorec+shards+shared+shylu+stk+stokhos+stratimikos~strumpack~suite-sparse~superlu+superlu-dist+teko+tempus+tpetra+trilinoscouplings~wrapper~x11+zoltan+zoltan2 build_type=RelWithDebInfo cuda_arch=none cxxstd=14 gotype=long_long
umpire@6.0.0%gcc%9.3.0 /nr4jdia:+c+cuda~deviceconst+examples~fortran~ipo~numa~openmp~rocm~shared amdgpu_target=none build_type=RelWithDebInfo cuda_arch=80 tests=none
umpire@6.0.0%nvhpc%21.9 /jwjpkpb:+c+cuda~deviceconst+examples~fortran~ipo~numa~openmp~rocm~shared amdgpu_target=none build_type=RelWithDebInfo cuda_arch=80 tests=none
upcxx@2021.9.0%gcc%9.3.0 /mv7smui:~cuda+gasnet+mpi cross=none
upcxx@2021.9.0%gcc%9.3.0 /ngys4nf:+cuda~gasnet~mpi cross=none
upcxx@2021.9.0%nvhpc%21.9 /526i4jo:+cuda~gasnet~mpi cross=none
wannier90@3.1.0%gcc%9.3.0 /rnknj3s:+shared
zfp@0.5.5%gcc%9.3.0 /rjjnagm:~aligned~c+cuda~fasthash~fortran~ipo~openmp~profile~python+shared~strided~twoway bsws=64 build_type=RelWithDebInfo cuda_arch=80
zfp@0.5.5%nvhpc%21.9 /s4ii7k6:~aligned~c+cuda~fasthash~fortran~ipo~openmp~profile~python+shared~strided~twoway bsws=64 build_type=RelWithDebInfo cuda_arch=80

Shown below are few useful spack find commands you can use to filter output

All root specs built with nvhpc compiler
elvis@perlmutter> spack find -x --format "{name}" %nvhpc
cray-mpich  hypre  umpire  upcxx  zfp

We have optimized our cuda builds for the NVIDIA A100 gpus which is set in spack packages using the cuda_arch=80 variant. If you want to find out all specs built for cuda you can use the following command shown below.

All specs built with variant cuda_arch=80
elvis@perlmutter> spack find --format "{name}@{version}%{compiler.name}@{compiler.version}: {variants}" cuda_arch=80    
camp@0.2.2%gcc@9.3.0: +cuda~ipo~rocm~tests amdgpu_target=none build_type=RelWithDebInfo cuda_arch=80
camp@0.2.2%nvhpc@21.9: +cuda~ipo~rocm~tests amdgpu_target=none build_type=RelWithDebInfo cuda_arch=80
hypre@2.23.0%gcc@9.3.0: ~complex+cuda~debug+fortran~int64~internal-superlu~mixedint+mpi+openmp+shared+superlu-dist~unified-memory cuda_arch=80
hypre@2.23.0%nvhpc@21.9: ~complex+cuda~debug+fortran~int64~internal-superlu~mixedint+mpi+openmp+shared+superlu-dist~unified-memory cuda_arch=80
kokkos@3.4.01%gcc@9.3.0: ~aggressive_vectorization~compiler_warnings+cuda~cuda_constexpr~cuda_lambda~cuda_ldg_intrinsic~cuda_relocatable_device_code~cuda_uvm~debug~debug_bounds_check~debug_dualview_modify_check~deprecated_code~examples~explicit_instantiation~hpx~hpx_async_dispatch~hwloc~ipo~memkind~numactl+openmp~pic+profiling~profiling_load_print~pthread~qthread~rocm+serial+shared~sycl~tests~tuning+wrapper amdgpu_target=none build_type=RelWithDebInfo cuda_arch=80 std=14
kokkos-kernels@3.4.01%gcc@9.3.0: ~blas~cblas~cublas+cuda~cusparse~ipo~lapack~lapacke~mkl+openmp~pthread~serial+shared~superlu build_type=RelWithDebInfo cuda_arch=80 execspace_cuda=auto execspace_openmp=auto execspace_serial=auto execspace_threads=auto layouts=left memspace_cudaspace=auto memspace_cudauvmspace=auto offsets=int,size_t ordinals=int scalars=double
petsc@3.16.1%gcc@9.3.0: ~X~batch~cgns~complex+cuda~debug+double~exodusii~fftw~giflib+hdf5~hpddm~hwloc+hypre~int64~jpeg~knl~libpng~libyaml~memkind+metis~mkl-pardiso~mmg~moab~mpfr+mpi~mumps+openmp~p4est~parmmg~ptscotch~random123~rocm~saws~scalapack+shared+strumpack~suite-sparse+superlu-dist~tetgen~trilinos~valgrind amdgpu_target=none clanguage=C cuda_arch=80
raja@0.14.0%gcc@9.3.0: +cuda+examples+exercises~ipo+openmp~rocm+shared~tests amdgpu_target=none build_type=RelWithDebInfo cuda_arch=80
slepc@3.16.0%gcc@9.3.0: +arpack~blopex+cuda~rocm amdgpu_target=none cuda_arch=80
umpire@6.0.0%gcc@9.3.0: +c+cuda~deviceconst+examples~fortran~ipo~numa~openmp~rocm~shared amdgpu_target=none build_type=RelWithDebInfo cuda_arch=80 tests=none
umpire@6.0.0%nvhpc@21.9: +c+cuda~deviceconst+examples~fortran~ipo~numa~openmp~rocm~shared amdgpu_target=none build_type=RelWithDebInfo cuda_arch=80 tests=none
zfp@0.5.5%gcc@9.3.0: ~aligned~c+cuda~fasthash~fortran~ipo~openmp~profile~python+shared~strided~twoway bsws=64 build_type=RelWithDebInfo cuda_arch=80
zfp@0.5.5%nvhpc@21.9: ~aligned~c+cuda~fasthash~fortran~ipo~openmp~profile~python+shared~strided~twoway bsws=64 build_type=RelWithDebInfo cuda_arch=80    

Spack allows one to query specs by tags which can be useful. The e4s tag was added to several spack packages in spack/23382 which allows one to query specs using spack find -t e4s. In order to see available tags you can run the following and specify name of tag in spack find.

elvis@perlmutter> spack tags
27 available package tags:
    amr          ecp            finite-elements  hep         i/o            neutrino   spectral-elements
    build-tools  ecp-apps       flow             heps        key4hep        proxy-app  storage
    cfd          ecp-proxy-app  fluid            high-order  mcmc           radiuss    trace-tools
    e4s          fem            gibbs-sampler    hpc         navier-stokes  solver

For instance if you want to know number of E4S packages in our spack environment you can run the following command.

Number of spack packages with e4s tag
elvis@perlmutter> spack find -t e4s --format "{name}" | uniq | wc -l
21

Tcl Modules - Flat Hierarchy

The e4s/21.11-tcl module provides Tcl modules that display the E4S stack in a Flat Hierarchy which means you can see entire stack in a single module tree. Shown below is the module tree for Tcl modules. We have added suffix for some modulefiles such as -mpi for packages built with MPI and -cuda for cuda built packages.

module avail for Tcl modules
elvis@perlmutter> ml av

---------------------------------------------------- /global/common/software/spackecp/perlmutter/e4s-21.11/modules/tcl/cray-sles15-zen2 -----------------------------------------------------
adios2/2.7.1-gcc-9.3.0-mpi                      kokkos/3.4.01-gcc-9.3.0                slepc/3.16.0-gcc-9.3.0-mpi    (D)    upcxx/2021.9.0-gcc-9.3.0-mpi
amrex/21.11-gcc-9.3.0-mpi                       papi/6.0.0.1-gcc-9.3.0                 sundials/5.8.0-gcc-9.3.0-mpi         upcxx/2021.9.0-nvhpc-21.9-cuda (D)
gasnet/2021.9.0-gcc-9.3.0                       petsc/3.16.1-gcc-9.3.0-mpi-cuda        superlu/5.3.0-gcc-9.3.0              wannier90/3.1.0-gcc-9.3.0-mpi
hypre/2.23.0-gcc-9.3.0-mpi-cuda                 plumed/2.6.3-gcc-9.3.0-mpi             trilinos/13.0.1-gcc-9.3.0-mpi        zfp/0.5.5-gcc-9.3.0-cuda
hypre/2.23.0-nvhpc-21.9-mpi-cuda         (D)    raja/0.14.0-gcc-9.3.0-cuda             umpire/6.0.0-gcc-9.3.0-cuda          zfp/0.5.5-nvhpc-21.9-cuda      (D)
kokkos-kernels/3.4.01-gcc-9.3.0-mpi-cuda        raja/0.14.0-gcc-9.3.0           (D)    umpire/6.0.0-nvhpc-21.9-cuda  (D)
kokkos-kernels/3.4.01-gcc-9.3.0          (D)    slepc/3.16.0-gcc-9.3.0-mpi-cuda        upcxx/2021.9.0-gcc-9.3.0-cuda

We have excluded modulefile generation for implicit specs to avoid explosion of modules. The implicit specs are lower level libraries not used by users however there are some specs that we explicitly generated modules because they could be useful for users.

Lmod Modules - Hierarchical Modules

We have deployed the e4s stack with Lua modules using the Lmod Hierarchy. This can be accessed via e4s/21.11-lmod modulefile. In this stack, the software stack is organized hierarchy based on compilers and mpi. In order to access the software one must load the compiler or mpi module in-order to get to desired application.

You can navigate between e4s/21.11-tcl and e4s/21.11-lmod stack if you prefer however only one stack can be loaded at a time

elivs@perlmutter> ml e4s/21.11-lmod

The following have been reloaded with a version change:
  1) e4s/21.11-tcl => e4s/21.11-lmod

Once you load this stack we will expose the MODULEPATH for the Core directory (/global/common/software/spackecp/perlmutter/e4s-21.11/modules/lmod/cray-sles15-x86_64/Core) which will contain the base compilers.

elvis@perlmutter> ml av

------------ /global/common/software/spackecp/perlmutter/e4s-21.11/modules/lmod/cray-sles15-x86_64/Core -------------
   gcc/9.3.0    nvhpc/21.9

For instance, if you load the gcc/9.3.0 module, this will load another module tree for specs built with gcc@9.3.0 along with another moduletree for specs built with MPI provider cray-mpich. By default we load dependent modules hence cray-mpich was loaded.

elvis@perlmutter> ml gcc/9.3.0

Lmod is automatically replacing "nvidia/21.9" with "gcc/9.3.0".


Due to MODULEPATH changes, the following have been reloaded:
  1) cray-mpich/8.1.12


elvis@perlmutter> ml av

--------------------------------- /global/common/software/spackecp/perlmutter/e4s-21.11/modules/lmod/cray-sles15-x86_64/cray-mpich/8.1.12-mvyqt7u/gcc/9.3.0 ---------------------------------
   adios2/2.7.1    hypre/2.23.0-cuda             petsc/3.16.1-cuda    slepc/3.16.0-cuda        sundials/5.8.0     upcxx/2021.9.0  (D)
   amrex/21.11     kokkos-kernels/3.4.01-cuda    plumed/2.6.3         slepc/3.16.0      (D)    trilinos/13.0.1    wannier90/3.1.0

---------------------------------------------- /global/common/software/spackecp/perlmutter/e4s-21.11/modules/lmod/cray-sles15-x86_64/gcc/9.3.0 ----------------------------------------------
   cray-mpich/8.1.12 (L)    kokkos-kernels/3.4.01 (D)    papi/6.0.0.1        raja/0.14.0   (D)    umpire/6.0.0-cuda      zfp/0.5.5-cuda
   gasnet/2021.9.0          kokkos/3.4.01                raja/0.14.0-cuda    superlu/5.3.0        upcxx/2021.9.0-cuda

You can switch between gcc and nvhpc compiler however this will change the MODULEPATH but also auto-swap the compilers which means you can't load modules from gcc and nvhpc moduletree at same time. Shown below is the module stack for nvhpc compiler.

elvis@perlmutter> ml nvhpc/21.9

Lmod is automatically replacing "gcc/9.3.0" with "nvhpc/21.9".


Due to MODULEPATH changes, the following have been reloaded:
  1) cray-mpich/8.1.12

elvis@login33> ml av

--------------------------------------------- /global/common/software/spackecp/perlmutter/e4s-21.11/modules/lmod/cray-sles15-x86_64/nvhpc/21.9 ----------------------------------------------
   cray-mpich/8.1.12 (L)    umpire/6.0.0-cuda    upcxx/2021.9.0-cuda    zfp/0.5.5-cuda

------------------------------------------------ /global/common/software/spackecp/perlmutter/e4s-21.11/modules/lmod/cray-sles15-x86_64/Core -------------------------------------------------
   gcc/9.3.0    nvhpc/21.9 (L)

Spack Developer Workflow

There is a known issue with spack unable to bootstrap clingo on Perlmutter see https://github.com/spack/spack/issues/28315. The main issue is getting clingo installed which must be in your python environment. In order to get around this we have provided a script you can source which will create a python environment in your $HOME directory and install clingo. You can source the following script based on your shell type.

!!! note: If you already have clingo in your python environment you don't need to source this script.

# bash/sh/zsh users
source /global/common/software/spackecp/perlmutter/e4s-21.11/spack/bin/setup-spack.sh

# csh/tcsh users
source /global/common/software/spackecp/perlmutter/e4s-21.11/spack/bin/setup-spack.csh

Shown below is an output of what would happen if you source this script.

elvis@login08> source /global/common/software/spackecp/perlmutter/e4s-21.11/spack/bin/setup-spack.sh
Creating python virtual environment in /global/homes/s/elvis/.spack-pyenv
WARNING: You are using pip version 20.2.3; however, version 21.3.1 is available.
You should consider upgrading via the '/global/homes/s/elvis/.spack-pyenv/bin/python3 -m pip install --upgrade pip' command.
Package    Version
---------- -------
cffi       1.15.0
clingo     5.5.1
pip        20.2.3
pycparser  2.21
setuptools 44.1.1
WARNING: You are using pip version 20.2.3; however, version 21.3.1 is available.
You should consider upgrading via the '/global/homes/s/elvis/.spack-pyenv/bin/python3 -m pip install --upgrade pip' command.
Your python interpreter used by spack is /global/homes/s/elvis/.spack-pyenv/bin/python

We recommend you run the following to see if spack will concretize a spec and not try to bootstrap clingo. Note we must exit the e4s environment via despactivate since spack configuration will use the production path which will cause some issues.

(.spack-pyenv) elvis@login08> despactivate
(.spack-pyenv) elvis@login08> spack spec zlib
Input spec
--------------------------------
zlib

Concretized
--------------------------------
zlib@1.2.11%gcc@9.3.0+optimize+pic+shared arch=cray-sles15-zen2

If you want to install some spec we recommend you create a spack environment in your user space.

First we need to run despactivate to get out of the e4s environment. You can confirm your active environment by running spack env st.

Next we create a spack environment in our home directory and activate the environment

(.spack-pyenv) elvis@login08> spack env create -d $HOME/spack-demo
==> Updating view at /global/homes/s/elvis/spack-demo/.spack-env/view
==> Created environment in /global/homes/s/elvis/spack-demo
==> You can activate this environment with:
==>   spack env activate /global/homes/s/elvis/spack-demo

(.spack-pyenv) elvis@login08> spack env activate $HOME/spack-demo

Next, let's install m4, as it turns out m4 is an external package because our site configuration mimics the deployment configuration so that you can leverage our recommendation when install packages.

(.spack-pyenv) elvis@login08> spack install m4
[+] /usr (external m4-1.4.18-v2jy6pw2c4qi35cekefo7foryzoislc2)
==> Updating view at /global/homes/s/elvis/spack-demo/.spack-env/view
==> Warning: [/global/homes/s/elvis/spack-demo/.spack-env/._view/nzffkglxmsrngdyixy3adkdryrbdnedj] Skipping external package: m4@1.4.18%gcc@9.3.0+sigsegv patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 arch=cray-sles15-zen2/v2jy6pw

You can retrieve the site scope via spack config --scope site get packages.

The spack configuration will install packages in your $HOME/spack-workspace but this can be overridden in your spack.yaml. The site default indicate the root of the software tree is $HOME/spack-workspace/software and root of tcl module tree is $HOME/spack-workspace/modules/tcl

(.spack-pyenv) elvis@login08> spack config --scope=site get config | head -n 8
config:
  install_tree:
    root: $HOME/spack-workspace/software
    projections:
      all: ${ARCHITECTURE}/${COMPILERNAME}-${COMPILERVER}/${PACKAGE}-${VERSION}-${HASH}
  source_cache: $HOME/.spack/cache
  module_roots:
    tcl: $HOME/spack-workspace/modules/tcl

Let's try installing papi which should install from source and will relatively quick.

(.spack-pyenv) elvis@login08> spack install papi
==> Installing papi-6.0.0.1-my65i474uh2qzp76mkta2u2o46qqrjev
==> No binary for papi-6.0.0.1-my65i474uh2qzp76mkta2u2o46qqrjev found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/3c/3cd7ed50c65b0d21d66e46d0ba34cd171178af4bbf9d94e693915c1aca1e287f.tar.gz
==> No patches needed for papi
==> papi: Executing phase: 'autoreconf'
==> papi: Executing phase: 'configure'
==> papi: Executing phase: 'build'
==> papi: Executing phase: 'install'
==> papi: Successfully installed papi-6.0.0.1-my65i474uh2qzp76mkta2u2o46qqrjev
  Fetch: 0.60s.  Build: 23.78s.  Total: 24.38s.
[+] /global/homes/s/elvis/spack-workspace/software/cray-sles15-zen2/gcc-9.3.0/papi-6.0.0.1-my65i474uh2qzp76mkta2u2o46qqrjev
==> Updating view at /global/homes/s/elvis/spack-demo/.spack-env/view

We can find the root directory for papi by running the following. We see the package was installed in our home directory.

(.spack-pyenv) elvis@login08> spack location -i papi
/global/homes/s/elvis/spack-workspace/software/cray-sles15-zen2/gcc-9.3.0/papi-6.0.0.1-my65i474uh2qzp76mkta2u2o46qqrjev

We can create the module tree and retrieve location of our generated module as shown below.

(.spack-pyenv) elvis@login08> spack module tcl refresh --delete-tree -y
==> Regenerating tcl module files
(.spack-pyenv) elvis@login08> spack module tcl find --full-path papi
/global/homes/s/elvis/spack-workspace/modules/tcl/cray-sles15-zen2/papi-6.0.0.1-gcc-9.3.0-my65i47

If you want to add this to MODULEPATH you can simply run the following

(.spack-pyenv) elvis@login08> module use $HOME/spack-workspace/modules/tcl/cray-sles15-zen2

Now we see the modulefile papi-6.0.0.1-gcc-9.3.0-my65i47 is visible and can be loaded as any other software package.

(.spack-pyenv) elvis@login08> ml -t av papi
/global/homes/s/elvis/spack-workspace/modules/tcl/cray-sles15-zen2:
papi-6.0.0.1-gcc-9.3.0-my65i47
/global/common/software/spackecp/perlmutter/e4s-21.11/modules/tcl/cray-sles15-zen2:
papi/6.0.0.1-gcc-9.3.0
/opt/cray/pe/lmod/modulefiles/core:
papi/6.0.0.10
papi/6.0.0.12

References

We have published our e4s/21.11 spack configuration at https://github.com/spack/spack-configs/tree/main/NERSC/perlmutter/e4s-21.11 along with other relevant scripts.