Skip to content

MPI and GPUs in Containers

Both podman-hpc and Shifter can use the optimized Cray MPICH and the NVIDIA drivers on Perlmutter from inside a container. The approach is the same for both runtimes:

  1. Build your application against a standard, shared-library build of MPICH inside the image.
  2. At run time, NERSC's container modules swap the Cray MPICH libraries into the container in place of the MPICH you built. This works because the two are ABI-compatible.

Install MPICH from source, not with apt-get

The library swap only works when MPICH is built from source with shared libraries, as in the example below. MPICH installed from a distribution package (for example apt-get install mpich) is not swapped.

container modules

The term modules is overloaded. Both shifter and podman-hpc include "modules" which can provide modified functionality. They should not be confused with Lmod environment modules which are provided by the Cray Programming Environment.

Example image

This Containerfile builds MPICH 3.4.3 from source and installs mpi4py and works with either runtime.

Use an MPICH version no newer than the injected libraries

The MPICH libraries injected at run time are based on MPICH 3.4. An application built against a newer MPICH can fail to start if it uses functions the injected libraries do not have (for example, mpi4py 4.0.3 built against MPICH 4.2.2 fails with undefined symbol: MPI_Buffer_iflush). Build against MPICH 3.4.3 as shown below.

FROM docker.io/library/ubuntu:24.04
WORKDIR /opt

ENV DEBIAN_FRONTEND=noninteractive

RUN apt-get update && \
    apt-get install --yes --no-install-recommends \
        build-essential \
        ca-certificates \
        gfortran \
        python3-dev \
        python3-venv \
        wget && \
    rm -rf /var/lib/apt/lists/*

ARG MPICH_VERSION=3.4.3
ARG MPICH_PREFIX=mpich-${MPICH_VERSION}

RUN wget https://www.mpich.org/static/downloads/${MPICH_VERSION}/${MPICH_PREFIX}.tar.gz && \
    tar xzf ${MPICH_PREFIX}.tar.gz && \
    cd ${MPICH_PREFIX} && \
    FFLAGS=-fallow-argument-mismatch FCFLAGS=-fallow-argument-mismatch \
        ./configure --with-device=ch4:ofi && \
    make -j 8 && \
    make install && \
    cd .. && \
    rm -rf ${MPICH_PREFIX} ${MPICH_PREFIX}.tar.gz && \
    ldconfig

ARG MPI4PY_VERSION=4.0.3
RUN python3 -m venv /opt/venv && \
    /opt/venv/bin/pip install --no-cache-dir mpi4py==${MPI4PY_VERSION}
ENV PATH=/opt/venv/bin:$PATH

Build

Both runtimes start from an image built with podman-hpc.

Build with podman-hpc

Build the image on a Perlmutter login node:

podman-hpc build -t mpich-example:1.0 .

Tip

If downloads during a podman-hpc build are slow, add --net slirp4netns. See the Known issues.

The image is then ready to run with podman-hpc.

Shifter

Shifter cannot use the locally built image directly. After building it with podman-hpc as above, push the image to a registry, then pull it with shifterimg:

shifterimg pull docker:<your-user>/mpich-example:1.0

Running with MPI

The two runtimes differ in one important way: Shifter enables the mpich and gpu modules by default, while podman-hpc requires you to ask for MPI (--mpi) and GPU (--gpu) support.

srun -N 2 -n 4 podman-hpc run --rm --mpi mpich-example:1.0 \
    python3 -m mpi4py.bench helloworld
srun -N 2 -n 4 shifter --image=docker:<your-user>/mpich-example:1.0 \
    python3 -m mpi4py.bench helloworld

Shifter also works from a batch script:

#!/bin/bash
#SBATCH --image=docker:<your-user>/mpich-example:1.0
#SBATCH --qos=regular
#SBATCH -N 2
#SBATCH -C cpu

srun -n 4 shifter python3 -m mpi4py.bench helloworld

Note

The image must contain a glibc at or above the version required by the injected MPICH libraries. If not then you may see an error like version GLIBC_2.25 not found.

Using GPUs

To use the NVIDIA drivers and tools (such as nvidia-smi) inside the container, podman-hpc needs the --gpu flag. Shifter enables GPU support by default, so no flag is needed.

srun -n 1 -G 1 -C gpu podman-hpc run --rm --gpu <image> nvidia-smi
srun -n 1 -G 1 -C gpu shifter --image=<image> nvidia-smi

Shifter modules replace the defaults

In Shifter, --module=... replaces the default modules (mpich and gpu) instead of adding to them. If you pass --module, list every module you need: --module=gpu,mpich is the same as the default, and --module=gpu,cuda-mpich enables CUDA-aware MPI. --module=none turns all modules off.

CUDA-aware MPI

To pass GPU buffers directly to MPI calls, use Shifter with the cuda-mpich module.

srun -n 4 -G 4 -C gpu shifter --image=<image> --module=gpu,cuda-mpich ./my_app

CUDA-aware MPI requires an image built on CUDA 11

The CUDA-aware MPI libraries that Shifter injects need the CUDA 11 runtime (libcudart.so.11.0) in the image. An image built on a CUDA 12 or 13 base fails with libcudart.so.11.0: cannot open shared object file. Use a CUDA 11 development image such as nvcr.io/nvidia/cuda:11.8.0-devel-ubuntu22.04 (the last CUDA 11 release) as the base. GPU access without CUDA-aware MPI (--gpu in podman-hpc, or Shifter's default gpu module) works with newer CUDA images.

CUDA-aware MPI is not currently available with podman-hpc.

See Building Custom Images for a full CUDA and MPICH Containerfile and an example.

Other libraries