Building Custom Images¶
Users with the need to build their custom images can use one of the base images provided by NERSC. Before starting custom builds, please refer to the container building basics provided at our shifter-for-beginners-tutorial page for build terminology and instructions. This instruction page assumes that you either have podman-desktop (free) or docker-desktop (may require payment) installed on your local system. It is possible to build a custom container image on Perlmutter using podman-hpc. However, there are additional needed to then push the image to a container repository and pull it from there using shifter. Therefore, we have used podman desktop for all the examples discussed on this page.
Container base images¶
Depending on the use case, there are several base images that are available to users to build their images on. Popular among them are the following:
FROM ubuntu:22.04
which is typically used as a base image for CPU only application builds.
FROM nvcr.io/nvidia/cuda:11.8.0-cudnn8-devel-ubuntu22.04
which is our preferred base image to build GPU enabled applications. There are several other base images that are available as a starting point for GPU enabled applications from the NGC repository
NERSC base image¶
However, any container image can be used as a base image and therefore, as a starting point. We suggest our users to start their builds with the following NERSC supported base image:
FROM docker.io/nersc/base_cuda_mpich:11.8x4.2.2
The recipe for which is as follows:
FROM nvcr.io/nvidia/cuda:11.8.0-cudnn8-devel-ubuntu22.04
WORKDIR /opt
ENV DEBIAN_FRONTEND noninteractive
RUN \
apt-get update && \
apt-get install --yes \
build-essential autoconf cmake flex bison zlib1g-dev \
fftw-dev fftw3 apbs libicu-dev libbz2-dev libgmp-dev \
libboost-all-dev bc libblas-dev liblapack-dev git \
libfftw3-dev automake lsb-core libxc-dev libgsl-dev \
unzip libhdf5-serial-dev ffmpeg libcurl4-openssl-dev \
libboost-dev libboost-system-dev libtool automake \
libboost-filesystem-dev libboost-graph-dev uuid-dev \
libboost-regex-dev libedit-dev libyaml-cpp-dev make \
python3-dev python3-cffi python3-ply python3-pip \
aspell aspell-en valgrind libyaml-cpp-dev wget vim \
libzmq3-dev python3-yaml time valgrind libevent-dev \
mlocate python3-jsonschema python-is-python3 &&\
apt-get clean all
# Install MPICH
WORKDIR /opt
ARG mpich=4.2.2
ARG mpich_prefix=mpich-$mpich
RUN \
wget https://www.mpich.org/static/downloads/$mpich/$mpich_prefix.tar.gz && \
tar xvzf $mpich_prefix.tar.gz && \
cd $mpich_prefix && \
./configure FFLAGS=-fallow-argument-mismatch FCFLAGS=-fallow-argument-mismatch \
--prefix=/opt/mpich/install && \
make -j 16 && \
make install && \
make clean && \
cd .. && \
rm -rf $mpich_prefix.tar.gz
ENV PATH=$PATH:/opt/mpich/install/bin
ENV PATH=$PATH:/opt/mpich/install/include
ENV LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/mpich/install/lib
RUN /sbin/ldconfig
ENV PATH=$PATH:/usr/local/cuda/lib64/stubs
ENV LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/lib64/stubs
ENV PATH=$PATH:/usr/local/cuda-11.8/targets/x86_64-linux/lib/stubs
ENV LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-11.8/targets/x86_64-linux/lib/stubs
# Install Eigen and Blas
WORKDIR /opt
RUN git clone https://gitlab.com/libeigen/eigen.git
RUN cd /opt/eigen && \
mkdir build && \
cd /opt/eigen/build && \
cmake -DCMAKE_INSTALL_PREFIX=/opt/eigen/install .. && \
make blas && \
make lapack && \
make install
ENV PATH=$PATH:/opt/eigen/install/bin
ENV PATH=$PATH:/opt/eigen/install/include
ENV LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/eigen/install/lib
This base image should provide most of the needed libraries and compilers needed to build a GPU enabled application. It is built off of a CUDA 11.8 base image and includes MPICH 4.2.2, which will be replaced using the Cray-MPICH module injection during runtime on Perlmutter using shifter modules. This image is freely available to all NERSC users.
Guide to using custom images¶
To build a custom container, user can use the following Containerfile (the name of the file) template:
#Containerfile template to build your custom image
FROM docker.io/nersc/base_cuda_mpich:11.8x4.2.2
WORKDIR /opt
ENV DEBIAN_FRONTEND noninteractive
ENV CUDA_INSTALL_PATH "/usr/local/cuda-11.8/targets/x86_64-linux"
ENV MPI_HOME "/opt/mpich/install"
#Install your application here
WORKDIR /opt
RUN <either git clone or wget commands go here>
RUN cd /opt/<application-folder> && \
mkdir build && \
cd /build && \
<build instructions go here>
ENV PATH=$PATH:<full path to application install bin dir>
ENV PATH=$PATH:<full path to application install include dir>
ENV LD_LIBRARY_PATH=$LD_LIBRARY_PATH:<full path to application install lib dir>
The custom image has to be built first on the local machine before it can be deployed on Perlmutter. To build this application on your local machine using podman, use a terminal window pointing to directory containing the Containerfile above and use the following:
user@localhost:~> podman build -t <username>/<application name>:<tag> -f Containerfile --progress=plain .
Upon completion of the build process, you can check the availability of this image on the local host using:
user@localhost:~> podman images
which will provide the user with the repository name, image tag, image ID, date it was created, and the size of the image. To use this image on Perlmutter, it has to be pushed using image ID to either dockerhub, or quay.io hub, or registry.nersc.io. For this test case, We are using dockerhub repository:
user@localhost:~> podman push <image ID> docker://docker.io/<your dockerhub username>/<application name>:<tag>
Users can now pull this image on Perlmutter using shifter.
user@loginXXXX:~> shifterimg -v pull docker:<your dockerhub username>/<application name>:<tag>
To verify if the image was pulled successfully, use the following:
user@loginXXXX:~> shifterimg images | grep '<application name>'
Example 1¶
In the following example, we are using the NERSC provided base image to build custom image to test CUDA aware MPI performance. The recipe of this image build is as follows:
#Containerfile for CUDA-aware-MPI using NERSC cuda-mpich base image
FROM docker.io/nersc/base_cuda_mpich:11.8x4.2.2
WORKDIR /opt
ENV DEBIAN_FRONTEND noninteractive
ENV CUDA_INSTALL_PATH "/usr/local/cuda-11.8/targets/x86_64-linux"
ENV MPI_HOME "/opt/mpich/install"
# Install MPI-ping-pong from OLCF
WORKDIR /opt
RUN git clone https://github.com/olcf-tutorials/MPI_ping_pong.git test_code
RUN cd /opt/test_code/cpu && \
make
RUN cd /opt/test_code/cuda_staged && \
sed -i 's:-arch=sm_70:-arch=sm_80:g' Makefile && \
sed -i 's:OMPI_DIR:MPI_HOME:g' Makefile && \
sed -i 's:-lmpi_ibm:-lmpi:g' Makefile && \
make
RUN cd /opt/test_code/cuda_aware && \
sed -i 's:-arch=sm_70:-arch=sm_80:g' Makefile && \
sed -i 's:OMPI_DIR:MPI_HOME:g' Makefile && \
sed -i 's:-lmpi_ibm:-lmpi:g' Makefile && \
make
ENV PATH=$PATH:/opt/test_code/cpu
ENV PATH=$PATH:/opt/test_code/cuda_staged
ENV PATH=$PATH:/opt/test_code/cuda_aware
After pushing the custom image to dockerhub repository and then pulling it to Perlmutter, we can test it on Perlmutter using an interactive node and running the container image using shifter.
user@loginXXXX:~> salloc -N 2 -G 2 -C gpu -t 01:00:00 -A nstaff -q interactive --image=neilmehta87/cuda_aware_mpi:0.1
user@nid00XXXX:~> srun -n 2 --module cuda-mpich shifter pp_cuda_aware
Transfer size (B): 8, Transfer Time (s): 0.000061704, Bandwidth (GB/s): 0.000120748
Transfer size (B): 16, Transfer Time (s): 0.000062580, Bandwidth (GB/s): 0.000238114
Transfer size (B): 32, Transfer Time (s): 0.000062027, Bandwidth (GB/s): 0.000480470
Transfer size (B): 64, Transfer Time (s): 0.000062328, Bandwidth (GB/s): 0.000956308
Transfer size (B): 128, Transfer Time (s): 0.000063280, Bandwidth (GB/s): 0.001883841
Transfer size (B): 256, Transfer Time (s): 0.000109115, Bandwidth (GB/s): 0.002185023
Transfer size (B): 512, Transfer Time (s): 0.000135204, Bandwidth (GB/s): 0.003526788
Transfer size (B): 1024, Transfer Time (s): 0.000105962, Bandwidth (GB/s): 0.009000118
Transfer size (B): 2048, Transfer Time (s): 0.000107372, Bandwidth (GB/s): 0.017763905
Transfer size (B): 4096, Transfer Time (s): 0.000107688, Bandwidth (GB/s): 0.035423685
Transfer size (B): 8192, Transfer Time (s): 0.000107290, Bandwidth (GB/s): 0.071110223
Transfer size (B): 16384, Transfer Time (s): 0.000105464, Bandwidth (GB/s): 0.144681948
Transfer size (B): 32768, Transfer Time (s): 0.000108190, Bandwidth (GB/s): 0.282073040
Transfer size (B): 65536, Transfer Time (s): 0.000108950, Bandwidth (GB/s): 0.560210946
Transfer size (B): 131072, Transfer Time (s): 0.000111772, Bandwidth (GB/s): 1.092132093
Transfer size (B): 262144, Transfer Time (s): 0.000131579, Bandwidth (GB/s): 1.855467162
Transfer size (B): 524288, Transfer Time (s): 0.000828882, Bandwidth (GB/s): 0.589084209
Transfer size (B): 1048576, Transfer Time (s): 0.000294777, Bandwidth (GB/s): 3.312890393
Transfer size (B): 2097152, Transfer Time (s): 0.000863627, Bandwidth (GB/s): 2.261537311
Transfer size (B): 4194304, Transfer Time (s): 0.000297197, Bandwidth (GB/s): 13.143621926
Transfer size (B): 8388608, Transfer Time (s): 0.000483843, Bandwidth (GB/s): 16.146768619
Transfer size (B): 16777216, Transfer Time (s): 0.001428390, Bandwidth (GB/s): 10.938886329
Transfer size (B): 33554432, Transfer Time (s): 0.001944343, Bandwidth (GB/s): 16.072270060
Transfer size (B): 67108864, Transfer Time (s): 0.004310126, Bandwidth (GB/s): 14.500735198
Transfer size (B): 134217728, Transfer Time (s): 0.007852202, Bandwidth (GB/s): 15.919101115
Transfer size (B): 268435456, Transfer Time (s): 0.014746705, Bandwidth (GB/s): 16.952939254
Transfer size (B): 536870912, Transfer Time (s): 0.028526683, Bandwidth (GB/s): 17.527449602
Transfer size (B): 1073741824, Transfer Time (s): 0.058527615, Bandwidth (GB/s): 17.085951704
Example 2¶
To use HDF5 in a custom image, the user is required to only append source and build instructions on top of the base NERSC provided image as shown:
#Containerfile for HDF5 using NERSC cuda-mpich base image
FROM docker.io/nersc/base_cuda_mpich:11.8x4.2.2
WORKDIR /opt
ENV DEBIAN_FRONTEND noninteractive
ENV CUDA_INSTALL_PATH "/usr/local/cuda-11.8/targets/x86_64-linux"
ENV MPI_HOME "/opt/mpich/install"
#Install HDF5
WORKDIR /opt
RUN git clone -b hdf5_1_14_3 https://github.com/HDFGroup/hdf5.git hdf5
RUN cd hdf5 && \
mkdir build && \
cd build && \
cmake -G "Unix Makefiles" -DCMAKE_C_COMPILER=mpicc -DCMAKE_CXX_COMPILER=mpicxx \
-DCMAKE_Fortran_COMPILER=mpif90 \
-DCMAKE_BUILD_TYPE:STRING=Release -DBUILD_SHARED_LIBS:BOOL=ON \
-DBUILD_TESTING:BOOL=ON -DHDF5_BUILD_TOOLS:BOOL=ON -DHDF5_BUILD_FORTRAN:BOOL=ON \
-DHDF5_ENABLE_PARALLEL=ON -D CMAKE_INSTALL_PREFIX=/opt/hdf5/install ../ && \
cmake --build . --config Release && \
cpack -C Release CPackConfig.cmake && \
make -j 4 && \
make install
ENV PATH=$PATH:/opt/hdf5/install/bin
ENV PATH=$PATH:/opt/hdf5/install/lib
ENV PATH=$PATH:/opt/hdf5/install/include
ENV PATH=$PATH:/opt/hdf5/install/share
ENV LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/hdf5/install/lib
Please help us improve this page
Users are invited to contribute helpful information and corrections through our GitLab repository.