Skip to content

Podman-hpc for Beginners Tutorial

This tutorial is meant to demonstrate all the steps you need to build and run a container at NERSC. We will outline the steps using the podman-hpc container runtime, but since podman-hpc is OCI-compliant, the same commands/workflow should be possible with both Podman and Docker on your local machine.

Helpful vocabulary

  • Containerfile (or less generically, Dockerfile)- Recipe for building an image, including software within the image.
  • Image- The product of building the recipe spelled out in the Containerfile. Can be thought of a blueprint.
  • Container- The running instantiation of an image. If we think of the image as a blueprint, we can think of a container as a building constructed with the blueprint.
  • Container runtime- Framework used for building and running containers. Ex: Podman, Docker, Singularity, CharlieCloud. Note that podman-hpc is a wrapper that extends the capabilities of Podman for HPC.
  • OCI (Open Container Initiative)- common standard for container runtimes. Podman and Docker are OCI-compliant, meaning their syntax is generally interchangeable.
  • Registry- Online storage area for images. Can be public or private.

Build your hello-world image on a Perlmutter login node

Let's create our first image that will print hello world from inside a container! when we run it. Paste the following contents into a file called Containerfile:

FROM docker.io/library/ubuntu:22.04

ENTRYPOINT echo "hello from inside a container!"

What do these commands do?

FROM docker.io/library/ubuntu:22.04 downloads an existing Docker OCI base image that already contains the Ubuntu operating system (note that we use a fixed version instead of ubuntu:latest which would be a moving target). In general it's a good idea to take advantage of the many existing building blocks within the wider container ecosystem. For example, if you're planning to run on an NVIDIA gpu, you may wish to start with an NVIDIA CUDA base image.

ENTRYPOINT echo "hello from inside a container!" will automatically execute your echo statement to print hello when the container runs.

Now let's build your container using podman-hpc, making sure you are in the same directory as your Containerfile.

podman-hpc build -t hello-world:1.0 .

What does this do? It simultaneously builds your podman-hpc image based on what you specified in your Dockerfile and also names and tags your image with hello-world (name) and 1.0 (tag).

If you do a podman-hpc images command, you should now see this image listed.

elvis@perlmutter:login32:~/example> podman-hpc images
REPOSITORY                               TAG         IMAGE ID      CREATED         SIZE        R/O
localhost/hello-world                    1.0         e2c0e704cb8b  26 seconds ago  80.3 MB     false

Note that podman-hpc will prepend localhost/ to the image name since the image we built did not come from an external registry. However you do not need to use localhost in any podman-hpc commands.

Other useful Containerfile commands:

RUN will run a command inside the container environment to modify its contents. This is usually used for installing and configuring software.

COPY will copy files into your image.

WORKDIR will establish or change the current working directory during the image build. It will also affect the default directory that used when you run a container with this image.

ENV can be used for setting or appending to existing environment variables.

Run your hello-world container

Your hello-world image has been built and is now ready to run as a container.

elvis@perlmutter:login32:~/example> podman-hpc run --rm hello-world:1.0
hello from inside a container!

By default, the entrypoint runs automatically, printing our hello world message. The --rm flag instructs podman-hpc to clean up the container after you are done running it. You don't need this, but if you don't use it, you'll have a lot of unused containers accumulating on your system.

Congratulations, you have now built and run your first container!

Migrate your image from your login node to $SCRATCH

The image you just built is stored locally on the login node. If you'd like to use this image in a job (or access it on any other login node), you'll need to migrate your image onto the $SCRATCH filesystem.

podman-hpc migrate hello-world:1.0

Now the podman-hpc images command will show the R/O copy that has been created on $SCRATCH.

elvis@perlmutter:login32:~/example> podman-hpc images
REPOSITORY                               TAG         IMAGE ID      CREATED        SIZE        R/O
localhost/hello-world                    1.0         e2c0e704cb8b  7 minutes ago  80.3 MB     true
localhost/hello-world                    1.0         e2c0e704cb8b  7 minutes ago  80.3 MB     false

Note that if you modify your image, you'll need to re-migrate it.

Interactively inspect your container

You may find it useful to run your container interactively, especially for debugging purposes. The podman-hpc -it interactive flag allows you to enter the running container. You may wish to specify the type of shell, i.e. bash. Note that the requested shell must be present in your image.

In this example, we enter the container to check the OS. For comparison, note that NERSC's operating system is SUSE Linux, so this is additional verification that we are in fact executing this command inside our container. When you are done inspecting your image, type exit to leave your container.

Important: we have to disable the image entrypoint since it interferes with interactive mode.

elvis@perlmutter:login32:~/example> podman-hpc run --rm -it --entrypoint= hello-world:1.0 bash
root@48ecef60abb5:/# cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.2 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
root@48ecef60abb5:/# exit
exit
elvis@perlmutter:login32:~/example>

Submit an interactive job using your image

To use your image inside an interactive job, you can submit as you normally would

salloc -N 1 -t 60 -C cpu -q interactive

The only requirement for using your image in a job is that you have done a podman-hpc migrate. This has made a copy of the image in $SCRATCH.

Once inside your interactive job, you can issue the same podman-hpc run command we used on the login node.

elvis@nid200021:~> podman-hpc run --rm hello-world:1.0
hello from inside a container!
elvis@nid200021:~>

Submit a batch job using your image

Submitting a batch job with a podman-hpc image is very similar. Here is an example jobscript we'll call submit-podman-hpc.slurm.

#!/bin/bash

#SBATCH --qos=debug
#SBATCH --constraint=cpu
#SBATCH -t 00:02:00
#SBATCH -N 1
#SBATCH -o output.o%j

echo "slurm job id:" $SLURM_JOB_ID

podman-hpc run --rm hello-world:1.0

We can submit this job with sbatch submit-podman-hpc.slurm.

Let's check the output:

elvis@perlmutter:login16:~/example> cat output.o19050827 
slurm job id: 19050827
hello from inside a container!
elvis@perlmutter:login16:~/example>

Let's say we don't want to use the ENTRYPOINT we have specified in our image. Instead let's go ahead and print our OS information like we did above.

#!/bin/bash

#SBATCH --qos=debug
#SBATCH --constraint=cpu
#SBATCH -t 00:02:00
#SBATCH -N 1
#SBATCH -o output.o%j

echo "slurm job id:" $SLURM_JOB_ID

podman-hpc run --rm --entrypoint= hello-world:1.0 bash -c "cat /etc/os-release"

The output of this job looks like

elvis@perlmutter:login16:~/example> cat output.o19051343 
slurm job id: 19051343
PRETTY_NAME="Ubuntu 22.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.2 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
elvis@perlmutter:login16:~/example>

This should look familiar-- it should be same output you saw earlier. You can substitute in any command you like after the image:tag (hello-world:1.0). This command will run inside your container. For multiple commands, it is probably easiest to pass the name of a script that will execute them. Keep in mind that you may need to COPY or bind-mount a directory if you would like a file to be accessible inside your container.

Optional- Create an account on an external registry

If you plan to share your images or would like to store them in an additional place, you should consider creating an account on an external registry.

One popular example is Docker Hub. You can get a free account there, although you should note that your images will be publicly accessible.

For private images or images for use within a collaboration, NERSC offers our own image registry. If you would like to store images there, please open a ticket.

Optional- Install Podman/Docker on your local system

Users who wish to build container images on their local system can download free versions of Podman Desktop or Docker desktop. Note that this will likely require you to have admin permissions on your system.

Once installed, make sure the Podman/Docker service is running. You can then build, test, and push container images to an external registry.

Users with non-x86 hardware (i.e. Mac M1) will find that images they build on their local systems won't work out of the box on Perlmutter's x86 hardware. This will require a multi-arch build.

You may notice slow upload times for large images depending on your internet speed. This is one of the benefits of building images at NERSC rather than an external system.

General advice about building your application in a container

  1. Pick an appropriate base image depending on your target hardware and preferred OS
  2. Install the application dependencies yourself (lapack, fftw, etc.) either using the OS package manager or from source if required
  3. Start with an existing build recipe, but expect to adjust it for the new OS and package installation paths

Go forth and be productive with containers

Now that you have finished this tutorial, you might like some more detailed information about the different ways to use containers at NERSC. Here are several resources: