Kokkos¶

Kokkos Core implements a programming model in C++ for writing performance portable applications targeting all major HPC platforms. For that purpose it provides abstractions for both parallel execution of code and data management. Kokkos is designed to target complex node architectures with N-level memory hierarchies and multiple types of execution resources. It currently can use OpenMP, Threads, CUDA, HIP, SYCL and OpenMPTarget as backend programming models. On Perlmutter GPUs, one can use CUDA and OpenMPTarget backends with nvcc, nvc++ and llvm compilers. On Perlmutter CPUs, one can use OpenMP and Threads backends with all PrgEnv compilers.

Building Kokkos on Perlmutter¶

There are many combinations to build Kokkos on Perlmutter. We list the most commonly used combinations here. Most of the options such as build type and architecture options are common across all combinations.

CMAKE_COMMON_OPTIONS="\
    -D CMAKE_INSTALL_PREFIX=$INSTALL_PREFIX \
    -D CMAKE_BUILD_TYPE=Release \
    -D CMAKE_CXX_STANDARD=17 \
    -D Kokkos_ENABLE_TESTS=ON \
    -D Kokkos_ARCH_AMPERE80=ON \
    -D Kokkos_ARCH_ZEN3=ON "

PrgEnv-gnu: CUDA backend for device and OpenMP backend for host¶

This is the most widely used Kokkos configuration for NVIDIA GPUs and works with the default modules, i.e., PrgEnv-gnu and cudatoolkit. The configuration uses nvcc as the device compiler and g++ as the host compiler. CMake command:

cmake -DCMAKE_CXX_COMPILER=$KOKKOS_SRC/bin/nvcc_wrapper \
    ${CMAKE_COMMON_OPTIONS} \
    -D Kokkos_ENABLE_CUDA=ON \
    -D Kokkos_ENABLE_OPENMP=ON

PrgEnv-nvhpc: CUDA backend for device and OpenMP backend for host¶

This configuration needs PrgEnv-nvhpc and uses nvc++ for both host and device compilation. CMake command:

cmake -DCMAKE_CXX_COMPILER=nvc++ \
    ${CMAKE_COMMON_OPTIONS} \
    -D Kokkos_ENABLE_CUDA=ON \
    -D Kokkos_ENABLE_OPENMP=ON

PrgEnv-llvm: CUDA and OpenMPTarget backends for device and OpenMP backend for host¶

This is an NPE and is currently in the experimental phase. It needs the llvm/16 module (which is the latest llvm release as of editing the current page) available on Perlmutter. It uses clang++ for both host and device compilation. CMake command:

cmake -DCMAKE_CXX_COMPILER=clang++ \
    ${CMAKE_COMMON_OPTIONS} \
    -D Kokkos_ENABLE_CUDA=ON \
    -D Kokkos_ENABLE_OPENMP=ON

clang++ compiler can also be used to build the OpenMPTarget backend of Kokkos instead of the CUDA backend for GPUs. CMake command:

cmake -DCMAKE_CXX_COMPILER=clang++ \
    ${CMAKE_COMMON_OPTIONS} \
    -D Kokkos_ENABLE_OPENMPTARGET=ON \
    -D Kokkos_ENABLE_OPENMP=ON

Threads backend with every PrgEnv¶

Kokkos allows a host backend based on std::threads from C++. The build command for this configuration is similar with all PrgEnvs. Using CC as the C++ compiler can generate a portable build command across all PrgEnvs. CMake command:

cmake -DCMAKE_CXX_COMPILER=CC \
    ${CMAKE_COMMON_OPTIONS} \
    -D Kokkos_ENABLE_THREADS=ON

Additional information¶

For applications using Kokkos, pass the INSTALL_PATH of Kokkos as KOKKOS_ROOT to the respective build systems. For the latest information on using kokkos please see:

• https://github.com/kokkos/kokkos
• Perlmutter Readiness page