CMake¶

CMake is an open-source, cross-platform family of tools designed to build, test, and package software. It is build-system generator -- on NERSC machines, CMake will generate UNIX Makefiles, by default -- and there is no need to enable CMake in cross-compilation mode, e.g. cmake -DCMAKE_SYSTEM_NAME=CrayLinuxEnvironment <etc>.

CMake Recommendations¶

Use modern CMake interface libraries. "INTERFACE" libraries are libraries that do not create an output but instead hold specific build features, e.g. include paths, build flags, link flags, link libraries, compile definitions, etc.

In order to "use" interface libraries, just "link" the interface library against the target you want the properties to build properties to be applied to.

It is possible to provide the path to the required libraries using the variable CMAKE_PREFIX_PATH in cases where cmake fails to find them on its own, e.g. export CMAKE_PREFIX_PATH=</path/to/folder>

If the CMake output includes error message(s) stating that the target platform does not support dynamic linking, this may be resolved by executing export CRAYPE_LINK_TYPE=dynamic.

NOTE: in below, replace foo- with your project name or custom prefix. A prefix is not necessary but prevents potential conflicts with other targets, e.g. if you use target compile-options in one of your projects named foo and also use compile-options as a target in another project named bar, then when using foo and bar in the same project, one of the add_library(compile-options INTERFACE) commands will fail because the target compile-option already exists.

Use the Cray Compiler Wrappers¶

If you see a line similar to:

-- Cray Programming Environment <version and compiler>

at the beginning of the configuration step, then CMake has correctly identified the cray programming environment. By following the recommendations in this guide, you will not need to tell CMake how to find the programming environment manually. However, some CMakeLists.txt are configured to look for compilers in pre-defined places. This can lead to CMake not detecting the Cray programming environment. In this case, please specify the C, C++, and Fortran compilers manually:

$ CC=$(which cc) CXX=$(which CC) FC=$(which ftn) <cmake command>
...
<cmake output>

MPI¶

Provided by Cray compiler wrappers
- There is no need to do anything unique
- If you have an existing find_package(MPI), this will succeed (i.e. MPI_FOUND will be true) but variables such as MPI_CXX_FLAGS, etc. will be empty.

For general MPI support when not using the Cray compiler wrappers, you can use the following:

# FindMPI.cmake is provided by CMake
find_package(MPI REQUIRED)

# Later, when creating an actual library or executable:
add_executable(bar bar.cpp)
target_link_libraries(bar PUBLIC MPI::MPI_CXX)

If your target includes C sources, then also link to MPI::MPI_C. Similarly, link to MPI::MPI_Fortran for Fortran sources.

See the CMake documentation for more details: https://cmake.org/cmake/help/latest/module/FindMPI.html

OpenMP¶

Use the standard package for finding OpenMP:

find_package(OpenMP REQUIRED)

# Later, when creating an actual library or executable:
add_library(bar SHARED bar.cpp)
target_link_libraries(bar PUBLIC OpenMP::OpenMP_CXX)

If your target includes C sources, then also link to OpenMP::OpenMP_C. Similarly, link to OpenMP::OpenMP_Fortran for Fortran sources.

See the CMake documentation for more details: https://cmake.org/cmake/help/latest/module/FindOpenMP.html

Threading¶

If using non-OpenMP threading, the Intel compilers require adding the -pthread compilation flag.

# FindThreads.cmake options, provided by CMake
set(THREADS_PREFER_PTHREAD_FLAG ON)
find_package(Threads REQUIRED)

# Later, when creating an actual library or executable:
add_library(bar SHARED bar.cpp)
target_link_libraries(bar PUBLIC Threads::Threads)

See the CMake documentation for more details: https://cmake.org/cmake/help/latest/module/FindThreads.html

ScaLAPACK, BLACS etc. Packages with Cray Libsci¶

The Cray compiler wrappers implicitly link Cray LibSci which contains math libraries such as BLAS, LAPACK, ScaLAPACK, BLACS.
If your CMakeLists.txt file has a find_package command searching for a library contained in Cray LibSci and if CMake cannot find it, you can create a module file named Find<PackageName>.cmake in a directory specified by the CMAKE_MODULE_PATH variable and set the library and include directory paths there to an empty string. For example, if cmake fails with find_package(ScaLAPACK REQUIRED), the FindScaLAPACK.cmake file can be created with the contents:

set(${CMAKE_FIND_PACKAGE_NAME}_LIBRARIES "")
set(${CMAKE_FIND_PACKAGE_NAME}_INCLUDE_DIRS "")

Note that CMake should be able to "find" BLAS and LAPACK.

Linking CUDA Math Libraries¶

On Perlmutter, when the cudatoolkit module is loaded, CUDA math libraries are stored in the directory, ${CUDA_HOME}/../../math_libs. If you are using CMake to build a software package and you encounter error messages that contain statements such as,CUDA_curand_LIBRARY=NOTFOUND, it may be necessary to explicitly tell CMake to search the math_libs directory. This can be done by setting two environment variables before calling CMake as shown below:

module load cudatoolkit

export CMAKE_PREFIX_PATH=${CUDA_HOME}/../../math_libs:${CMAKE_PREFIX_PATH}
export CPATH=${CUDA_HOME}/../../math_libs/include:${CPATH}

Language Standards and Features¶

CMake knows the appropriate build flag(s) for the different language standards of "first-class" languages, e.g. C, C++, Fortran, and (in CMake 3.8+) CUDA.

These can be set globally:

# helpful for setting the language standard
set(CMAKE_C_STANDARD    11 CACHE STRING "C language standard")
set(CMAKE_CXX_STANDARD  11 CACHE STRING "C++ language standard")
set(CMAKE_CUDA_STANDARD 11 CACHE STRING "CUDA language standard")

option(CMAKE_C_STANDARD_REQUIRED    "Require the C language standard to set"    ON)
option(CMAKE_CXX_STANDARD_REQUIRED  "Require the C++ language standard to set"  ON)
option(CMAKE_CUDA_STANDARD_REQUIRED "Require the CUDA language standard to set" ON)

option(CMAKE_C_EXTENSIONS    "Enable/disable extensions, e.g. -std=gnu11 vs. -std=c11"     OFF)
option(CMAKE_CXX_EXTENSIONS  "Enable/disable extensions, e.g. -std=gnu++11 vs. -std=c++11" OFF)
option(CMAKE_CUDA_EXTENSIONS "Enable/disable extensions" OFF)

Or on a per-target basis:

add_library(foo SHARED foo.cu foo.cpp foo.c)
set_target_properties(foo PROPERTIES
    C_STANDARD                    99
    C_STANDARD_REQUIRED           ON
    C_EXTENSIONS                  OFF
    CXX_STANDARD                  11
    CXX_STANDARD_REQUIRED         ON
    CXX_EXTENSIONS                OFF
    CUDA_STANDARD                 11
    CUDA_STANDARD_REQUIRED        ON
    CUDA_EXTENSIONS               OFF
    CUDA_RESOLVE_DEVICE_SYMBOLS   ON
    CUDA_SEPARABLE_COMPILATION    ON
    CUDA_PTX_COMPILATION          OFF)
)

Build flag recommendations¶

CMake has the ability to check whether compiler flags are supported
- Use this in combination with INTERFACE libraries and you can use target_link_libraries(foo PUBLIC foo-compile-options) to ensure other CMake projects linking against foo will inherit the compile options or target_link_libraries(foo PRIVATE foo-compile-options) to ensure that the compile options are not inherited when linking against foo.
- Use the most recent CMake release to ensure that the CMake version was released after the compiler version
The following macros are useful

include(CheckCCompilerFlag)
include(CheckCXXCompilerFlag)

# create interface target with compiler flags
add_library(foo-compile-options INTERFACE)

#----------------------------------------------------------------------------------------#
# macro that checks if flag if supported for C, if so add to foo-compile-options
#----------------------------------------------------------------------------------------#
macro(ADD_C_FLAG_IF_AVAIL FLAG)
    if(NOT "${FLAG}" STREQUAL "")
        # create a variable for checking the flag if supported, e.g.:
        #   -fp-model=precise --> c_fp_model_precise
        string(REGEX REPLACE "^-" "c_" FLAG_NAME "${FLAG}")
        string(REPLACE "-" "_" FLAG_NAME "${FLAG_NAME}")
        string(REPLACE " " "_" FLAG_NAME "${FLAG_NAME}")
        string(REPLACE "=" "_" FLAG_NAME "${FLAG_NAME}")

        check_c_compiler_flag("${FLAG}" ${FLAG_NAME})
        if(${FLAG_NAME})
            target_compile_options(foo-compile-options INTERFACE
                $<$<COMPILE_LANGUAGE:C>:${FLAG}>)
        endif()
    endif()
endmacro()

#----------------------------------------------------------------------------------------#
# macro that checks if flag if supported for C++, if so add to foo-compile-options
#----------------------------------------------------------------------------------------#
macro(ADD_CXX_FLAG_IF_AVAIL FLAG)
    if(NOT "${FLAG}" STREQUAL "")
        # create a variable for checking the flag if supported, e.g.:
        #   -fp-model=precise --> cxx_fp_model_precise
        string(REGEX REPLACE "^-" "cxx_" FLAG_NAME "${FLAG}")
        string(REPLACE "-" "_" FLAG_NAME "${FLAG_NAME}")
        string(REPLACE " " "_" FLAG_NAME "${FLAG_NAME}")
        string(REPLACE "=" "_" FLAG_NAME "${FLAG_NAME}")

        # runs check to see flag is supported by compiler
        check_cxx_compiler_flag("${FLAG}" ${FLAG_NAME})
        if(${FLAG_NAME})
            target_compile_options(foo-compile-options INTERFACE
                $<$<COMPILE_LANGUAGE:CXX>:${FLAG}>)
        endif()
    endif()
endmacro()

#----------------------------------------------------------------------------------------#
# macro that checks if flag if supported for C and C++
#----------------------------------------------------------------------------------------#
macro(ADD_FLAGS_IF_AVAIL)
    foreach(FLAG ${ARGN})
        add_c_flag_if_avail("${FLAG}")
        add_cxx_flag_if_avail("${FLAG}")
    endforeach()
endmacro()

Provide options for enable AVX-512 flags and leak-detection flags

# ---------------------------------------------------------------------------- #
# options
option(USE_AVX512 "Enable AVX-512 architecture flags" OFF)
option(USE_SANTITIZER "Enable leak detection" OFF)

With the above macros and options, check the flags and when available, the macro appends these flags to foo-compile-options which are later "linked" to your library and/or executable which inherits the compile-options

# standard flags for C and C++
add_flags_if_avail("-W" "-Wall" "-Wextra" "-Wshadow")

# "new" keyword doesn't exist in C so no need to check
add_cxx_if_avail("-faligned-new")

# OpenMP SIMD-only (supported by GCC)
add_flags_if_avail("-fopenmp-simd")

# enable runtime leak detection
if(USE_SANITIZER)
    add_flags_if_avail("-fsanitize=leak")

    # emit warnings that this feature is not available
    if(NOT c_fsanitize_leak)
        message(WARNING "Sanitizer is not available for selected C compiler")
    endif()

    if(NOT cxx_fsanitize_leak)
        message(WARNING "Sanitizer is not available for selected C++ compiler")
    endif()
endif()

# check for AVX-512 flags
if(USE_AVX512)
    if(CMAKE_C_COMPILER_ID MATCHES "Intel")
        add_flags_if_avail("-xMIC-AVX512")
    else()
        # these flags are supported by newer GCC versions
        add_flags_if_avail("-mavx512f" "-mavx512pf" "-mavx512er" "-mavx512cd")
    endif()
endif()