Compiling a code on NERSC resources¶

Purpose¶

This playbook will guide you through the steps to compile a code on NERSC's Perlmutter supercomputer.

Overview of process¶

From the perspective of a person compiling code, there are three phases in the code compilation process.

First, we will set up the environment to prepare it to compile the code, by figuring out the dependencies and other requirements, and then customizing the build settings to enable this.

Second, we will build the application by compiling human-readable code into machine-readable object files, and linking object files and libraries into a binary.

Finally, we will install the code in the appropriate location, where you can access it and run it within a job (batch or interactive).

How code is compiled (from a technical perspective)¶

Compiling a code is a multistep process. First, the compiler translates human-readable code into machine-readable object files. To do this, it may need additional inputs, called "include" or "header" files. The location of these files, if they are not included in the compiler's default search directories, must be provided to the compiler and are denoted with a -I flag on the compile line (e.g., -I /path/to/include). The products of this compilation process, called object files, are usually named with a .o suffix. They may be linked together into a library file in an intermediate step.

The final step is to combine object files and library files into the final executable. This step is known as linking. During this process, the linker makes sure that all variables and functions or subroutines that are used in the code are defined. It searches for the required libraries in default directories and directories provided by the user with the -L flag (e.g., -L /path/to/libs). The names of required libraries beyond the defaults are denoted by -lname (for a library file with the actual name libname.a or libname.so).

An executable may be statically linked or dynamically linked. A statically linked executable is self-contained; all object files and library files are bundled together into a single large file. During link time, the linker searches for library files terminating in .a to include in the executable. On the other hand, a dynamically linked executable is generally smaller and contains a list of directories where the shared-object libraries (generally terminating in a .so suffix) reside. When the dynamically linked executable is invoked, it loads the necessary libraries at runtime.

Steps¶

1. Determine dependencies and requirements.
2. Download code to be compiled.
3. Load modules as appropriate.
4. Set up build configuration.
5. Build and install the code.

Detailed instructions¶

Step 1: Determine dependencies and requirements¶

Look at the documentation for the code you want to compile. There may be documentation available online, or else there could be a file included in the package called README or INSTALL or something similar. This is a crucial first step -- usually the developers will explain how to compile the code. Even if it doesn't make much sense at first, reading the instructions and following that by reading this playbook should help to clarify matters.

Something important to examine is the code's dependency (if any) on other software. Does the code have any dependencies on other libraries? Does it use MPI for parallelization? Does it use any I/O libraries? Does it use numerical libraries? Does it require BLAS subroutines?

If your code uses MPI, then you will use the ftn, cc, and/or CC compiler wrappers for code written in Fortran, C, or C++ (respectively) when compiling.

NERSC may have other common dependencies available through its module system. Use module spider <example> to search for a package called <example>. If you find one you're looking for, make a note of it, as in the third step, we will load that module.

BLAS subroutines are available in the Cray libsci module and will be linked into the final executable automatically if you are using the compiler wrappers.

If you cannot find a particular package, you may need to compile it yourself. In this case, repeat this step focused on that dependency.

Step 2: Download code to be compiled¶

Download the packages that you need to install the code. Common commands for accessing code from a public repository include

• wget https://url.for.code/path/to/code/code-version.tar.gz (for a code that provides tar files as releases)
• git clone https://github.com/project/code.git (for a code with a repository on Github that can be cloned)
• scp elvis@another.cluster:mydir/myfile.tar.gz ./mydir (for copying a gzipped tar file from another.cluster to the directory mydir, invoking this command from a NERSC machine. You'll need to authenticate to the other cluster.)

If the code you've downloaded is a tar file or a zip file, you'll need to expand it. The following table provides useful commands for this.

Make sure your source code ends up in an appropriate location. You can download a tar file, cd to the directory where you want the source code, and then invoke the command to expand the downloaded file (be sure to provide the path to the downloaded file in your command!). Many users store source code in their home directory or in their project CFS directory. If you download it to scratch, it could be purged from the system!

Step 3: Load modules¶

In Step 1, we determined which modules would be relevant and useful for this code. At this point, we now load the modules. Note that some modules may require prerequisite modules to be loaded.

Step 4: Set up build configuration¶

There are many different build systems that are used to compile. The most common ones are Makefiles, Autoconf, and CMake.

Makefiles¶

A makefile orchestrates the compilation of object files and their linking into libraries or executables. For some applications, makefiles are provided that need to be customized for the platform. For tips on customizing makefiles, see the section on editing makefiles in the NERSC build tools documentation.

Autoconf¶

This is the canonical method for building code, and often the simplest. An automated script called configure sets up the configuration of makefiles.

The configure script takes options, which set variables in the makefile. For more information about configure as applied in the NERSC environment, please see the NERSC Autoconf Documentation

The configuration process could fail if settings are not quite right. Look for error outputs and adjust configuration settings accordingly. Reinvoking configure overwrites any previous settings.

CMake¶

CMake is similar to Autotools in the sense that it is an automation that sets up the configuration of Makefiles. But, it is a much more powerful tool that can do much more complex processes. For a typical user, we can ignore most of its features and focus on a minimum set.

CMake can sometimes have trouble finding the Cray compiler wrappers, so we recommend invoking it in the following way:

CC=$(which cc) CXX=$(which CC) FC=$(which ftn) <cmake command>

For more information on using CMake, please see the NERSC CMake Documentation.

Step 5: Build and install the code¶

For most codes that used makefiles or autoconf, simply typing make on the command line begins the build process. We recommend keeping a record of what took place during the compilation (especially handy when there's been a build error!) by redirecting the output of the make command to a file.

make > make.log &

(The optional ampersand backgrounds the process, which is handy if you want to do something else in the shell while you wait for your code to compile.) If you want to watch the text scroll by as it happens, use the tee command to send the make output to both a file and the screen:

make | tee make.log

If there are issues in the compilation, you can generally wipe the slate clean by invoking make clean, which removes object files that were compiled, then start over.

For CMake, the command could instead be cmake --build . (or something similar) and as the compilation progresses a percent progress is reported.

After our executable has been successfully compiled, it can be installed with the make install command. (Check your instructions -- this step is not always required.) The most common error in this phase occurs when the install directory has not been set properly. Build systems often default to a system-level installation directory, which is not accessible to NERSC users. Correcting the installation directory (often called prefix in the case of autobuild) in the configuration stage will solve this problem.

Troubleshooting¶

Even the most experienced NERSC staff encounter errors as they proceed to build a new piece of software the first time. Sometimes instructions are incomplete, or the uniqueness of the system environment results in a unique situation. Fortunately, error messages from build tools and compilers are often helpful.

Because build errors are common, you can often find out how to solve your problem by using your favorite search engine to browse for matches to key words in your error message. And of course, you can submit a ticket for any issue that has you truly stumped.