Sanitizers and sanitizers4hpc¶

Introduction¶

LLVM Sanitizers are a group of debugging tools for detecting various kinds of bugs in C and C++ codes. There are multiple tools, including AddressSanitizer, LeakSanitizer, ThreadSanitizer, MemorySanitizer, each with a specific debugging capability.

A sanitizer consists of a compiler instrumentation module and a runtime library. To use a sanitizer, you first build an executable instrumented for the sanitizer, by specifying a compile flag. When the instrumented executable is run, the runtime intercepts relevant operations and inspects them. When it detects a problem, it generates a warning message.

Because of the instrumentation and the way how the debugging work is played out, memory usage can become several times bigger and the instrumented code can run several times slower. Therefore, it is important to rebuild your code without instrumentation after debugging is complete.

Supported Compilers¶

These tools can be used with more than just LLVM compilers: they are compatible with all compilers provided on Perlmutter, except the Nvidia compiler.

You don't need to change the way you compile your MPI code in order to use these tools (i.e., you can still use the Cray compiler wrappers cc/CC as normal). For a non-MPI code, the following C/C++ base compilers can be used, too.

GNU	Cray	Intel	AOCC	LLVM
`gcc`/`g++`	`craycc`/`craycxx`	`icx`/`icpx`	`clang`/`clang++`	`clang`/`clang++`

Note that Intel's icc and icpc do not work for the sanitizer tools as they are not Clang-based.

Sanitizer Flags¶

These compilers accept many LLVM sanitizer compile flags. Use the ones for your needs. For example, you don't have to instrument the entire code. Instead, you can exclude certain functions or source files from instrumentation with the -fsanitize-blacklist= or -fsanitize-ignorelist= option.

Runtime behavior of a tool can be controlled by setting the santizer environment variable to certain runtime flags. The variable is ASAN_OPTIONS for AddressSantizer, LSAN_OPTIONS for LeakSanitizer, TSAN_OPTIONS for ThreadSantizer, MSAN_OPTIONS for MemorySanitizer, etc.

You can find compile and runtime flags at the following web pages:

Below we show how to use some popular Sanitizers.

AddressSanitizer¶

AddressSanitizer is a memory error detector for C/C++. The tool can detect the following types of bugs:

Use after free (dangling pointer dereference)
Out of bounds array accesses to heap, stack and globals (heap buffer underflow/overflow, stack buffer underflow/overflow, global buffer underflow/overflow, respectively)
Use after return: use of a stack object after returning from the function where this object is defined
Use after scope: use of a stack object outside the scope it was defined
Initialization order bugs: non-deterministic outcome due to unspecified order in which constructors for global objects in different source files run
Double-free, invalid free
Memory leaks

To instrument for AddressSanitizer, compile and link your code with -fsantizer=address. Below is an example in the PrgEnv-gnu environment:

$ cat illegalmemoryaccess.cpp
#include <iostream>

int main(int argc, char **argv) {
  int *array = new int[10];

  for (int i = 0; i < 11; ++i) // Access more than allocated memory.
      array[i] = i+1;

  delete [] array;

  return 0;
}

$ g++ -O0 -g -fsanitize=address -fno-omit-frame-pointer illegalmemoryaccess.cpp

$ ./a.out
=================================================================
==2267569==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x604000000038 at pc 0x0000004009df bp 0x7ffe9e373680 sp 0x7ffe9e373678
WRITE of size 4 at 0x604000000038 thread T0
    #0 0x4009de in main /pscratch/sd/e/elvis/addresssanitizer/illegalmemoryaccess.cpp:7
    #1 0x7fbf17c3c24c in __libc_start_main (/lib64/libc.so.6+0x3524c)
    #2 0x4008b9 in _start ../sysdeps/x86_64/start.S:120

0x604000000038 is located 0 bytes to the right of 40-byte region [0x604000000010,0x604000000038)
allocated by thread T0 here:
    #0 0x7fbf188bba88 in operator new[](unsigned long) (/usr/lib64/libasan.so.8+0xbba88)
    #1 0x40097e in main /pscratch/sd/e/elvis/addresssanitizer/illegalmemoryaccess.cpp:4
    #2 0x7fbf17c3c24c in __libc_start_main (/lib64/libc.so.6+0x3524c)

SUMMARY: AddressSanitizer: heap-buffer-overflow /pscratch/sd/e/elvis/addresssanitizer/illegalmemoryaccess.cpp:7 in main
Shadow bytes around the buggy address:
  0x0c087fff7fb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c087fff7fc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c087fff7fd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c087fff7fe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c087fff7ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c087fff8000: fa fa 00 00 00 00 00[fa]fa fa fa fa fa fa fa fa
  0x0c087fff8010: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c087fff8020: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c087fff8030: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c087fff8040: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c087fff8050: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  ...
  Right alloca redzone:    cb
==2267569==ABORTING

The tool uses an extra memory block called shadow bytes for tracking the state of allocated memory blocks. With that, the tool correctly detects heap buffer overflow by 4 bytes. The line with the arrow in the shadow bytes map has 5 00's, meaning addressable 40bytes. The next shadow byte in the red zone is marked with the [fa] symbol, indicating out of bound access.

The next example shows detection of a "use after free" bug:

$ cat accessafterdelete.cpp
#include <iostream>

int main(int argc, char **argv) {
 int *array = new int[100];

  delete [] array;
  return array[10]; // access after delete.
}

$ g++ -O1 -g -fsanitize=address -fno-omit-frame-pointer accessafterdelete.cpp

$ ./a.out
=================================================================
==2315893==ERROR: AddressSanitizer: heap-use-after-free on address 0x614000000068 at pc 0x0000004009b6 bp 0x7ffc732d66f0 sp 0x7ffc732d66e8
READ of size 4 at 0x614000000068 thread T0
    #0 0x4009b5 in main /pscratch/sd/e/elvis/addresssanitizer/accessafterdelete.cpp:7
    #1 0x7f1e67a3c24c in __libc_start_main (/lib64/libc.so.6+0x3524c)
    #2 0x4008b9 in _start ../sysdeps/x86_64/start.S:120

0x614000000068 is located 40 bytes inside of 400-byte region [0x614000000040,0x6140000001d0)
freed by thread T0 here:
    #0 0x7f1e686bc498 in operator delete[](void*) (/usr/lib64/libasan.so.8+0xbc498)
    #1 0x400983 in main /pscratch/sd/e/elvis/addresssanitizer/accessafterdelete.cpp:6
    #2 0x7f1e67a3c24c in __libc_start_main (/lib64/libc.so.6+0x3524c)

previously allocated by thread T0 here:
    #0 0x7f1e686bba88 in operator new[](unsigned long) (/usr/lib64/libasan.so.8+0xbba88)
    #1 0x400978 in main /pscratch/sd/e/elvis/addresssanitizer/accessafterdelete.cpp:4
    #2 0x7f1e67a3c24c in __libc_start_main (/lib64/libc.so.6+0x3524c)

SUMMARY: AddressSanitizer: heap-use-after-free /pscratch/sd/e/elvis/addresssanitizer/accessafterdelete.cpp:7 in main
Shadow bytes around the buggy address:
  0x0c287fff7fb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c287fff7fc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c287fff7fd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c287fff7fe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c287fff7ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c287fff8000: fa fa fa fa fa fa fa fa fd fd fd fd fd[fd]fd fd
  0x0c287fff8010: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c287fff8020: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c287fff8030: fd fd fd fd fd fd fd fd fd fd fa fa fa fa fa fa
  0x0c287fff8040: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c287fff8050: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  ...
==2315893==ABORTING

Note that the freed 400-byte region is marked with fd and the memory block accessed erroneously is marked with [fd].

LeakSanitizer¶

LeakSanitizer is a run-time memory leak detector. It can be combined with AddressSanitizer to get both memory error and leak detection, or used in a stand-alone mode.

To use LeakSanitizer in stand-alone mode, link your program with -fsanitize=leak flag. Make sure to use clang (not ld) for the link step, so that it will link in the proper LeakSanitizer run-time library into the final executable.

For the examples with this tool, let's use the locally developed PrgEnv-llvm environment.

$ cat memory-leak.c
#include <stdlib.h>
void *p;
int main() {
  p = malloc(7);
  p = 0; // The memory is leaked here.
  return 0;
}

$ clang -fsanitize=leak -g -O0 memory-leak.c

$ ./a.out
=================================================================
==2335900==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 7 byte(s) in 1 object(s) allocated from:
    #0 0x55966653a842 in malloc /.../nersc/nersc-user-env/prgenv/llvm_src_17.0.6/compiler-rt/lib/lsan/lsan_interceptors.cpp:75:3
    #1 0x559666565898 in main /pscratch/sd/e/elvis/addresssanitizer/memory-leak.c:4:7
    #2 0x7efe8f83e24c in __libc_start_main (/lib64/libc.so.6+0x3524c) (BuildId: ddc393ac74ed8f90d4fdfff796432fbafd281e1b)

SUMMARY: LeakSanitizer: 7 byte(s) leaked in 1 allocation(s)

AddressSanitizer integrates LeakSanitizer and enables it by default. The next example shows this:

$ clang -fsanitize=address -g memory-leak.c

$ ASAN_OPTIONS=detect_leaks=1 ./a.out
=================================================================
==2339511==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 7 byte(s) in 1 object(s) allocated from:
    #0 0x56040740afde in malloc /.../nersc/nersc-user-env/prgenv/llvm_src_17.0.6/compiler-rt/lib/asan/asan_malloc_linux.cpp:69:3
    #1 0x560407447a68 in main /pscratch/sd/e/elvis/addresssanitizer/memory-leak.c:4:7
    #2 0x7fdab443e24c in __libc_start_main (/lib64/libc.so.6+0x3524c) (BuildId: ddc393ac74ed8f90d4fdfff796432fbafd281e1b)

SUMMARY: AddressSanitizer: 7 byte(s) leaked in 1 allocation(s)

ThreadSanitizer¶

ThreadSanitizer (TSan for short) detects data races among threads.

To instrument for ThreadSanitizer, compile and link your code with -fsantizer=thread. To get file names and line number in output, you need to add -g. For better performance, you can add -O1 or higher.

Below is an example of the tool catching a data race among OpenMP threads in the PrgEnv-gnu environment:

$ cat buggyreduction_omp.c
#include <stdio.h>

int main (int argc, char **argv) {
  int sum = 0;
  #pragma omp parallel for shared(sum)
  for (int i=0; i<1000; i++)
    sum += i;

  printf("sum = %d\n", sum);
  return 0;
}

$ cc -fsanitize=thread -g -O1 -fopenmp buggyreduction_omp.c

$ export OMP_NUM_THREADS=8
$ ./a.out
=================
WARNING: ThreadSanitizer: data race (pid=2240264)
  Read of size 4 at 0x7ffdf6e678bc by thread T1:
    #0 main._omp_fn.0 /pscratch/sd/e/elvis/sanitizers/buggyreduction_omp.c:6 (a.out+0x400895)
    #1 <null> <null> (libgomp.so.1+0x1dd4d)

  Previous write of size 4 at 0x7ffdf6e678bc by main thread:
    #0 main._omp_fn.0 /pscratch/sd/e/elvis/sanitizers/buggyreduction_omp.c:7 (a.out+0x4008aa)
    #1 GOMP_parallel <null> (libgomp.so.1+0x14e95)

  Location is stack of main thread.

  Location is global '<null>' at 0x000000000000 ([stack]+0x1e8bc)

  Thread T1 (tid=2240266, running) created by main thread at:
    #0 pthread_create <null> (libtsan.so.2+0x61be6)
    #1 <null> <null> (libgomp.so.1+0x1e38f)

SUMMARY: ThreadSanitizer: data race /pscratch/sd/e/elvis/sanitizers/buggyreduction_omp.c:6 in main._omp_fn.0
==================
sum = 335625
ThreadSanitizer: reported 1 warnings

Runtime behavior is controlled with the TSAN_OPTIONS environment variable. For info on runtime flags that can go with it, please see ThreadSanitizer Flags.

You may have to run an instrumented executable a few times because, when a race condition doesn't happen during a run, you will see no warning message even with a buggy code.

For more info on the tool, please check

According to a webpage above, the cost of race detection varies by program. But for a typical program, memory usage may increase by 5-10x and execution time by 2-20x.

MemorySanitizer¶

MemorySantizer (MSan) warns when a variable that has not been initialized is read.

To instrument for MemorySanitizer, compile and link your code with -fsantizer=memory. To get file names and line number in output, you need to add -g but use -O1 or higher optimization level for better performance. To get meaningful stack traces in error messages, add -fno-omit-frame-pointer. To get the accurate stack traces, you may need to disable inlining and use -fno-optimize-sibling-calls (for example, not to allow a recursive call to itself to be replaced with a for loop construct).

MemorySanitizer can track origins of uninitialized values. This feature is enabled by -fsanitize-memory-track-origins=2 (or simply -fsanitize-memory-track-origins) Clang option.

Note

The GNU compilers don't support MemorySanitizer.

Here's an example with the AOCC compilers using a test code found in the Clang documentation page:

$ cat umr.cc
#include <stdio.h>

int main(int argc, char** argv) {
  int* a = new int[10];
  a[5] = 0;
  if (a[argc])
    printf("xx\n");
  return 0;
}

$ CC -fsanitize=memory -fno-omit-frame-pointer -g -O1 umr.cc

$ ./a.out
==578284==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x2cf202 in main /pscratch/sd/e/elvis/sanitizers/umr.cc:6:7
    #1 0x7fc4fa63e24c in __libc_start_main (/lib64/libc.so.6+0x3524c)
    #2 0x24e4b9 in _start /home/abuild/rpmbuild/BUILD/glibc-2.31/csu/../sysdeps/x86_64/start.S:120

SUMMARY: MemorySanitizer: use-of-uninitialized-value /pscratch/sd/e/elvis/sanitizers/umr.cc:6:7 in main
Exiting

The MSAN_OPTIONS environment variable can control runtime behavior of the instrumented executable. Please see Sanitizer Flags for common flags.

Note that, with PrgEnv-cray and PrgEnv-intel, the example doesn't display the source code line info and this problem was reported to HPE. The problem can be fixed with

export MSAN_OPTIONS="allow_addr2line=true"

For more info on the tool, please check

according to which typical slowdown introduced by MemorySanitizer can be 3x.

sanitizers4hpc¶

HPE's sanitizers4hpc is an aggregation tool to collect and analyze LLVM Sanitizer output from a distributed-memory parallel (e.g., MPI) code at scale. It makes sanitizer's result easier to understand, by presenting output by group of MPI tasks sharing the same pattern.

Currently it supports

AddressSanitizer
LeakSanitizer
ThreadSanitizer

with the Cray and the GNU compilers. It also supports Nvidia Compute Sanitizer's Memcheck tool for CUDA codes (an example below).

To run an app with the tool, load the sanitizers4hpc module and then launch as follows:

sanitizers4hpc <sanitizers4hpc options> -- ./a.out <application arguments>

CPU Sanitizers¶

The following is to launch, using the launch argument flag -l (short for --launcher-args=), 2 MPI tasks (-n 2) with 2 OpenMP threads each over 2 compute nodes with an executable instrumented for ThreadSanitizer:

$ cat buggyreduction_mpiomp.c
#include "mpi.h"
#include <stdio.h>

int main (int argc, char **argv) {
  int rank;

  MPI_Init(&argc, &argv);
  MPI_Comm_rank(MPI_COMM_WORLD, &rank);

  int sum = 0;
  #pragma omp parallel for shared(sum)
  for (int i=0; i<1000; i++)
    sum += i;

  printf("%d: sum = %d\n", rank, sum);

  MPI_Finalize();
  return 0;
}

$ cc -fsanitize=thread -g -O1 -fopenmp buggyreduction_mpiomp.c

$ salloc -C cpu -N 2 -q debug -t 30
...

$ module load saniters4hpc

$ export OMP_NUM_THREADS=2

$ sanitizers4hpc -l "-n 2" -- ./a.out
0: sum = 499500
1: sum = 499500
RANKS: <1>
ThreadSanitizer: data race
  Read of size 4 at 0x7fff57a3313c by thread T1:
    #0  main._omp_fn.0 /pscratch/sd/e/elvis/sanitizers/buggyreduction_mpiomp.c:12 (a.out+0x400a55)
    #1    (libgomp.so.1+0x1dd4d)
...
RANKS: <0>
ThreadSanitizer: data race
  Read of size 4 at 0x7ffddd5bccfc by thread T1:
    #0  main._omp_fn.0 /pscratch/sd/e/elvis/sanitizers/buggyreduction_mpiomp.c:12 (a.out+0x400a55)
    #1    (libgomp.so.1+0x1dd4d)
...

GPU Sanitizer¶

The tool supports Compute Santizer's Memcheck tool only. Racecheck, Initcheck and Synccheck are not supported.

Here is an example of running a MPI+CUDA code in the PrgEnv-gnu environment. It launches 4 MPI tasks (-n 4), each with 1 GPU (--gpus-per-task=1) on a single GPU node.

$ salloc -C gpu -N 1 --gpus-per-node=4 -q debug -t 30 -A <allocation>
...

$ module load sanitizers4hpc

$ sanitizers4hpc -l "-n 4 -c 32 --cpu-bind=cores --gpus-per-task=1 --gpu-bind=none" -m ${CUDA_HOME}/compute-sanitizer/compute-sanitizer -f -- ./a.out
RANKS: <2,3>
...
Saved host backtrace up to driver entry point at error
    #0 0x2eae6f in /usr/local/cuda-12.2/compat/libcuda.so.1
    #1 0xd8f0 in /home/jenkins/src/gtlt/cuda/gtlt_cuda_query.c:325:gtlt_cuda_pointer_type /opt/cray/pe/lib64/libmpi_gtl_cuda.so.0
...
RANKS: <0-1>
...
Saved host backtrace up to driver entry point at error
    #0 0x2eae6f in /usr/local/cuda-12.2/compat/libcuda.so.1
    #1 0xd8f0 in /home/jenkins/src/gtlt/cuda/gtlt_cuda_query.c:325:gtlt_cuda_pointer_type /opt/cray/pe/lib64/libmpi_gtl_cuda.so.0
...

The -f flag is to bypass sanitizers4hpc's check that the executable is instrumented with a LLVM sanitizer. Since the code above is not and the intention is to use Compute Sanitizer's Memcheck tool only, the flag is used here.

Note

Aggregation of run output is not perfect (especially with ThreadSanitizer and Compute Sanitizer) and we reported this to HPE.

Please check the man page (man sanitizers4hpc) for more information.