Darshan eXtended Tracing (DXT)¶

Darshan provides an extended tracing module (DXT) that can be enabled at runtime to capture fine-grain I/O traces. DXT is not enabled by default when using Darshan because it imposes an additional overhead as any other tracing tool. Nonetheless, such additional information has proven useful in understanding I/O behavior and pinpointing root causes of performance inefficiencies.

Once enabled, DXT collects detailed traces from the POSIX and MPI-IO layers reporting the operation (write/read), the rank that issued the call, the segment, the offset in the file, and the size of each request. It also captures the start and end timestamps of all the operations issued by each rank. To enable DXT explorer at NERSC, make sure to follow the guidelines available at the Darshan docs (how to compile your application with darshan, how to enable it for non-MPI applications, etc), then define the DXT_ENABLE_IO_TRACE environment variable in your submission script like so:

export DXT_ENABLE_IO_TRACE=1

Accessing I/O Traces¶

The DXT module saves the tracing information inside the same .darshan profiling log file; refer to the Darshan documentation to find the logs produced by your applications.

Similar to regular profiling logs, Darshan DXT traces can be processed to produce a plain text report. For example, given $LOGFILE, an environment variable storing the .darshan file, you can parse it with darshan-dxt-parser, a command available in the darshan module loaded by default:

darshan-dxt-parser $LOGFILE

!!! tip As a trace, the output will be quite long, especially if the application has accessed several files or made small I/O requests during a long run. Redirect the output to a file (e.g., > $PARSED_LOGFILE) or pipe it to other commands for better reading (e.g., | less).

Visualizing I/O Traces with DXT Explorer¶

DXT Explorer is an interactive web-based log analysis tool to visualize Darshan DXT logs, and helps understanding the I/O behavior of applications. It adds an interactive component to Darshan which can aid researchers, developers, and end-users visually inspect their applications’ I/O behavior, zoom-in on areas of interest and have a clear picture of where I/O problems may be. The complete documentation can be accessed at dxt-explorer.readthedocs.io.

To create interactive visualizations with DXT Explorer at NERSC you can use a Shifter container:

shifter --image=docker:hpcio/dxt-explorer -- dxt-explorer $LOGFILE

Several options are available in dxt-explorer:

usage: dxt-explorer [-h] [-o OUTPUT] [-t] [-s] [-d] [-l] [--start START] [--end END] [--from START_RANK] [--to END_RANK] darshan

DXT Explorer:

positional arguments:
  darshan               Input .darshan file

optional arguments:
  -h, --help            show this help message and exit
  -o OUTPUT, --output OUTPUT
                        Name of the output file
  -t, --transfer        Generate an interactive data transfer explorer
  -s, --spatiality      Generate an interactive spatiality explorer
  -d, --debug           Enable debug mode
  -l, --list            List all the files with tracing data
  --start START         Report starts from X seconds (e.g., 3.7) from the beginning of the job
  --end END             Report ends at X seconds (e.g., 3.9) from the beginning of the job
  --from START_RANK     Report start from rank N
  --to END_RANK         Report up to rank M

DXT Explorer will generate some .html files you can copy to your local device and open in any web browser.

@inproceedings{dxt-explorer,
   title = {{I/O Bottleneck Detection and Tuning: Connecting the Dots using Interactive Log Analysis}},
   author = {Bez, Jean Luca and Tang, Houjun and Xie, Bing and Williams-Young, David and Latham, Rob and Ross, Rob and Oral, Sarp and Byna, Suren},
   booktitle = {2021 IEEE/ACM Sixth International Parallel Data Systems Workshop (PDSW)},
   year = {2021},
   volume = {},
   number = {},
   pages = {15-22},
   doi = {10.1109/PDSW54622.2021.00008}
}