CTest/CDash without CMake

In general, four files are needed to test and submit to a CDash dashboard:

  1. CTestConfig.cmake
  2. CTestCustom.cmake
  3. CTestTestfile.cmake

These files can be generated by hand. The detailed contents and format of the above files is beyond the scope of this documentation but can be found here.

NERSC provides a Python package PyCTest to streamline the generation of the above files. This package enables easily wrapping any build system into a CDash submission without CMake and generate tests. For example:

  • Makefiles
    • pyctest.pyctest.CONFIGURE_COMMAND = "./configure"
    • pyctest.pyctest.BUILD_COMMAND = "make -j8"
  • Python Project
    • pyctest.pyctest.BUILD_COMMAND = "python setup.py install"

Using PyCTest

Name Version Platforms Downloads
Conda Recipe Conda Version Conda Platforms Conda Downloads

Install pyctest from Anaconda or PyPi

  • conda install -c conda-forge pyctest
  • python -m pip install -v pyctest

PyCTest does not simply implement routines for writing the necessary files, instead the package creates direct bindings to the CMake/CTest source code. The Anaconda distributions are pre-compiled and recommended. The PyPi distribution is a source distribution that will require building CMake from scratch -- which can take a significant amount of time depending on your system specifications.

Write a primary driver script for the project, e.g. pyctest-runner.py, and place in the top-level directory of the project.

PyCTest Example for Python Project

#!/usr/bin/env python

import os
import sys
import platform
import pyctest.pyctest as pyctest
import pyctest.helpers as helpers

parser = helpers.ArgumentParser("ProjectName", source_dir=os.getcwd(),binary_dir=os.getcwd(), vcs_type="git")
args = parser.parse_args()

pyctest.BUILD_NAME = "{}".format(args.build)
pyctest.BUILD_COMMAND = "python setup.py install"

test = pyctest.test()
test.SetName("unittest")
test.SetCommand(["nosetests"])

pyctest.run()

PyCTest Example for autotools Project

#!/usr/bin/env python

import os
import sys
import platform
import multiprocessing as mp
import pyctest.pyctest as pyctest
import pyctest.helpers as helpers

parser = helpers.ArgumentParser("ProjectName", source_dir=os.getcwd(), binary_dir=os.getcwd(),
                                vcs_type="git")
parser.add_argument("-n", "--build", type=str, required=True, help="Build name for identification")
args = parser.parse_args()

# CONFIGURE_COMMAND can only run one command so if autogen is required, just execute it here
cmd = pyctest.command(["./autogen.sh"])
cmd.SetWorkingDirectory(pyctest.SOURCE_DIRECTORY)
cmd.SetErrorQuiet(False)
cmd.Execute()

pyctest.BUILD_NAME = "{}".format(args.build)
pyctest.CONFIGURE_COMMAND = "./configure"
pyctest.BUILD_COMMAND = "make -j{}".format(mp.cpu_count())

# alternate test declaration format
pyctest.test("unittest", ["./run-testing.sh"])

pyctest.run()

PyCTest Example for CMake Project

#!/usr/bin/env python

import os
import sys
import platform
import multiprocessing as mp
import pyctest.pyctest as pyctest
import pyctest.helpers as helpers

project = "PyCTestDemo"
binary_dir = os.path.join(os.getcwd(), "{}-build".format(project))
parser = helpers.ArgumentParser("ProjectName", os.getcwd(), binary_dir)
parser.add_argument("-n", "--build", type=str, required=True, help="Build name for identification")
args = parser.parse_args()

pyctest.BUILD_NAME = "{}".format(args.build)
pyctest.UPDATE_COMMAND = "git"
pyctest.CONFIGURE_COMMAND = "cmake {}".format(pyctest.SOURCE_DIRECTORY)
pyctest.BUILD_COMMAND = "cmake --build {} --target all -- -j{}".format(
    pyctest.BINARY_DIRECTORY, mp.cpu_count())

pyctest.test("unittest", ["./run-testing.sh"])

pyctest.run()

PyCTest Example for CMake Project

Write a primary driver script for the project, e.g. pyctest-runner.py, and place in the top-level directory of the project.

#!/usr/bin/env python

import os
import sys
import platform
import multiprocessing as mp
import pyctest.pyctest as pyctest
import pyctest.helpers as helpers

project = "PyCTestDemo"
binary_dir = os.path.join(os.getcwd(), "{}-build".format(project))
parser = helpers.ArgumentParser("ProjectName", os.getcwd(), binary_dir)
parser.add_argument("-n", "--build", type=str, required=True, help="Build name for identification")
args = parser.parse_args()

pyctest.BUILD_NAME = "{}".format(args.build)
pyctest.UPDATE_COMMAND = "git"
pyctest.CONFIGURE_COMMAND = "cmake {}".format(pyctest.SOURCE_DIRECTORY)
pyctest.BUILD_COMMAND = "cmake --build {} --target all -- -j{}".format(
    pyctest.BINARY_DIRECTORY, mp.cpu_count())

pyctest.test("unittest", ["./run-testing.sh"])

pyctest.run()