Articles

Case study: From “We need builds” to nightly releases on every system

A build script that builds Python, installs Conan packages, and configures a C++ project does three jobs in one language. A second operating system brings a second script in another language. A CMake superbuild moves that orchestration into one top-level project that compiles nothing and only sequences the stages, so the same code runs on Windows and Linux.

We specified, built, and run the release system for a customer’s large simulation framework and the customer’s plugins, libraries, and demos. One superbuild pulls all of them in and ships nightly releases for Windows and RHEL 7, 8, and 9. The case study is From “We need builds” to nightly releases on every system.

Put the orchestration in CMake, not in shell scripts

Make and a shell script run commands; neither knows anything about compilers, platforms, or dependency packages. CMake is a generator: it writes Makefiles, Ninja files, or Visual Studio projects for each platform from one description.

A superbuild applies that idea one level up. The top-level CMake project builds other projects in order and records when each step is complete. That gives the orchestration properties a script has to grow by hand:

• One language on every host. CMake runs the same code on Windows and Linux, so there is one copy to edit and test.
• Incremental steps. ExternalProject writes a stamp after each step, and a step runs again only when one of its inputs changes.
• Correct ordering of configure steps. The inner project calls find_package(Python), so Python has to exist before that configure starts. In a superbuild the inner project configures at build time of the outer one, after the embedded Python is in place.

That last point is what “modern CMake” means in practice here. The dependency graph is data CMake can see and enforce, not an order hidden in the lines of a script.

Sequence three stages under a project that compiles nothing

The superbuild builds its own Python and then everything that depends on it. The top-level project declares no languages, project(app_superbuild NONE), and compiles nothing itself.

Stage 1 builds CPython from a submodule, and a second step installs the pip requirements into it. Both steps have stamps. The pip step lists the requirements files as dependencies, so editing one runs provisioning again and nothing else.

Stage 2 runs conan install with the Conan that lives inside the embedded Python, and writes the Conan toolchain file for the inner project.

Stage 3 is the inner C++ project. It receives the Conan toolchain file and a preload cache script that points every Python lookup at the embedded interpreter. Our packaging code enters through an extension-path variable the inner project already supported, so the inner project itself is untouched.

A simplified top level, written for this post:

cmake_minimum_required(VERSION 3.29)
project(app_superbuild NONE)
include(ExternalProject)

file(READ "${CMAKE_SOURCE_DIR}/build_config.json" cfg)
string(JSON py_version GET "${cfg}" python version)
# Sets profile and lockfile (the Linux part is in the next section)
include("${CMAKE_SOURCE_DIR}/cmake/select_profile.cmake")
set(APP_BIN "${CMAKE_BINARY_DIR}/app")

ExternalProject_Add(embedded_python
  SOURCE_DIR "${CMAKE_SOURCE_DIR}/external/cpython"
  INSTALL_DIR "${CMAKE_BINARY_DIR}/python"
  CONFIGURE_COMMAND <SOURCE_DIR>/configure --prefix=<INSTALL_DIR>
  BUILD_COMMAND make
  INSTALL_COMMAND make install)
ExternalProject_Add_Step(embedded_python provision
  COMMAND "${CMAKE_BINARY_DIR}/python/bin/python3" -m pip install
          -r "${CMAKE_SOURCE_DIR}/requirements.txt"
  DEPENDS "${CMAKE_SOURCE_DIR}/requirements.txt"
  DEPENDEES install)

add_custom_command(OUTPUT "${APP_BIN}/conan_toolchain.cmake"
  COMMAND "${CMAKE_COMMAND}" -DPROFILE=${profile} -DLOCKFILE=${lockfile}
          -DOUTPUT_DIR=${APP_BIN}
          -P "${CMAKE_SOURCE_DIR}/cmake/run_conan_install.cmake"
  DEPENDS "${CMAKE_SOURCE_DIR}/conanfile.py" "${profile}")
add_custom_target(conan_install DEPENDS "${APP_BIN}/conan_toolchain.cmake")
add_dependencies(conan_install embedded_python)

ExternalProject_Add(app
  SOURCE_DIR "${CMAKE_SOURCE_DIR}/core"
  BINARY_DIR "${APP_BIN}"
  CMAKE_ARGS -DCMAKE_TOOLCHAIN_FILE=${APP_BIN}/conan_toolchain.cmake
             -DAPP_EXTENSION_PATH=${CMAKE_SOURCE_DIR}/cmake/packaging
  DEPENDS conan_install)

Because every host builds and runs the same interpreter, every host also runs the same Conan. With the packaging in CMake, a Python upgrade is a small, testable edit: bump the submodule and rebuild.

Select the profile, lockfile, and build directory from the OS

A single JSON file holds the build config. The wrapper scripts read it, and CMake reads the same file with string(JSON), so there is no second copy of the version number or the profile names to drift.

On Linux, CMake reads /etc/os-release and extracts the Enterprise Linux major version. That one value selects the Conan profile, the lockfile, and the build directory name. The profile and lockfile names below are illustrative:

# RHEL 8 and 9 set PLATFORM_ID, for example "platform:el9"
file(STRINGS "/etc/os-release" platform REGEX "^PLATFORM_ID=")
string(REGEX MATCH "el([0-9]+)" _ "${platform}")
set(el_major "${CMAKE_MATCH_1}")
if(NOT el_major)
  # RHEL 7 has no PLATFORM_ID; take the major version from VERSION_ID
  file(STRINGS "/etc/os-release" version REGEX "^VERSION_ID=")
  string(REGEX MATCH "[0-9]+" el_major "${version}")
endif()
set(profile "${CMAKE_SOURCE_DIR}/profiles/linux-el${el_major}-release")
set(lockfile "${CMAKE_SOURCE_DIR}/locks/linux-el${el_major}.lock")

That is how one source tree serves a RHEL 8 build container and a RHEL 9 build container. Adding RHEL 9 took a pair of profiles, a lockfile, and a container definition: about a day from the decision to a shipped release. The RHEL 8 names stayed as they were, so RHEL 8 users saw no change.

The build directory name itself comes from one place. A single CMake script computes it from the version and the platform, and CMake, the wrapper scripts, and a DLL-check script all call that script, so the logic exists once.

Upgrade the framework with a version swap

The framework core is one entry in the superbuild, pinned to an upstream release. An upgrade swaps that version. The superbuild pulls the new release and builds every plugin, library, and demo against it, on every operating system, in the next pipeline.

The customer carries only a couple of its own commits on the framework. We cherry-pick them onto the latest framework release branch, then update any plugin that used an interface the new version deprecated. We have upgraded the framework four times this way.

A simplified sequence, written for this post:

# Carry the customer's commits onto the new framework release branch
git -C core fetch upstream --tags
git -C core switch -c customer/next upstream/release-next
git -C core cherry-pick <customer-commit-1> <customer-commit-2>
git add core && git commit -m "Framework: move to the next upstream release"

Keep the customer’s changes to the framework this small on purpose. Every commit carried on top of upstream is a commit to cherry-pick at each upgrade; code that belongs to the customer lives in its own plugin and library repositories.

Pin the Conan graph: lockfiles, os.distro, skipped binaries

These Conan settings keep the dependency graph identical from one host to the next. Each one lives in the profiles or in a file the superbuild copies into place, so a fresh cache starts from the same state as every other.

Lockfile per OS and distro

A strict lockfile rejects any requirement it does not contain. Qt pulls X11 and xcb packages on Linux only, so one Windows lockfile cannot serve a Linux install.

On Linux we treat the Windows lockfile as a partial lock and derive a Linux lockfile from it:

conan install . --lockfile=app.lock --lockfile-partial \
                --lockfile-out=app.linux.lock

Distro in the package ID

Every Linux program depends on glibc, the system C library, and a binary built against a newer glibc will not load on an older one. RHEL 9 ships glibc 2.34 and RHEL 8 ships glibc 2.28, so a RHEL 9 binary that uses newer glibc symbols will not load on RHEL 8. Conan’s default settings still give a RHEL 8 build and a RHEL 9 build the same package ID.

Without a distro setting, whichever build uploads last replaces the other on the remote, with no error at upload time. A user setting in settings_user.yml puts the distro into the package ID (values illustrative):

os:
  Linux:
    distro: [null, el8, el9]

Each Linux profile then sets os.distro, and the superbuild copies settings_user.yml into the Conan home before every install, so a fresh cache always has it.

Skipped static binaries

Conan 2 skips downloading the binary of a static library when only shared libraries consume it, and the double-conversion library inside Qt 6 is one such case. Without that binary, the inner project’s code generator finds an empty package folder and the configure stops, far from anything that looks like a Conan setting. The tools.graph:skip_binaries=False setting makes Conan fetch those binaries; it lives in the profiles and on the command line.

Package for a host with nothing installed

CPack produces the release archives. The embedded Python installs into bin/, with its DLLs and a ._pth file next to the executable on Windows. The application therefore runs on a target with no Python at all, which an air-gapped environment requires.

Ship the GCC runtime beside the binaries

The C++ runtime needs the same treatment, because GCC 13 binaries can require GLIBCXX_3.4.32. RHEL 8 ships GCC 8, whose runtime stops at GLIBCXX_3.4.25, and RHEL 9 ships GCC 11, which stops at GLIBCXX_3.4.29. We bundle the GCC runtime libraries in a lib/ directory and point each binary’s RPATH at it.

We bundle the runtime rather than link libstdc++ statically into each library. The application loads plugins with dlopen, so each plugin would otherwise carry its own copy of the runtime and its own type-info tables. Cross-library dynamic_cast and exception propagation need one shared runtime.

The embedded CPython builds with --enable-shared, so its executable needs an RPATH of its own, or the loader can pick up a different libpython from the host. We set it through LDFLAGS at configure time as $ORIGIN/../lib, relative to the executable, so the install can move. readelf -d python3 | grep -i path confirms it.

Smoke-test the package before it ships

The build finishes with a smoke test. It hides the build-time GCC from the linker cache, clears LD_LIBRARY_PATH, and runs the packaged executable. A missing runtime library then shows up in the pipeline rather than on a user’s machine:

# Run the packaged binary with no help from the build host
env -u LD_LIBRARY_PATH "${PKG}/bin/app" --version
readelf -d "${PKG}/bin/python3" | grep -iE 'rpath|runpath'

RHEL 7 releases come from the same superbuild, run inside a toolchain container where every tool is built from source against the cluster’s system libraries. That path has its own guide: Running modern software on an operating system past end of maintenance.

Release nightly from one CI template

A scheduled pipeline builds a release every night from everything developers merged that day, so a change reaches users the day it lands. It first moves each plugin and library repository to the head of its branch, with overrides for any branch a release pins. The release report records the commit every repository built from, so any release can be rebuilt exactly.

The Linux CI has one job template for RHEL 8 and RHEL 9. Only the container image differs, because the superbuild selects everything else from /etc/os-release. Each job keeps three caches: the Conan cache keyed on the hash of the conanfile, a ccache directory, and the build directory.

On every branch pipeline, the job script first checks that every submodule sits at the commit the branch records. A submodule left at an older commit compiles old code without any warning:

if git submodule status --recursive | grep '^+'; then
  echo "submodule not at recorded commit" >&2; exit 1
fi

The pipeline runs the unit, integration, and scenario tests and packages their reports into the release. A coverage job builds the whole product instrumented and runs the unit and integration tests inside it; every pipeline reports coverage, and the main branch publishes the browsable report.

Keep the Windows workspace on a fixed path

The Windows build tree is 38 GB. A tree that size belongs in a kept workspace on the runner disk, not in a CI cache archive, so the Windows job pins one workspace at a fixed GIT_CLONE_PATH.

The path has to be fixed because CMake stores absolute paths in the build tree. The default path contains the runner token, which changes whenever someone registers the runner again. Incremental MSBuild in a tree that never moves keeps Windows builds fast.

Results

How we measured: the effort, cadence, RHEL 9 time, and delivery reach come from our own record of the work; the operating systems, framework versions, and report contents come from the release records of the system we built and run; the coverage report is the one the pipeline publishes.

Recommendations

• Release nightly from the head of every branch and record each repository’s commit in the release, so every release is current and can be rebuilt exactly.
• Keep your changes to an upstream framework to a few commits, so each upgrade is a version swap and a short cherry-pick.
• Build the interpreter inside the superbuild, so every host runs the same Python and the same Conan, and a Python upgrade is a small, testable edit.
• Add os.distro to the Conan package ID before the first upload, so two distros never share an ID; the cost is one settings_user.yml and a line per profile.
• Set tools.graph:skip_binaries=False in every profile before the first build, because the missing binary otherwise appears as a missing helper call in a generated CMake file, two layers from the setting.
• Smoke-test the packaged binary with the build-time toolchain hidden and LD_LIBRARY_PATH cleared, so a missing runtime library shows up in the pipeline, not on a user’s machine.

Commercial teams: Industry

References

• CMake ExternalProject module
• CMake string(JSON) command
• CMake CPack module
• Conan 2 lockfiles
• Conan 2 settings and settings_user.yml
• Conan 2 global configuration, including tools.graph:skip_binaries
• GCC: libstdc++ ABI policy and symbol versions
• GitLab Runner custom build directories
• Using Python on Windows: the embeddable layout and ._pth files

If your software has to build and release on Windows, RHEL, Ubuntu, or whatever else your program runs, see Build and release engineering or get a free estimate.

Related case studies