Articles

Case study: Current software running on RHEL 7, past end of maintenance

Red Hat has ended the maintenance phase of Red Hat Enterprise Linux (RHEL) 7. Not every system can move when it does: a customer’s HPC cluster still ran RHEL 7, and a C++20 simulation framework had to run there.

This guide shows how we got current software onto that cluster with nothing installed on it. It covers why new binaries will not load on an old system, the container that builds every tool from source, packaging, testing, and delivery. The customer side is in the case study Current software running on RHEL 7, past end of maintenance.

Find the newest library version a binary needs

Every Linux program calls the system C library, glibc (the GNU C library), for basics such as opening files and allocating memory. Each function in it carries a version tag, and a binary records the tag of each library function it uses. The loader refuses to start a binary whose tags the system does not provide.

RHEL 7 ships glibc 2.17 and the C++ runtime of GCC 4.8. A binary built on a current system asks for newer tags, so on RHEL 7 it stops with an error such as version `GLIBC_2.34' not found or version `GLIBCXX_3.4.32' not found. This command prints the newest C library tag a binary needs:

# Newest glibc symbol version each file asks for; RHEL 7 provides up to 2.17
objdump -T ./app ./lib/*.so* 2>/dev/null \
  | grep -o 'GLIBC_[0-9.]*' | sort -uV | tail -1

The C++ runtime works the same way. The libstdc++ ABI table lists the tag each GCC release adds: GCC 4.8.3 and later provide up to GLIBCXX_3.4.19, while GCC 13.2.0 adds GLIBCXX_3.4.32.

Inventory the target before you build

Write down what the cluster has before you choose any version. Four facts decide the toolchain: the C library version, the newest C++ runtime tag, the CPU features of the oldest node, and what the scheduler expects of a job. Run these checks on a login node and on a compute node, because the two can differ:

cat /etc/redhat-release                     # exact operating system release
ldd --version | head -1                     # C library version
strings /usr/lib64/libstdc++.so.6 \
  | grep -o 'GLIBCXX_[0-9.]*' | sort -uV | tail -1   # newest C++ runtime tag
grep -o -w -m1 'avx2' /proc/cpuinfo         # vector features on this node

Compile for the oldest CPU in the cluster. A binary built with -march=native on a new workstation can use instructions an older node lacks, and it stops with an illegal-instruction error there. The generic x86-64 target, or an instruction-set level every node supports, avoids that.

Choose where the new toolchain lives

There are three common ways to run new code on an old system. Each puts the new parts in a different place:

The third approach suits a cluster that must stay as it is. We built the whole toolchain from source (compilers, linker, Git, CMake, Python, and others) in a container anyone can rebuild, and built the release flow around it.

Build every tool from source in a matching container

The build container starts from a CentOS 7.9 base image. CentOS 7 is rebuilt from the RHEL 7 sources, so it has the same glibc 2.17. Anything compiled inside it links against the libraries the cluster has.

Inside that container we built the toolchain from source, in order: binutils 2.40 (the assembler and linker), then GCC 13.2.0, then Python, Git, and CMake 4.0.0 with the new compiler. GCC releases before GCC 15 can be bootstrapped with a C++11 compiler, so the base image’s GCC 4.8 builds GCC 13. CentOS 7’s package mirrors were retired with the operating system, and its archive is at vault.centos.org, so the sketch points yum there first:

# Illustrative sketch of the first two stages, written for this article
FROM centos:7.9.2009
# Point yum at the CentOS archive; the regular mirrors are retired
RUN sed -i -e 's|^mirrorlist=|#mirrorlist=|' -e 's|^#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|' /etc/yum.repos.d/CentOS-*.repo
RUN yum -y install gcc gcc-c++ make bzip2 xz perl m4 zlib-devel
ARG PREFIX=/opt/toolchain
ENV PATH=${PREFIX}/bin:${PATH}
COPY binutils-2.40.tar.xz gcc-13.2.0.tar.xz /src/

# Stage 1: assembler and linker
RUN cd /src && tar xf binutils-2.40.tar.xz && cd binutils-2.40 \
 && ./configure --prefix=${PREFIX} --disable-nls \
 && make -j"$(nproc)" && make install

# Stage 2: the compilers, bootstrapped with the base image's GCC 4.8
RUN cd /src && tar xf gcc-13.2.0.tar.xz && cd gcc-13.2.0 \
 && ./contrib/download_prerequisites && mkdir build && cd build \
 && ../configure --prefix=${PREFIX} --enable-languages=c,c++ \
      --disable-multilib \
 && make -j"$(nproc)" && make install

# Stage 3 (not shown): Python, Git, CMake and the rest with the new GCC
ENV CC=${PREFIX}/bin/gcc CXX=${PREFIX}/bin/g++

The same pattern extends to any other compiler or tool your program needs. A tool built here inherits the old system’s library versions, so it runs wherever the cluster’s operating system runs.

Build the dependencies with that toolchain

A toolchain alone is not enough: every library the framework links must also be built in the container. We built all of the project’s Conan packages from source with the new GCC. A Conan profile points the package builds at the new compiler:

[settings]
os=Linux
arch=x86_64
build_type=Release
compiler=gcc
compiler.version=13
compiler.libcxx=libstdc++11
compiler.cppstd=20

[buildenv]
CC=/opt/toolchain/bin/gcc
CXX=/opt/toolchain/bin/g++

Conan’s default package ID does not record the system C library version. Nothing in that ID stops a prebuilt binary from a newer system entering the build, and on RHEL 7 such a binary will not load. Build every package from source once with conan install . --profile=rhel7 --build="*", then upload the results to a remote that serves only this target.

Package the release so nothing is installed on the cluster

The release has to run on a cluster where nobody installs compilers or libraries. Programs built with GCC 13 need its newer C++ runtime, which RHEL 7 does not have. There are two standard ways to carry it.

• Ship the shared runtime: copy libstdc++.so.6 and libgcc_s.so.1 into the release’s lib/ folder, and give each binary an RPATH of $ORIGIN/../lib.
• Link the runtime statically with -static-libstdc++ -static-libgcc.

Code that loads plugins at run time needs the shared form, so that every plugin uses one runtime and one set of type information.

An interpreter needs the same care. If the release carries Python, build it in the same container and give it its own RPATH, so the release never depends on a Python on the cluster. Check the result in a stock RHEL 7 container before anything leaves the build machine:

# In a clean container: every library must resolve from the release itself
for f in release/bin/* release/lib/*.so*; do
  if ldd "$f" 2>&1 | grep -q 'not found'; then echo "MISSING: $f"; fi
done

Test on a surrogate with the same OS and scheduler

We tested the full RHEL 7 release on a surrogate cluster with the same operating system and the same scheduler as the customer’s cluster. Real jobs on the surrogate show what unit tests miss: the scheduler’s environment, file paths on compute nodes, and the loader’s search order.

Treat the surrogate as the gate before any disc is cut: the release must run there from a clean unpack, with nothing installed.

Deliver on discs into the air-gapped environment

The customer’s cluster is air-gapped, so the release went in on DVDs. Unpack the release on the surrogate from the same archive that goes onto the discs, so the copy you tested is the copy you ship. The method for packing, splitting, and checking a large delivery on optical discs is in Delivering software to air-gapped systems and SCIFs on optical discs.

Let any developer rebuild: publish the pre-built image

A toolchain that only one machine can rebuild is a risk to the customer. We published a pre-built image with the toolchain and the dependency cache to the customer’s artifact repository and documented it. Any developer can reproduce the build from a Windows workstation with a container engine.

The same image can be exported as a single .tar file for systems that cannot reach the repository:

docker save toolchain-rhel7:latest | gzip > toolchain-rhel7.tar.gz
# on the receiving machine
docker load < toolchain-rhel7.tar.gz

Carry the same flow to newer systems

The container approach does not tie a customer to RHEL 7. As the customer moved forward, the same superbuild produced releases for RHEL 8 and RHEL 9, each built in a container that matches its target. One code base serves every generation the customer still runs.

Keep each target’s library versions in one place, such as the base image tag of its container. Adding a generation then means adding a container and a profile, not a second build system. One source tree, every operating system: a CMake superbuild shows how one tree selects its profile from the system it builds on.

Results

How we measured: the cluster result is our record of the deployment on the customer’s cluster; the release and the surrogate test come from the release records; versions come from the toolchain image’s build files.

Recommendations

• Print the newest GLIBC_ and GLIBCXX_ tags of every binary before release, because the loader rejects any tag the target lacks.
• Build the toolchain inside a container whose base matches the target, so every tool links against the target’s own libraries.
• Build every dependency from source with that toolchain, because a prebuilt package can carry newer library tags than the target has.
• Ship the C++ runtime with the release and point binaries at it with $ORIGIN, so nothing is installed on the cluster.
• Test on a surrogate with the same operating system and scheduler before the discs are cut.
• Publish the toolchain image, so any developer can reproduce the build without the original machine.

References

• Red Hat: Red Hat Enterprise Linux 7 reaches End of Maintenance Phase
• GCC: Prerequisites for GCC
• GCC: libstdc++ ABI policy and symbol versions
• The GNU C Library manual
• ld.so(8): the dynamic loader, RPATH and $ORIGIN
• Conan 2: profiles
• Docker: docker image save

If your software has to run on a system you cannot upgrade, see Build and release engineering or get a free estimate.

Related case studies