Articles

Case study: Developers inside a SCIF build releases with no network

Software for an air-gapped network or a SCIF arrives one way: on media that someone carries in. A build that reaches out to a package registry, a Git server, or a container registry stops at the first download. The kit has to hold everything.

This guide shows what goes into such a kit, how to prove it before it leaves, and how to split it across optical discs and put it back together inside. It is the method behind the case study Developers inside a SCIF build releases with no network.

List everything the build fetches

Start by finding every download the normal build makes. A typical C++ and Python project fetches from at least four places. They are Git servers for source and submodules, a package manager’s remote for libraries, a container registry for the build image, and the Python package index.

Run one clean build on a connected machine with logging on, and record every host it contacts. One way is to give the build container a single route out, through a proxy that logs each request. Each of those hosts becomes a part of the kit.

Anything left off the list stops the build inside the SCIF, where nobody can fetch the missing piece. Keep the list in the kit’s README as well, so the people inside know what the kit replaces.

Pack the kit: source, caches, image, and packages

The kit we built carries five things, plus documentation for the people inside:

• every source repository, as a Git bundle;
• the dependency caches for Windows and Linux;
• a saved build image;
• the Python packages;
• scripts that unpack it all and build a release.

A Git bundle is a whole repository, history included, in one file. Each submodule gets its own bundle:

# Source: one bundle per repository, named like the last part of its URL
export KIT="$PWD/kit"
git -C framework bundle create "$KIT/src/framework.git" --all
git -C framework submodule foreach --recursive \
  'git bundle create "$KIT/src/$(basename "$(git remote get-url origin)")" --all'

# Libraries: export the Conan cache, once per operating system
conan cache save "*:*" --file=kit/deps/conan-linux.tgz

# Build image: save it as one archive
docker save build-env:release | gzip > kit/image/build-env.tar.gz

# Python: download every package as a file, for the target's platform
pip download -r requirements.txt -d kit/wheels \
  --only-binary=:all: --platform manylinux2014_x86_64 --python-version "$PY_VER"

Run the Conan export on each operating system you build for, because a Windows cache and a Linux cache hold different binaries. To export only what the build uses, start Conan from an empty home. Run the normal install for every configuration you ship, then save that cache:

export CONAN_HOME="$PWD/kit-conan-home"     # starts empty
conan profile detect
conan install . --profile=release --build=missing
conan install . --profile=debug --build=missing
conan cache save "*:*" --file=kit/deps/conan-linux.tgz

That cache then holds every recipe and binary the release needs and nothing left over from other projects. The pip download platform and Python version must match the interpreter inside the SCIF, not the machine that packs the kit.

Write the scripts that unpack and build

Inside, the scripts reverse every step and point each tool at the local copy. Git learns to read submodules from the bundles, Conan restores its cache, Docker loads the image, and pip installs only from the local folder:

# Inside: clone from the bundle and redirect submodule URLs to bundles
git clone kit/src/framework.git framework
git config --global url."$PWD/kit/src/".insteadOf "https://git.example.org/group/"
git -C framework -c protocol.file.allow=always \
  submodule update --init --recursive

conan cache restore kit/deps/conan-linux.tgz
gunzip -c kit/image/build-env.tar.gz | docker load
pip install --no-index --find-links kit/wheels -r requirements.txt

The insteadOf rule maps each URL recorded in .gitmodules to a bundle in the kit, so no project file changes. Current Git refuses local paths for submodules by default, which is why the update allows the file protocol for that one command.

Prove it with networking turned off

We proved the kit on a machine with networking turned off before it left. The test is simple and strict: start from an empty machine, copy the kit in, run the scripts, and build a release. Any step that tries to reach the network stops there, on our side of the wall.

A container gives a quick first check, because Docker can start one with no network at all. The full check still belongs on a separate machine or virtual machine with its network adapter removed, which starts from nothing but the kit:

# Quick first check: build inside a container that has no network
docker run --rm --network none -v "$PWD/kit:/kit:ro" build-env:release \
  /kit/scripts/unpack-and-build.sh

Treat this test like a release gate. After any fix to the kit, run the whole test again from an empty machine.

Then compare the release built offline with one built on a connected machine from the same commit. The file lists should match, and so should the test reports. A difference points at something the connected build fetched that the kit does not carry.

Split the kit across discs and reassemble it inside

A large project does not fit on one disc. We split large projects across several DVD or Blu-ray discs and reassembled them inside the SCIF.

Archive the kit as one file, cut it into parts smaller than one disc, and write a checksum file beside the parts. A single-layer DVD holds 4.7 GB (ECMA-267), so 4,000 MB parts leave room for the checksum file and a short README on every disc:

tar -cf kit.tar kit/
split --bytes=4000M --numeric-suffixes=1 kit.tar kit.tar.part-
sha256sum kit.tar kit.tar.part-* > SHA256SUMS

Inside, copy every part from its disc into one folder, check each part, join them, and check the result:

grep 'part-' SHA256SUMS | sha256sum -c -        # every part, before joining
cat kit.tar.part-* > kit.tar
grep ' kit.tar$' SHA256SUMS | sha256sum -c -    # the joined archive
tar -xf kit.tar

Number the discs on their labels in the order the parts were cut. Put the README and the checksum file on every disc, so any single disc tells the person inside what it belongs to.

Choose the discs and label them for review

Pick the disc type from the size of the kit and from the drives inside. A Blu-ray disc holds several times what a DVD holds, so a large kit needs fewer discs, fewer burns, and fewer checks. DVDs read in almost any optical drive, so confirm the machine inside has a reader for the larger discs before you choose them.

Expect the facility’s staff to review media before it goes in. Make that review easy: plain archive formats, a README on every disc, and a manifest that lists every file in the kit with its checksum and where it came from. Generate the manifest from the kit folder itself, so it can never drift from what was burned:

( cd kit && find . -type f -print0 | sort -z \
    | xargs -0 sha256sum ) > MANIFEST.sha256

Keep the same manifest with the next kit. Comparing the two shows exactly which files changed between transfers.

Keep a transportable mirror current

A kit is a snapshot, and the customer’s source keeps moving. A mirror service keeps a transportable copy of every repository current, refreshing it every 2 hours, so each new kit starts from current source.

Build the next kit from the mirror, not from developers’ clones. A mirror holds every branch and tag the teams have pushed, so nothing depends on what one person had checked out.

Send later updates as incremental bundles

After the first full kit, most transfers only need the commits made since the last one. A Git bundle can hold just those commits, as long as the receiving side already has the commits the bundle builds on. Tag what each transfer carried, so the next bundle knows where to start:

# Outside: everything on main since the last transfer
git bundle create update-07.bundle transfer-06..main
git tag transfer-07 main

# Inside: check that the bundle applies, then fetch from it
git bundle verify update-07.bundle
git fetch update-07.bundle main:refs/remotes/transfer/main

git bundle verify reports any commit the bundle needs that the repository inside lacks, before anything changes. Send a full bundle again whenever that check reports a missing commit or the history was rewritten.

Validate third-party components on the air-gapped system

Software from another organization needs the same proof, run on the air-gapped system itself. We validated a third-party plugin on an air-gapped system by running its demonstration case and comparing the output with the reference output. The difference in mean and in standard deviation was 0.0.

Keep the reference output in the kit too, so the same comparison can run again inside after every update. The people inside can then repeat the check themselves, with no help from outside the facility.

Before a third-party component goes on a disc, list the shared libraries each of its binaries needs and check that the kit provides them:

readelf -d plugin/lib/*.so | grep NEEDED | sort -u

Results

How we measured: the offline proof is a full release build on a machine with networking turned off; the mirror interval is the service’s schedule; the plugin comparison ran on the air-gapped system against the plugin’s reference output.

Recommendations

• Record every host a clean build contacts before you pack anything, because each one is a part the kit must carry.
• Carry caches and images, not instructions, because nobody inside can download what an instruction asks for.
• Prove the kit on a machine with networking turned off before any disc is cut, and repeat the proof after every fix.
• Write a checksum file for the whole archive and every part, and check both inside before you unpack.
• Build each kit from a mirror of every repository, so the kit never depends on one developer’s clone.
• Run each third-party component’s own test case on the air-gapped system and compare it with the reference output.

References

• Git: git bundle
• Git: git config, url.<base>.insteadOf
• Conan 2: conan cache save and restore
• Docker: docker image save and docker image load
• pip: pip download
• GNU coreutils: split and sha2 utilities
• ECMA-267: 120 mm DVD, read-only disc

If your software has to reach an air-gapped network or a SCIF, see Build and release engineering or get a free estimate.

Related case studies