Practice a Build-System Interview With a Hidden Dependency
Find a hidden build dependency by comparing declared inputs, generated files, toolchains and cache invalidation in a fresh environment.
TL;DR
- In this fictional build-system exercise, a local build passes while a clean CI build fails.
- Compare action inputs and dependencies, then examine cache correctness and nondeterministic output.
- Make the missing dependency explicit and prove the repair in a fresh environment.
Investigate why the clean build fails
In this fictional build-system interview, a project compiles on one developer's machine but fails in a clean continuous-integration environment. The source commit and declared dependencies appear identical. The interviewer asks how you would find the hidden dependency.
Start by defining the build action's intended inputs and outputs. The source tree is only part of the input set. Toolchain versions, generated files, environment variables, system libraries and network responses can affect the result if the build reads them.
Do not immediately copy the developer's entire home directory into the build image. That might reproduce the symptom while preserving the hidden dependency and exposing unrelated data. The exercise is about making the relevant inputs explicit, not preserving every accidental condition of one workstation.
Compare the action boundaries
Create an inventory of the exact failing action: command, working directory, declared input files, executable versions and permitted environment. Record sensitive values privately or by safe identifiers rather than dumping secrets into public logs.
| Possible hidden input | Investigation | Durable direction |
|---|---|---|
| Local generated file | Remove it in a controlled copy | Declare generation and dependency |
| System tool on PATH | Compare executable identity | Use an explicit toolchain |
| Undeclared environment value | Run with a minimal environment | Declare required configuration |
| Network-fetched mutable file | Inspect download and version selection | Pin or otherwise control the input |
The Bazel hermeticity documentation explains the relationship between isolated builds and declared inputs. The broader interview lesson is to account for what an action can observe; using a named build tool does not automatically make every custom action hermetic.
Follow a concrete failure
Suppose the fictional project imports a generated schema file. On the developer's machine, a previous command created it, but the build graph does not declare that generation step. The clean environment correctly lacks the file.
The repair is not merely to commit whatever happens to be on the workstation. Decide how generated artifacts are managed in this project. If generated during the build, declare the generator, source schema and output relationship. If committed by policy, add a reliable check that the committed output matches its authoritative source.
Then delete the local artifact in a controlled test and rebuild from the declared inputs. This experiment distinguishes a repaired dependency graph from a build that still succeeds only because stale output remains nearby.
Treat cache correctness as part of the problem
A cache key must represent the inputs that can affect the action's output. If a generator reads an undeclared template, changing that template might incorrectly reuse an old result. A fast cache hit can therefore hide a correctness defect.
Explain what belongs in the action identity: relevant file contents, toolchain and declared configuration, according to the build system's model. Avoid adding the entire environment indiscriminately; that can create unnecessary misses while still failing to capture some real input.
Test a meaningful input change and confirm that the relevant action reruns. Also change an unrelated file and check whether unnecessary work occurs. Correctness comes first, but a useful dependency graph should describe scope accurately enough to support efficient rebuilding.
Examine nondeterministic output
Even declared inputs can produce different bytes if the action embeds current timestamps, unstable ordering or other uncontrolled state. Ask whether reproducible output is required and which differences matter to the artifact consumers.
If an archive's file order changes without changing its logical contents, the difference may still disrupt hashes or caches. If a build injects a release timestamp intentionally, define how that value is supplied and represented rather than letting every action read the wall clock independently.
Keep functional correctness and byte-for-byte reproducibility distinct. A reproducible build can consistently produce the wrong program, and a functioning artifact may still be difficult to verify if its construction varies unexpectedly. Our system design frameworks guide can help articulate those separate requirements.
Record the repaired dependency alongside the owning build rule so the next contributor understands why that input must remain explicit.
Prove the repair in a fresh environment
Build from a clean checkout with the declared toolchain and configuration, then repeat under the relevant isolation conditions. Confirm the generated dependency and inspect whether the action performs unexpected network or filesystem reads where the environment allows that observation.
Have a practice partner introduce a toolchain update or a changed template after the cache is populated. Explain why the correct actions should invalidate and how you would detect a stale reuse. Do not respond by disabling all caching unless that is a deliberate temporary diagnostic step.
Finish with the missing input, the change to the build graph and the experiment that supports the diagnosis. Use the mock interview strategy to practise this causal explanation. The desired outcome is a build whose dependencies are explicit enough that a new machine does not need the original developer's history to reproduce it.