How Develocity Found a Bug in the Java Compiler

A while back I was chasing a Build Cache miss that made no sense.

The Build Cache works by hashing everything that goes into a piece of work. Same inputs, same hash, reuse the previous result instead of doing it again. It is one of the larger reasons a big build finishes in minutes rather than hours, and when it stops working you notice, because the build gets slower for no visible reason.

That is what was happening to us. Intermittently, across continuous integration (CI) agents, work that should have been reused was being redone.

And the thing that made it genuinely maddening rather than merely annoying was that nothing was wrong. No test failed. No task failed. Every build was green. Every log was clean.

It turned out to be a bug in the Java compiler. Not in our code, not in our build, not in our CI setup — in javac itself, shipped and in production and used by essentially everyone who writes Java. It has since been fixed.

This is the story of how it got found, because I think that is the more interesting half.

Open the build that missed the cache — every task succeeded. Open the one before it — every task succeeded. Read the logs and there is nothing to read: no warning, no stack trace, no anomaly. The compiler was not complaining, and it had no reason to, because as far as it was concerned it had done its job correctly. By almost every definition, it had.

The problem did not exist inside a build. It existed as a difference between two builds — and that distinction decides whether you can find it at all.

Every build we run publishes a Build Scan to Develocity: a full record of what the build actually did, captured from inside the build tool rather than scraped out of the console output. Not just what was printed, but every task, every input to every task, every file hash, every cache key.

Develocity can put two of those records side by side and show what differs. That is the entire investigation.

I picked two builds — same commit, same branch, both green, both on CI, 24 seconds apart, one of them a Build Cache miss — and asked what was different. First I looked at the task that compiles the module in question:

Zero differences. Every source file, every classpath entry, every compiler setting — identical across the two machines.

Then I looked at the task that packages up what the compiler produced:

One difference, and it names the two culprits: two compiled class files, with two different resulting cache keys. And on both sides, the outcome is SUCCESS.

Read those two screens together and the conclusion is unavoidable and slightly absurd. The compiler was handed identical inputs and produced different outputs. Same source, same settings, different bytes coming out. Everything downstream that consumed those bytes was — entirely correctly — treating them as new work and redoing it.

Task inputs are only half of what gets recorded. Develocity also captures the machine each build ran on:

Twelve items compared, and three of them differ: the two agents are one kernel patch apart, one JVM has 5.7 GiB of heap and the other 5.3, and the machines have different public hostnames.

Two of those go nowhere. A few hundred megabytes of heap headroom and a machine name have no path to the order in which javac writes a permits clause — the list of subtypes a sealed type allows. What counts is everything the screen reports as identical: the JVM that actually ran the build matches down to the patch and build number, as do the core count, the worker count, the locale and the charset.

The kernel patch level was the one I could not let go of, because it was the only difference anywhere in the comparison that tracked the bug. The class that came out wrong came from the 5.4.0-166 node, and it kept coming out wrong on every node running that kernel. The kernel patch level was the only signal I had, and I said so when I reported the problem. It was also the wrong one.

The task comparison did something else, too. It killed the other theory anyone sensible would reach for. We run annotation processors over that source, and a processor rewriting the file would explain everything. But the processors run as part of that compile task, and the compile task's inputs and configuration were identical on both machines.

Whatever the processors did, they did it the same way twice. I did not have to design an experiment to establish that. It was already recorded, from a build that had finished days earlier.

Now I had two specific files that disagreed, so I compared them directly.

One line of difference. Two bytes, swapped. Out of a class file, out of a module of some 1,700 classes, out of an entire build.

The standard tool for inspecting a class file reports the two as identical. What shows the difference is decompiling both and putting them side by side:

The class is a sealed interface — a type that declares, up front, exactly which other types are allowed to implement it. That list of permitted subtypes is written into the class file. And the compiler was writing it in a different order depending on which machine ran the build.

The order carries no meaning. The JVM does not care. Nothing about the running program changes.

Which is exactly why it had gone unreported ever since sealed types were finalized in Java 17. The reordering is invisible to the language, invisible at runtime, and invisible to the usual tooling. But it is perfectly visible to anything that hashes the file. Which is to say, to every build cache in existence.

It would be easy to shrug at this. Compiling that module takes seconds. Who cares if it occasionally gets done twice?

That is the wrong unit of measurement, because compilation is not the thing that gets repeated. Compiling sits near the root of the build graph, and a cache key is computed from a task's inputs. So the key of every task downstream is derived, transitively, from the bytes this one produced.

When those bytes change, the packaging step's key changes. Then the key of everything consuming that package changes. Then the keys below those. The invalidation does not stay where it started. It propagates to the leaves.

So two bytes in the wrong order, in a step that takes seconds, discard the cached result of everything that comes after. Unit tests rerun. Integration tests rerun. Static analysis, license checks, security scanning, artifact assembly, every verification gate in the pipeline — all of it reruns. Not because a single one of them would reach a different conclusion, but because the hashes say the inputs changed, and a cache that ignored that would not be a cache worth having.

Now multiply it by every branch, every pull request, every agent, every day. The cost was never the recompilation. It was a pipeline repeatedly re-proving things it had already proven, spending developer waiting time and CI compute to arrive at answers it was already holding. That is the real bill for two bytes, and it is why a cache miss nobody can explain is worth weeks of someone's attention.

I posted to the OpenJDK compiler-dev mailing list. The first reply from the javac maintainers at Oracle was a fair brush-off: deterministic output is desirable in a compiler, not guaranteed. And I could not reproduce the problem outside our own enormous build. What changed that was the evidence. Once "two bytes differ" became "the permitted subtypes are in the wrong order," it was a checkable claim, and it was filed as JDK-8322477 the next day.

So we inverted the usual arrangement: rather than us shrinking our build, they sent instrumented JDKs up to it, the last of them carrying a proposed fix. Each round trip was judged the way the bug had been found — by comparing two complete records to see whether the difference had gone. "Did it work" meant two machines now produce identical bytes, which is not something you can eyeball.

That last JDK made the difference go away, which eliminated the one explanation still standing: nothing was reordering our sources. The maintainer's reaction, in a mail to me:

Understanding why it happened was, in the maintainer's own words, in the area of impossible — and it had been hiding in our builds, and everyone else's, the whole time.

The explanation came from Octavia Togami, a colleague on Gradle's JVM Platform team, who succeeded where I had failed and produced a reproducer three files long. It hinges on whether some other, unrelated file in the compilation happens to mention the two subtypes, and in what order. That is legitimate behavior for a build tool doing incremental compilation, and it is enough to flip the compiler's output. Their verdict:

That also disposed of the kernel version. What flips the output is the order of the files in the compilation, and that order is decided by the build, not by the machine underneath it — so the kernel patch level was at most a bystander that happened to track the symptom. It was the only environmental difference that ever correlated with the bug, and correlating was all it did. The comparison had reported that difference accurately. Reading a cause into it was mine to get wrong.

The fix sorts the list by source position, so encounter order stops mattering. Integrated 23 days after that first mailing list post, shipped in JDK 23, backported to 21.0.4.

Nobody was looking for this bug. It was not on anyone's backlog. No test we could have written would have caught it, because the compiler's output was correct — byte ordering aside, both files described exactly the same program. Both builds passed. If you had opened either one and read it end to end, you would have found nothing, because there was nothing in either one to find.

It only ever existed as a difference between two builds. Finding it required having both of those builds recorded in enough detail to be compared. Not the tail end of two log files, but the complete set of inputs to every task on both sides, down to individual file hashes. The distance between "these two builds differ somewhere" and "these two class files differ, and here are their cache keys" is the whole investigation. Most of what followed was pulling on a thread that was already in my hand.

That is the part that generalizes. The Build Cache told me only that something was wrong: a miss where there should have been a hit. That is an alarm, not an answer. What turned it into a defect report was Develocity — capturing each build from inside the build tool as it runs, and keeping that record whether or not anyone expects to need it.

Because the records exist, any two builds can be set side by side, and a question nobody thought to ask while they were running — why did these two produce different bytes from identical source? — is answerable from data captured days earlier. That is the capability, and it is indifferent to where the answer turns out to live. This time it lived in the Java compiler.

Request a trial