A Month of Red X's: What Your CI History Says When Nobody's Watching

Every pull request to your repository leaves a record. Not the diff, not the description — the run. The wall of green checks, the occasional red X, the re-run triggered by someone typing 'retry' in a Slack channel at 4:47 PM Pacific. Most maintainers treat CI logs as exhaust: something produced on the way to a merge and never looked at again.

That's a mistake, because your CI history is one of the most honest artifacts your project publishes. Unlike the README, nobody polishes it. Unlike the roadmap, nobody curates it. It simply accumulates, and it documents what your project actually values — not what it claims to.

We decided to read one like a stranger.

The Exercise: Thirty Days, No Context

The rules were simple. Pick a mid-sized open-source repository — ours, in this case, though the method works on any project with a public pipeline. Pull every workflow run from a thirty-day window. Then read them the way an outside contributor evaluating whether to invest their weekends would: with no inside knowledge, no benefit of the doubt, and no patience for ambiguity.

The stranger's questions are blunt. How often do builds fail for reasons unrelated to the change being tested? How many runs are retries of runs that already happened? How many tests are quietly skipped, and does anyone notice? Each answer is a data point about the project's relationship with its own quality bar.

What we found matched patterns we've since observed across dozens of repositories.

The Flaky Tax, Itemized

Roughly 11 percent of the failed runs in our sample were retried and passed without a code change. That number — the flaky rate — is the single most revealing statistic in a CI log, and almost no dashboard surfaces it by default.

A flaky rate in the low single digits is noise. Double digits is a message to every prospective contributor: your time here will be spent re-running pipelines, not writing code. And because flakiness clusters around specific tests, the fix is rarely a mystery. Sort failures by test name and the usual suspects appear immediately — timing assertions, unordered map iteration, integration tests sharing a database without isolation.

The deeper signal is behavioral. In our sample, the same three tests accounted for over 60 percent of all retries, and the retry pattern had been stable for months. Nobody had filed an issue, because the people hitting the flake were all maintainers who knew the workaround. New contributors, encountering the same failure, had simply left. The log recorded the attrition; nobody was reading it.

The Quiet Skips

Between the red X's and green checks lives the third state: skipped. Skipped tests are where projects negotiate with themselves in public.

Some skips are legitimate — platform-specific behavior, tests gated behind credentials nobody should have in CI. But a skip with a stale TODO from two years ago is a small, dated confession that a gap in coverage was noticed, deprioritized, and forgotten. Read in aggregate, the skip history maps exactly where a project's ambition and its capacity diverged.

Our audit found 47 distinct skip reasons, of which 19 referenced issues that had since been closed without the tests being restored. The issues were resolved; the tests were never un-skipped. That's not malice — it's the ordinary sediment of a busy project. But an outside contributor doing due diligence reads those skips and recalibrates their trust in the coverage percentage accordingly.

Retry Culture

The most uncomfortable finding was how normal re-running had become. In the sample month, 23 percent of merged pull requests had at least one re-run triggered by a maintainer, not by a code change. The pipeline had effectively become a slot machine: pull the lever until the checkmark appears.

Every re-run teaches the team the same lesson — that failure is ambiguous, that waiting is a valid fix, that the red X doesn't mean what it says. That culture leaks. Contributors start re-running instead of investigating, and genuine signal drowns in polite, habitual retries.

From Archaeology to Fixes

The audit is only worth doing if it changes something. Our checklist, distilled from the exercise:

• Compute your flaky rate. Retried-and-passed runs divided by total runs, tracked monthly. Publish it if you're brave.
• Name your top five offenders. Quarantine flaky tests in a separate job that doesn't block merges, then fix or delete them on a deadline.
• Triage every skip older than 90 days. Restore it, fix it, or delete it with a comment explaining why. Stale skips are debt with a timestamp.
• Require a note on every manual re-run. Even one line. Friction here converts habitual retries into investigations.
• Read your own history quarterly. Thirty minutes, once a quarter, looking at the pipeline the way a stranger would.

The Logs Are the Documentation

There's a line we keep returning to: your CI logs are documentation of your engineering culture, written automatically and read by everyone deciding whether to trust you. They can't be faked, which is precisely why they're persuasive.

A project that audits its own pipeline history — that treats a month of red X's as evidence rather than exhaust — is telling a story about itself that no landing page can. The contributors reading those logs are already drawing conclusions. The only question is whether the story they find is one you'd tell on purpose.