Ten Thousand Questions Don't Lie: Turning Ticket Volume Into a Documentation Roadmap
Every open-source project has two documentation systems. The first lives in /docs, is versioned, reviewed, and deliberate. The second lives in Discord, Stack Overflow, and a mailing list archive nobody indexes — and it's written one answer at a time by whichever maintainer happens to be online.
The second system is more honest than the first. It records what users actually failed to understand, not what the maintainers hoped they would. Over the past quarter, we audited roughly ten thousand support interactions across three mid-sized open-source projects to test a simple hypothesis: every question asked more than a handful of times is a missing documentation page.
The data backed it up — with some uncomfortable caveats.
Counting What Keeps Getting Asked
The methodology was deliberately low-tech. We exported a year of Discord messages from designated support channels, pulled every question tagged on Stack Overflow, and scraped the mailing list archives. After deduplication and bot filtering, each thread was classified by intent: installation, configuration, debugging, API usage, conceptual misunderstanding, or bug report in disguise.
The distribution surprised us. Fewer than 15% of questions were genuinely novel — edge cases, real bugs, or requests for capabilities that don't exist. The remaining 85% clustered into patterns so tight they were almost embarrassing. One project had 340 separate threads asking the same question about environment variable precedence, each answered individually, none of it reflected in the configuration guide.
Multiply that across a project's lifetime and the arithmetic gets grim. Each repeated question costs a maintainer ten to twenty minutes. A cluster of 340 threads is roughly two working weeks of unpaid labor — spent answering a question that one well-placed paragraph would have retired permanently.
The Three Shapes of a Documentation Gap
Not all gaps are equal, and the ticket data revealed three distinct failure modes.
The missing page. The documentation simply doesn't cover the topic. Environment variable precedence fell here. The fix is straightforward: write the page, and mine the best community answer for its first draft. The person who answered the question on Discord the twentieth time has already done most of the writing.
The buried page. The documentation exists, but the user never found it. This is the more insurious failure, because it feels like the docs team already did the work. In practice, we found buried pages shared a trait: they answered the question the user had, but were titled after the internal concept the maintainers used. Users searching for "why isn't my config being picked up" will never click a page called "Resolution Order in the Settings Pipeline." Search analytics and ticket phrasing tell you the user's vocabulary; the docs should adopt it.
The page that answers the wrong question. The docs explain how a feature works, but the user's ticket reveals they're stuck one step earlier — on why they'd want it, or what error message to expect when it fails. These conceptual gaps show up as debugging tickets that resolve themselves once someone explains the underlying model. They're the hardest to spot from inside the project, because to a maintainer the concept is obvious. To a newcomer arriving at 11 p.m. with a production incident, it is not.
Building the Feedback Loop
Auditing once is useful. The real win is making ticket volume a standing input to the documentation roadmap, the same way CI failures are a standing input to release engineering.
The projects that did this well shared a lightweight pipeline:
- Tag, don't triage. Volunteers or bots tag every support thread with a topic label. No deep analysis at intake — classification happens in bulk, monthly.
- Set a threshold. Any topic exceeding five identical questions in a quarter automatically opens a documentation issue. The issue body is pre-populated with links to the threads, which doubles as a requirements document written by users.
- Answer with a link, log the miss. When a maintainer answers a question that should have been covered, they file a one-line note. Recurring notes on the same topic are the earliest signal that a page is buried or mistitled.
- Measure the decay. After a new page ships, track whether the corresponding ticket cluster shrinks. If it doesn't, the page exists but isn't being found — which sends you back to step two's vocabulary problem.
One project we examined cut its top-five recurring question clusters by roughly 60% within two release cycles using nothing more elaborate than this. The maintainer hours reclaimed funded the documentation work several times over.
The Objection, and Why It Mostly Fails
The standard pushback is that noisy support channels are the price of a growing community, and that some users simply won't read documentation. Both are partly true. But the framing gets the causality backwards. Users who ask the same question three hundred times aren't refusing to read — they're demonstrating, at scale, exactly where the reading experience breaks down. A project that treats ticket volume as user noise is discarding its highest-signal user research for free.
There's a governance dimension, too. Documentation driven by ticket data is documentation driven by evidence of user need rather than maintainer intuition — which tends to survive maintainer turnover better. The next person to inherit the project finds a paper trail of what users struggled with, not just what the previous team thought was important.
Start With Last Quarter
You don't need tooling to begin. Export last quarter's support channel, sort by topic, and count. If the top ten clusters don't map to pages that should exist, you have unusually good documentation. If they do — and in our sample they always did — you have a roadmap that wrote itself, sourced from the most honest documentation your project has ever produced.
The question your users keep asking is the page you haven't written. The data has been sitting in your Discord the whole time.
