AI made writing code cheap. Reviewing it just got expensive.

We analyzed 99 open-source SaaS products to see what changed after a year of AI coding adoption. The bugs didn't go away. Teams are just paying for quality in a different way. Here's the data, and what we're building to fix it.

We wanted numbers, not vibes

Everyone has an opinion about what AI coding agents are doing to software quality. We wanted data.

So we analyzed 99 open-source B2B SaaS projects: CRMs, ERPs, e-commerce platforms, communication tools, analytics products. Think Odoo, Cal.com, PostHog, n8n, Mattermost, WooCommerce. We compared the first half of 2025 against the first half of 2026. Real commits, real pull requests, real bug reports.

A quick note on method, because it matters.

We did not trust GitHub labels to count bugs. Label conventions vary wildly between projects, and some projects changed them mid-year. Instead, an LLM read the actual content of sampled issues and classified each one: real bug report, or something else (feature request, question, chore).

To detect AI usage, we looked for co-authorship trailers in commit history, like Co-authored-by: Claude. We searched for fifteen agents in total — Claude, Copilot, Codex, Cursor, Devin, OpenCode, Aider, Gemini, Cline, Windsurf, and five more — but nearly every match came from the first five. This only catches agents that sign their commits. So every adoption number below is a floor, not a ceiling.

What we found

1. AI adoption exploded.

Commits co-authored by AI agents went from 220 to 8,260 in one year. That is a 37x jump. The number of projects with any AI-agent activity tripled, from 21 to 63 out of 99.

2. Bugs did not go away.

In the 44 repos actively using AI agents (at least 10 AI-co-authored commits in H1 2026), the bug rate went from 18.9 to 17.7 bugs per 100 commits. That is a 6.4% improvement. Better, but small.

Put another way: these teams still ship about 1 bug for every 6 commits, in codebases where AI now writes a big share of the code.

3. Merging got slower and PRs got bigger.

Median PR merge time in those same repos rose from 26.2 to 37.8 hours (+44%). Median PR size grew from 68 to 90 lines (+32%). Two thirds of the AI-adopting repos got slower at merging, not faster.

Here is the full picture for the 44 repos actively using AI agents:

The uncomfortable read

Put these three findings together and a picture emerges.

The bug rate stayed roughly flat because merging got slower. Teams absorb bigger, AI-assisted changes by spending more human review time on each PR. Quality did not come for free. It is being paid for with reviewer hours.

The bottleneck did not disappear. It moved from writing code to checking it.

The real goal is not "fewer bugs" as a slogan. The goal is to keep your defect escape rate flat, or better, without slowing down your PRs.

Here is the problem with that goal today. CI checks that your code compiles, passes lint, and passes the tests you wrote. Your reviewer checks something much harder: whether the code still does what the product intended. They do that check in their head, from memory, while reading a 600-line diff.

That is exactly the check that AI-generated code stresses the most. The code looks plausible. It is well formatted. It passes the linter. And it can quietly disagree with the spec.

"But we already have an AI reviewing our PRs"

Fair question. AI code review tools exist, and many teams already use one. They do catch real bugs. The issue is which kind.

Bugs come in two classes. The first is code bugs: null dereferences, off-by-one errors, race conditions, injection risks. These are wrong no matter what your product is supposed to do, and AI reviewers are genuinely useful at catching them.

The second class is intent bugs: code that is clean, safe, and internally correct, but does the wrong thing. The discount applies to the wrong customer tier. The cancellation flow skips a step your docs promise. The API returns a field the spec says was removed. Nothing about the code is broken. It just disagrees with what the product intended.

An AI reviewer is structurally blind to that second class, because it has no source of truth for intent. Your product's intended behavior lives in PRDs, design docs, tickets, and people's heads. The reviewer sees, at best, the PR description. And the PR description was written by the same person, or the same AI, that wrote the code. If both are wrong in the same way, the review passes.

They are also probabilistic, and they start from zero on every PR. An LLM reviews each diff fresh. Different runs produce different comments. Some are sharp, many are nitpicks, and teams learn to skim past them. Nothing accumulates. There is no persistent record that says "this documented behavior is verified, and stays verified."

And they are scoped to the changed lines. Business logic often breaks outside the diff: a modified function quietly violates a behavior that is documented, and depended on, somewhere else entirely. A diff-scoped reviewer never sees it.

There is a deeper problem when the code itself was written by AI. Asking a model to review a model's code is not an independent check. The reviewer shares the writer's blind spots. If the model misunderstood what the feature was supposed to do, a second model reading the same context tends to misunderstand it the same way. The code looks right to the reviewer for the same reason it looked right to the writer.

Verification only works when the check comes from somewhere independent of the thing being checked. Tests derived from the spec give you that. A model grading its own homework does not.

So AI review adds another opinion to the PR. Useful, but it is one more thing a human has to sort out. What is missing is a gate with ground truth: the spec itself, turned into checks that run on every change.

What we're building: make the spec part of CI

This is the problem TrueCourse goes after. It is open source, and it works like this.

Your docs become a spec corpus. truecourse spec scan walks the Markdown files you already have: PRDs, ADRs, RFCs, READMEs, design notes. It filters out the noise, groups the rest by product area, and flags places where two docs contradict each other. (You would be surprised how often they already do.)

That conflict detection matters more than it used to. When a human writes code, a stale or contradictory doc is an annoyance they route around from memory. When an AI agent writes code, the spec is the only source of truth it has to reason from. If two docs disagree, the agent picks one, silently, and the choice shows up as a bug two sprints later. A coherent spec corpus is not just prep for verification. It is prep for every prompt an AI agent reads before it writes a line of code.

Every spec section gets a guard. truecourse guard generate reads each section that makes a testable claim and writes a scenario test bound to that exact section. An LLM writes the test once, up front. If the spec and the code already disagree at that moment, you find out immediately.

Runs are deterministic. No model in the loop. truecourse guard run executes the scenarios like a normal test suite. No LLM calls. No flakiness. No per-run token bill. A failing scenario means one precise thing: this spec section and the code disagree. Maybe the code drifted. Maybe the spec is stale. Either way, a gate told you, not a customer.

It works in both directions. If the code changes, its scenarios fail. If a spec section changes, its scenarios go stale and tell you. Your spec becomes a living coverage map, where every section carries its own proof and its current status.

And yes, an LLM writes each scenario. The difference is what it reads and what it leaves behind. It works from the spec section alone, not from the code changes it will later judge, so the check comes from an independent source. And its output is not an opinion in a PR comment. It is a plain, human-reviewable test, committed to your repo, that runs deterministically from then on. Review it once, like any other test a teammate wrote. After that, the check is code, not a model's mood.

Today the guard tests CLI tools and APIs: it boots your server or runs your command, and checks the real response against the spec. Web, mobile, and desktop UI surfaces are next.

The point is not to replace human review. The point is to take the part of review humans are worst at (holding the whole product spec in their head while reading a big diff) and turn it into an automatic check that runs on every PR.

Reviewers go back to reviewing design and judgment. The "does this still do what we meant?" question gets answered by machinery.

That is how you keep the defect escape rate flat without the 44% merge-time tax.

Here's what it caught

We didn't want to just describe how this works, so we pointed the guard at a real, popular open-source API: Cal.com.

Cal.com was not a random pick. It is one of the heaviest AI adopters in our sample: 45% of its H1 2026 commits are AI-co-authored, the second-highest share of the 99 repos we analyzed. Heavy AI usage, and documentation the code no longer follows. The thesis of this article, in miniature.

Cal.com's own documentation states that every API response carries three headers: X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset. The guard read that claim straight out of the docs, generated a scenario test asserting it, booted Cal.com's API v2 server, and ran the test against it.

It failed. None of those headers exist.

Every documented name is missing an undocumented -Default suffix. A client built by reading the docs, whether a human integrating the API or an AI agent doing the same, reads nothing. We confirmed it independently with curl.

This isn't a stale doc nobody noticed. It's a known issue, open on Cal.com's own tracker since 2023: calcom/cal.diy#12306. The guard found it in a single run, from the documentation alone, and pointed straight at the section responsible.

Try it

TrueCourse is open source and runs locally in your repo. No server, no database, no account: results are plain JSON files inside your project.

npm install -g truecourse
cd your-repo
truecourse spec scan        # curate your docs into a spec corpus
truecourse guard generate   # author scenario tests bound to spec sections
truecourse guard run        # deterministic: spec vs code, pass or fail

A note on cost, since scan and generate call an LLM: the open-source version runs on top of Claude Code, so it uses your existing Claude subscription — no separate API key to set up or pay for. Before either command spends anything, TrueCourse still shows you a token and cost ceiling and asks you to confirm. The real bill lands at or below it. Re-runs only pay for spec sections that actually changed; everything unchanged is a cache hit. And guard run is free, every time.

Star or contribute on GitHub: github.com/truecourse-ai/truecourse

Want this as a gate in your CI/CD pipeline, checking spec drift on every PR before merge? Want your specs pulled straight from Confluence, Jira, Linear, and Google Docs instead of just the Markdown in your repo? That is what the commercial version does. Join the waitlist.

Methodology notes: 99 actively developed open-source B2B SaaS repos across 19 categories, comparing H1 2025 with H1 2026. "Actively using AI agents" means at least 10 AI-co-authored commits in H1 2026; 44 repos met that bar. Bug counts come from LLM classification of sampled issue content (up to 100 issues per repo per period), scaled to each period's total issue volume. PR merge time comes from up to 100 merged PRs per repo per period. PR size comes from evenly spaced samples of merged PRs with per-PR line counts. AI usage comes from commit co-authorship trailers; we searched fifteen agents (Claude, Copilot, Codex, Cursor, Devin, OpenCode, Aider, Gemini, Cline, Windsurf, Amp, Jules, Factory, Roo, Goose) — the first five account for all but 5 of the 8,260 matched commits. This is a conservative floor, since inline assistants like autocomplete leave no trace in commit history.