Swiss cheese for software: the test stack that gets thicker every time a bug gets through

Every test suite in Modulo catches a different class of bug, and no single layer catches everything. The pre-commit hooks catch the cheap stuff in under five seconds. The deterministic suites catch logic and contract errors at increasing cost. The AI reviewer catches the things no test can encode. And when a bug gets through all of them, it becomes a new rule in a lower layer, so the stack gets thicker over time. This post maps the layers and shows how the feedback loop closes.

The model

The Swiss cheese model of accident causation says failures happen when the holes in several layers of defense line up. Modulo applies the same idea to development. Each layer has holes. A typo sails past the linter, a logic error sails past the unit tests, a contract mismatch sails past integration. The layers work because their holes rarely line up, and because every miss fills a hole in one of them.

Layer one: pre-commit

Every commit runs 17 hooks. Ruff fixes and formats the staged Python, Bandit scans for security issues, Semgrep applies custom rules, Gitleaks blocks secrets, import-linter enforces architecture contracts, ESLint checks the frontend. Most are generic. The Semgrep rules are not: more than 50 of them at the time of writing were written by us, and most exist because a specific bug happened once.

One example. A log call passed the key name inside extra={}. Python's logging reserves that key, and the KeyError only fires at INFO level. Unit tests run at WARNING, so they stayed green while production crashed silently - a boot-time seed skipped every org's cost seeding for weeks before anyone noticed. The fix shipped, and a Semgrep rule shipped with it. That class of bug can never merge again.

Layer two: fast CI

Unit tests run on every push and every PR. About 14,000 backend unit tests execute in under 30 seconds with the database mocked and a StubModelBackend standing in for every LLM call. Per-module coverage thresholds block merges: modulo.auth at 90 percent, pipeline_engine at 85, db.rls at 95. The frontend runs 478 Vitest tests and a WCAG contrast check. Architecture tests enforce that modulo.api never imports LangGraph directly. All of these are deterministic, all run on every push, and all gate the merge.

Layer three: heavy CI

Integration tests run against real Postgres in Docker with Alembic migrations applied first, not mocks. The BDD suite runs 261 Gherkin feature files through pytest-bdd and Playwright against the full stack. Thirty-five Playwright specs can target localhost, staging, or production through the E2E_TARGET env var. These suites are too slow for the merge gate, so they run on pushes to main and against staging after every deploy. The merge gate is deliberately fast; the expensive suites are the net behind it.

Layer four: a real deployment

The same 35 Playwright specs run against staging with real authentication - no API mocking - on every push to main, after every deploy, and on a daily schedule. Staging WCAG runs contrast checks against the real UI. This layer catches what mocks cannot: the deployed frontend talking to the deployed backend with real credentials. A login flow that works in unit tests and fails in production is caught here, before the same failure reaches production.

Layer five: AI review as a hard gate

The PR Reviewer pipeline reviews every pull request against 12 quality lenses: correctness, security, code quality, error handling, edge cases, testing, performance, style, dependencies, documentation, UX conformance, and infrastructure. It runs the Definition of Done checklist, verifies the implementer considered every test suite the change could affect, and can run the impact subset itself. It can also run three checks the lenses do not cover: prove-the-fix (the test must fail without the change), contract round-trip (frontend keys must match backend Pydantic fields, verified against the OpenAPI types), and surrounding-code exploration (grep the wider codebase for duplication instead of reviewing only the diff).

The review is a gate, not a suggestion. The merge queue refuses to merge without an APPROVED review, and it re-checks the review decision immediately before merging to close the race where a reviewer posts changes after collection. The testing lens is the largest source of change requests in the review history: over-mocking, weak assertions, and tests that never exercise the real code path.

Layer six: the collection

PR review is scoped to one pull request. Nothing reviews main as a whole - two PRs can each merge green while both adding the same utility, or each introducing a slightly different error-handling pattern. The improve-architecture pipeline walks the product map graph and reviews delivered work as a collection, which is the first mechanism that sees beyond PR scope. A daily reviewer that scans every commit merged in the last 24 hours and checks for logical conflicts, duplicated utilities, and missed abstractions is next on the roadmap.

Layer seven: the feedback loop

This is the layer that makes the other six compound. Every QA finding and every production incident ends in one of two places: a rule in the Semgrep set, or a lesson in AGENTS.md that every future agent reads. The reserved-log-key bug became a Semgrep rule. A deploy-throttle cascade became a codified lesson. A merge-queue race became a hard gate in the workflow. The stack is not static. Each miss fills a hole, and the holes that remain get harder to line up.

What we have not automated

The AI reviewer asserts tests are valid and appropriate for the change. What it does not do is generate tests for a change and block on their quality. AI writes tests as part of implementation, but the gate is the test passing, not an AI judgment of test sufficiency. We have deliberately kept that step manual for now.

Modulo runs its own delivery on Modulo, and the numbers are public, so you can check whether the process catches problems before merge.