This follows the production readiness review practice established by SRE teams, applied here to systems that call language models. This is the rubric we commit to applying on every engagement, so coverage is consistent and every finding traces back to a named check. This is that checklist — six sections, what each looks for, and why each one earns its place. We ran it on our own platform first.
The rule
Every finding must cite observable evidence — a file and line, a command and its output, a request and its response. "This looks fragile" is not a finding. Every finding carries a concrete failure scenario: what input, in what state, produces what wrong result. And we say plainly when something is done well — it builds trust in the criticism, and it tells you what not to break.
A. Automated first pass
2.5 hours
What it looks for
Dependency and vulnerability scan, every language in the repository — not just the primary one.
Public attack surface enumeration: list every unauthenticated route, then check each one individually for signature verification, tenant scoping, rate limiting.
Route contract diff: every API path the frontend calls, against every route the backend registers.
Secret scan across committed code, image layers, and CI logs.
Does a clean checkout build, and does the stack actually start and serve — not just build an image.
Test signal: do tests assert behaviour, or only that code runs without throwing.
Docs versus reality: spot-check three checkable claims in the README against the code.
Why it is here
Mechanical checks run first because they are cheap and they do not get tired. Each step below earned its place on our own codebase. Dependency scanning is step one because ninety seconds of it surfaced two reachable CVEs. Attack surface enumeration is its own step because enumerating every unauthenticated route is what surfaced two Critical vulnerabilities — reading the code without a list had not reached those handlers. The route contract diff surfaced two broken product surfaces: a dashboard calling three endpoints that were never built, and a settings page calling six. "Does the stack actually run" is separate from "does it build" because we shipped a compose file that crash-looped on a missing environment variable while CI reported green, having only built the image.
B. Failure behaviour
3 hours
What it looks for
Restart mid-operation: kill the process during an external write and see whether it retries blindly, loses the work, or marks it for review.
Idempotency: can the same request execute twice, and is there a key enforced at the boundary that matters.
Timeout coherence: client timeouts, server timeouts, and how long the operation actually takes, compared.
Resource lifetime: whether long operations hold a database connection or handler for their whole duration.
Retry and backoff: present, bounded, and distinguishing retryable failures from permanent ones.
Unbounded growth in in-memory maps, caches, and rate-limit windows, especially where the key space is caller-controlled.
Failure surfacing: are errors reported to the user, or caught and logged to a console nobody reads.
Why it is here
Most systems are correct on the happy path. This section tests behaviour outside it — retries, timeouts, duplicate requests, resource exhaustion, and how failures reach operators and users. A blind retry against a non-idempotent endpoint is a Critical finding, and mismatched timeout layers make working operations look broken to the people using them.
C. AI-specific
3 hours, more if decisions affect people
What it looks for
Consequence classification: does the system make or materially influence a decision about a person.
Human oversight: is a human genuinely in the execution path, or is oversight asserted in a policy document while the code writes regardless.
Oversight quality: decision latency, whether the reviewer opened the reasoning, override rate, consecutive-approval streaks.
Automatic logging and log retention against the relevant AI Act articles.
Data path to the model: what personal data reaches the provider, in which country, under what transfer mechanism.
Evaluation soundness: if prompt or model changes are claimed to be evaluated, is the input actually held fixed.
Autonomy boundaries: what the model can trigger without a human, and whether that boundary is enforced in code or only by convention.
Why it is here
If a system makes or influences a decision about a person, Annex III high-risk classification is worth checking: it depends on the specific use, and we do not give legal advice. Under the enacted EU AI Act, Annex III high-risk obligations apply from 2 August 2026. A deferral to 2 December 2027 was politically agreed in May 2026 and is expected to be adopted, but is not yet in force; check the current position with your own advisers. Reviewers reliably drift into rubber-stamping while the paperwork still records a human choice, so oversight quality is checked, not assumed. Evaluation comparisons can be silently invalidated by non-deterministic prompt construction — interpolating a map into a prompt string produces a different prompt every run in a language that randomises map iteration order.
D. Data protection
2 hours
What it looks for
Lawful basis, and whether any special-category data under Article 9 is in play.
Retention that runs: is deletion implemented and scheduled, or only promised in a policy.
Erasure completeness: is there a path that removes everything for one customer, covering tables added since it was written.
Sub-processors and residency: do the documented processors match what the code actually calls, and where is each hosted.
Credential encryption: third-party tokens encrypted at rest, bound to the tenant, key stored separately from backups.
Backups: exist, restore tested, and the encryption key is not stored only alongside the backup it decrypts.
Why it is here
A deletion commitment with no purge job is a live compliance exposure, not a documentation gap. This section checks what actually runs against what is written down, because the two drift.
E. Security
2 hours
What it looks for
Token verification: algorithm pinned, issuer checked, expiry checked, keys fetched from a trusted source and cached safely.
Tenant isolation: every query scoped by tenant, especially ones that take an identifier from the request body rather than the session.
Authorisation: is the privileged action gated server-side, or only hidden in the UI.
Least privilege at integration boundaries — OAuth scopes, API permissions, read-only where reads suffice.
Deploy account: deploying as root over SSH is common, and worth flagging when found.
Rate limiting and abuse protection on unauthenticated routes.
CORS and headers: wildcard origins on an authenticated API.
Why it is here
This section is where a working demo and an auditable system diverge. A tenant identifier trusted from a request body instead of a verified session is the single most common Critical we look for.
F. Operability
2 hours
What it looks for
Liveness versus readiness distinguished, and whether readiness checks the database.
Diagnostics: can someone without shell access tell which subsystem is broken.
Migrations: transactional, applied automatically, and does the service refuse to serve if they fail.
Upgrade path: what happens if one half of the system deploys and the other does not.
Observability: standard metrics format, and whether "why did this specific request fail yesterday" is answerable.
Configuration: deployable to a new environment from environment variables alone, or dependent on hand-edited files on the host.
Why it is here
Correctness on day one is not the same question as operability on day two hundred. This section is what tells you whether the system can be run by someone other than the person who built it.
What the report looks like
One document: an executive summary a non-engineer can act on, a sorted findings table, findings in detail with evidence and a failure scenario for each, an honest statement of what we did not cover, and next steps sized so remediation can be quoted separately. If we find nothing serious, we say so and charge the fee — that is a valid outcome, and pretending otherwise would destroy the practice.
See the full shape of the review, including price.