AI Pipeline Audit
We won’t build your dark factory. We’ll tune the one you have.
We run our own: a delivery loop where agents plan, build, review and merge — numbers on the homepage, book on the shelf, one real company shipped through it end to end. That took years and every failure mode in the catalog.
You don’t need to repeat that. You need someone who already has to read your pipeline — the prompts nobody’s opened in four months, the retrieval call returning the wrong chunk, the agent one weird input from a public apology, the token bill 40% above necessary — and tell you what to tune first.
Scope
Six areas. Nothing invented — same six every time.
A
Context Engineering
Prompt inventory, context assembly, RAG quality, instruction drift.
B
Evaluation & Regression Safety
Do evals exist, do they measure user-relevant things, is there a gate before prod.
C
Cost & Latency
Spend per feature, model routing, caching, retry storms. Typical outcome: 30–60% reduction.
D
Failure Modes & Reliability
Hallucination surface, agent loop guards, degradation, silent-failure detection.
E
Security & Abuse
Prompt-injection surface, exfiltration paths, tool over-permissioning, PII flow.
F
Architecture & Operability
Provider abstraction, observability, prompt rollback story, ownership topology — benchmarked against our own published operating loop.
Where the fix is something we’ve already open-sourced, that’s a line in the report with a reference implementation attached — never a pitch inside the findings. The report has to stand without it, or it reads as an ad.
Process
Read-only access. A findings register with a dollar figure on every line.
Focused — 5 days
Read-only access, trace samples, invoices, and 60 minutes of one engineer’s time. Enough to find every CRITICAL and HIGH.
Deep — 2 weeks
A live-traffic week, an eval set built from your own edge cases, a re-test after the first round of fixes lands.
The deliverable
Exec summary → findings register (severity CRITICAL / HIGH / MEDIUM / LOW, evidence, $ impact, scoped fix) → cost model → 90-day plan → appendix.
Run the loop. Keep the memory.
The audit compounds the same way the loop does.
Every audit’s anonymized findings become a field note on /blog by default — unless you opt out. Your edge case makes the next audit sharper, and you’re cited as a design partner if you want to be. Nothing client-identifying goes out without sign-off.
Pricing
One ladder. One guarantee.
Focused Audit
5 days, six areas, findings register + call.
$3,500
Deep Audit
2 weeks, live-traffic week, eval set built, re-test.
$7,500
Fix Sprint
We implement the CRITICAL/HIGH fixes, fixed scope.
$10–25K
Operating Retainer
Ongoing tuning of your loop, 10–15 h/wk.
$6–10K/mo
Dark Factory, turnkey
We build you the full loop. The first call is us trying to talk you out of it.
from $150K
Guarantee
If the findings don’t exceed the fee in annualized savings or risk reduction — half back.
The turnkey tier exists to be declined. It makes the audit read as trivial, and — because we openly discourage the big build — it buys a kind of trust no one anchoring on implementation revenue can afford. If you insist anyway, $150K+ is a fine problem to have.
Intake
Worth 20 minutes?
Write to a person, not a form. Tell us what you’ve built and what worries you about it — we’ll tell you within a day whether Focused or Deep is the right entry point.