AI I built and run

Production AI, with a budget and a gate.

Written and run at Chainlabs on the Node, Postgres and pgvector stack the vertical already runs, at no new running cost. 2,278 of the vertical repo’s 2,386 commits are mine. How I work with the agents, and how I make sure I understand what they build, is on the About page.

Three patterns
Under the hood
Measured on public data
Also built
Public code

Gateway

A multi-provider gateway with a cost ledger and a kill switch

OpenAI, Anthropic and Gemini behind one interface, used by two production services: automatic fallback, a 15-minute breaker that skips a dead provider and retries it later, 60-second timeouts and a daily limit of 20 AI questions per user. Every call, embeddings included, is priced per provider and model into a shared cost ledger; unpriced calls are flagged rather than counted as zero.

Every call, embeddings included, goes through the gateway and into the ledger.

Retrieval

Hybrid-search RAG

pgvector similarity plus Postgres full-text search, merged by reciprocal rank fusion, over the team’s own documentation. The knowledge base is split by audience.

Hybrid search over the team’s own documentation. No cross-encoder reranker in production.

Evaluation

Evaluation sets with a regression gate

Eval sets built from human-verified QC results, with client details removed, and LLM judges validated against human labels before use. I am not quoting production recall or pass rates until I can publish them with a date and a sample size.

A prompt change cannot merge if the pass rate drops.

Under the hood

Spend control and the tool loop.

The two-tier budget trip. It holds until a human resets it.

The ops assistant the 43-person network uses. About 17,000 lines of Node; 197 of its 200 commits are mine.

Fifteen production patterns

One platform, five jobs.

Retrieval

Hybrid-search RAGTool-call retrievalDocument ingestionGuarded text-to-SQL

Control

LLM gatewaySemantic cacheFeature flagsJob queue

Quality

Eval setsRegression gatePrompt registry and A/BSecond opinionTraces and replay

Docs

Self-healing docsDrift check

Decisions

Fine-tuning deferredVision check retired

Measured on public data

The same pattern, rebuilt and measured.

My team’s hybrid-search pattern rebuilt clean-room on a public finance benchmark, BEIR FiQA, 648 queries. Rank fusion lifted nDCG@10 from 0.25 to 0.37 over keyword search, but vectors alone scored 0.40, so fusion has to be measured per corpus.

Fusion beat keyword search on every metric; vectors alone beat fusion (p<0.0001). A cross-encoder rerank added nothing significant and 374 ms per query. Public benchmark numbers, not employer data.

github.com/mallyma1/hybrid-search-measured

Keyword search (BM25)0.25

Rank fusion0.37

Fusion plus rerank0.37

Vectors alone0.40

nDCG@10, BEIR FiQA test set, 648 queries

Also built

What keeps it running.

Keep it answering

Every call traced; any trace can be replayed against another model and diffed; p50 and p95 latency per feature in a weekly digest.
Per-provider failure reasons, a live provider-health page, and a log scanner that flags outage signatures such as rate limits.
A Postgres job queue with leases, retries, dead-lettering and a dollar cap per job.

Keep it affordable

A semantic cache with a 30-day expiry: exact matches first, then matches by meaning only within the same entity, so an answer about one exchange is never reused for a similarly named one.
Feature flags in one table: every capability ships off or in shadow mode, and I switch it on from Telegram once it has earned it.
Built, measured and retired a GPT-4o vision check on payment screenshots when it proved unreliable and costly, with a decision record so nobody rebuilds it.

Keep it useful

A feedback loop: a thumbs-up or a staff correction becomes a stored answer for later similar questions, and a thumbs-down retires it.
A research pipeline: Gemini search with citations, then a separate call extracts structured JSON.
Document ingestion of PDF, Word and Excel as text, images deliberately left out.
Guarded text-to-SQL: the model sees table structure only, never rows, and one read-only SELECT over approved tables.

Keep it honest

Self-updating docs: an LLM reads merged pull requests and proposes FAQ updates as reviewable PRs, applying only simple new entries itself.
A prompt registry with hash-checked versions and a stable per-user A/B split, reported with the sample size.
A second opinion on manual-review QC rows: two models advise, the rules engine decides.
Fine-tuning, on purpose not yet: nothing is trained until evals show a failure retrieval cannot fix. About 20 Claude Code subagent definitions, with a fact-checker as the final gate.

Before Chainlabs: LLM-based quality checks on contract summaries and compliance documentation at the Starknet Foundation.

Public code

Built on public data. Employer code stays private.

Python · benchmark