Building and testing software with LLMs — evals, AI testing, and CI for AI apps

Testing AI-written code without wrecking your CI signal

AI assistants generate plausible tests at a volume your reviewers can't absorb. The fix isn't better prompts. It's CI gates that treat flakiness as a first-class signal.

A triage playbook for flaky evals: detect, stabilize, quarantine

You reran the eval and it passed. That's a coin flip, not a test. A triage playbook: measure flake rates, stabilize with seeds and tolerance bands, quarantine the rest.

LLM evals are flaky tests: how to build CI for AI you can trust

Temperature zero won't save you. LLM evals are flaky tests wearing a lab coat, and the fix is the same discipline CI teams learned years ago: measure, baseline, quarantine.

Your LLM evals are flaky tests. Run them like it.

Non-deterministic evals will wreck your CI signal the same way flaky tests do. Here's how to gate LLM changes statistically instead of rerunning until green.

MCP for CI: ask your test suite a question from Claude or Cursor

The Model Context Protocol lets your AI editor query your CI reliability data directly. Here's how to wire up an MCP server for CI and actually ask useful questions.

Flaky LLM evals: how to test AI apps without your CI lying to you

LLM apps break CI in a way normal code never does: the same input can pass or fail on a whim. Here's how to build evals you can actually gate on.

Setting pass/fail thresholds for LLM evals in CI without gaslighting yourself

A hard-coded score threshold on a non-deterministic eval is a coin flip wearing a suit. Here's how to gate LLM changes in CI without lying to yourself.

Your LLM evals are flaky, and your CI is lying to you

Non-deterministic LLM evals wreck your CI signal the same way flaky unit tests do — worse, actually. Here's how to build eval suites you can gate on.

CI for AI agents: testing non-deterministic systems

Agents are the hardest software to test. A practical blueprint for CI for AI agents: trajectory assertions, tool mocking, eval harnesses, and what 'green' actually attests to.

Keeping AI-generated code from breaking your test suite

AI coding assistants raised your PR throughput — and quietly raised your flaky-test rate with it. Here's how to keep your CI signal trustworthy at the new volume.

Testing RAG pipelines: what actually breaks

RAG systems break in two very different places — retrieval and generation — and treating them as one testable unit is how you end up debugging blindly in production.

LLM-as-judge: how to use it without fooling yourself

LLM-as-judge can replace hours of human annotation — or silently validate garbage. Here's how to tell which one you're getting.

Setting up CI for prompt and eval regression testing

A prompt change that tanks quality is the silent killer of LLM features. Here's how to run evals in CI so every PR gets a score — and failing scores block the merge.

Flaky evals: when your LLM tests are non-deterministic

Your LLM eval passes on Monday, fails on Wednesday, and you changed nothing. That's not a model problem — that's a flaky eval. Here's how to fix it.

How to test LLM-powered applications

Traditional assertions fall apart when your function returns something different every time. Here's how to build a test suite that actually catches regressions in LLM apps.

How to test LLM-powered applications

Testing LLM-powered applications is genuinely hard — but most teams are making it harder than it needs to be. Here's what actually works.