Engineering best practices for fast-moving teams

Flaky LLM evals: how to test AI apps without your CI lying to you

LLM apps break CI in a way normal code never does: the same input can pass or fail on a whim. Here's how to build evals you can actually gate on.

Flaky async tests: why fake timers beat sleep() every time

The setTimeout(500) in your test isn't a wait — it's a wager. Here's how to kill the single most common class of flaky async tests for good.

Vitest isolate: when turning it off speeds up CI and when it breaks your tests

Disabling Vitest isolate can cut your test suite time in half — or surface a dozen flaky tests you never knew you had. Here's the tradeoff, spelled out.

Cycle time: the DORA metric that actually changes behavior

Cycle time is the DORA-adjacent metric leaders love and misread constantly. Here's how to measure it, what it hides, and how teams game it.

Setting pass/fail thresholds for LLM evals in CI without gaslighting yourself

A hard-coded score threshold on a non-deterministic eval is a coin flip wearing a suit. Here's how to gate LLM changes in CI without lying to yourself.

Jest JUnit output: how to make Jest results readable in CI

Jest prints beautiful output in your terminal and useless output in CI. Here's how to emit JUnit XML that your pipeline can actually parse — and what to watch for.

Change failure rate: the DORA metric everyone quotes and nobody measures right

Change failure rate is the DORA metric leaders love to cite and teams love to game. Here's how to measure it honestly — and why flaky tests corrupt it.

Code coverage policy that survives contact with reality

Coverage is a proxy, not a goal. Here's when test coverage actually tracks quality, when it doesn't, and how to write a policy your team won't game.

The flakiest test in your suite is fighting over a database row

Order-dependent tests that fight over the same database rows are one of the sneakiest sources of CI flakiness. Here's how to find and kill them.

Your LLM evals are flaky, and your CI is lying to you

Non-deterministic LLM evals wreck your CI signal the same way flaky unit tests do — worse, actually. Here's how to build eval suites you can gate on.

Stop paying for cold caches: GitHub Actions runner cache benchmarks

The cache/save step looks free in your logs. It isn't. Real numbers on GitHub Actions cache restore times, layer reuse, and when self-hosted runners cut them to zero.

The async race condition is why your tests are flaky

The single biggest source of flaky tests isn't infrastructure — it's async code racing your assertions. Here's how to find and kill it.

Change failure rate: the DORA metric everyone measures wrong

Of the four DORA metrics, change failure rate is the one teams most often measure wrong — and the one most contaminated by flaky CI. Here's how to get it right.

Why 80% coverage mandates produce worse tests

A hard coverage number sounds like a policy. What it actually produces is a new category of test: one that executes code without checking anything.

Merge queues don't fix flaky tests — they industrialize them

Merge queues make CI failures more expensive, not less. One flaky test can serialize an entire team's output. Here's the math — and the fix.

Your test suite is a distributed system: the race conditions that aren't in your code

Your application code isn't the only place race conditions live. Your test suite runs like a distributed system, and it has all the same failure modes.

Test evidence for regulated teams: what auditors actually want from your CI

Auditors don't care that your tests passed. They care that you can prove which tests ran, against which code, and that the results weren't quietly massaged before you shipped.

Change failure rate is lying to you if your tests are flaky

If flaky tests trigger rollbacks or reruns, your change failure rate is counting noise as incidents. Here's how to fix the signal before it reaches leadership.

The real cost of a 5% test failure rate

A 5% flaky test rate sounds manageable. Run the numbers on rerun compute, engineer interruptions, and delayed releases, and it stops sounding that way.

The Vitest cache, explained: what it stores, when to keep it, and when it lies to you

Vitest has two caches, and most teams cache the wrong one in CI. Here's what each does, how to configure them, and how one of them can make your test order nondeterministic.

CI for AI agents: testing non-deterministic systems

Agents are the hardest software to test. A practical blueprint for CI for AI agents: trajectory assertions, tool mocking, eval harnesses, and what 'green' actually attests to.

Keeping AI-generated code from breaking your test suite

AI coding assistants raised your PR throughput — and quietly raised your flaky-test rate with it. Here's how to keep your CI signal trustworthy at the new volume.

Testing RAG pipelines: what actually breaks

RAG systems break in two very different places — retrieval and generation — and treating them as one testable unit is how you end up debugging blindly in production.

LLM-as-judge: how to use it without fooling yourself

LLM-as-judge can replace hours of human annotation — or silently validate garbage. Here's how to tell which one you're getting.

Setting up CI for prompt and eval regression testing

A prompt change that tanks quality is the silent killer of LLM features. Here's how to run evals in CI so every PR gets a score — and failing scores block the merge.

Flaky evals: when your LLM tests are non-deterministic

Your LLM eval passes on Monday, fails on Wednesday, and you changed nothing. That's not a model problem — that's a flaky eval. Here's how to fix it.

How to test LLM-powered applications

Traditional assertions fall apart when your function returns something different every time. Here's how to build a test suite that actually catches regressions in LLM apps.

Git metrics that actually tell you something useful

Most teams collecting git metrics end up measuring the wrong things. Here's how to cut through the noise and focus on signals that actually reflect engineering health.

Vitest performance: why your test suite is slow and how to fix it

Vitest is fast by default, but default isn't always fast enough. Here's how to actually diagnose and fix a slow Vitest suite.

How to test LLM-powered applications

Testing LLM-powered applications is genuinely hard — but most teams are making it harder than it needs to be. Here's what actually works.

docker/setup-buildx-action@v3: what it does, why it matters, and how to use it right

docker/setup-buildx-action@v3 is two lines of YAML that unlock multi-platform builds, layer caching, and faster CI — if you configure it correctly.

Code coverage: what it actually tells you (and what it doesn't)

A 90% coverage badge doesn't mean your code is well-tested. Here's what code coverage actually measures, where it misleads you, and how to use it without fooling yourself.

jest-junit: generate JUnit XML reports from Jest and surface them in CI

jest-junit transforms Jest's output into JUnit XML — the format every CI platform and test analytics tool understands. Here's how to configure it correctly.

How to speed up pytest

Runners: Slow test suites aren’t just annoying; they can grind your team’s momentum to a halt, delay releases, and make everyone grumpy.

How to speed Up Vitest

Runners: There are always room to squeeze out more performance.

How to speed up Playwright tests

Runners: We’ve seen a few tricks to help frontend teams and distilled findings down to a few concrete strategies without sacrificing coverage or reliability.

Non-obvious ways self-hosting GitHub Actions can increase CI cost.

Runners: With great power comes with great responsibility — it's not all sunshine and rainbows, there are some unobvious challenges.

BuildPulse Runners: Run your GitHub Actions 2x faster, at half the cost.

Runners: Run GitHub Actions builds on our optimized-infrastructure, geared to cut build times and cost.

Replit Case Study

Flaky Tests: How Replit crushes flakiness with BuildPulse

Auditboard Case Study

Flaky Tests:

Ultimate Guide to Fixing Flaky Tests

Flaky Tests: We see all kinds of flaky tests - see insider tips

Where do flaky tests come from?

Flaky Tests: To actually address the issue, it becomes imperative to uncover the root causes that underlie this mysterious phenomenon.

Minimizing flaky test impact

Flaky Tests: This tutorial outlines strategies to minimize their impact and enhance the functionality of your test suite.

What are the different types of tests

Flaky Tests: There are various types of software testing, each serving a specific purpose on the journey to a high-quality, bug-free software product.

TestOps in continuous testing

Flaky Tests: TestOps is a methodology that integrates software testing activities into the DevOps workflow seamlessly. It aligns testing with development...

Unit testing vs. integration testing

Flaky Tests: Unit testing and integration testing are fundamental types of testing strategies that help developers identify and fix issues at different stages

Effective unit testing

Flaky Tests: To excel in creating high-quality unit tests, adhering to a set of best practices and principles is vital.

Causes of flaky tests

Flaky Tests: While flaky tests might appear as insurmountable challenges, dedicated efforts in understanding and mitigating their underlying causes can pave the way to more reliable tests.

The impact of flaky tests

Flaky Tests: They derail a software engineer's workflow by causing non-deterministic behaviour, complicating debugging and making test results unreliable.

How to mitigate and prevent flaky tests

Flaky Tests: How do we mitigate the impact of flaky tests and prevent their recurrence?

How to fix flaky tests

Flaky Tests: In this guide, we'll discuss how to find and fix flaky tests in your test suite.

What are flaky tests?

Flaky Tests: The unique characteristic of flaky tests—producing inconsistent outcomes—leads to debugging difficulties and can considerably undermine the confidence

Code resilience through static analysis and code coverage

Code Coverage: As projects grow in complexity, managing source code and guarding against vulnerabilities become even more crucial.

What is dependency coverage?

Code Coverage: One crucial aspect of this management is understanding and evaluating dependency coverage - an often overlooked area of code coverage...

What are code coverage metrics?

Code Coverage: How you quantify, however, makes all the difference - let's explore the different types of coverage metrics.

Why code coverage tools can make or break your application

Code Coverage: Automated testing is a good first step in reducing the chances that your users encounter a bug, however, it's important to understand what you are testing in the first place.

How is code coverage measured?

Code Coverage: Code quality stands as a paramount factor in determining the success and sustainability of any project.

The difference between code coverage and code quality

Code Coverage: While code coverage metrics provide insights into the extent of testing, code quality metrics highlight the maintainability and robustness of the application.

What is the purpose of code coverage?

Code Coverage: Code coverage provides valuable insights into the thoroughness of your testing efforts, enabling you to identify areas of your codebase that might require further attention.

Building culture with cycle times

Engineering Metrics: How cycle time can be a cornerstone in building a robust engineering culture.

Code Readability

Engineering Metrics: How does it impact the broader development process?

Quality gates in pull requests

Engineering Metrics: Quality gates play a pivotal role in ensuring that the code meets the desired standards before it's merged.

Different ways of measuring code complexity

Engineering Metrics: Add a sentence here that will hook readers

Which git metrics should I be looking at?

Engineering Metrics: With a plethora of metrics available, which ones truly matter?

What is tech debt?

Engineering Metrics: Imagine borrowing time from the future to meet present needs.

How engineering leaders maintain sync with stakeholders

Engineering Metrics: This bridge is built on trust, transparency, and most importantly, metrics.

Preventing performance review bias

Engineering Metrics: Unconscious preferences can skew perceptions, biases can lead to unfair and unrepresentative assessments.

Why does deploy frequency matter?

Engineering Metrics: Let's explore the significance of deploy frequency and its implications in the software development landscape.

Efficient pull requests: the pathway to enhanced software quality

Engineering Metrics: We dive into best practices for pull requests, emphasizing the importance of a meticulous yet swift code review process.

Benefits of engineering intelligence

Engineering Metrics: The phrase "what gets measured gets managed" holds profound significance.

Building a data-driven engineering culture

Engineering Metrics: Let's explore the steps and strategies that can help teams harness the power of metrics to drive excellence.

Beyond story points and ticket status

Engineering Metrics: Traditional project management tools, which often revolve around story points and ticket statuses, might not be the best indicators of genuine progress.

The power of sprint velocity

Engineering Metrics: What is sprint velocity, and how does it fit into a broader, data-driven engineering culture?

Measuring your software delivery pipeline effectively

Engineering Metrics: Sofware delivery pipelines encompass both the 'how' - the stages like coding time, pickup time, and review time, and the 'what' - outcomes.

What is merge time?

Engineering Metrics: Among these guiding stars, merge time shines brightly, offering a glimpse into the efficiency of the engineering process.

How to accelerate engineering velocity?

Engineering Metrics: How can velocity be accelerated? Let's delve into the strategies, metrics, and tools that can supercharge your engineering velocity.

Measuring engineering performance without damaging culture

Engineering Metrics: Let's explore how engineering leaders can harness metrics to boost performance without compromising team culture.

Understanding engineering effectiveness

Engineering Metrics: Let's delve into the world of engineering metrics and explore how they can be harnessed to understand effectiveness without damaging the culture.

The 4 pillars of engineering productivity

Engineering Metrics: Enter the four pillars of engineering productivity: code velocity metrics, throughput metrics, quality metrics, and goals.

Engineering Productivity: how to measure and improve it

Engineering Metrics: Metrics like cycle time, lead time, and lines of code offer invaluable insights into the efficiency of the development process.

Improve productivity with just these 5 metrics

Engineering Metrics: While there are numerous metrics that engineering teams can track, focusing on the right ones can make all the difference.

How to improve cycle time and estimation

Engineering Metrics: As your organization grows, understanding and optimizing the performance of your engineering team becomes a non-negotiable.

Why start measuring engineering metrics and impact today

Engineering Metrics: As the digital realm evolves, understanding and optimizing the performance of your engineering team becomes a non-negotiable.

Enhancing engineering impact with metrics

Engineering Metrics: Organizations can enhance engineering impact through the strategic use of metrics, process optimization, and fostering a culture of continuous improvement.