Expect vs Agent-Browser vs Stagehand vs Passmark: picking the right AI testing tool in 2026

tldr: The AI testing market has split into four distinct philosophies. Expect generates tests from your git diffs inside coding agents. Agent-Browser gives AI models a fast Rust-powered browser. Stagehand adds natural-language primitives on top of Playwright. Passmark caches AI-discovered actions so regression suites run at zero LLM cost after the first pass. Each tool is genuinely good at what it does. The right choice depends on whether you need test generation, browser automation, hybrid scripting, or cost-effective regression at scale.

---

The AI testing stack is fracturing

A year ago, the conversation was simple: "Should we use AI for testing?" In 2026, that question has been replaced by a harder one: "Which AI testing tool fits our workflow?"

The tools have diverged around a few core philosophies. Some focus on generating tests. Others focus on giving AI models direct browser control. A few try to augment Playwright with intelligence. And at least one is purpose-built for the economics of regression testing.

The market for ai browser automation tools has split into at least four categories, each solving a genuinely different problem.

We evaluated all four tools covered in this post. We ran them against real applications, measured their costs, and stress-tested their CI integration. This is what we found.

If you have been following our writing on why AI testing tools alone won't fix QA and Playwright test agents, this post goes deeper into the specific tools shaping the space right now.

Expect: test generation from code diffs

Expect (GitHub) is built by Million Software and takes a fundamentally different approach from the other tools here. It does not automate a browser directly. Instead, it reads your git diff, generates a test plan, and executes that plan in a real browser via Playwright.

What it does: You change code, Expect figures out what to test. No test authoring required.

It ships as a CLI testing skill that plugs into AI coding agents like Claude Code, Codex, Cursor, and Gemini CLI. When you invoke it, Expect analyzes the code you changed, reasons about what could break, and spins up a browser session to verify. It checks performance (LCP, INP), security (npm dependency vulnerabilities, CSRF), UI correctness, and feature completeness.

Expect ships as a CLI package but the primary way to invoke it is as a slash command inside a coding agent:

# Install globally (one-time)
npm install -g expect-cli

# Inside Claude Code, Codex, Cursor, Gemini CLI, etc.
/expect -m "test the checkout flow" -u http://localhost:3000

# CI mode for automated pipelines
/expect --ci -u https://staging.yourapp.com

Expect supports Chrome profile reuse and CDP connections to already-running browsers, which makes it practical in development workflows where you already have a session open.

Where Expect shines: Zero-config test generation. You do not write tests. You do not maintain tests. You change code, and Expect derives what needs checking. For teams that ship fast and have no test coverage at all, this is a big deal.

Where it falls short: Expect is designed for change validation, not regression testing. It tests what changed, not what might have broken elsewhere. The FSL-1.1-MIT license is more restrictive than MIT or Apache-2.0. And because it depends on external AI coding agents for its runtime, you are adding a dependency on those systems. It launched in March 2026, so the community and documentation are still maturing.

Agent-Browser: the fastest way to give AI a browser

Agent-Browser comes from Vercel Labs and has quickly become the most-starred tool in this comparison at 29,500+ GitHub stars. It is written in Rust, which tells you everything about its priorities: raw speed and minimal overhead.

What it does: A native CLI that gives any AI model composable browser control via CDP, with accessibility-tree snapshots that Vercel claims are 93% smaller than Playwright MCP equivalents. (Independent verification by paddo.dev shows the savings are real but vary widely by page complexity, so treat the number as directional, not guaranteed.)

Agent-Browser is not a testing framework. It is browser infrastructure. You get composable commands that an AI model can call to navigate, interact with, and observe web pages.

# Open a page
agent-browser open "https://demo.vercel.store"

# Take an accessibility snapshot (compact refs like @e1, @e2)
agent-browser snapshot

# Click an element by its ref
agent-browser click @e5

# Fill a form field
agent-browser fill @e12 "test@example.com"

# Take a screenshot
agent-browser screenshot

The accessibility-tree snapshot format is what makes Agent-Browser special for AI integrations. Instead of sending full DOM or pixel screenshots to a model, it sends a compact tree with numeric references. This slashes token usage dramatically, which matters when you are paying per token for model calls.

It also supports network interception, multi-tab workflows, device emulation, and ships with a dashboard on port 4848 for visual debugging.

Where Agent-Browser shines: Speed. Community (Apache-2.0 license, massive adoption). The accessibility-tree format is genuinely elegant, and other tools will likely adopt similar approaches. If you are building custom AI agents that need browser access, Agent-Browser is the best foundation available.

Where it falls short: It is explicitly not a testing framework. There are no assertions, no test plans, no pass/fail semantics. You need to build all of that on top. It is Chrome-only. And while the Rust binary eliminates Node.js as a dependency, you still need an AI model layer above it to make decisions. Agent-Browser is infrastructure, not a solution.

Stagehand: the hybrid Playwright+AI SDK

Stagehand (GitHub) is the most mature tool in this comparison. Built by Browserbase, it has been available since March 2024, has over 22,000 GitHub stars, and pulls 700,000+ weekly npm downloads. It is the incumbent.

What it does: Keep writing Playwright scripts, but replace brittle selectors with natural-language actions that self-heal when the UI changes.

Stagehand adds four AI primitives on top of Playwright:

import { Stagehand } from "@browserbasehq/stagehand";
import { z } from "zod";

const stagehand = new Stagehand({
  env: "LOCAL",
  modelName: "claude-sonnet-4-20250514",
});

await stagehand.init();
await stagehand.page.goto("https://demo.vercel.store");

// Natural language action
await stagehand.page.act("click on the Acme Circles T-Shirt");

// Structured data extraction
const product = await stagehand.page.extract({
  instruction: "get the product name and price",
  schema: z.object({
    name: z.string(),
    price: z.string(),
  }),
});

// Observe available actions
const actions = await stagehand.page.observe("what can I do on this page?");

// High-level agent execution
await stagehand.agent().execute("add the shirt to cart and go to checkout");

Stagehand's auto-caching is worth highlighting: when an action has been resolved before, it replays the cached selector without calling the LLM. When the UI changes and the cache misses, it re-engages the AI to find the new selector. The hybrid approach is genuinely clever in theory. In practice, teams have reported that server-side caching for act(), extract(), and observe() sometimes fails silently despite the docs claiming it works (issue #1767). Verify caching actually hits before relying on it for cost estimates.

It supports multiple AI providers (OpenAI, Anthropic, Google) through the Vercel AI SDK, so you are not locked into one model vendor.

Where Stagehand shines: Maturity. The MIT license. Multi-model flexibility. The extract() primitive is excellent for scraping structured data. The hybrid caching approach works well for tests that run against slowly-evolving UIs. If you need a general-purpose AI browser SDK, Stagehand is the safest bet in 2026.

Where it falls short: Each AI action takes 1-3 seconds, which adds up across large test suites. LLM costs scale linearly with test volume because caching is per-action, not per-flow. The ~75% success rate on novel tasks means you will hit flaky steps in complex workflows. And Stagehand is a general-purpose SDK, not a testing framework. You still need to structure your own test plans, assertions, and reporting.

Passmark: purpose-built regression testing

Passmark (GitHub) is our open-source framework, built specifically for one use case: running AI-powered regression tests at scale without the costs spiraling.

What it does: Write tests in plain English. The first run uses AI to discover actions and cache them. Every subsequent run replays cached Playwright actions with zero LLM calls.

The core insight behind Passmark is that regression tests are repetitive by nature. The same flows run hundreds of times. Paying for AI on every run is wasteful. So Passmark separates discovery (AI-powered, first run) from execution (cached Playwright actions, every run after).

import { test, expect } from "@playwright/test";
import { runSteps } from "passmark";

test("shopping cart flow", async ({ page }) => {
  await runSteps({
    page,
    userFlow: "Shopping cart flow",
    steps: [
      { description: "Navigate to https://demo.vercel.store" },
      { description: "Click Acme Circles T-Shirt" },
      { description: "Select color", data: { value: "White" } },
      { description: "Add to cart", waitUntil: "My Cart sidebar is visible" },
    ],
    assertions: [
      { assertion: "Cart shows Acme Circles T-Shirt" },
      { assertion: "Selected color is White" },
    ],
    test,
    expect,
  });
});

The caching layer uses Redis. On the first run, Passmark sends each step description to an AI model, which resolves it into concrete Playwright actions (selectors, clicks, fills). Those actions get cached per step. On run two and beyond, Passmark replays the cached steps directly. If the UI changes and a cached action fails, Passmark re-engages the AI to discover the new action and updates the cache.

One caveat on caching scope: today, caching is per single step. Multi-action sequences within one step description still re-execute via AI on every run. Flow-level caching that memoizes an entire sequence once and replays it wholesale is on the roadmap (issue #8) but not shipped. For workloads where each step is a discrete action, the current implementation already produces the cost curve below. For long multi-action steps, expect some AI calls on repeat runs.

Assertions use multi-model consensus: Claude, Gemini, and an arbiter model all evaluate the assertion independently. This reduces false positives significantly compared to single-model evaluation.

Other features include dynamic placeholders ({{run.email}} for disposable email addresses), 8 configurable model slots for fine-tuning cost vs. quality, and OpenTelemetry tracing for observability.

For the full backstory on why we built this, see Why we open-sourced Passmark.

Where Passmark shines: Regression economics. After the first run, your AI bill for test execution drops to near-zero. Multi-model assertions are more reliable than single-model checks. The Playwright foundation means you get all of Playwright's browser support, parallelism, and CI integration. Natural-language test authoring means non-engineers can read and write tests.

Where Passmark falls short: The community is small (219 stars). The FSL-1.1-ALv2 license may be a concern for some organizations. You need API keys for both Anthropic and Google (at minimum). The Redis dependency adds infrastructure. And the initial discovery run is slower and more expensive than subsequent cached runs, which means the economics only pay off if you are running tests repeatedly.

Known failure modes

No tool review is honest without a list of places each one actually breaks. These are real issues from the public GitHub trackers as of April 2026, not marketing-adjacent gripes.

Expect

• Process leak on macOS (#98): orphaned Playwright, ffmpeg, and Chromium processes can pin CPU at 150–400% per process. Watch your CI runner load average.

• Silent stalls (#80): 0.0.24/0.0.25 can hang with "Agent produced no output for 180s" against localhost apps.

• Cookies not injected (#89): the vaunted Chrome profile reuse has inconsistent results. Auth-dependent flows may fail in CI despite working locally.

Agent-Browser

• Windows ARM64 broken install (#1256): 0kb binary on Windows ARM. Use x86 Windows runners or Linux/macOS.

• CDP attach hangs on macOS Chrome 139 (#1193, fixed): --cdp used to hang indefinitely on specific Chrome versions. Resolved in newer releases, but worth checking if you're on an older agent-browser build.

• Profile session loses active page (#1211): --profile sessions drop track of the active page after the first command in some workflows.

Stagehand

• Anthropic models break act() (#1986): Claude wraps responses in $PARAMETER_NAME, breaking Zod validation. Affects the exact Claude Sonnet 4 model shown in most tutorials.

• Server-side cache not working (#1767): silent cache failures for extract(), act(), observe() despite docs claiming otherwise. Cost estimates depend on caching actually working, so verify before you budget.

• CUA CDP race conditions (#1778): v3 can throw -32000 Cannot find context during page navigation.

Passmark

• Multi-action caching gap (#8): flow-level caching that replays an entire multi-step sequence wholesale is on the roadmap, not shipped. Per-step caching works today.

• Requires two API keys (#25): Anthropic and Google both required for multi-model assertion consensus. OpenAI-only teams need to issue a Google key before they can run it.

• No configurable assertion retry (#6): failed assertions fail the test. Noisy flows may want retry semantics that aren't there yet.

Every tool on this list is under a year old except Stagehand. Expect these lists to change. Check the issue trackers before you commit.

A note on Browser Use

Readers are likely to confuse Agent-Browser (Vercel Labs, Rust CLI, browser infrastructure for AI models) with Browser Use (open-source Python library for building autonomous browsing agents, 80k+ stars). They are different projects solving related but distinct problems. Browser Use is the closest analog to a high-level autonomous agent in this category. It is intentionally excluded from this comparison because it is not positioned for testing: it is for building agents that browse the web for general tasks. Every "Stagehand vs Browser Use" post you have seen is comparing Stagehand's SDK to Browser Use's autonomous agent, not to Agent-Browser.

Head-to-head comparison

[image 1 here](Four-panel architecture diagram with one column per tool showing the stack for each: Expect sits inside an AI coding agent above Playwright; Agent-Browser is a Rust CLI calling Chrome directly via CDP; Stagehand is an SDK wrapping Playwright with AI primitives; Passmark is a Playwright library with a Redis cache layer and multi-model assertion consensus)

The "vs" breakdown

Expect vs Stagehand

These tools solve different problems at different stages of the development lifecycle. Expect operates at code-change time. It reads your diff, decides what to test, and runs those tests automatically. You never write a test file. Stagehand operates at test-authoring time. You write scripts that mix Playwright code with natural-language actions.

If your team has zero test coverage and ships daily, Expect gets you validation immediately with no upfront investment. If your team needs durable test suites that persist across sprints, Stagehand gives you the building blocks. Expect tests are ephemeral by design. Stagehand tests live in your codebase.

The cost profiles differ too. Expect calls an LLM on every invocation because it generates fresh test plans each time. Stagehand caches resolved selectors, so repeat runs against unchanged UIs skip the LLM. But Stagehand's caching is per-action, not per-flow, so complex suites still accumulate significant model costs.

For most teams, these tools are complementary rather than competitive. Use Expect in your coding agent for immediate PR validation, and Stagehand (or another framework) for your persistent regression suite.

Agent-Browser vs Stagehand

This is the most common comparison in the space, and it is somewhat misleading. Agent-Browser is infrastructure. Stagehand is a SDK. They operate at different layers of the stack.

Agent-Browser gives you raw browser control primitives that are optimized for AI consumption. Its accessibility-tree snapshots are dramatically more token-efficient than alternatives. It is fast because it is Rust, and it is flexible because it imposes no opinions about how you structure tests or workflows.

Stagehand gives you higher-level abstractions. act("click the submit button") is more expressive than agent-browser click @e14, and Stagehand handles the AI resolution internally. You also get extract() for structured data and observe() for page understanding, which Agent-Browser does not offer.

If you are building a custom AI agent that needs browser access, start with Agent-Browser. If you are writing tests or automation scripts and want AI-enhanced selectors, start with Stagehand. Many teams will end up using both: Agent-Browser for their agent infrastructure, Stagehand for their test authoring.

Passmark vs Stagehand

This comparison gets to the heart of the cost question in AI testing. Both tools use AI to resolve natural-language instructions into browser actions. Both cache those resolutions. The difference is scope and strategy.

Stagehand caches at the action level. Each act() call caches its resolved selector. But every action still gets evaluated independently, and cache invalidation is per-action. In a 50-step regression flow, you might have 50 separate cache entries, each with its own invalidation lifecycle.

Passmark caches at the flow level. The entire sequence of steps gets resolved once, and the full set of Playwright actions is cached together. On subsequent runs, the entire flow replays without any AI involvement. When a step fails, Passmark re-discovers just that step and updates the cache.

The economic difference is stark at scale. A 100-test regression suite running twice daily with Stagehand might cost $200-500/month in LLM fees even with caching, because novel actions, cache misses, and assertion evaluations all hit the model. The same suite with Passmark costs the AI budget for the first run plus near-zero for every subsequent run, unless UI changes force re-discovery.

Stagehand is the better choice if you need a general-purpose AI browser SDK for scraping, monitoring, or exploratory testing. Passmark is the better choice if your primary goal is regression testing at predictable cost.

Passmark vs Expect

Expect and Passmark are bookends of the testing lifecycle. Expect generates tests when code changes. Passmark runs tests to catch regressions across the entire application.

Expect does not produce persistent test artifacts. Each run is a fresh analysis of the current diff. This is powerful for PR validation but means you cannot build a growing regression suite with Expect alone. Passmark is the opposite: you author tests once in plain English, and they persist and run indefinitely with cached execution.

The ideal workflow uses both. Expect validates the specific changes in a PR. Passmark runs the full regression suite to catch unintended side effects. Expect catches "did I break what I changed?" Passmark catches "did I break something else?"

Licensing is similar (both use FSL variants), and both are relatively new to the market. The main practical difference: Expect requires an AI coding agent environment, while Passmark runs standalone in any Playwright-compatible CI setup.

Agent-Browser vs Passmark

These tools barely overlap. Agent-Browser is a browser control primitive for building AI agents. Passmark is a regression testing framework. Comparing them is like comparing a database driver to an ORM.

Where the comparison gets interesting is in how they relate to the broader AI testing stack. Agent-Browser could theoretically serve as the browser layer underneath a testing framework. Its compact accessibility-tree snapshots would be excellent for reducing the token cost of AI-driven test discovery. But today, Passmark uses Playwright directly, which gives it cross-browser support and a mature toolkit that Agent-Browser (Chrome-only) cannot match.

If you are building a custom testing agent from scratch and want maximum control over the browser layer, Agent-Browser is a strong foundation. If you want a working regression testing solution today, Passmark is ready out of the box.

When to use what

Use Expect when:

• You have minimal or no test coverage

• You want instant PR validation inside your coding agent

• Your team ships fast and cannot afford to write test plans

• You need security and performance checks alongside functional testing

Use Agent-Browser when:

• You are building a custom AI agent that needs browser access

• Token efficiency is critical for your AI pipeline

• You need the fastest possible browser control layer

• You want Apache-2.0 licensing and a large community

Use Stagehand when:

• You need a general-purpose AI browser SDK

• Your use case spans testing, scraping, and monitoring

• Multi-model flexibility matters (swap providers easily)

• You want the most mature and battle-tested option

Use Passmark when:

• Regression testing is your primary concern

• You run large test suites daily and LLM costs matter

• You want plain-English test authoring with reliable assertions

• You need multi-model assertion consensus for fewer false positives

[image 2 here](Decision flowchart starting with "What are you building?" branching through regression testing, PR validation, custom AI agent, and browser automation SDK to recommend Passmark, Expect, Agent-Browser, or Stagehand respectively)

Can they work together?

Yes, and this is probably the most important takeaway. These tools are not mutually exclusive.

A practical stack for a mid-size engineering team might look like:

1. Expect runs inside your coding agent during development. Every PR gets automatic test generation and validation.

2. Passmark runs your full regression suite in CI on every merge to main. Cached execution keeps costs predictable.

3. Agent-Browser powers any custom AI agents your team builds for monitoring, internal tools, or customer-facing automation.

4. Stagehand handles one-off automation tasks, data extraction, or exploratory testing where you need the flexibility of a general-purpose SDK.

The tools address different phases of the software lifecycle. Forcing a single tool to cover all phases is how you end up with an expensive, fragile testing setup.

The cost equation at scale

Here are the economics made concrete. Assume a team running 200 regression tests, twice per day, with an average of 10 AI actions per test.

Raw AI actions per month: 200 tests x 10 actions x 2 runs x 30 days = 120,000 actions.

These are estimates based on public pricing and observed token usage, not measured benchmarks. None of these four tools has published an independent, third-party cost study on a standardized workload. Your actual numbers will vary significantly based on model choice (GPT-5 vs Claude Sonnet 4.6 vs Gemini 2.5), prompt length, application complexity, and how often your UI changes. Treat the table as a relative ordering, not absolute dollars.

The pattern holds regardless: if your primary use case is running the same tests repeatedly, step-level caching (Passmark's approach) produces a dramatically lower cost curve than action-level caching (Stagehand) or no caching (Agent-Browser, Expect). The tradeoff is upfront investment. The first run is the most expensive because every step requires AI discovery. That cost amortizes over every subsequent cached run.

For teams that also want fully managed QA without maintaining any of this infrastructure, Bug0 Managed handles the entire regression suite for you. You can book a demo to see it in action.

Two risks nobody talks about

License nuance. Expect and Passmark both ship under Functional Source License (FSL) variants that revert to MIT or Apache-2.0 after a two-year delay. FSL is not OSI-approved. Legal teams at larger organizations will flag it as non-standard and may block internal use, even though the eventual conversion makes it functionally open source. Stagehand (MIT) and Agent-Browser (Apache-2.0) avoid this friction entirely. If you are evaluating for a regulated industry or a company with a strict OSS policy, start the legal conversation early.

Model churn risk. Every tool on this list is coupled to specific model versions and behaviors. When OpenAI ships GPT-5.4 or Anthropic ships Claude Sonnet 5, some act() calls, prompt templates, or tool descriptions will stop working as expected. Stagehand issue #1870 (an invalid reasoningEffort error after a model upgrade) is the kind of break to expect — that one is fixed, but new variants ship every quarter. Caching helps insulate you: cached tests do not care about model behavior until the cache misses. Uncached tools (Expect, Agent-Browser) feel model updates immediately. Plan for a maintenance window each time your upstream model rolls.

FAQs

What is the best AI testing tool in 2026?

There is no single best tool. Stagehand is the most mature and general-purpose. Expect is the fastest path to test coverage if you have none. Agent-Browser is the best browser infrastructure for building custom AI agents. Passmark is the most cost-effective for regression testing at scale. The right choice depends on your primary use case.

Is Stagehand free?

Yes. Stagehand is open source under the MIT license and free to use. You will need API keys for at least one AI provider (OpenAI, Anthropic, or Google), and those providers charge for model usage. Browserbase offers a hosted runtime if you do not want to manage your own browser infrastructure, and that is a paid service.

What is Agent-Browser used for?

Agent-Browser is a CLI tool that gives AI models fast, composable control over a Chrome browser. It is used as the browser layer in AI agent pipelines, automation workflows, and custom testing setups. It is not a testing framework by itself. You need to build test logic, assertions, and reporting on top of it.

How does Passmark reduce AI testing costs?

Passmark caches each step's resolved Playwright actions in Redis after the first AI-powered discovery run. On every subsequent run, it replays those cached steps directly without calling any AI model. The AI is only re-engaged when a cached step fails (usually because the UI changed), at which point Passmark re-discovers that specific step. Your LLM costs end up proportional to how often your UI changes, not how often your tests run. Caching is per-step today; flow-level caching that memoizes multi-action sequences wholesale is on the roadmap.

Can I use Expect in CI/CD?

Yes. Expect supports a --ci flag that runs it in headless mode suitable for CI pipelines. It can connect to a running Chrome instance via CDP or launch its own browser. Since Expect analyzes git diffs, it works best in CI environments where it can access the current changeset.

Does Stagehand work with Anthropic models?

Yes. Stagehand supports Claude models through the Vercel AI SDK. You can configure it to use claude-sonnet-4-20250514 or other Anthropic models as the AI provider for action resolution and data extraction.

How is Passmark related to Bug0?

Passmark is the open-source AI regression testing framework built by Bug0. It powers both Bug0 Studio (self-serve testing, from $250/mo) and Bug0 Managed (done-for-you QA, from $2,500/mo). We open-sourced Passmark because we believe the testing framework itself should be transparent and community-driven. Read the full story in Why we open-sourced Passmark.

How do I get started with Bug0?

If you want to run Passmark yourself, install it from GitHub and follow the documentation. If you want a fully managed QA solution where Bug0 engineers build and maintain your regression suite, book a demo to learn about Bug0 Managed.