tldr: Most AI testing tools hardcode you to one model provider. Passmark routes every AI call through a gateway, so a provider outage or price hike doesn't take your test suite down with it.
The lock-in nobody mentions when you buy an AI testing tool
When you pick an AI testing vendor, you're also picking their model provider. Nobody puts that on the pricing page.
Most tools call one provider's SDK directly. It works fine, until the day it doesn't. Model deprecations happen often enough that a dedicated tracking service exists just to keep up with shutdown dates across providers. Google's own Gemini deprecations page lists retirement dates on a rolling basis, and Anthropic's Claude 3.5 Haiku went from deprecation notice to scheduled shutdown in about six months. None of that is your product's fault. It's your testing vendor's dependency, and now it's your problem too.
You'd never build your app this way. You wouldn't hardcode a single payment processor with no fallback, or a single cloud region with no failover. Yet most AI testing tools do exactly that with their model provider, because it's the fastest way to ship a demo.
This is why Passmark, the open-source engine behind Bug0's done-for-you QA service, routes every AI call through a gateway instead of calling one provider's API directly.
What an AI gateway actually is
An AI gateway (sometimes called an LLM gateway) is a single API layer that sits between your application and multiple LLM providers. Instead of calling Anthropic's SDK or Google's SDK directly, you call the gateway. It handles routing, retries, rate limits, and observability, and can fail over to a different provider without you rewriting any code.
Contrast that with the common alternative: importing a provider SDK straight into your test runner. It's simpler to build. It's also a single point of failure with no ejector seat. This isn't a niche concern either. AI infrastructure vendors like TrueFoundry write about vendor lock-in prevention through AI gateways as a standard pattern for any team building on LLMs, not just testing tools. Testing infrastructure just hasn't caught up to that standard yet.
Why most AI testing tools skip this
Wiring up a gateway is extra work that doesn't show up in a demo. Calling one provider directly is faster to ship, and most teams don't feel the cost until something breaks.
What breaks, in practice: a model gets deprecated with a short migration window. A provider has an outage during your deploy freeze. A model update changes how it grades your assertions, and your pass rate shifts for no reason you changed. If your testing tool is wired to one provider with no fallback, every one of these becomes an incident for you, not a routing decision for your vendor.
How Passmark routes around it
Passmark connects through multiple gateways instead of one hardcoded provider: Vercel AI Gateway, OpenRouter, OpenCode Zen, and Cloudflare AI Gateway. It also supports direct provider SDKs when you want to bring your own keys. Passmark's product page puts it simply: bring your own Anthropic and Google API keys, or route through a gateway. This article is the explanation of why that choice exists.
Passmark also splits its AI calls across eight dedicated model slots covering step execution, assertions (primary, secondary, and arbiter), extraction, and user-flow exploration at different effort levels. That matters here because it means a single provider hiccup doesn't take down every capability at once. Step execution can keep running on a healthy provider while assertions fail over independently.
Gateway routing is one of two resilience layers in Passmark. The other is multi-model consensus, where Claude and Gemini evaluate assertions in parallel and an arbiter settles disagreements. Consensus is about accuracy, catching one model's hallucination before it decides your pass or fail. Gateway routing is about availability, keeping the suite running when a provider itself is degraded or unreachable. They solve different problems, and you need both.
What to ask before you buy an AI testing tool
Before you commit to an AI testing vendor, ask what happens on a bad day for their model provider:
- Are they calling one provider directly, or routing through a gateway with failover?
- If a model is deprecated, do you have to wait on their roadmap, or can they swap it without touching your tests?
- If a provider has an outage mid-release, does your entire suite stop, or does it degrade gracefully?
Most vendors haven't had to answer these questions yet, because most AI testing tools are young enough that they haven't hit a real provider outage during a customer's release window. That day is coming for all of them.
Why this gets more important, not less
Model releases aren't slowing down. Every quarter brings a faster or cheaper model worth switching to, and every switch is a chance for something to break if your testing infrastructure is wired to one provider. Betting your test suite on a single model choice made in 2026 is a bet that ages badly by 2027. Routing through a gateway is what makes swapping models a config change instead of a re-platform.
FAQs
What is an AI gateway?
A single API layer that routes requests to multiple LLM providers, instead of calling one provider's SDK directly. It handles failover, rate limiting, and observability across providers from one interface.
Why does AI testing need a gateway if the tests already work?
Because "already works" assumes the model provider never has an outage, never deprecates a model, and never changes pricing. A gateway is what keeps your tests running when one of those assumptions breaks, which eventually, one of them will.
What's the difference between an AI gateway and multi-model consensus?
A gateway is about availability: keeping your test suite running when a provider is down or degraded. Multi-model consensus is about accuracy: running Claude and Gemini in parallel on assertions so one model's mistake doesn't decide your pass or fail. Passmark uses both, for different reasons.
Does Bug0 lock me into one AI model?
No. Passmark, the open-source engine that powers Bug0's managed QA service, routes through Vercel AI Gateway, OpenRouter, OpenCode Zen, and Cloudflare AI Gateway, or direct provider SDKs if you bring your own keys. The engine is public at github.com/bug0inc/passmark, so you can read exactly how the routing works. Your tests aren't tied to one provider's uptime.
What happens if an AI provider has an outage during a test run?
With a gateway in place, Passmark can route around the degraded provider instead of failing every test that depends on it. Without one, a testing tool hardcoded to a single provider just goes down with it.
