Allowlist-gated browser harness for testing conversational AI agents

Problem

A no-code voice and chat agent platform exposes its test chat, call flows and agent settings only in its web app, not in its API, so an AI coding agent could not test or adjust those agents directly. Letting an AI click around an account where live, customer-facing agents sit one click away from test agents is also genuinely risky. A chat test also cannot show how a voice agent copes on a real phone line, where callers interrupt, go quiet, change their minds and read out numbers that can be misheard.

Goal

Give a coding agent reliable hands: talk to a test agent one message at a time, read its workflow and settings, and request changes through the platform's built-in assistant, while making it impossible to touch anything that has not been explicitly allowed. Then test the same kind of agent the way its callers meet it: on a real phone call, with a recording of both sides and a verdict backed by evidence.

My role

Sole designer and engineer of the tool, its agent-facing skill sheet, its test suite and its documentation. Designer and engineer of the outbound phone caller, its scenarios, its scoring rules and its tests.

Solution

A command-line program with no dependencies that attaches to a browser the operator has already signed into and exposes a handful of commands that return JSON. A short skill sheet teaches a coding agent when to use it. The safety rules live in the program, not in the instructions, because a model can argue its way past advice but not past code.

A second tool, now an internal project at an early stage, is an outbound AI caller. It phones a voice agent deployed on a hosted voice-agent platform, plays a scripted test caller, records both sides of the call on separate channels, and scores the agent on that evidence. It is also built with no dependencies, and its safety limits live in configuration that the program checks before every call.

How it works

The operator signs in once. The tool attaches to that browser over its debugging protocol through a websocket, using a handful of protocol methods, and never handles a password.

Before touching a page, every command calls one policy check. The agent must be on a local allowlist; the specific capability must be enabled for it (chatting, reading the platform assistant's proposals, or applying a proposal, kept separate because applying one changes the agent); and the agent must have no phone number attached, checked live against the platform's API. If that field is missing or has an unexpected type, the check refuses instead of assuming safety.

Commands open a clean conversation, send one message and return the reply together with the flow block that produced it; read the whole workflow, including each block's form-control state; repeat a scenario many times to count how often a behaviour occurs; and send instructions to the platform's assistant with a before and after fingerprint of the workflow, so the exact change is reported.

The calling agent decides what to test and judges the answers. The tool only types, reads and refuses.

Credentials come from the operating system's keychain at the moment of use and are never logged.

The phone caller works like this. Gates first: a call is refused unless the target is marked authorized with a record of who approved it, the scenario is on that target's allowed list, every limit is set (call length, a call cap and a calling window), and no kill-switch file is present. Every live run also needs the operator's go for that run.

The call: a cloud telephony service places it with two-channel recording from the first second and connects it to the tool's own WebSocket server; the telephony service turns the agent's speech into text and the caller's replies into speech. The test caller says up front that this is an automated, recorded test call; that sentence is added by the program itself.

Each scenario gives the caller a goal and a fictional identity, and the scenarios cover scheduling, identity checks, privacy, interruptions, silences, off-topic questions and self-corrections. Scripted disruptions (talking over the agent, going quiet, correcting itself) are timed against the conversation. Replies come from hosted language models, with a backup route and a rule that only one route may speak per reply. If a reply fails, the caller tries once more before ending the call.

Test levels. Calls are grouped into two levels: a short check before an agent is shown in a demo, and a full stress test that has not yet run on a live line. Each level is designed to end in a go or no-go decision with its reasons, and one serious failure is enough for a no-go.

Evidence: after the call the tool downloads the recording, transcribes the agent's side with a local speech model, measures from the audio who spoke when, and builds a timed two-speaker transcript. It also reads the platform's own record of which steps the agent actually ran, such as booking or taking a message.

The verdict comes from rules written as pure functions, in three sections: agent performance (the only part that can fail a call), improvement flags, and test caller diagnostics. When the caller is at fault, for example it was not quiet long enough for a silence test, the call is marked inconclusive and never counted against the agent. Missing evidence is never a pass. Earlier verdicts are kept and never edited, and each call gets a replay page and a plain-language report.

Technical challenges

Driving a complex single-page app reliably (reply detection, reading flow nodes) without an automation framework.

A guard that fails closed when the platform changes its data shape.

Separating low-risk actions from ones that change the agent, and treating chat as not free of consequences, because a test message can fire webhooks, create records or book slots.

Building with only the runtime's built-in fetch, WebSocket and crypto.

A test suite that runs its own headless browser against a local fixture imitating the product's markup, so tests touch nothing real.

Turn-taking on a live phone call: the test caller has to wait for a pause, answer once and drop replies that went stale while the agent kept talking. The first real call showed the caller talking too long, and the fix was confirmed on the next call.

Judging timing from the recording itself rather than from when transcript text arrived, and judging spoken numbers on a local transcription of the recording, because live speech-to-text can mishear them.

Keeping the tester's own faults apart from the agent's, so a scripted disruption that did not really happen makes a call inconclusive instead of failing the agent.

Keeping the test caller to its own goal: it agrees only to what its scenario asks for and politely declines anything else the agent offers, so each call tests only the agent.

Recognising the different ways an agent confirms a booking, to cut false misses in scoring.

Capabilities

One-message-at-a-time chat testing
Workflow and settings reading
Variance measurement across repeated runs
Assistant-driven edits with before and after diffs
Allowlist and per-capability gating
Live refusal of any agent with a phone number attached
A status command
JSON output designed for AI agents
AI test calls placed to a deployed voice agent
Scripted interruptions, silences and self-corrections
Two-channel call recording and a timed two-speaker transcript
Evidence-based verdicts that never blame the agent for the tester's faults
Per-call gates: authorization, allowed scenarios, limits and a kill switch
Replay pages and plain-language call reports
A short check before demos, plus a full stress test with go or no-go rules

Technology

Impact

Manual click-through checks of voice and chat agents became repeatable test runs driven by an AI agent, with measured variance. The harness was used in real testing of several conversational agents, where approved changes were applied and retested with no risk to live agents. No metrics are claimed.

The phone caller is early. Its first test rounds ran on real calls against a demo receptionist line. They surfaced a phone number read back wrongly and an off-topic question the agent never answered, and they surfaced a turn-taking problem in the tester itself, which was found and fixed. Since then that short check has run on the same line, booking calls included. The first run ended early because of a telephony problem, and a retest of the failed and unfinished checks led to fixes in the tester. The full stress test has not yet run on a live line.

Skills demonstrated

Protocol-level browser automation
Safety engineering for AI agents: policy in code, fail-closed guards and separated capabilities
Tool design for LLM agents
Fixture-based testing
Dependency-free engineering
Voice agent testing over real phone calls
Evidence-based evaluation of AI agents