Allowlist-gated browser harness for testing conversational AI agents

Problem

A no-code voice and chat agent platform exposes its test chat, call flows and agent settings only in its web app, not in its API, so an AI coding agent could not test or adjust those agents directly. Letting an AI click around an account where live, customer-facing agents sit one click away from test agents is also genuinely risky.

Goal

Give a coding agent reliable hands: talk to a test agent one message at a time, read its workflow and settings, and request changes through the platform's built-in assistant, while making it impossible to touch anything that has not been explicitly allowed.

My role

Sole designer and engineer of the tool, its agent-facing skill sheet, its test suite and its documentation.

Solution

A command-line program with no dependencies that attaches to a browser the operator has already signed into and exposes a handful of commands that return JSON. A short skill sheet teaches a coding agent when to use it. The safety rules live in the program, not in the instructions, because a model can argue its way past advice but not past code.

How it works

The operator signs in once. The tool attaches to that browser over its debugging protocol through a websocket, using a handful of protocol methods, and never handles a password.

Before touching a page, every command calls one policy check. The agent must be on a local allowlist; the specific capability must be enabled for it (chatting, reading the platform assistant's proposals, or applying a proposal, kept separate because applying one changes the agent); and the agent must have no phone number attached, checked live against the platform's API. If that field is missing or has an unexpected type, the check refuses instead of assuming safety.

Commands open a clean conversation, send one message and return the reply together with the flow block that produced it; read the whole workflow, including each block's form-control state; repeat a scenario many times to count how often a behaviour occurs; and send instructions to the platform's assistant with a before and after fingerprint of the workflow, so the exact change is reported.

The calling agent decides what to test and judges the answers. The tool only types, reads and refuses.

Credentials come from the operating system's keychain at the moment of use and are never logged.

Technical challenges

Driving a complex single-page app reliably (reply detection, reading flow nodes) without an automation framework.

A guard that fails closed when the platform changes its data shape.

Separating low-risk actions from ones that change the agent, and treating chat as not free of consequences, because a test message can fire webhooks, create records or book slots.

Building with only the runtime's built-in fetch, WebSocket and crypto.

A test suite that runs its own headless browser against a local fixture imitating the product's markup, so tests touch nothing real.

Capabilities

One-message-at-a-time chat testing
Workflow and settings reading
Variance measurement across repeated runs
Assistant-driven edits with before and after diffs
Allowlist and per-capability gating
Live refusal of any agent with a phone number attached
A status command
JSON output designed for AI agents

Technology

Impact

Manual click-through checks of voice and chat agents became repeatable test runs driven by an AI agent, with measured variance. The harness was used in real testing of several conversational agents, where approved changes were applied and retested with no risk to live agents. No metrics are claimed.

Skills demonstrated

Protocol-level browser automation
Safety engineering for AI agents: policy in code, fail-closed guards and separated capabilities
Tool design for LLM agents
Fixture-based testing
Dependency-free engineering