Verdict
An AI agent that tests Jira tickets, fixes them and proves the fix
Give it a ticket. It tests the live product in a real browser and returns a verdict with screenshot evidence, then finds the root cause, writes a minimal fix, boots the app to prove it works, and opens a draft PR. The QA team uses it today to resolve real tickets, which people then review, approve and ship to production.
- Of its results approved by a person
- 91%
- Of its results approved by a person
- AI cost per approved ticket
- ~$3.87
- AI cost per approved ticket
- Products set up: two web apps and a mobile app
- 3
- Products set up: two web apps and a mobile app
- From first research to a tool the team uses
- ~3 weeks
- From first research to a tool the team uses
// how it works
The core loop
Each step is a separate command with its own output, so a person can stop, read and decide at any point. Nothing reaches the shared repo without proof.
Check
Reads the ticket, comments and attachments, writes acceptance criteria, and tests them on the live product with a real browser or HTTP.
Verdict
FIXED, WORKS, STILL_FAILING, NEEDS_INFO, BLOCKED or NOT_TESTABLE, judged only against its criteria, with screenshots as evidence.
Fix
A coding agent finds the root cause in a local clone and commits a minimal change, inside file and line limits.
Verify
Checks out that commit, boots the app's dev server and runs the same check again, now against the local build.
Draft PR
Only after a passing verify on that exact commit: push, open a draft pull request and link it on the ticket.
Review
People review the PR, approve or ask for a follow-up round on the same branch, and ship. Verdict never merges.
// screenshots
What it looks like
// architecture
A CLI that does the work, a control plane that runs it
The core is a small Python CLI on the Claude Agent SDK. The rule that shapes everything: the agents only test or edit a local clone; anything with write access to shared systems (Jira comments, git push, pull requests) is done by deterministic code after validation. On top, a FastAPI control plane queues runs, streams their logs live and serves the team's board. It never imports the agents: it spawns the CLI exactly as a person would.
Check agent
Claude Agent SDK · Playwright MCP
- Real browser through Playwright MCP, plus an HTTP tool limited to the environment's hosts.
- Tools: read its run folder, the browser and HTTP. No shell, no file writes, no Jira tools.
- Returns a typed JSON verdict: criteria, steps, evidence, confidence.
- Login secrets are substituted by the browser layer; the agent never sees them.
Fix agent
Persistent workspace per repo
- One locked git tree per repo; dependencies reinstall only when the lockfile changes.
- Hooks keep every path inside the clone; protected paths and diff size limits are enforced.
- Secrets are scrubbed from its environment, and the push token never reaches .git/config.
- Follow-up rounds continue on the same branch and update the same PR.
Verify & PR gate
Proof before a pull request
- Boots the dev server in its own process group, waits for it, tests, then kills the whole group.
- A failed boot is BLOCKED, never STILL_FAILING: no false failures from a broken environment.
- The gate allows a PR only for a passing verify on the fix's exact commit.
- A forced PR needs a reason and opens with a "Not verified locally" line.
Control plane & board
FastAPI · SSE · React 19
- Serial run queue, live logs over server-sent events, cancel that really reaps the browser.
- Sign in with Atlassian: every comment, clone, push and PR is made as the person who asked.
- Tabs for the board, PRs, learning, metrics and activity; a cost ledger per card.
- Files are the record; SQLite is only an index that can be rebuilt from them.
// pipeline
Where the AI acts, and where it doesn't
The agents read untrusted input: ticket text, attachments, web pages, repo code. So they get the minimum tools, and every step that changes a shared system is plain code or a person.
- TicketJira, read only
- Checkbrowser + HTTP
- Fixlocal clone only
- Verifylocal build
- Gatepassing verify?
- Draft PRpush + comment
- Reviewapprove, ship
- Verify still failing, or a reviewer asks for changes: a follow-up round fixes again on the same branch and updates the same PR.
- Environment or login down: the verdict is BLOCKED, never a false fail.
- Agents propose what they learned about the product; a note reaches later prompts only after a person approves it, so a ticket can't plant instructions.
// design principles
Agents test, code writes, people decide
An AI agent that edits code and talks to shared systems has to be boring to trust. Three rules made that possible.
Agents test
least privilege
Each agent gets only the tools its job needs: the checker can browse but not write, the fixer can edit but only inside its clone. Everything they return is a typed schema.
Code writes
proof before a PR
Jira comments, pushes and pull requests come from deterministic code, behind a gate that needs a passing verify on the exact commit. Nothing is ever merged automatically.
People decide
measured against their calls
Every Done card carries a person's approve or reject with a reason. Metrics, confidence calibration and learning all start from those calls.
// engineering decisions
How it was built
Researched first, designed in writing, then built in small specs and plans, all with Claude Code as a pair. I made the calls and reviewed every change.
Research before code
Studied existing ticket-to-browser agents, production repro bots and coding agents. The key lesson taken: false greens and false fails from expired logins or a down environment.
Prompt-injection aware
Tickets and web pages are treated as data. Tools are restricted, paths are confined by hooks, project settings can't leak in, and learnings need human approval.
Code-only mode for mobile
A mobile app can't be opened in a browser, so its check reads code instead, verify runs static checks on the fix and its base, and every result says "code review only".
Several repos per product
A ticket can live in the web app, a GraphQL gateway or a shared package. The check suggests the repo, the fix works there, and verify and PR follow it.
Honest metrics
Approval rate, cost per approved ticket, and calibration: if high-confidence results are approved no more often than low ones, the confidence is noise, and the page says so.
Adopted by QA
Started as a CLI for me and became a tool the QA team uses: they run checks and fixes from the board, review the PRs, approve them and ship to production.
// what's in it
Feature set
- Six verdicts with screenshot and video evidence, posted to Jira on request
- Ticket intake: rich text, comments, links, images and video frames
- Fix with follow-up rounds and guidance, on the same branch and PR
- Verify against a local build of the fix's exact commit
- PR gate with forced overrides that must give a reason
- Board with drag-and-drop stages, live logs and cancel
- Sign in with Atlassian; runs act as the person who asked
- Cost per run and per card, kept apart from plan estimates
- Learning tab: keep or dismiss what the agents propose
- Metrics: approval rate and confidence calibration per agent
- Code-only mode for the mobile app
// tech stack
Built with
AI
Backend
Frontend
Integrations
Quality
My role
Creator and lead engineer: research, design, architecture, agents, guardrails, backend, board UI and rollout to the QA team. Built with Claude Code and the Claude Agent SDK.