I Built an AI Design Agent. It Taught Me to Optimize the Workflow Instead.

I expected wiring the builder to be the hard part. I never wired it. I stopped one step short of finishing the agent — not because it broke, but because I measured the workflow first, and the measurement told me not to.

This is the second post in a series about redesigning a client’s site with Claude Code. The first post ended on a thread I said I most wanted to pull: halfway through the project I built a dedicated design agent to automate the loop, and by the end I’d almost stopped using it. This is why.

The pain that made me build it

The feedback never arrived the same way twice. It came through Figma from the client side, and Figma has no fixed shape for it: sometimes a comment pinned to a frame, sometimes an arrow drawn on the canvas pointing at nothing in particular, sometimes text typed straight onto the artboard, floating next to the thing it referred to. Every round, I opened the file, read the feedback in whatever shape it arrived, figured out which node it was about, and translated it into a code change by hand.

I was spending more time translating feedback than writing the code that answered it. And more pages were coming — main, then the product pages, then the institution variants. Same loop, again and again, page after page.

So I set out to build hands-off delegation: hand the agent a Figma link with feedback on it, and get back the code changes. I’d just review.

What the agent was

The plan was a pipeline. A feedback-laden Figma link goes in; the agent analyzes the screens, collects the matching feedback, and emits a clean screen → to-do list; a builder applies it.

I split it across cheap open models to keep the cost down — a hands/eyes split. The judgment ran on mimo-v2.5-pro (text-to-text, so no vision), and the actual seeing — reading a screenshot to find what an arrow was pointing at — got delegated to mimo-v2.5. Claude Code was the hands: file edits, the dev server, the loop. (Model IDs as of June 2026, run through OpenRouter.)

The intake side worked. Give it a link, it pulled the feedback out and paired each note to a node by coordinates. I was one wire away from full delegation — connect the builder, and the loop runs itself.

Then I measured it before wiring that last piece. The measurement is why the builder never got connected.

The measurement I ran instead of finishing

I didn’t want to guess where to automate. Claude Code keeps every session on disk, so I broke the workflow into steps and timed each one against the real history. (The first post counted 3,297 prompts across five worktrees; for this analysis I worked from the 1,359 I could cleanly tag and classify.) The plan was simple: find the longest step, automate that.

The longest step was human visual judgment. Not a step the agent ran slowly — a step only I could run. Roughly half of the back-and-forth loop — 49% — was me looking at a render and deciding it was wrong. That’s the exact thing I was trying to automate away, and it was the single biggest block of time.

That could have been a fluke of measurement, so I checked it three ways, and all three said the same thing.

First, I went hunting for time-saving levers, and every one came back small. Re-orientation waste — the agent re-reading a file it had just edited, re-running a command it just ran — was about 2% of turns. Context-keeping tricks I’d hoped would cut the ping-pong: near zero. There was nothing to optimize because nothing was broken. The turns were long because the work is genuinely multi-step, not because it was leaking time.

Second, I tested whether the bottleneck was information or judgment. If the agent just needed the exact values, feeding them in should reduce corrections. I injected the precise Figma values — colors, coordinates — and the visual corrections went from zero savings to zero savings. 0→0. The agent wasn’t stuck for lack of data. It was stuck because “this hierarchy reads wrong” isn’t a data problem.

Third, this wasn’t only me. Precise design-to-code is the thing nobody has cracked — Lovable, v0, Bolt all produce fast first drafts and all leave the pixel-level correction to a human. I was hitting the same wall the whole category hits.

The ROI made the decision concrete. An autonomous tool loop would spend around 20 steps and 431K tokens to arrive at a change that one deterministic coordinate calculation produced in a single line. I was about to spend real engineering effort automating the cheap part of the loop, while the expensive part — the judgment — was structurally un-automatable.

So I changed the question

The mistake was optimizing for time. Time was already near-optimal; there was nothing left to cut. The question that actually had a lever behind it was different: I’m a resource too. How often am I pulled in, and how much of that is avoidable?

Time optimization was 0–2%. Intervention optimization was 32%.

I’d been pulled into the loop 1,092 times. Sorting those interventions:

• Roughly 41% were irreducible. New direction, visual diagnosis — “this looks cheap,” “use that reference.” Only I can produce those. That’s not waste; that’s the job.
• Roughly 32% were avoidable. Two kinds. About 16% were approval stops — “ok, commit,” “yes, continue” — the agent pausing to ask permission for something it never needed to ask about. Another 16% were corrections cleaning up the agent over-reaching: I said move it, not resize it; do the one thing I asked, not the three things nearby.

The reducible part isn’t judgment. It’s the agent stopping when it shouldn’t and doing more than it was asked. That’s a real lever, and — unlike “make the AI have better taste” — it’s one I could actually pull.

The smallest right fix for each waste

Here’s what I got wrong for weeks: I thought the answer was one smarter agent. It wasn’t. Each kind of repeated waste wanted a different, smaller fix. Once I named them, the fixes stopped looking like “build a bigger agent” and started looking like five unrelated things:

• Approval stops → an auto-approve setting. The 173 “ok, commit” turns weren’t a tooling problem. They were a permissions problem. Letting the agent commit and run the dev server without asking removed them. No code.
• Over-reach → a behavior rule. “Do only what’s asked, one step at a time” as a standing instruction cut a chunk of the “I said move it, not resize it” corrections.
• Coordinate ambiguity → read the geometry, don’t guess. The arrows-on-canvas feedback had no structure to parse. Instead of teaching an AI to guess intent — or asking the designer to change how they work — I built the intake around how the feedback already arrived: pull the raw coordinates over the Figma REST API and decide what’s feedback by position (notes sit outside the section’s right edge, matched to screens by y). Intent became a calculation, not a guess.
• Cutout guesswork → a narrow skill. Images came in with backgrounds that needed removing, so I built a small figma-cutout skill. (This one has a twist — see below.)
• Every-turn review → section-level review. The biggest one. Instead of being pulled in on every turn, the agent runs a whole section end-to-end and stops once, for review. 1,092 interventions collapse toward roughly one per section.

A setting, a rule, a coordinate reader, a narrow script, a review cadence. Not one of them is “a more autonomous agent.” Every one of them targets a specific measured waste.

What that leaves the AI doing well

Strip out the judgment and what’s left is the deterministic slice — and there, the AI is genuinely good. It maps arrow-tip coordinates to the right node. It cut a WCAG contrast fail I’d have shipped: a mustard #A68200 at 3.61:1, below AA, swapped for a burnt orange #C2410C at 5.18:1. These are calculations, and it does calculations well.

The figma-cutout skill is the sharpest lesson in the whole project. I built it to remove image backgrounds, and it worked. Then I found the real problem: the images looked dirty because they were being exported through the wrong path. Fetch them the right way — rawImages — and there was nothing to cut out. The processing I’d automated wasn’t needed at all.

That’s the seam. The AI will solve the problem you hand it, precisely and fast. It won’t notice the problem is framed wrong. Framing stays human; the slice goes to the machine.

Where it landed, and proof it held

What survived isn’t an agent. It’s narrow deterministic tools (read Figma over REST, map feedback coordinates, cut images), auto-approve for anything reversible, section-level delegation with review, and me as the taste gate. Every-turn collaboration collapsed toward a handful of stops per page.

The strongest evidence that this was a real conclusion and not a rationalization is that it held. Here’s every tool call from my own logs, accumulated over the project — the autonomous agent I built against the narrow scripts that stayed:

The shape is the whole argument — and it’s a cumulative count, so a flat curve means I stopped reaching for that tool. The agent’s line goes almost flat right after I measured the workflow and barely grows after, ending at 132 total calls. The deterministic tools never stop climbing, ending at 382, nearly three times as much. I didn’t decide to abandon the agent; I just quietly stopped reaching for it, and the logs caught me doing it. And three weeks after the measurement, in the last week of the project, the pattern still holds: zero Workflow or Agent calls, screenshot-eyeballing down and Figma-coordinate fetching up, and me still typing pixel deltas by hand — “left: 6%”, “shrink that 10%.” The measurement predicted exactly the workflow I ended up with.

The takeaway

When a task requires human judgment, stop optimizing the agent and start optimizing the collaboration.

The narrow deterministic tools that survived each got their own writeup, if you want the mechanics rather than the conclusion: reading messy Figma comments by matching feedback to coordinates, getting Claude Code to migrate an Astro site to Next.js cleanly, and — on the “auto-approve for anything reversible” question — where Claude’s newer Cowork actually draws the folder-access boundary.

The agent isn’t where the time goes. The human-in-the-loop is, and you can’t automate that away — but you can measure around it. Three questions did the work:

1. How often does the human actually need to step in? Cut the interventions that aren’t judgment (the approval stops, the over-reach cleanup). Keep the ones that are.
2. Where is the agent repeating waste? Diagnose it, then fit the smallest thing that removes it — a setting, a rule, a coordinate reader, a narrow script.
3. What’s the smallest deterministic slice worth automating? Hand the AI the calculations. Keep the framing and the taste.

I set out to automate the human out of the loop. The data said the human is the one part you can’t remove — so I automated everything around them instead. That’s the win the agent I didn’t finish couldn’t have given me.

This post is the why and the what. The rest of the series is the how — each of these pieces, built out in full:

• Messy, non-uniform feedback → read it by coordinates. How you get an AI to apply a designer’s marked-up Figma feedback without it hallucinating — by building the tool around how the feedback arrives instead of asking the designer to change. (next)
• Every-turn review → section-level stop points. Where to let the agent run and where to make it stop — the delegation-and-review loop.
• Running it cheap → the hands/eyes model split. Putting judgment and vision on different open models to keep the cost down.
• The deterministic slice → what Claude is quietly good at. Contrast, coordinates, cutouts — and the rawImages twist, in full.
• Drift and inconsistency → a design-system doc the AI works from. Setting the rules up front so the agent stops reinventing them every section.

Same project, same logs. This post was where I decided what to optimize; those are how I did it.

Researched and drafted by me and Claude, from my own logs.