The Missing Layer After Launch

You built an agent. You launched it. Everything works in the demo. Everyone is happy.

Now a few simple questions.

How do you know it is actually working out there? How do you watch across hundreds of real conversations every day? How do you feel the health of the system? How do you find the holes you do not know are there yet?

Most stories about agents stop at the moment you ship. We built it, it works, the end. I think shipping is the moment when the real work begins. Somehow only a few people talk about that part.

I call it the missing layer. I gave a talk about it at the AI Engineer World's Fair, and this post is the written version, with more detail on what we actually run at Wandero today.

Here is the world we live in now. You can build a whole product in a couple of weeks. You can write 100,000 lines of code with agents. We rebuilt our entire platform in three weeks, and that was months ago, with weaker models. Building has become the easy part. Even Cognition, the company selling the coding agent, puts writing code at about 20% of the problem. The rest is testing it, reviewing it, deploying it, and maintaining it.

Nobody tells you what happens after. The moment you launch, you start losing control. Not because the system got worse. Because you can no longer see it.

---

Why you lose control

An agent is not normal software.

Normal software has a few features and several flows. You can walk every flow before release, write tests for each one, and know what you shipped. The coverage is finite.

An agent has no predefined flows. Think about Claude Code or Codex. They do whatever the user needs. Most products built on agents work this way. You give instructions, the agent handles the request, and every customer asks for something different. You cannot write all the conversations in advance. The coverage is endless.

That is the part that kept me up at night after we launched. You lose the feel for your own system. Is it getting better or worse this week? Are customers happy? Is the agent struggling somewhere, quietly? You do not know. And the normal safety nets do not save you here.

We tried them. Unit tests. Rule-based checks. Scripts that simulate customer conversations. They help, and we still use them. But each one covers one slice of a problem that has no edges. Customers always do something you did not write down.

Three things make this a new problem.

Same input, different path. Models are not deterministic. A slight change in the input can send the agent down a completely different trajectory. You cannot enumerate the paths, so you cannot pretest them.

Failures hide. An agent can struggle in the middle of a long task, hit a problem, get lucky, find a workaround, and finish. No red alert. No error on the dashboard. Everything looks fine. But that hidden struggle was an early warning. The problem is still living in your codebase, and a reliable system should not depend on luck. Anthropic hit this too running long coding agents: "a major failure mode was Claude marking features complete without proper testing."

Finished does not mean helpful. One of our customers asked for a trip plan. The agent ran the whole flow and produced the itinerary. It also picked the wrong supplier service and made mistakes in the pricing. Technically, the session was a success. The task still failed, because a human looking at the result would not be happy with it. No log line captures that. LangChain wrote this failure down as a rule: "an agent can return a technically successful response and still fail the task itself."

Harrison Chase wrote the other rule, the one we learned the hard way: you do not know what your agent will do until it is in production. His reason matches ours. "You cannot fully predict how your agent will be used until real users start interacting with it." The inputs are infinite, and the model is sensitive to small changes. Production is where you learn what you needed to test in the first place.

---

What closing the loop means

So what do you actually need after launch?

For us it came down to four jobs. Not tools, jobs. The tools change, the jobs stay.

Feel the system. A zoomed-out view of the whole thing. How many sessions, how healthy, which way is it trending, where are the patterns. Without it you are nervous all the time, because anything could be happening and you would not know. This is the job that gives you back the pulse you lost at launch, and it is where every other job starts. You cannot fix what you cannot see, and you cannot prioritize what you cannot compare.

Maintain it. The classical job every software team knows. Watch the logs, catch the alert, find the root cause, fix it fast. With agents the job changes shape: the detection, the diagnosis, and the first draft of the fix can all be automated, so maintaining becomes reviewing fixes instead of hunting for them. This is the fastest loop in the system, and speed here is the point. A bug that lives for an hour costs you one bad session. A bug that lives for a month teaches your customers to stop trusting the product.

Watch how customers use it. This one surprised us. The models are powerful, but people do not know what the tools can do. They use the agent in ways that make their own life harder, and they do not complain. Somebody has to notice, because what comes out of this job is usually not code. It is onboarding, education, sometimes a product change. The maintenance job protects the system. This one protects the value the customer actually gets from it.

Keep the codebase healthy. When agents fix agents, PRs merge fast. Dead code accumulates. Documents go stale. And the codebase is the ground every other job stands on: the monitoring agents read it to diagnose, the fixing agents change it, the reviewer judges against it. If it rots, every loop above it gets slower and dumber. Nobody stares at that whole picture unless you make it someone's job. So we made it a routine's job.

Together the four jobs close the cycle. One sees, one fixes, one protects the customer, one protects the ground the others stand on. Something changes in production, the system notices, diagnoses, proposes a fix or raises a hand, and a human decides. The tighter that cycle runs, the faster the product improves and the calmer you are.

There is a simpler way to say all of this. Agents made shipping many times faster. They did not make your feedback faster. That gap between the two speeds is where products rot after launch. Closing the loop means one thing: making the feedback as fast as the shipping.

We are not the only ones who landed here. Replit published their version of this loop: cluster the production traces, catch the degradations that aggregate metrics miss, propose the fix, ship it the same day. Humans stay on four explicit gates: hypothesis, architecture, eval curation, launch approval. Much bigger scale, same shape.

One thing makes this loop possible at all. We do not own the model, so nothing here retrains weights. It does not need to. Everything around the model is changeable: the tools, the prompts, the skills, the docs the agent reads, the checks it runs. Almost every fix the loop produces is a change to that layer, and that layer is where the leverage lives anyway. OpenAI's Codex team asks one question when an agent fails: what capability, context, or structure is missing? Our loop exists to ask that question automatically, every hour, against production.

From my experience, the loop is at least as important as the product itself. Sometimes more. Everyone can call the same model. The feedback loop around it is yours.

---

The routines we run

Here is what this looks like for us today. Seven routines. The team that keeps our agents healthy is also made of agents.

This is not a recipe. It is what we run right now, and we keep changing it. I am sharing the shape, not the sauce.

The fast loop

Log monitoring. Runs every hour against production logs. It reads the errors and the warnings, but also the unusual: something slow, something that smells like a security or scaling problem, anything a person scanning the logs would raise an eyebrow at. When something needs attention, it digs into the codebase the way an on-call engineer would, looking for the root cause rather than the first suspicious line. Sometimes it catches a problem from the symptoms before anything crashes. When it is confident it found a real bug in our code, it opens a pull request with the fix. When it needs help, it sends a Slack message instead: here is what I found, someone should look at this.

Peter Steinberger runs a similar loop over his open-source repos on a five-minute tick. Ours is hourly on purpose. An hour of logs shows the same error hitting several sessions, which a single alert never could. Batches diagnose better, and the signal stays clean enough that we actually read it.

PR review. Every PR the log monitor opens gets reviewed by a different agent. This matters more than it sounds. The agent that wrote the fix is biased. It already believes in its own diagnosis, and agents are eager to ship. The reviewer starts with fresh context and a different goal: criticize. It checks whether the fix solves the root cause or just hides the symptom, whether it matches our coding practices, whether it breaks something nearby. It requests changes, and the author agent responds, and they loop until the PR is ready or closed. Only then does it reach us.

Cognition built their reviewer on the same rule: it works best when it shares no context with the coder. The fresh context is what does the work.

Session insights. Runs once a day over the latest customer conversations. Same instinct as log monitoring, different perspective. Logs tell you what the machines did. A session tells you what the customer experienced. This routine reads the trajectory of each conversation: what the user asked, what the agent did, whether the user was happy, whether the agent struggled or solved it cleanly. When something went wrong it investigates the codebase and, like the log monitor, ends in a PR or a Slack message.

One detail that took us time to appreciate: all of these agents have access to everything. Logs, database, sessions, the codebase, Slack. The same access a human debugging the problem would need. What makes them different routines is not the toolset. It is the goal and the starting perspective. One starts from the logs, one starts from the sessions, and they meet in the codebase.

The zoom-out

The daily session run also feeds a bigger picture: a health dashboard that scores every conversation. Not a sample. Every one. Per session it records what happened: the outcome, the number of turns and tool calls, the cost, what went wrong. Across sessions it finds the patterns. The rule that makes it useful: an issue that appears in five sessions is one pattern, not five problems. It names the pattern, counts the affected sessions, and suggests a fix.

We built the dashboard ourselves, because I know exactly which questions I want it to answer. This is the screen that gives me back the feel of the system. Health, trends, score distribution, sentiment, the AI insights connecting dots across the week. Before agents, this was simply impossible. You could not have someone read every conversation. Now you can.

Client usage

Usage monitoring. This routine zooms out from single sessions to customer behavior. It reads a few sessions from the same customer and asks a different question: are they using the product effectively?

The honest answer is often no, and it is nobody's fault. The tools are new. People do not know what agents can do. We found customers copy-pasting emails into the chat by hand, not knowing the integrations existed. We found someone asking the agent for the same thing every morning instead of turning on the automation that would do it daily. Nobody complains about this. They just get less value and slowly drift away.

This routine only notifies. There is nothing to fix in the code. It tells us: this customer does not know about this feature, this one is struggling with that workflow, someone should talk to them. It turned out to be as valuable as the bug-finding ones, because the biggest gap in agent products right now is not capability. It is education.

Anthropic's forward-deployed engineers run into this at every customer: nearly every platform is agentic now, every platform can do almost anything, and so "your customers have no idea what the heck it is that you actually do." You have to go look.

Codebase health

The last cluster watches the thing all the other routines write to: the codebase itself.

This part is a real debate right now. Dex Horthy argues that maintainability is the thing agent loops cannot fix: there is no way to penalize a model for eroding maintainability the way a failing test penalizes a wrong answer, so models never learned to care, and the fully automated software factories that ignored this have already collapsed once. I take the objection seriously. Our answer, for now, is to stop hoping maintainability emerges and make it an explicit job with its own routines.

Dead code cleanup. When a system improves itself this quickly, it leaves things behind. Old paths nobody calls, flags nobody flips. A routine sweeps for dead code and opens the cleanup PRs.

Docs and tests freshness. Agents sometimes update the code and miss the documentation, or leave a test asserting last month's behavior. This routine checks that what the docs and tests say still matches what the code does, and fixes what drifted.

The Claude Code team runs this pattern on itself: 20 or 30 routines maintaining their own codebases, including a one-sentence prompt that sweeps for dead code every day and opens the PR.

The weekly audit. Once a week, a routine looks at everything that changed and reviews it the way a principal engineer would. Did the new features break our system design? Did we introduce a security problem? Is this still scalable, still flexible, still the codebase we want? It is the slowest loop and the highest altitude, and it regularly catches the kind of drift no single PR review can see, because no single PR is wrong. Only the sum is.

There is also a selfish reason to spend tokens on this. Sonar measured an identical agentic task against a clean codebase and a typical one: the clean one needs "a material reduction in the amount of tokens, reasoning, energy." A healthy codebase stopped being a human comfort. It is what keeps your own agents fast and cheap. Spotify reached the matching conclusion from the other side: most of their pull requests are now written by agents, and "the more consistency we have, the better our agents work."

---

We stopped reading diffs

Here is the part that actually changed how it feels to run the company.

These routines open more pull requests every day than the three of us do. In the talk I said ten times more, and that was not an exaggeration. If we read every diff line by line, we would be the bottleneck of our own system, and the loop would be worth nothing.

So the PRs meet us at a different level. Every one arrives labeled, with a real description, and with an HTML artifact: a small page that explains what the PR is, why it exists, what the root cause was, what the risks are, with a diagram when the flow is complicated. Two or three minutes and you understand it. You jump into the actual code only when something smells wrong.

This is the shift that I did not expect when we started. The human moves up a layer. You stop being the person who reads every line and become the person who watches the system that reads every line. You still decide. Every merge is still ours. But you make the decision from a report, the same way a manager reads a summary instead of sitting in every meeting.

Artifacts are becoming the standard review surface. Claude Code now ships them for this exact job: "a PR walkthrough or a living project dashboard," shared with your team at a link. Lilian Weng names the destination: humans move up the stack, not out of the loop.

People ask whether the human should be in the loop at all. Maybe not forever. But I think the order matters: close the loop first, make yourself the bottleneck, and then removing yourself becomes easy. Most teams argue about autonomy before they have a loop worth automating.

---

Trust is the whole game

None of these routines earned trust on day one. Each one took rounds of iteration: tightening the noise filters, adjusting what counts as reportable, checking its output against reality until we stopped double-checking it. That calibration is the real work. An operations agent that cries wolf gets ignored, and an ignored routine is worse than none, because you believe you are covered. The routines are cheap to write and expensive to trust, and the expensive part is the one worth doing. If you run something like this and learned something different, I want to hear it.

The argument is simple.

Agents made building easy. The same power that lets anyone ship a product in two weeks is what makes the product impossible to watch by hand after launch. The coverage is endless, the failures hide, and finished does not mean helpful. So the watching has to be built, and it is built from the material of the product itself: agents, with the right access, the right perspective, and a human at the gate.

In the last post I wrote that the model is interchangeable and the system around it is the product. This is the part of that system nobody sees in a demo.

Shipping is the easiest part today. The loop is the part that comes after, and almost nobody budgets for it.

Everyone can use the same model. Everyone can build the same features. The system that watches your product, improves it every day, and keeps you calm while it grows: that part only you can build.

The harness that watches itself is the moat.