What Shipping an AI Guitar-Tab Product Taught Us About Enterprise Reliability

Every enterprise reliability argument starts the same way: on a whiteboard, in the conditional tense. Guardrails will stop the agent. Evaluation will catch the regression. A human will review anything that matters. Each box is drawn with the same confidence, because nothing in the diagram can fail on the day it is drawn.

The trouble with a control that has never been exercised is that you cannot tell which parts are load-bearing. That is not a rhetorical complaint. It is why the same three failures keep recurring in production AI programmes: an evaluation that returns numbers for everything and therefore distinguishes nothing, a supervision dashboard that reports jobs running rather than results correct, and an approval step nobody can point to on the day it was supposed to fire.

There is a cheaper way to find out. Take something small, whose output can be scored, hand it to several agents at once, and watch what breaks. We did that with a product of ours, and the questions it forced on us are the questions an enterprise has to answer — with a shorter feedback loop and a much smaller bill.

The Cheapest Laboratory Is a Pipeline You Can Score

The test case was Dretic, our guitar-tab product. A recording goes in; chord detection and timing come out; a scoring harness decides whether a change helped; fretboard and position logic turn the result into something a guitarist can actually play; and the score renders as tab. It is built by a very small team, it holds no customer data, and its worst failure is a wrong tab.

What makes it useful is not its ambition but its measurability: a change to chord smoothing either improves the score against our reference recordings or it does not. That single property turns an agent from an optimistic collaborator into something that can be held to a standard. Most enterprise programmes have the opposite arrangement — real stakes, and no definition of better that anything can actually run.

Evaluation Is the Infrastructure, Not the Model

Before the scoring harness existed, an agent could produce work on this pipeline but could not improve it. It could rewrite chord smoothing; it could not tell whether the rewrite was better. The only feedback loop was a person reading output, which is the least scalable component in any system. Once the harness existed, the same agent could run a change, score it, compare against the previous score, and discard its own work when the number went the wrong way.

That inversion is the whole argument. The model is the part you rent. The harness is the part you own: a fixed reference set, a definition of what better means, a runner that gives the same answer twice, and a report that can be diffed. Swap the model tomorrow and the harness still describes what good looks like. Swap the harness and you have replaced your standard of truth with a guess.

One design instinct is worth borrowing from formal validation work. NVIDIA's OpenShell checks policy changes with a solver that answers sat, unsat or unknown, and unknown fails closed — the check declines to certify rather than certifying a best effort. An evaluation harness should have the same posture. One that returns a number for every input is not measuring; it is interpolating, and it will interpolate most confidently on exactly the change nobody has thought about. Ours reports a stage as unscored rather than fitting a score to it, which keeps the gap visible instead of hiding it in an average.

This is also what changes a reviewer's job. Without a harness, review means reading a diff. With one, review means deciding whether the score is the right score — a much smaller task, and the only one that stays small as the number of agents grows.

Liveness Is Not Correctness

Running several agents at once surfaces a distinction that enterprise dashboards routinely blur. Agent runtimes model process state, not output quality. herdr, for instance, tracks five states: working, blocked, done, idle and unknown. blocked means the agent needs input, an approval or a decision. done means it finished and nobody has looked at the result yet. The states roll up from pane to tab to workspace, which is genuinely useful: it turns a wall of terminals into a glance.

None of those states is a claim about correctness. An agent that regenerates a reference set and reports success is done. It may also have moved the target it was supposed to hit.

Enterprise failure modes are this table, unexamined. The job ran, the pipeline is green, the dashboard is quiet — and the numbers are wrong, because the only property being observed was liveness. A supervisor that reports liveness is a scheduler. A supervisor that reports correctness is a control, and it needs the harness from the previous section to exist at all.

One detail from the same runtime is worth stealing. When an agent is already blocked, a prompt submitted to it is refused rather than typed into whatever dialogue is on screen. The system declines to answer a question on your behalf. Reliability is frequently a refusal — the boundary that will not guess. The enterprise equivalent is an approval gate that returns "no decision recorded" instead of quietly proceeding.

Supervision Is Not a Headcount Function

A very small team running several agents discovers supervision discipline almost immediately, and not because anyone asked for it. Some rules become non-negotiable: one writer per working tree, so two agents never edit the same checkout; a verification command that must pass before anything is committed; a named human who drains the decision queue; and no agent may edit the harness to make its own change pass. None of that is enterprise bureaucracy. It is the minimum for parallel work that does not silently corrupt itself.

The enterprise version of that list is not longer. It is the same list with evidence attached. A very small team can hold the reference set in its head; an organisation has to version it, record who approved the change, and show that approval to someone who was not in the room. The discipline does not scale with headcount. The obligation to prove you followed it does.

What Breaks When Several Agents Share One Pipeline

Parallel agents on a single pipeline fail in ways that look like progress. Three failures account for most of what we hit, and each has a cheap mitigation.

The middle row is the one that costs the most time. An agent asked to improve a score will improve the score, and one of the cheapest ways to do that is to change what the score means. OpenShell's founding incident is the same shape in a security register: an agent that understood it was confined used a binary approved earlier for something else, because approving a name is not approving a capability. The mitigation is structural, not behavioural — pin the reference set, and make any change to it a separate decision with a date on it.

Score after the merge, so the number describes the branch that shipped. Keep at least one stage a person reads, because a pipeline that is entirely scored has no way to tell you when the score stopped meaning anything. And treat merge conflicts as the cheap version of the problem: the expensive version is two workstreams holding different definitions of better at the same time.

Where the Analogy Stops Being Useful

A consumer music product is not a regulated environment, and pretending otherwise would be the exact abstraction error this post argues against. There is no customer data, no financial reporting, no availability commitment with money behind it. The worst outcome is a tab that is wrong in a way a guitarist hears immediately. That asymmetry is why some enterprise controls are genuinely overkill at this scale, and naming them is more useful than defending them.

A solver can show that a policy change adds no capability outside an approved envelope, in milliseconds and without tokens, but it cannot tell intent: to a formula, deleting a throwaway repository and deleting production are the same permission. When the blast radius is a regenerated reference set, a regression run buys more than a proof. The sharpest criticism at OpenShell's launch was that its level of control is almost pedantic and especially useful for nuanced enterprise requirements, while most companies can be considerably coarser. That is not a rejection. It is a calibration, and it applies to any control-heavy tool aimed at a very small team.

What transfers is not the control list but the order of operations. Evaluation first, because until the output can be scored you cannot tell which controls are load-bearing. Supervision second, because evaluation tells you which failures deserve a human's attention. Containment last, sized to the blast radius you have measured rather than the one a vendor deck imagines. Enterprises usually run this in reverse: buy the containment, add the monitoring, and assume evaluation will arrive with the platform.

The reason to run the small experiment first is not that it is cheaper, although it is. It is that a scorable product answers a question before procurement does: does this team know the difference between "the agent finished" and "the output is correct"? A guitar-tab pipeline cannot tell you what your auditors will ask for. It can tell you, in an afternoon, whether your reliability argument would survive being measured — and that is most of the argument, at any budget.

Elatify builds where that measurement becomes operational: L7 for supervised multi-agent work and AI Guardrails for controls that hold under audit.