Trust Is the Bottleneck Primitive
There's a growing argument in the infrastructure world that GitHub is the wrong shape for a world where agents write most of the code. That software delivery needs to stop being a sequence of human actions and become infrastructure: a small set of primitives like source control, execution, artifacts, caching, identity, policy. I think that's right, and I want to argue with it anyway.
Because the energy in that conversation goes to the throughput-shaped primitives, execution and caching and artifacts. Understandable, those are the ones you can sell compute for. But identity and policy, the two that always get listed and then walked past, are the ones that decide the actual question underneath all of this, which is: why have humans in the pipeline at all?
I spent a few evenings trying to answer that by designing the thing. A spec for autonomous merges. Attestations, policy engines, the works. I want to walk you through why every version of it failed, because the way it failed is more interesting than the design.
The ceremony
Strip a modern delivery pipeline down and the humans persist at three checkpoints: code review, deploy approval, incident response. Notice what these have in common. None of them are technical work anymore. Agents already write the code, review the diff, and roll back on alarms, often better than we do at 4pm on a Friday. The three checkpoints are trust ceremonies. A human signs so that, if it goes wrong, there is a human to ask.
We've seen this movie. Self-driving cars could handle Phoenix years before they were allowed to. Capability arrives early, permission arrives late, and the gap between them is where all the interesting engineering lives. Waymo didn't close the gap by making the car smarter; they closed it with geofences, remote operators, incident telemetry, and a decade of receipts. Bounded consequences and fast attribution, until the track record did the talking.
So I tried to build the receipts.
The spec, and how it died
Version one was the obvious design. Every agent-produced change ships with a signed attestation: which agent, which model, hash of the full prompt transcript, blast radius of the diff, eval scores. A policy engine reads the attestation and decides whether a human signature is required. Autonomy becomes a policy expression instead of a product mode. Clean.
# change-attestation v0.1
subject:
repo: github.com/org/repo
base: sha
head: sha
diff_digest: sha256 # binds attestation to exact content
producer:
agent: claude-code@2.x
model: claude-fable-5
identity: spiffe://org/agents/ci-bot # workload identity, not API key
session_digest: sha256 # hash of full prompt/context transcript
transcript_uri: s3://... # optional, for audit retrieval
human_principal: jared@ # who invoked/owns the agent, nullable
intent:
task_ref: JIRA-123 | issue URL | freeform
instruction_digest: sha256 # the original prompt, hashed
analysis:
blast_radius:
paths: [src/api/**]
surfaces: [public-api, db-migration, deps, iac, none]
computed_by: tool@version # blast radius must be tool-derived, not agent-claimed
evals:
- suite: golden-v3
score: 0.94
threshold: 0.9
runner_digest: sha256
checks: [lint, tests, sast] # standard CI results, referenced not duplicated
signatures:
- keyid: ...
sig: ... # sigstore-style, over the whole doc
It fell apart in about an hour of honest reading.
The blast radius field is computed by static analysis, because obviously you can't let the agent grade its own homework. Except blast radius is a semantic property and path analysis is syntactic. A one-line change to a serializer alters the wire behavior of every public endpoint while touching nothing your analyzer flags. Worse, the agent wrote the code, which means the agent can shape the code to fall under the analyzer's thresholds. That's Goodhart's law with a cryptographic signature on it.
Fine, add a second agent as reviewer. Chain the attestations. But your reviewer agent is probably the same model family as your author agent. Same training distribution, same blind spots, possibly the same contaminated context if it read the PR description. That is not two-party review. That is one party with two signatures. I had rebuilt LGTM-stamp culture, at machine speed, with better fonts. The AI safety literature studies this under the name "AI control," using untrusted models to monitor other untrusted models, and its central finding is roughly what you'd fear: monitoring helps, but correlated failure between author and monitor is the load-bearing risk [5].
Fine, hash the transcript so auditors can reconstruct what happened. Except the harness decides what goes in the transcript, and the harness is run by the party being audited. Fine, bind identity with workload certs. Except the cert authenticates the wrapper process, and the actual producer is a model behind an API whose weights and system prompt can change without the cert noticing.
The pattern, once you see it, is everywhere in the design: every mechanism pushes trust one layer down and declares victory. Agent claims defer to analyzers. Analyzers defer to attestations. Attestations defer to identities. Identities defer to a model provider who does not countersign anything. It's trust laundering, and I was the one running the laundromat.
None of this should have surprised me. Ken Thompson made the general form of the argument in Reflections on Trusting Trust in 1984: you cannot verify your way to trust in a system whose toolchain you didn't fully author, all the way down [1]. Forty years later I was rediscovering it with YAML. There's also a more flattering ancestor for the pre-merge dream: Necula's proof-carrying code, where the artifact ships with a machine-checkable proof of its own safety [2]. It worked beautifully for narrow properties like memory safety and never generalized to semantic ones, which is the blast-radius problem wearing an academic gown. And to be fair to the present: the supply-chain world has built real machinery here. in-toto and SLSA give you signed, chained attestations of who ran which step of a pipeline [3][4]. My spec was roughly in-toto extended to cognition, and that's where it breaks. in-toto can prove which process produced a change. It cannot prove the cognition behind the process was trustworthy, because nobody can, and the honest response to that is not a better attestation.
How humans actually do it
Here's the part that took me embarrassingly long to notice, given that I've spent seven years inside a company famous for its promotion process.
Humans don't solve this problem with attestations either. Nobody has ever verified my code semantically before trusting me with a deploy. What actually happened, at every job I've had, is slower and dumber and works: I showed up. I fixed a few small things without breaking anything. Someone got coffee with me and decided I wasn't reckless. I made small talk in the right hallways. My blast radius got quietly raised, not by a policy engine, but by a distributed consensus of people who had watched me operate. Tenure, coffee chats, and a rollback rate. Those are the credentials. The interview was theater; the trust came later, on the job, in tiny increments, and it was revocable the first time I paged someone at 2am for something stupid.
We extend trust to track record plus consequences, not to competence. The colleague we trust isn't the smartest one, it's the one who has been wrong in front of us and handled it well. The human-factors people have a name for the target state: calibrated trust, where your reliance on a system tracks its actual reliability rather than your feelings about it, and miscalibration in either direction (misuse or disuse) is the failure mode [6]. That paper is about cockpit automation from 2004. Swap "autopilot" for "coding agent" and it reads like it was written last month.
Agents cannot get coffee. But notice that nothing in the human mechanism actually requires the coffee. It requires bounded early responsibility, observed outcomes, and a memory of both. All three are buildable. Arguably they're more buildable for agents than for people, because agents don't get offended when you check.
The provenance problem
There's an objection here and it's a real one. People do not like AI output, categorically, before quality even enters the room. AI art, AI music, AI writing: label it, and a large share of your audience checks out on principle. I say this with some authority, because I make music with AI tools on the side, and I can report that the same track gets a very different reception depending on what you disclose about how it was made. The work didn't change. The provenance did.
You could dismiss this as sentiment that engineering will steamroll. I don't think you should, and the behavioral evidence backs the instinct. The algorithm aversion studies found something specific and damning: people abandon an algorithm after watching it err once, even when the algorithm demonstrably outperforms the humans they retreat to, and they extend no such death penalty to human forecasters making the same mistake [7]. That looks irrational until you read it as a provenance heuristic. It's the same instinct that makes your staff engineer uneasy about auto-merging agent PRs. For all of human history, output quality was evidence of a trustworthy process behind it: a person who cared, who practiced, who has something to lose. AI breaks the heuristic. The output can be excellent while the process behind it has no skin in the game whatsoever. People aren't rejecting the artifact. They're rejecting the missing accountability, and they're right to, because accountability is the thing we were actually using quality to estimate.
Which means the answer to "do people trust AI?" is: wrong question. People trust processes with consequences and memory. Today's AI has neither, so no. Give it both and the question changes shape.
What the primitives actually are
So here's my amendment to the primitive list. The bottlenecks are not execution and caching; those are bought problems, and several good companies will happily sell them to you. The bottleneck primitives are the machine equivalents of tenure and the coffee chat:
Exposure caps. Stop trying to prove a change is safe before merge. You can't; see above. Instead, make it structurally impossible for the change to exceed the consequences you've granted it. Progressive rollout with automated rollback isn't a deployment nicety in this world, it's the trust mechanism. You don't need to classify risk correctly if you can cap it: the geofence, not the smarter car. Yes, the pieces exist: error budgets from the SRE canon, automated canary analysis à la Kayenta, speculative merge queues of the kind Uber published for taming thousands of daily commits [8][9][10]. The claim isn't that the machinery is new. The claim is that its job changes: it was built as a reliability mechanism, and in an agent-authored world it gets promoted to the trust mechanism, because it's the only one left standing.
Reputation ledgers. Per agent, per repo, per class of change: rollback rate, incident attribution, divergence between eval scores and production outcomes. Autonomy expressed as policy over the ledger. "No human signature required where this agent's 90-day rollback rate in this repo is under half a percent and exposure is capped at five percent of traffic." Earned per-context. Revoked automatically on regression. Tenure, as a data structure.
Verified versus asserted, as a schema-level distinction. Anything the agent claims about itself is metadata. Policy may only gate on what the infrastructure observed. This sounds like a detail; it is the entire design. Every failure in my spec came from letting an assertion cosplay as a fact.
The uncomfortable implication, and here the infrastructure crowd wins the argument even where I'm quibbling with its emphasis: once trust lives in a runtime ledger and enforcement lives in the delivery pipeline, the pre-merge ceremony really is vestigial, and the forge really is just storage. The pull request becomes what the interview already is for humans. Theater. Pleasant, occasionally useful theater, but not where trust is decided.
Trust is decided the same place it always was. On the job, in small increments, with receipts.
To pin down what I'm claiming, since every ingredient above has prior art: the composition is the new part. Reputation-gated autonomy, evaluated only over infrastructure-verified fields, enforced by runtime exposure caps, applied to agent-produced changes. Attestation frameworks prove process, proof-carrying code proves narrow properties, progressive delivery bounds blast radius, calibrated trust describes the target. Nobody has stacked them into a delivery substrate where an agent's permission to act is a function of its ledger. That's the spec worth writing.
We just have to build the hallway.
I'm working on an open spec for the change-record and ledger formats described here. If you're wrestling with agent autonomy in your delivery pipeline, I'd like to compare notes.
References
[1] Thompson, K. Reflections on Trusting Trust. Turing Award lecture, CACM, 1984. [2] Necula, G. Proof-Carrying Code. POPL, 1997. [3] in-toto: A framework to secure the integrity of software supply chains. CNCF. https://in-toto.io [4] SLSA: Supply-chain Levels for Software Artifacts. https://slsa.dev [5] Greenblatt, R. et al. AI Control: Improving Safety Despite Intentional Subversion. 2023. [6] Lee, J.D. & See, K.A. Trust in Automation: Designing for Appropriate Reliance. Human Factors, 2004. [7] Dietvorst, B., Simmons, J. & Massey, C. Algorithm Aversion: People Erroneously Avoid Algorithms After Seeing Them Err. JEP: General, 2015. [8] Beyer, B. et al. Site Reliability Engineering. O'Reilly, 2016. (Error budgets.) [9] Kayenta: Automated canary analysis. Netflix/Google. https://github.com/spinnaker/kayenta [10] Ananthanarayanan, S. et al. Keeping Master Green at Scale. (Uber's SubmitQueue.) EuroSys, 2019.