Why we do not trust autonomous agents yet (and it has nothing to do with trust)

It's not that we're unwilling to trust agents. It's that nobody's built the infrastructure that would make it rational.

I’ve believed for a long time that every decision is a data decision. Even the ones that feel like gut calls. Your gut isn’t magic, it’s just your brain running a calculation on data you can’t consciously see. It’s mostly experience, which is really just data you collected without labeling it as such. Even something that feels brand new to you, the first time you encounter it, is still your brain matching it against everything already stored.

Follow that far enough and humans start looking a lot like agents. We take in a constant stream of input and act on it. I don’t merge a pull request because of some ineffable human instinct. I merge it because a few thousand small data points, collected over years, told me it’s fine.

So when I watch an agent run continuously, pulling in new data, deciding, acting, pulling in more, I don’t see something alien. I see something with the same shape as what I do all day.

Which is why the current state of things bugs me a bit. We’ve had the model capability for a while now. So why are we still mostly stuck with agents that need a human checking in every few steps, instead of ones that just run?

Everyone says trust. I don’t think that’s it.

Ask around and you’ll get the same answer: it’s a trust problem. People aren’t ready to hand over control yet. We need a human in the loop. Give it time.

I don’t buy it, or at least not as the root cause. I think trust is the symptom people point at because it’s the easy word, not because it explains anything.

Here’s the thing: we trust systems we don’t control all the time, without thinking twice about it.

During my master’s I worked on a project building a monitoring system for an airport. Fed it a lot of simulated data, built out the dashboards, watched it flag things. Fun project, honestly one of my favorites from that time. But the real thing I took away from it wasn’t technical. It was seeing, up close, just how much effort sits quietly behind the scenes to keep that kind of system running as close to error-free as possible. Redundant checks on top of redundant checks. Alerts for things that will probably never happen. Logs for everything, kept far longer than seems necessary, until you realize why they’re kept that long.

That project is the reason I don’t think trust in aviation has much to do with the people involved. You get on a plane and hand your safety to a pilot you’ve never met and know nothing about. No idea how experienced she is, how much sleep she got, whether today’s a good day for her, or whether she’s about to make a small mistake. Doesn’t matter. You get on anyway. Same with the person who inspected the plane that morning, or the engineer who signed off on the bridge you drove to the airport on.

You’re not actually trusting the person. Pilots make errors, same as anyone. You’re trusting the system built around them, the one I got a small look at building myself. The license she had to earn before she was ever allowed near the controls. The recurring checks that would catch it if she started slipping. The published schedule that makes a delay visible within minutes, not months. The maintenance logs. The black box that, if something does go wrong, gives you an exact, reconstructable answer for what happened and why, regardless of whether the person at the controls made a mistake.

Nobody thinks through this list before boarding a flight. But it’s doing the work in the background, and you’d notice the second it disappeared. Pull any one of those pieces away and trust doesn’t slowly erode. It just falls off a cliff. Not because the pilot got worse. Because you’d have no way of knowing if she did.

Same logic, applied to agents

I think this maps onto autonomous agents almost exactly, and it leads somewhere more useful than “give it time.” Trust isn’t a function of time. It’s a function of infrastructure.

Right now there’s no license for an agent. No recurring check that catches it degrading mid-run. No enforced ceiling on what it’s allowed to spend or do before someone has to approve the next step. No black box that tells you, after something breaks, exactly what happened.

That’s not a trust gap. That’s a missing stack.

And it’s missing for a fairly boring, structural reason: almost everything built in this space so far (the frameworks, the observability tools, the mental model most of us default to) assumes a short session. Prompt in, model thinks, a couple of tool calls, answer out, done. That’s the shape the last three years of tooling was built around.

That shape doesn’t hold anymore. 2022 to 2023 was mostly about teaching models to think. 2024 into 2025 was about teaching them to plan. Now, in 2026, we’re giving them a sandbox and letting them go, sometimes for hours, sometimes indefinitely, and finding out the infrastructure underneath was never designed to hold that kind of weight.

What that costs, concretely

Not abstract. In November 2025, four LangChain agents running a market-research pipeline got stuck in a loop. An Analyzer and a Verifier kept triggering each other, neither one able to break the cycle. It ran for 264 hours. Eleven days straight. $47,000 in API spend. Zero usable output. (Full case study)

Nobody caught it while it was happening. It surfaced the way a burst pipe surfaces: someone noticed the bill.

The line from that post-mortem has stuck with me more than anything else I’ve read on this topic: the team had observability, they did not have enforcement. They could see the agents running the whole time. They just had no way to stop them before the next call went out.

That’s not people trusting agents less afterward. That’s people realizing they had a dashboard and no brakes.

The six pieces I think are actually missing

Once you stop calling this a trust problem, you can start naming what’s actually missing, the same way you’d audit gaps in any production system. Here’s where I keep landing:

Durable execution, first. Long-running agents live inside a process that can crash, get evicted, lose its state, and there’s no standard way for it to pick back up where it left off, the way a database transaction would.

Context management, second. A long context window isn’t the same thing as an agent that knows what to do with a long history. One noisy tool response early on can quietly poison every decision that comes after it.

Tool design, third. Tools built for a quick call-and-return break down once an agent is hitting the same one hundreds of times, or looping with another agent across a shared tool.

Memory and skills, fourth. There’s no real infrastructure yet for storing and versioning reusable agent behavior, so agents keep re-deriving solutions that already exist somewhere as a “skill.”

Enforcement, fifth. This is the $47K story again. Seeing a problem and being able to stop it are two completely different capabilities, and most of the industry only has the first one.

Observability, sixth. Most existing tools assume a trace is short and finishes cleanly. They lose data on a crash and choke at scale, right when a long-running agent needs them the most.

All six trace back to the same root cause. Infrastructure built for short sessions, now being asked to carry long-running, autonomous ones. And the pressure isn’t holding steady, it’s compounding. METR’s research puts the length of task an AI system can reliably complete on a curve that’s roughly doubling every seven months.

(METR, “Measuring AI Ability to Complete Long Tasks”) The capability curve is outrunning the infrastructure curve, and the gap between them is where all six of these problems live.

I’ll go through each of these one at a time over the coming weeks. What actually breaks, what’s already being built to patch it, what’s still wide open. Some of it I have strong opinions on. Some of it I’m still working through myself, and I’ll say so when that’s the case.

Because my honest take is that the obstacle isn’t that we’re unwilling to trust autonomous agents.

It’s that nobody’s finished building the thing that would make trusting them the rational choice.