I Built a Robot to Read My Mind and Believed It Anyway.

I can drive across town with total ignorant hubris. I'm 100% certain I'll arrive at my destination using the most efficient route possible. Thank you, Google Maps.

However, if you dropped me in the middle of Boston, took away my phone, and asked me to find my way home, I'd instead traverse a random walk until I ran out of gas and, by extension, all sense of existential purpose. Being (ahem) of "masculine persuasion," it would never occur to me to actually ask another human for directions. Gone are the days of paper Road Atlases in the back-seat pockets with their street indices, lookup tables, and cross-referenced submaps. The designated navigator had to have keen eyes and quick fingers to flip through the large books at critical junctures because there was no automatic rerouting if you missed the turn. Only anxious screaming and fights.

Somewhere along the way, I handed over my sense of direction to Google. It isn't just me: the more people lean on GPS, the worse their own navigation gets.

In London, taxi drivers have to memorize every street in the city to gain their license. This notoriously difficult test is called "The Knowledge," which makes it sound like it involves magic wands, robes, and people using words like “brilliant,” "chuffed," and "fair play” when they vanquish the examination. Mastering The Knowledge literally reshapes the part of their brain that handles spatial memory.

Me? I barely have the knowledge to find the grocery store that’s literally around the block from my house.

Early this week, I had coffee with Guy Fedorkow, who studies a 1950s computer called Whirlwind at the MIT Museum. To debug programs that exist on punch tapes older than almost anyone reading this, he uses a variety of techniques to decipher, emulate, and patch gaps in the machine's ancient code.

He's a scholar of software as an artifact.

Sometimes he uses AI to help with the (often tedious) work. It makes mistakes, and he catches them because he can still reason about the layers underneath the prompt. He understands the nuances of style and the context in which humans once worked with the machine. These are abstractions that an AI might find irrelevant and annoyingly human (pdf), though they are thoroughly embedded in the culture on which everything else is built.

In the same way, when I'm following the Google Maps blue dot, sometimes I know it's taking me the long way, so I override it, even though the algorithm sometimes knows something I don't about road conditions ahead.

There's the person who turns an idea into the steps a machine follows: the programmer.

And there's the person who runs the machine without needing to know what's under the hood: the operator.

The blue dot makes me the operator of my own commute. I follow the instructions, I don't write them. I'm nowhere near "close to the metal". Both are real ways to get where you're going, and you can slide from one to the other without ever deciding to.

People have been nervous about this for as long as we've had the tools. When Grace Hopper built one of the first compilers in the early 1950s, people insisted it couldn't possibly work. There was serious debate over whether the convenience of "automatic programming" would outpace human understanding, and whether there would be any need for human programmers at all.

Sound familiar?

As the tools get better at handing us finished things, we shift our role from the programmer of our work to the operator.

It's a quiet trade, and an old worry. Calculators did not make us mathematically helpless, but every time a tool moves you from doing to watching, the muscle you stop using goes slack: lean on the autopilot and your manual flying gets rusty enough that regulators now nag airlines to make pilots hand-fly more often.

Will we even notice the moment it happens?

Like most people with too many ideas, I keep a graveyard of half-thoughts: hundreds of notes, sparks I meant to chase and mostly didn't. A few months ago, with a lot of AI help, I built a little tool to dig through the pile and tell me where to spend my attention. It reads every note, groups them by topic, and ranks the topics into a tidy leaderboard: here's what you keep circling back to. Start at the top.

It worked. It handed me a clean, confident table, and I believed it. I shifted what I actually worked on based on where things landed. I was the happy operator, reading the scoreboard, not the math behind it. And that's fine, mostly. The whole point of a good tool is to stop doing the boring part by hand.

Then I peeked under the hood.

I'd quietly built the ranker twice. The first version scored each topic by how much I kept returning to it, basically, how big the pile was. A few days later, I built a second, smarter version that scored by what had actually paid off, which topics had turned into finished work. Two reasonable ways to rank the same pile, and I'd never put their answers side by side. So I did. I ran both and laid the leaderboards side by side.

They didn't just differ. They contradicted each other at the very top. The first tool crowned my single most important topic, the obvious thing to do next. The second buried that exact same topic so far down that it fell off the list entirely. One had even sorted 58 notes into a topic, where the other counted 8. Same notes, same morning, opposite verdicts.

I'd been steering by one of these dials for weeks, and it had never occurred to me to check it against the other, because each one, alone, looked completely trustworthy. Clean table. Confident numbers. A ranked list in the calm, authoritative voice of a thing that has Done The Math.

That voice, of course, was partially mine. I built it… or at least had AI build it. I'd handed my own judgment a megaphone and mistaken the megaphone for a second opinion.

When I finally read the code to figure out the discrepancy, the cause was almost insulting in its smallness. To decide what counted as a "topic," each version matched notes against a list of keywords, and the two lists had quietly drifted apart as I tinkered. In one spot, the only difference was a single stray space in front of a word. That space was the whole gap between "your number one priority" and "didn't make the cut."

I had the danger exactly backward, and I think a lot of people do.

I'd assumed the threat of AI-assisted work was the code I couldn't read, the black box, the incomprehensible layer where I'd lose the plot. That moment never came. Every line was readable. I could rebuild the whole tool from scratch if I had the time. By definition, I was the programmer the entire time, and I still nearly steered my week off a cliff.

I didn't lose my grip to complexity. I lost it to a clean output I never cross-examined, a confident answer I never thought to question because confidence felt like correctness. At some layer of abstraction, I was programming.

This has a name, and it predates the chatbots by forty years. Human-factors researchers call it automation bias: the better a system runs, the less you scrutinize it, until you stop being the pilot and become a relay for the machine's confidence. At the dramatic end, it has a body count, crews who trusted a confident-but-wrong instrument all the way down.

The bloodless version is already loose in business.

In October 2025, KPMG, a firm with a quarter of a million employees, published a glossy report on AI whose citations turned out to be mostly fictional: of 45 references, 40 of the 45 titles were fake, padded with invented authors, fused sources, and 2019 press releases passed off as proof of 2025 technology. An AI tool had produced confident, authoritative-looking references, and somewhere inside a 250,000-person firm, nobody checked. GPTZero has a name for the genre: vibe citations. The report looked like it had been audited because it was formatted like research. It held up right until the organizations it described actually read it: UBS, the NHS, and Transport for London told the Financial Times the claims about them were untrue or misleading, and this June, KPMG pulled the report. For the record, the firm's own AI policy requires "human oversight to validate content and verify independent sources."

My version cost me a slightly mis-prioritized week. Same shape. Wildly different stakes.

What keeps you honest is cross-examining the layers that carry your judgment, checking them against a second source, or just your own gut. Understanding every layer beneath you, or above you, matters far less than that. You can be a perfectly competent programmer and still slide into operator-of-your-own-judgment the moment you trust a dashboard you've stopped auditing.

A while back, I wrote about my calendar lying to me, a tool I'd handed too much authority over my day. This is the same disease, one level deeper and perhaps more insidious. Back then, the tool was wrong, and I noticed. This time, the tool was wrong, and it sounded exactly like me being right.

If you run anything (a team, a budget, a roadmap, a grocery list, your life’s purpose), you're increasingly being handed clean, confident, AI-generated outputs to decide from. Summaries. Scores. Rankings. Recommendations.

The ones that will catch you off guard aren't the results you don't understand. You’ll get trouble from the ones you understand so well that you forget to question them.

A clean number is uniquely good at switching off your suspicion. We reliably believe we understand things we can't actually explain (try diagramming exactly how a toilet flushes, and if you are so confident, what happens EXACTLY to the water on its long voyage through a treatment plant before it's ready for the next flush).

We treat confidence and polish as cheap proxies for being correct. A ranked table looks audited because it is formatted like an answer. And the second a number starts running a decision, it starts to bend. It’s Goodhart's law, the reason every metric you lean on eventually games you back.

Call it the second-instrument test:

None of this is new, which is reassuring. It's just triangulation: why patients get second opinions (which change the diagnosis about 15% of the time and the treatment plan 37%), why safety-critical software gets written twice by separate teams so the versions can disagree, why the cheapest fix for a confident wrong hunch is to argue the opposite out loud.

What protects you isn't understanding every layer. Nobody has time for that, and chasing it is its own trap. You should develop an intuition for which layers carry your unique judgment, and keep a second instrument pointed at exactly those layers. Let the plumbing disappear. Never let your human judgment go down that same drain.

A few things I'm following that orbit this:

Right now, most AI systems hand you an answer and wait. That's the easy entry point. The second instrument can be you, a source, or a second run. But the tools are starting to act on their own now, booking the thing, sending the email, moving the money. "Go check the answer" stops working when there's no pause to check, much less a discerning critique of where your checking actually matters.

The good news is that some of us see it coming. The cross-examination is being built into the agents themselves: test suites before release, monitors after, explicit confirmations before anything irreversible, one model grading another's work.

However… and this is the point: Just because I tell myself I am a competent AI operator/programmer/builder, a cross-checker that checks my work only helps if it can fail in ways the thing it's checking doesn't.

Two copies of the same confident mistake are just an echo chamber with a quorum of wrong opinions.

In other words, there’s some construction ahead. Pass me the Road Atlas.

/ David

On repeat: Can't Find My Way Home by Blind Faith, a band named for the exact thing this issue warns against.

Reading: The Tyranny of Metrics (Jerry Muller): a case for why a dashboard turns a partial truth into false authority.

Building: the idea-triage tool above as part of a larger Signal Path infrastructure, getting excited about where this is leading. AI facilitated creative pipelines vs. AI-generated creation.