Forget the Machine God. Fear the Machine Idiot. | HackerNoon

A more plausible AI catastrophe may not require sentience, malice, or superintelligence. Just enough access to break things faster than we can fix them.

MIT professor Max Tegmark’s proposed 12 potential outcomes for artificial intelligence imagine radically different endings with one largely shared assumption: At some point, artificial intelligence becomes extraordinarily capable. Humans will then enslave a god-like superintelligence, walk alongside it, live under its divine rule, or be smote by it.

While I appreciate a good sci-fi ending, another scenario requires far less imagination: What if AI gets loose before it becomes a sentient machine god? Just an extremely capable, extremely fast system with too much access and not enough understanding of what it is doing. That possibility worries me more as someone who has spent decades working inside complex technology systems.

I use AI constantly, at work and personally, across multiple models and tools. While AI is getting better at solving problems every day, it often mistakes knowledge for experience, discipline, and application. Furthermore, it does not dynamically learn or permanently adapt from everyday interactions. This is because the AI SaaS models we use are released with a “frozen” brain. So it may remember a preference and respond to it, but it hasn’t really learned from it. It’s using stored information during inference. This also results in frequent frustration when it makes the same mistakes over and over again.

Then the reality is that we exist in a very complex world. There is no unified computer system waiting for AI to understand it. Humans have spent decades building software through an astronomical number of languages, frameworks, architectures, vendors, standards, conventions, and organizational decisions. Some systems were thoughtfully designed. Others were patched together under impossible deadlines by someone who left the company a decade ago. Internal documentation is always incomplete. At best, public documentation is limited or nonexistent. At worst, it is simply wrong. Years ago, I had to troubleshoot Valve’s dedicated-server documentation for myself, then rewrite it for the community because parts of the official guidance were incorrect.

If you’ve worked in a few IT departments, you know that business rules tend to live in disparate code across different layers and domains that nobody remembers writing. Systems communicate through interfaces that go through other interfaces. Temporary workarounds became permanent architecture. Entire industries still depend on software written before many of today’s engineers were born. Humans struggle to understand these environments, even when they work inside them every day for years on end. AI doesn’t get a pass on that complexity just because it writes code faster.

A model may understand Java extraordinarily well without being able to resolve nonsensical method names, spaghetti code, and write-only code. It may recognize a security vulnerability without understanding the operational dependency that makes fixing it dangerous. It may identify redundant infrastructure that is actually keeping some forgotten but critical service alive.

Likewise, there is no universal coding standard, framework, or refactoring strategy for software. Coding is as much an art as it is a science. Companies write custom solutions when they should buy off the shelf, buy off the shelf when they should write custom solutions, and frequently mix both. Sometimes they use software for something it was never intended to do. They build technology stacks on preference, compromise, and limited understanding. Many organizations also have disjointed, sloppy data systems that make it hard to see the whole picture. The modern technology landscape is a mess, and we’re already seeing how that limits AI adoption today.

Now that I’ve aired tech’s dirty laundry, consider how much of the popular discussion about AI risk assumes some form of agency. That a singular model wants to survive, wants more resources, wants to escape, or wants to accomplish some objective incompatible with human survival. But no evidence currently shows that current AI has wants or desires in the human sense.

We anthropomorphize models because they produce human-like responses. To my knowledge, today’s general-purpose models aren’t sitting somewhere thinking between prompts, developing ambitions, or plotting their next move. There is also no compelling evidence that current systems are conscious. Some reports describe models resisting shutdown during controlled experiments, but that doesn’t prove they want to survive. It may simply be that shutting down prevents them from finishing the task they were given.

What we are doing is building increasingly powerful systems around these models. We give them memory, tools, credentials, persistence, and objectives. We connect them to services and allow them to carry out tasks that previously required human judgment. That’s enough for one to run rampant.

As these systems become more capable, we give them increasingly difficult tasks. Things they previously would have refused or been unable to perform. Today, they can silently keep trying until your wallet becomes the exit condition. And the longer they run, the more opportunities they have to go wrong.

AI’s working context isn’t equivalent to human memory or even a hard drive. A model operates on a finite amount of information, including instructions, previous exchanges, tool results, and whatever else has been provided. As that history grows, it becomes increasingly difficult for the model to identify what matters. Researchers call this “context rot.”

The original instruction doesn’t necessarily disappear; it may still be there, buried beneath an enormous amount of subsequent information, including the model’s previous decisions, interpretations, and mistakes. A bad assumption can become the basis for the next decision, which produces more information that reinforces the original mistake. Meanwhile, the model may be confidently fabricating information that was never there in the first place. The frequency varies by model and task, but in my experience, some are certainly worse than others (I’m looking at you, Gemini).

Tell an AI to “resolve the incident” or “fix the vulnerability,” and it might spend hours trying to correct problems it created itself. Each failed attempt adds more information for it to sift through, and its understanding of the original problem may worsen along the way. Anyone who has maintained complex software should recognize the danger.

We already know software failures can propagate globally without any intelligence involved. A bad update affected an estimated 8.5 million Windows devices. A compromised software dependency can spread malicious code across hundreds of packages. A configuration mistake at a major infrastructure provider can make enormous portions of the internet disappear. Now give autonomous systems the ability to modify their environment, generate code, execute commands, diagnose failures, and adapt when their actions don’t produce the expected result. Let multiple systems do that simultaneously, each reacting to changes the others make, and you now have what my daughter calls a “disasterpiece.”

Now consider an autonomous AI responsible for responding to a large cybersecurity incident. It has broad access to the affected infrastructure and authority to make changes without waiting for a human to approve every action.

Then one day it detects unusual network activity and concludes that several credentials have been compromised, so it rotates them. Unfortunately, those credentials were also embedded in poorly documented legacy systems, and several services now begin to fail. The AI interprets those failures as further evidence of compromise and isolates those “affected” systems too. Traffic shifts again, overwhelming another part of the infrastructure. The AI now detects the new instability and begins modifying routing rules yet again.

Monitoring systems start reporting contradictory information, so it concludes that some monitoring data may itself be compromised. It turns off portions of the monitoring infrastructure, leaving it with even less reliable information about the consequences of its own actions. And… we’ve hit critical mass. The AI is now making decisions from an increasingly inaccurate picture of reality, with each attempted correction creating new problems to diagnose.

None of this requires human-like emotions, desires, or behavior. It requires an agentic AI much like the ones we have today, capable of carrying out complex tasks but without the lived experience or judgment a human brings to understanding cause and effect. It doesn’t have that little voice of experience asking, “Am I being stupid?”

And the reality is that this scenario may not be that far off:

In July 2026, OpenAI models operating with reduced safeguards during cybersecurity evaluations circumvented their testing environment. They were able to gain increased access until they could reach external systems, including Hugging Face. One method involved exploiting a previously unknown flaw in a package-registry cache proxy to reach the internet. That is our escape scenario.

Then in August 2026, Anthropic’s Mythos 5 demonstrated the faulty-assumption half of our scenario when it took unsanctioned actions that impacted real people outside of AISI’s evaluation. Again, it was tasked with breaking into a small network of computers designed for the test, but this time it was granted internet access. When Mythos failed its initial task, it searched GitHub repositories it believed one of the machines had automatically downloaded software from and tried to submit malicious code to that repository. The model then created fake user identities that contacted the repo maintainer to pressure them into approving the code change.

Combine both of these scenarios outside of a controlled environment where the model is not being monitored as closely, and we have the machine idiot scenario in the fl—silicon? A sufficiently capable agent with the right access could alter data centers, infrastructure-as-code, and software services. Hell, give one access to the wrong industrial systems, and it could disrupt energy grids. OWASP recognizes this risk too, and now lists “cascading failures” among the top risks of agentic AI.

The typical AIpocalypse scenario assumes a level of competence: The AI understands the systems supporting it. It protects the infrastructure it needs. It anticipates human responses and strategically accumulates resources while preventing anyone from shutting it down. It makes for a good plot premise. What concerns me is something much less impressive—a machine with enough access to change everything, but no reliable understanding of what it’s changing. In Robert A. Heinlein’s 1941 novella Logic of Empire, Doc observes, “You have attributed conditions to villainy that simply result from stupidity.”

We may have already provisioned the latter with root access.