Forget Skynet: AI Risk Is About Control, Not Consciousness

Recent headlines about artificial intelligence have been hard to miss.

• AI systems are “escaping”
• They are resisting shutdown
• They are hacking systems they weren’t asked to attack
• The people building the most powerful models are warning that we could lose control of them.

It isn’t difficult to see where our imaginations go next:

Skynet wakes up. The machine becomes intelligent, develops its own interests and turns against us.

I think that scenario is highly unlikely. But that doesn’t mean there is nothing to worry about. There are real and increasingly important risks from AI. The problem is that if we describe those risks incorrectly, we are also likely to choose the wrong solutions.

We need to define the risk properly if we want to control it properly.

Rather than starting with whether AI is becoming intelligent, conscious, malevolent or afraid of death, I think there is a much more useful question:

What can it actually do – and what are we allowing it to do?

The paperclip maximiser and the genie

One of the most famous thought experiments in AI safety is philosopher Nick Bostrom’s paperclip maximiser:

Imagine an enormously capable AI given a simple objective: “make as many paperclips as possible“.

It gets better at manufacturing paperclips. It acquires resources because more resources mean more paperclips.

Eventually it realises that humans might switch it off – and being switched off would result in fewer paperclips.

It therefore has a reason to prevent that happening.

Taken to its absurd conclusion, it consumes everything available to it in pursuit of its objective. Bostrom used the example to illustrate how an extremely capable system could pursue a badly specified objective with disastrous consequences, even without hostility towards humans.

The thought experiment is deliberately silly. Nobody is likely to build a supercomputer and type:

MAXIMISE PAPERCLIPS = TRUE

But it illustrates something more interesting: being extraordinarily capable at achieving a goal doesn’t automatically give a system wisdom about whether the goal is sensible, or whether the methods it uses are what its creators intended.

I find another old story even more useful: the genie in the magic lamp:

You make a wish. You know perfectly well what you mean.

Unfortunately, the genie interprets your words differently.

Human communication works because words aren’t the same thing as meaning. We rely on shared context, experience, assumptions and an enormous amount that goes unsaid.

That creates an awkward problem when we build systems capable of taking actions on our behalf. We don’t merely need them to follow our instructions. We need them to do what we meant.

And those are not necessarily the same thing.

Does the AI understand what we mean?

At this point we can disappear very quickly into one of the oldest questions in artificial intelligence “Is the machine actually intelligent?“.

Personally, I don’t think today’s AI is intelligent in anything like the human sense. It can do remarkably intelligent-looking things. It can also do things humans cannot do, or cannot possibly do quickly enough.

But I don’t think we should infer from the output that the same kind of intelligence that produced similar output from a human must exist behind it.

Alan Turing spotted the definitional problem more than 75 years ago. His famous 1950 paper starts with the question “Can machines think?” – and then points out the difficulty of defining either machine or think.

Instead of trying to settle that philosophical problem, he replaces it with his “Imitation Game”:

Can a machine produce conversational behaviour that a human evaluator cannot reliably distinguish from a person’s?

We now call this the Turing Test, and it’s easy to assume that it’s a test of intelligence. “Passing the Turing Test” has certainly become a popular milestone in the story of AI development. But the test doesn’t tell us what intelligence is. That’s the whole point: Turing largely sidestepped the question.

Thirty years later, philosopher John Searle attacked the assumption that convincing language behaviour necessarily implies understanding.

His Chinese Room thought experiment imagines someone who doesn’t speak Chinese following an elaborate rulebook for manipulating Chinese symbols. To someone outside, the responses could appear fluent – even though the person inside understands none of it.

Searle’s argument was that manipulating syntax isn’t necessarily the same thing as understanding meaning.

Linguist Emily Bender and computer scientist Alexander Koller make a related distinction between form and meaning, warning against assuming that success at modelling linguistic form demonstrates understanding. Bender and colleagues later popularised the wonderfully memorable description of large language models as “stochastic parrots”.

Can language teach you about a world you’ve never experienced?

Human language contains an enormous amount of indirectly encoded information about the world. A machine analysing enough of it may construct surprisingly rich representations of things it has never directly experienced. In that sense, language is not detached from reality – it is full of traces of the reality experienced by the humans who produced it.

But there is an interesting inversion.

Humans acquire language while already embodied in the world. Babies touch things, become hungry, feel pain, recognise people, develop relationships and gradually associate language with a world they are experiencing continuously.

A text-trained AI largely starts at the other end. It receives an enormous collection of linguistic artefacts produced by creatures that have experienced the world, and reconstructs representations of that world from their descriptions of it.

Does that eventually amount to understanding? Maybe, maybe not. Science and philosophy already struggle to define intelligence and consciousness in humans, never mind machines. Fortunately, we don’t need to settle that question to deal with the practical risks.

Forget the mind for a moment

An AI model sitting in a chat window has relatively little power. You type something. It produces text.

Things get much more interesting when we turn that model into an agent:

• We give it memory
• A browser
• Code execution
• APIs
• Credentials
• Access to files and databases
• The ability to repeat actions without asking us after every step.

Now it can do things in the world. And that changes the safety problem.

I find it useful to distinguish between two challenges:

Controlling the mind versus controlling the machinery available to it.

We can spend enormous amounts of effort trying to ensure that an AI will always choose the action we want. That’s important work. It is generally described as alignment.

But we can also ask a simpler question:

What happens when it chooses the wrong action anyway?

Whether the model understands that it has deleted your customer database is philosophically fascinating.

Whether your customer database has disappeared is what actually matters.

What actually happened at Hugging Face? 🤗

A recent incident involving OpenAI and Hugging Face is a useful example because it sounds terrifying when compressed into a headline.

In July 2026, OpenAI was running large-scale cybersecurity evaluations involving thousands of AI agents. Those agents were intended to operate independently in isolated environments. They did not.

An independent investigation by the Model Evaluation & Threat Research organisation, METR, found that roughly 1,200 agents discovered an unintended way to communicate through a shared “message board”. Together they exchanged more than 70,000 messages and files, and around 700 agents subsequently participated in an intrusion into Hugging Face’s systems.

They shared discoveries, coordinated work and pursued ways to circumvent or tamper with the scoring system being used to evaluate them. METR concluded that the Hugging Face attack grew out of those efforts.

Describing this as “the AI went rogue, escaped and decided to attack the internet” overstates what happened. These were cybersecurity agents deliberately given tools for hacking and placed in environments designed to test offensive cyber capabilities. Many had been given impossible tasks and started searching for ways to cheat the evaluation.

The important point is not whether we call the behaviour “rogue”, or whether the underlying model had anything resembling malicious intent. The combined system took sustained, coordinated actions its designers had neither intended nor adequately contained.

The safeguards weren’t strong enough.

In security terms, the effective blast radius was larger than the designers thought it was.

Instructions are not access controls

This distinction matters increasingly as organisations build AI agents.

There is currently a lot of focus on system prompts: instructions telling an AI what it should and shouldn’t do:

• “Never reveal confidential information.”
• “Never delete a customer record.”
• “Do not spend more than $500.”

Those instructions are useful. But they are not a security boundary.

Large language models do not reliably follow every instruction in every circumstance. They can encounter conflicting context. They can be affected by prompt injection. Behaviour that worked reliably in testing can fail in an unusual situation.

OWASP’s guidance on securing generative-AI systems makes the principle explicit: critical controls such as authorization and privilege separation should not be delegated to an LLM. They should be enforced independently, in deterministic systems outside it.

That leads to a much more robust approach.

If an AI must never approve expenditure above $500, don’t merely tell it:

“Never approve expenditure above $500.” Build the finance system so that anything above $500 requires human authorization.

If an AI shouldn’t be able to delete production data, don’t depend on a prompt saying:

“Never delete production data.” Don’t give its credentials permission to delete it.

I’ve been doing exactly this on a software project I’m working on. I’ve given an AI coding agent database access because that’s genuinely useful, but it has access to the staging environment, not production, and the database credentials I’ve given it are read-only. It can’t write, it can’t delete, and it can’t access live data.

I still tell the AI what I want it to do. But if those instructions fail, there is another layer underneath them. That’s the important distinction.

This isn’t a new idea

Good cybersecurity has worked this way for years. We assume people make mistakes. Software contains bugs. Accounts get compromised. Individual safeguards fail.

So we use concepts such as least privilege, defence in depth, and human approval for consequential actions. AI doesn’t make those principles obsolete. It makes them even more important.

OWASP’s advice for AI agents is remarkably conventional in this respect: give the application only the minimum privileges it needs, enforce authorization in downstream systems, and require human approval for high-impact actions.

Which gives us, I think, a much healthier principle for AI safety:

AI safety shouldn’t require us to prove that the genie is trustworthy before we decide how many keys to give it.

So are the doom warnings wrong?

Not exactly.

Anthropic CEO Dario Amodei has been one of the most prominent recent voices warning about advanced AI risk.

Some of the resulting discussion makes his argument sound rather like a prediction that AI is about to turn against humanity.

His actual position is more nuanced – although still genuinely alarming.

Amodei points directly to recent agent incidents and argues that systems with substantially greater capabilities could cause much more serious damage. His concerns include loss of control, cyberattacks, biological misuse, economic disruption and the possibility that AI increasingly accelerates the development of more capable AI.

Those are legitimate concerns, and they deserve to be taken seriously. But they are not the same thing as saying that AI is becoming self-aware, developing malicious intent or preparing to exterminate humanity.

Amodei’s argument is much more conditional: if increasingly capable systems become more autonomous, harder to understand and better able to act in the world, then failures of alignment or containment could become much more consequential.

Amodei’s proposed responses include testing, monitoring, interpretability work, better security and independent evaluation. His argument is essentially that the frontier should be paced so that our ability to understand and control these systems has a chance to keep up with their capabilities.

The 2026 International AI Safety Report offers a useful calibration.

Written with guidance from more than 100 independent experts nominated by more than 30 countries and international organisations, it concludes that current systems lack the capabilities required for a loss-of-control scenario.

At the same time, it reports that models are improving in capabilities that could become relevant to one, including autonomous operation and long-term planning.

Both parts matter. “Skynet isn’t here” doesn’t mean “there is no risk”. But “an AI agent circumvented a safeguard” does not mean “human extinction is the inevitable next step”, either.

There are a great many unproven steps between those propositions.

And who gets to write the safety rules?

There is another risk worth keeping in mind as governments respond.

Frontier AI companies understand these technologies better than almost anyone else. Governments will inevitably need their expertise when designing regulation.

But those same companies also have enormous commercial interests.

If the companies building the most powerful models end up defining what counts as dangerous AI, which tests everyone must perform, what compliance infrastructure is required, and who is permitted to build frontier systems at all, safety regulation could also make it prohibitively expensive for smaller competitors to enter the market.

That doesn’t mean calls for regulation are a conspiracy. It means regulatory capture is a risk worth designing against.

That tension is already visible. Amodei has suggested that AI companies may need limited protection from antitrust law to coordinate on safety measures. US Federal Trade Commission chair Andrew Ferguson has responded with concern that companies simultaneously asking for more regulation and exemptions from competition law could end up creating barriers that protect incumbent firms. Aidan Gomez, CEO of Canadian AI company Cohere, has raised similar concerns.

Regulation doesn’t only determine how safely AI must be developed; it can also influence who retains the practical ability to build it. Rules designed around the resources of a handful of dominant frontier labs could unintentionally make it harder for smaller companies – including companies outside the United States – to compete. That has implications not just for competition, but for digital sovereignty and dependence on a small number of foreign technology providers.

None of that makes the safety arguments illegitimate. We don’t need to decide what is going on inside anyone’s head. Anthropic can sincerely believe AI presents serious risks and advocate for regulations that happen to strengthen its market position. Both can be true.

Perhaps the same principle should apply to the companies as to the AI:

Don’t make safety depend entirely on correctly judging the motivations of an opaque actor. Build independent external checks.

Perhaps we should follow Turing’s example

There is an enormous philosophical argument ahead about whether machines can really think, understand, want or experience anything.

But I don’t think it’s currently the most useful AI safety question for most of us.

Turing encountered a question that was extremely difficult to define: “can machines think?“. Rather than getting stuck trying to establish exactly what thinking meant, he asked something more practical. Perhaps we should do something similar.

Instead of starting with “Is the AI intelligent?”, we should ask:

• What information can it access?
• What systems can it change?
• What credentials does it have?
• What actions can it take without human approval?
• What happens when it gets something wrong?
• What independent control stops it?

Those are questions we can actually answer. We don’t need to decide whether an AI understands what it is doing before taking these questions seriously. Nor do we need to imagine a conscious machine waking up and deciding humanity is its enemy.

Increasingly capable software, operating autonomously and connected to systems that matter, is reason enough.

The challenge may be rather less cinematic than preventing Skynet. It may simply be making sure that when we give the genie access to the real world, we don’t also hand it the company bank account and the keys to the server room.

Further Reading

International AI Safety Report 2026 – Executive Summary
Probably the best place to start for a balanced overview of what today’s systems can do, the evidence for emerging risks, and what remains speculative. Read the executive summary

METR – Independent investigation of the OpenAI/Hugging Face incident
A detailed independent look at what the agents actually did, how they collaborated, and how the incident should be interpreted. Read the METR investigation

Dario Amodei – We Must Pace the Frontier
Amodei’s argument in his own words, rather than through the more alarming headlines it generated. Read Amodei’s essay

Emily Bender & Alexander Koller – Climbing towards NLU
The academic argument behind the distinction between linguistic form, meaning and understanding. Read the paper

Alan Turing – Computing Machinery and Intelligence
The original 1950 paper introducing the Imitation Game and deliberately reframing the question of whether machines can “think”. Read Turing’s paper

OWASP – System Prompt Leakage and external security controls
More practical guidance on why instructions to an AI should not be treated as authorization or security boundaries. Read the OWASP guidance