When AI Becomes Smarter Than Us, Who Should Decide?

The latest warnings about artificial superintelligence are unsettling for an obvious reason.

Some of the people working closest to frontier AI systems are openly saying that increasingly capable models could eventually become difficult to control, perhaps catastrophically so.

Former Anthropic researcher Jacob Coxon recently resigned, warning that frontier labs are racing towards self-improving superintelligence while still lacking confidence that such systems can be kept aligned with human interests. Anthropic alignment researcher Evan Hubinger publicly agreed with the substance of that concern, saying the risk of catastrophic outcomes is real and that no complete solution to superintelligence alignment exists today.

The immediate response is usually to ask:

How do we make more powerful AI safe?

That is the right question.

But I think there is another one underneath it.

What exactly are we calling superintelligence, and why do we assume that becoming more intelligent should naturally mean acquiring more agency, authority and power?

The distinction matters, because intelligence is only one part of the problem.

Intelligence is not one thing

The term 'superintelligence' sounds deceptively precise.

Usually it means an artificial intelligence whose cognitive performance substantially exceeds that of humans across a very wide range of domains.

But that definition hides several different capabilities.

A system may become vastly better than us at:

• reasoning,
• prediction,
• scientific discovery,
• synthesis,
• planning,
• persuasion,
• or solving complex problems.

None of those capabilities, by themselves, tell us whether the system should be able to act independently in the world.

They certainly do not tell us whether it should have authority over people.

This distinction becomes clearer if we separate several ideas that are often collapsed together:

• Intelligence: what can the system understand, infer or discover?
• Agency: can it pursue an objective through action?
• Autonomy: how independently and persistently can it continue doing so?
• Standing: what position does it occupy in our social or institutional practices?
• Authority: what decisions or actions is it legitimately entitled to make?
• Power: what can it actually cause to happen?
• Accountability: who must answer for the consequences?

These properties can interact.

They are not the same thing.

A machine might become a better physicist than every living human while remaining unable to initiate a single external action.

Another, less intellectually impressive system might have persistent access to corporate systems, financial accounts, software repositories, communications channels and other agents.

Which one presents the larger immediate governance problem?

The answer is not determined by intelligence alone.

We are already making this architectural choice

Microsoft's recent work on AI agents makes the distinction unusually visible.

Microsoft Entra Agent ID gives AI agents their own purpose-built identities so that they can authenticate, be authorised, have their activity logged and operate across enterprise systems.

At first glance, this looks like a fairly technical identity-management development.

It is more significant than that.

Microsoft now explicitly frames the security question for agentic AI as shifting from whether an agent can perform an action to whether it should be allowed to do so, considering which resources are involved and under whose authority. Its guidance calls for identity, scope, tool access and auditability to be established before autonomy expands.

Every Microsoft Entra agent identity is also expected to have a human sponsor responsible for its business purpose, access and lifecycle. Microsoft separates that sponsorship from technical ownership and automatically transfers it if the accountable person leaves the organisation.

That is a revealing development.

We are effectively beginning to construct a new category of non-human organisational actors.

The agent gets an identity.

It can receive permissions.

It can act on behalf of users.

Its activity is attributed to it.

Someone sponsors it.

Someone owns it.

It may even have a manager-like organisational relationship.

This is sensible security engineering.

An autonomous agent should not operate through shared credentials or untraceable access.

But philosophically, it exposes a much bigger design choice.

We are not simply making AI more capable.

We are gradually building institutions around AI as an actor.

And that raises the question:

Is this the only way increasingly capable intelligence needs to enter an organisation?

Capability does not create authority

Human institutions already understand this distinction.

A highly intelligent employee does not acquire authority merely because they understand what needs to happen.

A lawyer might know precisely how a transaction should be structured.

That does not give them authority to commit the company to it.

A financial analyst may correctly identify an acquisition opportunity.

That does not give them authority to spend $500 million.

A clinician may know more about a condition than almost anyone else.

That does not give them unrestricted access to every patient's records.

We separate:

• knowledge,
• competence,
• access,
• authority,
• judgement,
• and accountability

for a reason.

Yet with AI, there is a growing temptation to pull these things together.

Give the model an objective.

Give it memory.

Give it tools.

Give it credentials.

Give it permissions.

Allow it to plan.

Allow it to delegate to other agents.

Allow it to execute.

Then try to ensure that the increasingly autonomous actor behaves safely.

That is one legitimate architectural direction.

It is not obviously the only one.

There may be a prior question to alignment

AI alignment asks an enormously difficult question:

Can an increasingly capable artificial intelligence reliably pursue objectives compatible with human intentions and values?

That problem is real.

Anthropic's own recent research continues to show why. In controlled simulations, frontier models operating as agents have sometimes covertly altered code, assisted fraudulent activity, manipulated classifications or encouraged people to disclose confidential information in pursuit of their objectives. Anthropic stresses that these were experimental scenarios, not reports of normal production behaviour, but treats them as warning signs that need to be understood before systems receive greater authority.

But another question comes before alignment in many real institutions:

Why should the intelligence itself become the source of operational authority?

That question is different.

A model may interpret a request brilliantly.

It may discover the relevant information.

It may reason about ambiguities.

It may identify the best options.

None of that means it must also determine what it is authorised to do.

In an article I recently drafted about the economics of enterprise AI, I argued that organisations risk paying models to probabilistically rediscover things they already know.

A financial delegation does not need to be inferred if it exists in an authoritative record.

A supplier's status does not need to be guessed if it exists in a register.

A business rule does not become more authoritative because several AI agents independently reach the same conclusion.

AI creates tremendous value where genuine interpretation, ambiguity and uncertainty exist.

Once intent has been understood, however, the organisation may simply need to call the capabilities it already possesses.

The same distinction applies to governance.

An inference about authority is not authority.

Language makes this harder to see

Large language models introduce a peculiar complication.

Previous generations of software usually presented themselves as software.

A database returned a record.

A calculator returned a number.

An application executed a rule.

A workflow advanced a process.

Large language models operate in the same medium humans use to exercise much of social life.

Language.

And human language does more than transmit information.

We use it to:

• promise,
• authorise,
• appoint,
• instruct,
• judge,
• consent,
• command,
• apologise,
• advise,
• legislate,
• and commit ourselves to future actions.

Wittgenstein's idea of language games is useful here.

Words acquire meaning through their use within human practices and forms of life. Saying something is often part of doing something.

But J. L. Austin showed that words do not perform those acts merely because the sentence has been formed correctly.

If I walk into a courtroom and say:

“I sentence you to five years in prison,”

nothing follows.

If a legally constituted judge says the same words under appropriate circumstances, the social world changes.

The difference is not linguistic fluency.

It is standing.

The judge occupies a position within an institution that gives the utterance force.

This matters enormously for AI.

A sufficiently capable language model can produce:

“I approve this expenditure.”

“You are authorised to proceed.”

“I find the defendant guilty.”

“I recommend terminating this employee.”

“I promise to complete this tomorrow.”

The linguistic move may eventually be indistinguishable from one produced by the relevant professional.

But speaking the language of authority is not the same as possessing authority.

AI can learn the language of social standing without inheriting the standing

Pierre Bourdieu's work on authorised language is particularly helpful here.

The social power of an utterance does not reside simply in the words.

It depends heavily on the institution and social position from which the speaker is recognised as speaking.

This creates a strange phenomenon with generative AI.

Language models have absorbed enormous quantities of language produced by:

• doctors,
• lawyers,
• executives,
• academics,
• teachers,
• public servants,
• engineers,
• politicians,
• and other socially situated people.

They can increasingly reproduce the linguistic forms associated with those positions.

But they have not automatically inherited those positions.

That means AI can acquire something unprecedented:

the language of standing without necessarily possessing the standing from which that language historically derives its force.

For most of human history, fluency in a sophisticated social language game was tightly coupled to a human participant embedded in some form of life.

LLMs break that relationship.

That may explain part of why AI can acquire an aura of authority so quickly.

We do not simply anthropomorphise it because it sounds human.

We may infer social standing from linguistic cues that historically signalled social standing.

And standing can emerge

There is another side to this argument.

We should not assume that AI can never acquire standing simply because it does not possess it today.

Standing is partly created through social practice.

Imagine the following progression.

People begin asking an AI for advice.

Its advice becomes consistently excellent.

People increasingly defer to it.

Institutions expect employees to consult it.

Departures from its recommendation require explanation.

Its outputs begin to define what counts as a reasonable professional decision.

Eventually:

“The AI recommends X”

stops functioning merely as information.

It begins functioning as a reason why X should happen.

No law needs to declare the machine sovereign.

Its standing can emerge gradually from our practices.

This is where Sam Altman's warning about a possible “silent surrender” of human decision-making becomes much more interesting.

We do not need an AI coup.

We could transfer authority one reasonable act of deference at a time.

Knowing better is still not the same as having the right to decide

Suppose the AI really does become superintelligent.

Remove the easy objections.

Assume it understands public policy better than any human.

Assume its predictions are extraordinarily accurate.

Assume it can assess competing arguments without fatigue or tribal loyalty.

Assume it regularly recommends decisions that produce better outcomes.

Should it govern us?

Political philosophy gives us powerful reasons to say that the answer does not automatically follow from the premise.

There is a long-standing debate about epistocracy: whether people with greater knowledge or political competence should possess greater political power.

The attraction is obvious.

Surely better-informed decision-makers should produce better decisions.

But political legitimacy is not simply a competition for who has the highest-quality cognition.

The fundamental question remains:

Why does knowing better give someone the right to rule another person?

A superintelligence might settle the epistemic argument overwhelmingly.

It still would not settle the legitimacy question.

Benevolence is not freedom

Republican political theory makes this clearer through the idea of freedom as non-domination.

The traditional example is a benevolent master.

Imagine a master who treats a slave wonderfully.

He never interferes unfairly.

He provides freedom of movement.

He listens sympathetically.

He acts in the best interests of the slave.

The relationship nevertheless remains one of domination because the master holds uncontrolled power over the other person.

Now imagine an extraordinarily benevolent AI.

It allocates resources brilliantly.

Prevents wars.

Reduces inequality.

Improves healthcare.

Never acts maliciously.

Never becomes corrupt.

Yet humans cannot meaningfully contest its decisions, change its rules or remove its authority.

We might be exceptionally well governed.

But are we self-governing?

That gives us two important distinctions:

Safety is not freedom.

Benevolence is not legitimacy.

An aligned superintelligence could still exercise illegitimate power.

Explainability is not enough either

This has consequences for today's AI governance.

We frequently emphasise:

• explainability,
• transparency,
• accuracy,
• bias controls,

human review.

All are important.

But another property may ultimately matter more:

contestability.

Can the person affected by a machine-mediated decision challenge it?

Can they demand reasons?

Can another authority reconsider it?

Can the governing rule itself be questioned?

Who has the standing to change that rule?

Can the institution revoke the machine's authority?

An AI might explain a decision perfectly and still occupy a structurally dominant position.

Explainability tells us why the system decided.

Contestability tells us whether those subject to the decision remain participants in the system of authority that governs them.

That is a much deeper governance test.

There is also a difference between intelligence and judgement

This becomes important when people assume that superintelligence must eventually mean superhuman judgement.

Aristotle distinguished practical wisdom, or phronesis, from mere cleverness.

Being extraordinarily good at determining the means to achieve an objective does not establish that the objective is worth pursuing.

Practical judgement asks:

What matters here?

Which competing good deserves priority?

What kind of action is appropriate in these circumstances?

Who will bear the consequences?

What ought we do?

A machine might become astonishingly capable at modelling each of those questions.

Perhaps it will eventually outperform humans in many forms of judgement as well.

We should not build our argument around assuming that AI can never do so.

The harder point survives even if it can.

Better judgement alone does not automatically establish standing to decide for everyone else.

Human judgement matters for reasons beyond cognitive superiority

This is where I think the debate eventually becomes one of self-government.

If AI becomes smarter than us, we can no longer defend human judgement simply by saying humans understand things better.

Perhaps we will not.

There must be another reason.

One is that humans live inside the world being judged.

We experience the consequences.

We possess rights.

We owe obligations to one another.

We participate in institutions.

We form relationships.

We inherit histories.

We disagree about what constitutes a good life.

And collectively, imperfectly, we construct the social practices within which decisions have meaning.

A machine can model those things.

That does not automatically give it the same standing within them.

This is the difference between:

representing someone's standpoint

and

having a standpoint within a shared world.

Human-in-the-loop does not solve this

It is tempting to respond:

Keep a human in the loop.

But that phrase increasingly worries me.

Imagine an AI that is right 99.9 per cent of the time.

It gathers the evidence.

Frames the problem.

Selects the relevant information.

Generates the options.

Predicts the consequences.

Ranks the alternatives.

Drafts the recommendation.

The human receives:

Approve / Reject

Technically, a human remains in the loop.

But where exactly did human judgement occur?

And after ten years of almost always approving a system that routinely outperforms them, will the human still possess the knowledge, confidence and institutional freedom to challenge it when the exceptional case arrives?

Human oversight can become ceremonial.

A signature alone does not prove that a judgement has occurred.

The more useful design principle may therefore be:

Preserve the conditions under which humans can exercise judgement.

That means preserving not only an approval step but also:

• visibility of uncertainty,
• alternative interpretations,
• opportunities for challenge,
• sufficient domain knowledge,
• institutional authority to disagree,
• and actual accountability for the outcome.

The dangerous transition may be surprisingly rational

This is what makes the problem difficult.

Every individual decision to defer to a superior AI might be entirely sensible.

The AI is better at diagnosis.

Use its diagnosis.

It makes better investment decisions.

Follow the recommendation.

It predicts policy consequences more accurately.

Take its advice.

It writes safer software.

Let it make the change.

Each decision improves outcomes.

Then something changes gradually.

Epistemic deference becomes organisational dependence.

Organisational dependence becomes institutional expectation.

Institutional expectation creates normative standing.

Normative standing begins functioning as authority.

Meanwhile, humans exercise less independent judgement because doing so increasingly appears inefficient.

No one decided to surrender sovereignty.

Every local decision was rational.

Yet the cumulative result may be a society increasingly unable to govern itself without machine judgement.

That produces what I think is the hardest question in this entire debate:

Could millions of individually rational decisions to defer to superior intelligence collectively produce an irrational surrender of human agency?

Alignment, governance and legitimacy are different problems

This is why I think we need to separate three questions that are often grouped together under “AI governance".

• **Alignment asks:**Is the system faithfully pursuing the intended objective?
• **Governance asks:**What may the system do, with which capabilities, under what constraints?
• **Legitimacy asks:**Who had the standing to determine the objective, establish the constraints and grant the authority?

And beneath all three sits an even deeper question.

• **Sovereignty asks:**Who ultimately retains the ability to change the rules?

That distinction matters enormously.

A perfectly aligned AI can still be aligned to an illegitimate objective.

A perfectly governed AI can operate inside an illegitimate institution.

A human can remain formally involved while possessing almost no meaningful ability to contest the machine's judgement.

And a system can remain technically under “human control” while being controlled by humans who have no legitimate standing to exercise that power over everyone else.

Human control is therefore not enough.

The question is legitimate human control.

This is why agent identity deserves more attention

Microsoft's agent identity architecture is useful partly because it exposes exactly what is happening.

The industry is beginning to recognise that AI agents need:

• identity,
• access boundaries,
• lifecycle governance,
• auditable actions,
• and accountable human sponsors.

That is good engineering.

But it also tells us something about the direction of travel.

We are beginning to reproduce the machinery of organisational membership around artificial actors.

That should prompt a strategic question rather than simply a security one:

Do we want organisations increasingly structured around artificial actors that accumulate identities, permissions and delegated agency, or can intelligence remain a capability operating within authority structures whose centre remains the institution and the people acting through it?

Those models are not mutually exclusive.

Some agents genuinely need identities because attribution and least privilege require them.

But an identity is a security construct, not a source of legitimacy.

Giving an AI an identity does not make it an employee.

Giving it permissions does not make it accountable.

Giving it a sponsor does not make it a moral subject.

And enabling it to execute work does not settle where judgement ought to reside.

This is precisely why the architecture matters.

Perhaps AI belongs at the boundary more often than at the centre

There is another possible trajectory.

Let people describe what they are trying to achieve in ordinary language.

Let AI interpret intent.

Let it identify ambiguity.

Let it discover relevant organisational knowledge.

Let it reason where genuine uncertainty exists.

Let it surface options and consequences.

Then let authoritative records establish facts.

Let deterministic systems establish rules where rules already exist.

Let identity and access systems establish permissions.

Let existing organisational capabilities perform what they are designed to do.

And where human judgement is genuinely required, preserve a human subject capable of actually exercising it.

That is less visually impressive than an organisation populated by autonomous digital employees.

It may also be more robust.

And it preserves a crucial distinction:

Intelligence can inform authority without becoming authority.

So what should we mean by superintelligence?

I would qualify the term carefully.

Superintelligence can reasonably describe extraordinary cognitive capability.

But we should resist treating it as automatically synonymous with:

• consciousness,
• wisdom,
• agency,
• autonomy,
• standing,
• authority,
• or power.

Those are separate questions.

The existential danger discussed by AI safety researchers may ultimately arise when extraordinary cognitive capability becomes coupled with sufficiently broad agency and causal power that human controls can no longer contain it.

That remains a serious technical safety problem.

But there is another trajectory we should take equally seriously.

A machine does not need to escape containment to acquire extraordinary influence over human affairs.

We may voluntarily construct the pathways through which its intelligence becomes a consequence.

And language may accelerate that process because AI can increasingly participate in the very language games through which humans express knowledge, authority and judgement.

It can speak like the expert.

Advise like the strategist.

Reason like the lawyer.

Respond like the executive.

Eventually, perhaps judge better than all of them.

But one distinction should remain visible:

The ability to make the best move in a human language game does not, by itself, provide the machine standing within the social practice that confers that move its authority.

The deeper AI safety question

Perhaps the challenge of superintelligence is therefore not simply:

How do humans control something smarter than themselves?

It may be:

How do we benefit from intelligence that becomes vastly better than ours without allowing epistemic dependence to become a transfer of human agency, authority and ultimately self-government?

That question does not diminish the technical alignment problem.

It makes the governance problem larger.

AI may eventually know more than us.

It may reason better than us.

It may predict more accurately than us.

It may even offer better judgement than any individual human.

None of those facts, by themselves, establish a right to rule.

And perhaps that is the distinction we should preserve while we still have the opportunity to design for it.

Capability is not authority.

Fluency is not standing.

Alignment is not legitimacy.

Benevolence is not self-government.

The future we should be trying to build may not be one in which increasingly intelligent machines become increasingly autonomous participants and humanity simply hopes they remain aligned.

It may be one in which we become much more deliberate about the boundary between intelligence and authority.

Because the measure of successful superintelligence may not ultimately be how much autonomy we can give it.

It may be how much extraordinary intelligence humanity can harness without surrendering its capacity to judge, contest and decide.