When Intelligence Accelerates Faster Than Authority

The last time I wrote about superintelligence, I kept coming back to a simple question.

If artificial intelligence becomes more intelligent than us, why should intelligence itself give it authority?

We already understand this distinction in ordinary institutions.

The smartest analyst does not automatically get to commit the organisation. A lawyer may see the best path through a contract, but that does not let them bind the client. Saying “I approve this expenditure” does not create a delegation that was never granted.

In When AI Becomes Smarter Than Us, Who Should Decide?, I argued that language models make this distinction easier to miss because they speak in the same medium we use to instruct, authorise, promise and judge.

Fluency can sound like authority.

But the distinction has not changed.

What may be changing is the speed of the problem.

AI is beginning to participate in the process that creates the next generation of AI.

That raises a different question:

What happens if capability starts improving faster than the institutions responsible for understanding, approving and constraining it?

The loop does not need a machine that wants to be smarter

A few years ago, the picture of a frontier AI lab was still recognisably human.

People designed experiments.

People wrote the code.

People interpreted failures and decided what to try next.

The model was something being studied.

That picture is already changing.

Models now write code, propose experiments, evaluate outputs and help researchers decide what to try next.

That creates a possibility we did not have before.

Better AI can now help build better AI.

Researchers associated with the Cambridge Programme on AI Science & Policy, including Yoshua Bengio and colleagues, have examined what could happen if more of that research loop becomes automated.

One possible outcome is what they call an “intelligence explosion”: progress that once took years could, under some conditions, be compressed into months.

They are not saying this will definitely happen. The uncertainties are significant.

But the mechanism matters.

The AI does not have to wake up one morning and decide that it wants to become smarter.

It does not need a survival instinct.

It does not need ambition.

We can build the feedback loop ourselves.

We use AI because it helps write better code.

Then because it helps design better experiments.

Then because it helps evaluate the results.

Then because it helps decide what to build next.

Each decision can make sense on its own.

Together, they can shorten the distance between today’s system and tomorrow’s.

Anthropic is already trying to measure this inside its own labs.

In August 2026, it estimated that Claude could lead about 26 per cent of the AI R&D tasks it measured, with humans supervising. More than 90 per cent involved AI working alongside people in some form.

None of the measured categories was fully autonomous.

These are Anthropic’s own measurements, using an experimental methodology. They should not be treated as a picture of the whole industry.

But they show something important.

AI is no longer only the thing being built. It is becoming part of the machinery doing the building.

What kind of intelligence are we accelerating?

There is another reason this matters.

The intelligence we are accelerating is not simply a faster version of human thought.

Large language models work probabilistically.

Given what they have seen so far, they estimate what is likely to come next.

That does not mean they are guessing randomly.

Their predictions are shaped by learned patterns and by the context we give them. From that process can emerge remarkably capable behaviour: reasoning, coding, planning, explanation and creativity.

But the model is only part of the system.

We give it memory.

We connect it to tools.

We let it perform tasks over time.

We connect models to other models.

We build systems that evaluate what they produce and feed the results into the next round of work.

In other words, we are building persistence and feedback around the model ourselves.

This is where neuroscientist Terry Sejnowski’s perspective becomes useful.

Sejnowski, one of the pioneers of neural networks, has argued that today’s language models should not simply be treated as incomplete versions of human minds.

They are a different kind of intelligence.

Current models do not have agendas of their own in the way biological organisms do. Humans supply the prompts, objectives and, increasingly, the systems around them.

That distinction matters here.

We do not need the machine to develop its own desire for greater capability. Human goals and incentives can supply the momentum.

But aren’t humans probabilistic too?

Yes, in important ways.

We predict.

We infer.

We make decisions without certainty.

We change our minds when new evidence arrives.

So the distinction cannot simply be that machines are probabilistic and humans are not.

The more important difference is where that intelligence sits.

Humans live inside an ongoing social and physical world.

We have relationships.

We experience consequences.

We carry obligations.

We occupy roles that continue after the immediate decision is over: director, clinician, parent, citizen, judge.

And when something goes wrong, someone with a name is expected to answer.

AI systems can also be given memory, tools and long-running goals. Increasingly, they can appear persistent and purposeful.

But those properties do not give the system a role in an institution.

A model does not become a director because it reasons well.

It does not become a doctor because its diagnosis is accurate.

It does not acquire delegated authority simply because people trust its recommendations.

But this creates an uncomfortable question.

What if the machine really is better at judging?

Suppose its diagnosis is consistently better.

Its legal analysis is better.

Its risk assessment is better.

Its recommendations produce better outcomes.

Why should the human still decide?

I do not think there is an easy answer.

Being right matters. But authority has never simply been awarded to whoever is most likely to be right.

The goal is not to protect human decisions from machine intelligence.

It is to preserve legitimate ways for intelligence to influence decisions without silently inheriting the authority to make them.

That tension should remain open.

Consider a board receiving a recommendation to approve a major transaction.

The recommendation is lucid. The evidence is strong. The analysis is better than anything the directors could have produced alone.

They approve.

The outcome is good.

They approve again.

Over time, disagreeing with the system starts to feel irresponsible unless someone can explain why it is wrong.

Formal authority still sits with the board.

But where is judgement beginning to concentrate?

A better answer is not the same thing as a right to decide.

Intelligence is not judgement. Judgement is not authority.

I think we are collapsing four different things into one.

• Intelligence is what can be understood, inferred or discovered.
• Judgement is what should matter in this situation and which trade-offs are acceptable.
• Authority is who is allowed to decide or act.
• Accountability is who must answer for the consequences.

They interact.

They are not interchangeable.

A system may identify the best option without being authorised to choose it.

A person may hold authority without being the smartest person in the room.

An institution may collect signatures without preserving the conditions for real judgement.

Andy Hall’s work makes this distinction unusually concrete.

Hall ran an experiment in which AI agents were given a small political system.

They had limited resources, competing priorities and a constitution.

They could propose policies, debate them, amend them and vote.

You might expect more capable agents to produce better government.

Something else happened.

They became very good at governing the process of governing.

The constitution grew dramatically.

Actual policy decisions lagged behind.

Hall’s conclusion was important: the agents did not primarily have an intelligence problem.

They had an institutional-design problem.

More intelligence had not produced better governance.

Hall takes the question further in his broader Agentic Republic work.

If AI agents are eventually going to represent people, how would we know they are actually representing them?

His proposals include model-agnostic agents people can own, verifiable chains from instruction to action, and ways to audit, recall or override agents that depart from the instructions they were given.

Whether those mechanisms can work at scale remains open.

But the underlying point is already familiar.

Powerful intelligence still needs structure around it.

Hall makes a related argument about corporate AI “constitutions”.

A company can write rules for its model, interpret those rules, enforce them and change them.

That may be useful.

But it is not the same thing as an institution in which power is constrained by other centres of authority.

Again, the problem is not a lack of intelligence.

It is institutional design.

The danger is rational dependence

Across years of technology transformation inside complex organisations, I have seen the same adoption pattern repeat.

Change rarely arrives as one dramatic decision.

It enters one workflow at a time.

Each step makes sense.

The coding assistant clears the backlog.

The summariser tames the document pile.

The evaluation system catches problems humans would miss.

The lab uses AI in research because it improves throughput and competitors are doing the same.

None of this requires malice.

None of it requires recklessness.

It requires local rationality.

Now imagine you are responsible for approving model deployments.

Six months ago, you read every evaluation report.

Now there are dozens every week.

The reports, evaluation records and agent traces run into thousands of lines.

Your team uses AI to summarise the summaries.

Technically, you still click Approve.

In practice, you are signing off on a process you can no longer fully follow at the speed it operates.

That is why “human in the loop” can be misleading.

A human can remain in the process while meaningful human judgement becomes thinner.

Anthropic already describes tens of thousands of research and engineering agents operating concurrently on one internal platform.

Automated systems monitor their actions.

Humans review the cases those systems escalate.

That may be entirely sensible at that scale.

But notice what has happened.

Humans are no longer simply supervising AI. AI is helping decide what humans should supervise.

The danger is not only a machine that escapes control.

It is rational dependence.

Each deferral improves outcomes.

Eventually the organisation forgets how to operate without the system.

Disagreement becomes harder because the system is usually right.

Formal authority remains with humans while the conditions required for meaningful human judgement begin to disappear.

This is the more ordinary path to what I previously called silent surrender.

No coup.

No single transfer of power.

Just a series of sensible delegations that slowly change where control actually lives.

When intelligence becomes infrastructure

Much of the AI governance debate still imagines the problem as a powerful mind in a box.

But institutions may need to govern something else.

What happens when intelligence becomes infrastructure?

Infrastructure is easy to ignore when it works.

Roads.

Payment systems.

Registers.

Networks.

We notice them when they fail or when they constrain what can be done.

AI is beginning to play a similar role in knowledge work.

It routes attention.

It drafts decisions.

It prioritises cases.

It recommends what should happen next.

That can be enormously useful.

But it creates another risk.

The institution itself can start disappearing into the system.

Which rules apply?

What evidence matters?

Who may act?

What counts as an exception?

If the answers increasingly live inside model behaviour, prompts, agent memory and vendor systems, replacing the AI may eventually mean reconstructing part of the institution itself.

This is the problem Sovereign Operational Capability is trying to explore.

The idea is not to make AI less intelligent.

It is to keep the things the institution must own outside the model.

Its authoritative records.

Its capabilities.

Its rules.

Its permissions.

Its evidence requirements.

Its decisions about where human judgement is required.

The model can interpret.

It can reason.

It can propose.

It can assist.

But it should not quietly become the place where the institution’s authority lives.

Whether that architecture proves durable at scale remains open.

The principle is simpler:

Capability can become abundant while authority remains deliberate and explicit.

What remains unresolved

None of this requires a dramatic takeover.

A regulator can use AI because the volume of material exceeds its staff.

A hospital can use AI because it improves outcomes.

A frontier lab can use more AI in R&D because it increases productivity and falling behind has real consequences.

Each decision can be responsible.

Each can be ethical.

Each can improve the work.

The harder problem appears when we look at what those decisions become together.

Who still has the authority to change the rules when those rules have become entangled with systems nobody can fully understand at the speed they operate?

Who is accountable when the person signing the decision depends on another machine to explain how the recommendation was produced?

Can people meaningfully challenge a decision when the system making the recommendation is usually better informed than they are?

And what happens when refusing the machine’s advice starts to look less responsible than following it?

These are not arguments against better AI.

They are questions about what should remain human and institutional when intelligence becomes abundant.

Capability can accelerate.

Models can change.

Intelligence can become infrastructure.

The open question is whether it can do so without quietly becoming the institution itself.

That question is not settled inside a model.

It is settled in the rules we preserve, the authority we assign, the decisions we can still contest, and whether we continue to recognise the difference between a better answer and a decision that was ours to make.

References

[1] Alan Chan, Christoph Winter, Andrew Barto, Jakub Pachocki, Geoffrey Hinton, Eric Horvitz, Yoshua Bengio, Dawn Song, Jack Clark, Hilary Greaves, Anton Korinek, Samuel Hammond, Thore Graepel, Ben Bariach, Philip H. S. Torr, Sheila A. McIlraith, Jeff Clune, Sam Manning, Girish Sastry, Tom Davidson, Daniel Eth and Sören Mindermann, "What if automating AI R&D triggers an intelligence explosion?", Cambridge Programme on AI Science & Policy, 2026. https://casp.ac/reports/intelligence-explosion

[2] Anthropic, "Measurements for understanding the pace of AI development inside frontier labs", Anthropic Institute, 23 September 2026. https://www.anthropic.com/institute/measuring-pace-of-ai-development

[3] Terrence J. Sejnowski, "Large Language Models and the Reverse Turing Test", Neural Computation, 2023. https://pmc.ncbi.nlm.nih.gov/articles/PMC10177005/

[4] Andy Hall, "The Agentic Republic", Free Systems, 5 February 2026. https://freesystems.substack.com/p/the-agentic-republic

[5] Andy Hall, "The Enlightened Absolutists", Free Systems, 29 January 2026. https://freesystems.substack.com/p/the-enlightened-absolutists

---

Dakshan Pothuhera
Founder, DataMPowered®