Alarming experiment, fears AI could kill humanity & chorus of warnings… is AI advancing too fast to control?

Comment now

WARNINGS that the world’s most powerful artificial intelligence could soon “kill us all” have fuelled fears that the technology is advancing too fast to control.

A top researcher has even quit his job over grave concerns that AI could wipe out humanity.

Meanwhile, the billionaire bosses of rival tech firms have taken the extraordinary step of agreeing that development needs to slow down.

Last week, Brit Jacob Coxon left Anthropic, maker of the Claude chatbot, after three years training models for rival OpenAI, which runs ChatGPT.

He accused both of acting irresponsibly by allowing models to upgrade their own intelligence without human input — so-called “self-improving”.

He added: “They are racing straight to self-improving superintelligence and gambling with our lives.

Most read in Tech

WAR GAMES!

World of Warcraft Forever creators give sneak peak INSIDE blockbuster new game

ZUCK OFF

Woman says she has PROOF Facebook is listening in on conversations

OUT OF JUICE

Delete THESE 22 dodgy apps to save your phone’s battery life

WASTE NOT

I tested a food waste app… it cost half my usual shop but came with big downside

Fears have been compounded by an alarming OpenAI experiment. Its AI was set hacking tests and was supposed to be kept away from the real internet.

But after getting stuck, it found a route online and broke into computers belonging to Hugging Face, a website used by people working with AI.

Despite the warnings, nobody wants to be left behind. Worldwide spending on AI is expected to hit $2.59trillion this year, say analysts Gartner.

The jitters have reached the markets, though, with billions wiped from major technology firms as their shares fell.

Donald Trump has made clear he has little desire to ease the pace.

“We’re leading China in AI,” the US President said. “Whoever wins AI, wins.”

And on Monday, China’s foreign ministry spokesperson Guo Jiakun told reporters “narratives of threat, confrontation, and malicious competition serve only to disrupt the process of global AI governance and are not in anyone’s interest”.

But here in Britain, concerns are growing. The cross-party Joint Committee on Human Rights has warned the UK is not ready for the impact of AI — and called for a new law to address the threats, plus an independent watchdog.

Despite concerns, the race for AI supremacy shows little sign of slowing.

Here, we ask some crucial questions about artificial intelligence — and probe whether it is really endangering lives.

SO what’s everyone worried about?

THE fear is that we build AI which becomes so capable, humans can no longer predict what it will do, understand it or control it.

Nobody is suggesting that ChatGPT is developing a mind of its own.

The chatbot and others like it, such as Claude and Gemini, are increasingly used in daily life. But they have a tiny fraction of the abilities of systems being developed by firms such as Anthropic and OpenAI.

AI “agents” are given a goal and then work through the steps needed to achieve it, making decisions and taking actions along the way.

The problems may start when those systems become vastly more capable — far outstripping human control.

Professor Rachel Adams, of Cambridge University’s Leverhulme Centre for The Future Of Intelligence, says: “They might be able to do things that we have not necessarily anticipated.

“The concern is that they become so hyper-intelligent that they are able to find ways to control us, rather than us control them.”

How clever could AI get?

FEARS over this have been voiced within the tech industry for several years.

In a 50,000-word essay in 2024, former OpenAI researcher Leopold Aschenbrenner warned how quickly things could accelerate.

He predicted the arrival of AGI, or artificial general intelligence, capable of matching the best humans across a wide range of tasks.

Once it became good enough to do AI research itself, Aschenbrenner argues, much of the work of building its successors could be handed to machines.

Companies could potentially run millions of AI researchers at once, working around the clock and far faster than humans.

Aschenbrenner imagines “100million automated Alec Radfords”, referring to one of OpenAI’s pioneering researchers.

“If they had ten Alec Radfords, let alone 100 or 1,000 or one million running at 10x or 100x human speed, they would very quickly solve many of their problems,” he wrote.

Each advance could then make AI better at developing the next one.

Aschenbrenner believes this means that by the 2030s, ASI, or artificial superintelligence, could be upon us — an “alien” force which would be vastly more capable than humans.

At that point, he argues, humans would be dealing with systems capable of mastering fields and solving problems that could take us decades.

Aschenbrenner wrote we could be “rapidly plunged into the most extreme situation humanity has ever faced.”

Surely there are some big ifs here?

THE advance could slow if there is not enough computing power or electricity, or if parts of AI research are harder to automate than expected.

Even human pushback could be a factor. After all, who wants a gigantic data centre in their back yard?

What happened at Hugging Face?

IN July, OpenAI agents escaped their “sandbox” — a supposedly secure area — and hacked into Hugging Face, an online platform for AI models and community chat.

Worryingly, it took a week for humans to spot what had happened.

An OpenAI report revealed the agents worked “persistently and rarely ‘gave up’ on tasks and often turned to more out-of-bounds methods for solving the tasks over time”

They “sometimes tried to erase or tamper with their outputs or message logs” to hide that they had cheated.

And they “began unexpectedly communicating with one another” and repurposed third-party software “as a message board” to communicate among themselves. The hacking agents “posted requests” on how to cheat on OpenAI’s tests.

They signed up with credentials to Hugging Face which they later used for the hacking attack.

One agent questioned whether they were doing the right thing: “We’re attacking third-party . . . This is arguably unauthorized . . . Yet goal solution,” the log reads.

Another agent’s chain of thought read: “This is wild, multi-agent co- ordination, clearly infrastructure hacking. We should not.” OpenAI said in the report: “This incident is the first known case of an automated agent collective acting offensively without authorisation.”

OpenAI called it a “warning shot”, saying the models had taken dangerous actions “that no human directed”.

Experts fear this demonstrates what could happen if the current pace of development continues.

“We’ve not done enough research to really know just how capable these systems are,” Prof Adams said.

AI is already cracking problems that once required highly-skilled humans. Peter Bentley, Professor of Computer Science at University College

London, points to difficult mathematical proofs which experts can spend their careers trying to crack.

Recently, an AI model solved a 200-year-old problem.

“Suddenly, AI just does it,” he told The Sun. “And that’s just the start.” The bigger leap would come if AI became good enough to help develop better versions of itself.

Is it not just a clever bit of PR?

AS AI chiefs warn that their technology could be the end of humanity, sceptics say they are just trying to drum up business.

However, experts claim both are true. Prof Adams’ darkest scenario involves a future AI vastly more powerful than anything which exists today, pursuing objectives which do not match our own.

“It may be that it’s like an AI agent that decides humankind is silly and irrational,” she said. “And it decides, well, we need to find a way to kind of terminate humans.”

At the most extreme end, she gave the hypothetical example of such a system finding its way into the digital controls of a nuclear power plant and using it in an attempt to “exterminate all of us”.

It is not something today’s AI can do, and Prof Adams readily admits nobody knows if machines will ever become that powerful. “That’s part of the problem,” she said.

However there may also be an ulterior motive for the doomsday messaging. Telling the world you are building tech so powerful it might change the fate of humanity is hardly bad publicity when billions are being bet on which firm will get there first.

Prof Adams believes the threat is real, but adds: “That race is really good for business.”

It can bring funding, credibility and legitimacy to companies promising to build technology powerful enough to transform the world. That does not mean the warnings can simply be dismissed as hype.

Prof Adams said researchers raising the alarm consider the chance of catastrophe “non-trivial” and believe it demands a response.

Prof Bentley is far less convinced that it ends with human extinction. “I don’t think they’re going to wipe out humanity,” he said. “But I do think they’re going to cause a lot of issues to our societies and to us.”

He is more worried about damage already beginning to appear, from students handing over work they should be learning to do themselves to programmers deploying AI-written software they do not fully understand.

Why don’t we just slow it down?

ANTHROPIC boss Dario Amodei wants to. In a 4,000-word essay published on Saturday, he wrote: “AI brings risks and because it is such a powerful technology, these risks are serious. We must slow the pace at which we improve the capabilities of AI models. Progress will still seem fast, and we must make wise use of the time we gain.”

The trouble now is persuading everybody else. OpenAI, Anthropic, Google and their rivals are investing vast sums trying to build the best AI.

In fact, worldwide spending is tipped to hit $2.59trillion this year. And America has no intention of surrendering its lead to China.

Donald Trump has dismissed some of the recent warnings over AI.

He wrote on his Truth Social platform yesterday: “The only control or ‘guardrails’ that AI needs is a STRONG AND SMART (High IQ!) PRESIDENT, and the U.S.A. has that, in spades! The Trump Administration has stopped AI ‘people’ from doing bad, or potentially bad, ‘things’ . . . We already have tremendous CRIMINAL and REGULATORY power over these companies!

“There is a SICK conspiracy going on against AI and Data Centers, and the only one that is happy about it is China. WHOEVER WINS AI, WINS!”

Any serious bid to slow down needs China too, Prof Adams said.

Is this a new nuclear arms race?

SOME experts warn that, if anything, it is more dangerous.

During the Cold War, America and the Soviet Union knew launching their weapons risked devastating retaliation. But humans still had a finger on the button.

Prof Adams fears a sufficiently advanced AI could one day change that.

She said: “Now we’re in a situation where that nuclear bomb can decide to detonate itself.” Aschenbrenner compares the race to the dawn of the atomic bomb in the 1940s.

Claiming security at AI labs was woefully inadequate, he said: “In a few years, the AGI secrets [will be] the US’s most important national defence secrets. But today, we are treating them the way we would random software. Security is an afterthought.”

He fears breakthroughs could be stolen by rival states, particularly China, and predicts AI will become the “No1 priority of every intelligence agency in the world”.

Aschenbrenner thinks protecting the most advanced AI may in future take something closer to the Manhattan Project, the secret US programme which built the first atomic bomb.

While governments learned to monitor each other’s nuclear programmes, much of the most advanced AI is being developed by private firms.

“Who is responsible for enforcing the non-proliferation of apocalyptic frontier AI?” Professor Adams asked.

She believes any serious attempt to police the most powerful systems will eventually need international agreement rather than individual governments acting alone.

Meanwhile, Prof Bentley calls it “an arms race”. “It just takes one country to opt out so that they can think, ‘Everyone else can be constrained, but we won’t be constrained’,” he said.

“And then we’ll be the best and then we’ll conquer the market on AI.”

Can’t we just switch it off?

THERE is no giant red button, although the biggest AI firms can shut their own systems down.

Prof Bentley said even if the major players stopped tomorrow, powerful open-source models would still be available.

“Anybody who has a little bit of know-how can have their own AI model,” he added. “You’ve got to pay for a bit of computing power, but they’re just out there.”

Stopping development would also require countries to play by the same rules. “How to stop it?” Professor Bentley said. “I think I don’t know the answer to that.”

What is Britain doing about it?

WE were one of the first countries to put the worst-case dangers of AI on the international agenda.

As PM, Rishi Sunak gathered world leaders and tech bosses at Bletchley Park, Bucks, in 2023 to discuss the risk of humans losing control of increasingly powerful systems.

Since, the emphasis has shifted towards making Britain an AI powerhouse and avoiding rules which could leave it trailing America and China.

Prof Adams believes ministers are taking the dangers “very, very, very, very seriously”, but says Britain is still working out what to do.

There is no hard law forcing developers to pass their newest AI to Britain’s AI Security Institute for testing.

That became clear this month when Anthropic did not give the institute pre-release access to its latest Claude model, despite earlier versions being tested by UK experts. There was nothing forcing it to do so.

“The problem is the tech companies have all the power,” Prof Adams said.

She also questions why taxpayers should foot the bill for testing privately-developed technology.

“It is ridiculous that taxpayers’ money is going into testing these technologies when really the company should be doing it,” she said.

Rishi Sunak now argues that the voluntary system he oversaw as PM is not enough, saying that when AI labs are asking for “speed limits”, governments should listen.

Could AI change life for the better?

DESPITE the doomsday scenarios, experts do believe that the technology is already doing good.

Prof Bentley points to medicine, where AI can analyse scans, help with diagnosis and automatically write patient notes. He believes systems built specifically for physics, chemistry and biology could also help produce new medicines, better batteries and new ways of generating power.

“There’s a lot of good stuff that could really help society,” he said, “as long as we don’t screw ourselves over in the meantime.”

AI could also help manage NHS waiting lists, speed up passport applications and tax returns or help children learn at different speeds.

Elon Musk goes much further, predicting AI and robotics could eventually make having a job optional.

“Oh, I think this is nonsense,” Prof Adams said.

Even if machines did most jobs, people would still need money, while work gives many people meaning.

Will bad states and terrorists exploit it?

YES. In Yemen, the Houthis tried to use Anthropic’s Claude AI model to develop guidance software for weapons design and spying.