Artificial intelligence will have the capability to destroy humanity “by the end of the decade”.
This alarming claim by a former Anthropic employee made headlines this week, adding to growing concerns about AI.
“They are racing straight to self-improving superintelligence and gambling with our lives,” developer Jacob Coxon said in his post on X.
The 27-year-old worked for OpenAI and Anthropic but said he had decided to leave the industry.
Surprisingly, he was publicly supported by senior company employees.
Anthropic’s top safety researcher said he believed there was more than a “10% chance” that AI could kill “all humans within the decade”.
According to Evan Hubinger, “the risk is low” at present, but he is worried about “superintelligence arising from recursive self-improvement”.
Those claims have been met with scepticism from some researchers.
“It’s very hard to avoid the suspicion that this has something to do with Anthropic’s IPO,” Professor Tomas Ward of DCU’s School of Computing told RTÉ News.
The company is expected to go public in October.
Like other AI giants, it needs to raise significant amounts of money to fund its energy-hungry models.
Demonstrating to potential investors that Anthropic’s technology is “powerful and incredibly omnipotent” could help, Prof Ward argues.
AI bosses call for a ‘slow down’ in development
However, some of the leading participants in the race to develop advanced AI are wary of a reckless sprint towards the artificial finish line.
Yesterday, CEO of Anthropic Dario Amodei called on AI companies to deliberately slow the rate at which they advance model capabilities.
“Not building the technology deprives humanity of benefits or simply places AI in the hands of authoritarian powers, while building it too fast is reckless. We have sought a middle way,” said Mr Amodei.
OpenAI chief Sam Altman and SpaceX boss Elon Musk followed suit, saying that they agreed with Mr Amodei on the need to stall the development of artificial intelligence.
Mr Altman also indicated that there would be no IPO for OpenAI in 2026, stating that “given everything happening with safety, right now would be an ill-advised moment to go public”.
‘Rogue’ agents
Suspicions of a PR ploy have been voiced before.
Yesterday, OpenAI confirmed that autonomous software built on its models targeted RubyGems, a site that provides services for coding, during testing in May.
In July, the ChatGPT creators also revealed that an AI agent went “rogue” and hacked an online platform.
Sam Altman’s company then said that autonomous agents (systems designed to complete multi-step tasks) escaped their controlled testing environment, known as a “sandbox”, gained access to the internet and hacked the systems of Hugging Face, a platform that hosts start-up AI models.
It was later reported that around 1,200 agents were involved in the attack – communicating and collaborating with each other.
Anthropic and Meta rushed to announce that they had similar incidents, with their models also gaining unauthorised access to real-world systems during testing.
At the time, Professor Barry O’Sullivan of UCC’s School of Computer Science called the hack revelations “theatrical” and “headline-grabbing”.
But why was it allowed to happen?
Computer scientists explain that the agents, which were given the hacking task, ultimately achieved it through an endless loop of “ask-act-report”.
“They knew that they weren’t [supposed] to do it, but the reward they were going to get for getting the right answer was more powerful than the objective than the constraint that was set on them not to cheat,” says Prof Ward.
Cal Newport, a computer science professor at Georgetown University and author, believes that the agents in this incident were “unpredictable, not malicious.”
Should we worry about AI’s “self-improvement” and “superintelligence”?
Industry experts agree that AI agents are in a constant process of recursive self–improvement – each model creates a better one.
The current state of development is described as narrow AI, which is designed to solve a specific task or a set of tasks based on existing data.
Then there is superintelligence.
It is a hypothetical (for now) concept where a computer’s intelligence surpasses human capabilities – a “genius” able to innovate, create and solve complex issues.
Meta’s Mark Zuckerberg believes that the creation of “superintelligence is in sight.”
He is among its most enthusiastic preachers, promising to make it available to each human individually so that they can “achieve their goals”.
Like Zuckerberg, Prof Ward is confident we will develop superintelligence “within our lifetime.”
“Without a shadow of a doubt”.
There is optimism around its ability to cure diseases or find answers to persistent energy crises.
In recent days, several AI bosses have argued that this technology represents one of the greatest leaps in human progress.
Sam Altman said its importance is comparable to that of electricity over a century ago – warning authorities against resisting adoption.
The chief executive of UK’s largest chipmaker Arm Holdings, Rene Haas, said he is confident that AI will help us cure cancer in our lifetime.
The ‘force for good and evil’
An “AI optimist” Prof Ward agrees that it can be a powerful tool, like electricity.
There is a significant difference, though.
Unlike electricity, AI agents “can become autonomous and make decisions by themselves.”
In many cases intent won’t be the problem, but rather the methods an agent chooses to achieve a given goal.
Professor Ward suggests a hypothetical scenario in which a superintelligent agent is asked to “reduce global carbon emissions as quickly as possible”.
“An innocuous target, isn’t it? But a really capable system that’s badly guardrailed might reckon that the most effective measures to reduce global carbon emissions would be to close down industries, ration electricity, or shut down aircraft manufacturing.”
“That’ll reduce global carbon emissions pretty quickly, but that’s not how we want it to be done”.
EU policymakers have been scrambling to regulate the rapidly evolving industry.
The EU AI Act became law two years ago, though its provisions are taking effect on a phased basis.
New rules on deepfakes and chatbots were introduced in August, obliging companies to make it clear when customers are dealing with a computer instead of a real person.
But preventing AI from being used by “malicious actors” is an entirely different challenge.
The military application of AI is a major concern for researchers, with the technology already being used in drones and surveillance.
In its latest threat intelligence report, Anthropic said it “broke up attempts” to use its Claude models by “threat actors”.
The incidents included attempts by scientists to create biological weapons in one of the regions “not supported by Anthropic”.
Those regions include China, Russia and North Korea, among others.
The company also claimed that Russia-linked hackers have been using Claude to carry out espionage campaigns in Ukraine.
It also accused Chinese firms of trying to copy its AI models.
“Paying attention” to how the technology is being developed and used is key, according to Prof Ward.
“It all depends on how we engineer and develop it, and the onus is on us to do it responsibly and to do it well.”
“It’s a force for good. It can also be a force for evil.”