![\"\"](\"https:\/\/images.newscientist.com\/wp-content\/uploads\/2024\/12\/20192226\/SEI_233869500.jpg\")
<\/div>OpenAI announced a breakthrough achievement for its new o3 AI model<\/p>\n
Rokas Tenys \/ Alamy<\/p>\n<\/div>\n<\/figcaption><\/figure>\n<\/p>\n
OpenAI\u2019s new o3 artificial intelligence model has achieved a breakthrough high score on a prestigious AI reasoning test<\/a> called the ARC Challenge, inspiring some AI fans to speculate that o3 has achieved artificial general intelligence<\/a> (AGI). But even as ARC Challenge organisers described o3\u2019s achievement as a major milestone, they also cautioned that it has not won the competition\u2019s grand prize \u2013 and it is only one step on the path towards AGI, a term for hypothetical future AI with human-like intelligence.<\/p>\n The o3 model is the latest in a line of AI releases that follow on from the large language models powering ChatGPT. \u201cThis is a surprising and important step-function increase in AI capabilities, showing novel task adaptation ability never seen before in the GPT-family models,\u201d said Fran\u00e7ois Chollet<\/a>, an engineer at Google and the main creator of the ARC Challenge, in a blog post<\/a>.<\/p>\n <\/p>\n Chollet designed the Abstraction and Reasoning Corpus<\/a> (ARC) Challenge in 2019 to test how well AIs can find correct patterns linking pairs of coloured grids. Such visual puzzles are intended to make AIs demonstrate a form of general intelligence with basic reasoning capabilities. But throwing enough computing power at the puzzles could let even a non-reasoning program simply solve them through brute force. To prevent this, the competition also requires official score submissions to meet certain limits on computing power.<\/p>\n OpenAI\u2019s newly announced o3 model \u2013 which is scheduled for release in early 2025 \u2013 achieved its official breakthrough score of 75.7 per cent on the ARC Challenge\u2019s \u201csemi-private\u201d test, which is used for ranking competitors on a public leaderboard. The computing cost of its achievement was approximately $20 for each visual puzzle task, meeting the competition\u2019s limit of less than $10,000 total. However, the harder \u201cprivate\u201d test that is used to determine grand prize winners has an even more stringent computing power limit, equivalent to spending just 10 cents on each task, which OpenAI did not meet.<\/p>\n The o3 model also achieved an unofficial score of 87.5 per cent by applying approximately 172 times more computing power than it did on the official score. For comparison, the typical human score is 84 per cent, and an 85 per cent score is enough to win the ARC Challenge\u2019s $600,000 grand prize \u2013 if the model can also keep its computing costs within the required limits.<\/p>\n But to reach its unofficial score, o3\u2019s cost soared to thousands of dollars spent solving each task. OpenAI requested that the challenge organisers not publish the exact computing costs.<\/p>\n No, the ARC challenge organisers have specifically said they do not consider beating this competition benchmark to be an indicator of having achieved AGI.<\/p>\n The o3 model also failed to solve more than 100 visual puzzle tasks, even when OpenAI applied a very large amount of computing power toward the unofficial score, said Mike Knoop, an ARC Challenge organiser at software company Zapier, in a social media post<\/a> on X.<\/p>\n In a social media post<\/a> on Bluesky, Melanie Mitchell<\/a> at the Santa Fe Institute in New Mexico said the following about o3\u2019s progress on the ARC benchmark: \u201cI think solving these tasks by brute-force compute defeats the original purpose\u201d.<\/p>\n \u201cWhile the new model is very impressive and represents a big milestone on the way towards AGI, I don\u2019t believe this is AGI \u2013 there\u2019s still a fair number of very easy [ARC Challenge] tasks that o3 can\u2019t solve,\u201d said Chollet in another X post<\/a>.<\/p>\n However, Chollet described how we might know when human-level intelligence has been demonstrated by some form of AGI. \u201cYou\u2019ll know AGI is here when the exercise of creating tasks that are easy for regular humans but hard for AI becomes simply impossible,\u201d he said in the blog post.<\/p>\n Thomas Dietterich<\/a> at Oregon State University suggests another way to recognise AGI. \u201cThose architectures claim to include all of the functional components required for human cognition,\u201d he says. \u201cBy this measure, the commercial AI systems are missing episodic memory, planning, logical reasoning and, most importantly, meta-cognition.\u201d<\/p>\n The o3 model\u2019s high score comes as the tech industry and AI researchers have been reckoning with a slower pace of progress<\/a> in the latest AI models for 2024, compared with the initial explosive developments of 2023.<\/p>\n Although it did not win the ARC Challenge, o3\u2019s high score indicates that AI models could beat the competition benchmark in the near future. Beyond its unofficial high score, Chollet says many official low-compute submissions have already scored above 81 per cent on the private evaluation test set.<\/p>\n Dietterich also thinks that \u201cthis is a very impressive leap in performance\u201d. However, he cautions that, without knowing more about how OpenAI\u2019s o1<\/a> and o3 models work, it is impossible to evaluate just how impressive the high score is. For instance, if o3 was able to practise the ARC problems in advance, then that would make its achievement easier. \u201cWe will need to await an open-source replication to understand the full significance of this,\u201d says Dietterich.<\/p>\n The ARC Challenge organisers are already looking to launch a second and more difficult set of benchmark tests sometime in 2025. They will also keep the ARC Prize 2025 challenge running until someone achieves the grand prize and open-sources their solution.<\/p>\n Topics:<\/p>\nWhat did OpenAI\u2019s o3 model actually do?<\/h2>\n
Does this o3 achievement show that AGI has been reached?<\/h2>\n
So what does o3\u2019s high score really mean?<\/h2>\n