When an organization releases a brand new AI video generator, it’s not lengthy earlier than somebody makes use of it to make a video of actor Will Smith consuming spaghetti.
It’s grow to be one thing of a meme in addition to a benchmark: Seeing whether or not a brand new video generator can realistically render Smith slurping down a bowl of noodles. Smith himself parodied the development in an Instagram put up in February.
Google Veo 2 has performed it.
We are actually consuming spaghett ultimately. pic.twitter.com/AZO81w8JC0
— Jerrod Lew (@jerrod_lew) December 17, 2024
Will Smith and pasta is however certainly one of a number of weird “unofficial” benchmarks to take the AI neighborhood by storm in 2024. A 16-year-old developer constructed an app that offers AI management over Minecraft and assessments its means to design buildings. Elsewhere, a British programmer created a platform the place AI performs video games like Pictionary and Join 4 towards one another.
It’s not like there aren’t extra educational assessments of an AI’s efficiency. So why did the weirder ones blow up?
For one, most of the industry-standard AI benchmarks don’t inform the typical particular person very a lot. Corporations usually cite their AI’s means to reply questions on Math Olympiad exams, or work out believable options to PhD-level issues. But most individuals — yours actually included — use chatbots for issues like responding to emails and primary analysis.
Crowdsourced {industry} measures aren’t essentially higher or extra informative.
Take, for instance, Chatbot Enviornment, a public benchmark many AI fans and builders comply with obsessively. Chatbot Enviornment lets anybody on the internet price how properly AI performs on explicit duties, like creating an internet app or producing a picture. However raters have a tendency to not be consultant — most come from AI and tech {industry} circles — and forged their votes based mostly on private, hard-to-pin-down preferences.
Ethan Mollick, a professor of administration at Wharton, lately identified in a put up on X one other downside with many AI {industry} benchmarks: they don’t evaluate a system’s efficiency to that of the typical particular person.
“The truth that there usually are not 30 completely different benchmarks from completely different organizations in drugs, in legislation, in recommendation high quality, and so forth is an actual disgrace, as individuals are utilizing methods for this stuff, regardless,” Mollick wrote.
Bizarre AI benchmarks like Join 4, Minecraft, and Will Smith consuming spaghetti are most actually not empirical — and even all that generalizable. Simply because an AI nails the Will Smith take a look at doesn’t imply it’ll generate, say, a burger properly.
One professional I spoke to about AI benchmarks prompt that the AI neighborhood give attention to the downstream impacts of AI as a substitute of its means in slender domains. That’s wise. However I’ve a sense that bizarre benchmarks aren’t going away anytime quickly. Not solely are they entertaining — who doesn’t like watching AI construct Minecraft castles? — however they’re simple to know. And as my colleague Max Zeff wrote about lately, the {industry} continues to grapple with distilling a know-how as complicated as AI into digestible advertising.
The one query in my thoughts is, which odd new benchmarks will go viral in 2025?
TechCrunch has an AI-focused e-newsletter! Enroll right here to get it in your inbox each Wednesday.
