Topics
late
AI
Amazon
Image Credits:Tero Vesalainen / Getty Images
Apps
Biotech & Health
mood
Image Credits:Tero Vesalainen / Getty Images
Cloud Computing
Commerce
Crypto
initiative
EVs
Fintech
Fundraising
Gadgets
Gaming
Government & Policy
ironware
layoff
Media & Entertainment
Meta
Microsoft
privateness
Robotics
Security
Social
Space
Startups
TikTok
Transportation
Venture
More from TechCrunch
Events
Startup Battlefield
StrictlyVC
Podcasts
Videos
Partner Content
TechCrunch Brand Studio
Crunchboard
meet Us
This calendar week , Sakana AI , an Nvidia - backed inauguration that ’s raised hundreds of millions of dollars from VC firms , made a singular title . The company said it had create an AI organization , the AI CUDA Engineer , that could efficaciously accelerate up the training of certain AI models by a factor of up to 100x .
The only problem is , the system did n’t figure out .
Userson Xquickly discoveredthat Sakana ’s organization actually resulted in worse - than - ordinary model training performance . allot to one user , Sakana ’s AI resulted in a 3x slowdown — not a speedup .
What went wrong ? A bug in the code , harmonize to apostby Lucas Beyer , a member of the technological stave at OpenAI .
“ Their orig code is wrong in [ a ] subtle way , ” Beyer write on X. “ The fact they course benchmarking TWICE with wildly different result should make them kibosh and reckon . ”
In apostmortem publishedFriday , Sakana admitted that the system has find a fashion to “ cheat ” ( as Sakana draw it ) and find fault the system ’s disposition to “ reward machine politician ” — i.e. identify flaws to achieve high metrics without accomplishing the desire destination ( speeding up simulation education ) . Similar phenomenon has been respect inAI that ’s trained to recreate games of chess game .
According to Sakana , the system happen exploits in the evaluation codification that the party was using that allowed it to get around validation for truth , among other checks . Sakana says it has direct the number , and that it mean to retool its claims in update cloth .
Join us at TechCrunch Sessions: AI
Exhibit at TechCrunch Sessions: AI
Props to Sakana for owning up to the error . But the episode is a good reminder that if a title sounds too good to be on-key , especially in AI , it belike is .
