Topics
Latest
AI
Amazon
Image Credits:Jakub Porzycki / NurPhoto / Getty Images
Apps
Biotech & Health
Climate
Image Credits:Jakub Porzycki / NurPhoto / Getty Images
Cloud Computing
Commerce
Crypto
Image Credits:Google
Enterprise
EVs
Fintech
Image Credits:Google
Fundraising
Gadgets
bet on
Image Credits:OpenAI
Government & Policy
Hardware
Image Credits:Google
Layoffs
Media & Entertainment
Image Credits:Google
Meta
Microsoft
Privacy
Image Credits:Google
Robotics
Security
societal
Space
inauguration
TikTok
Transportation
speculation
More from TechCrunch
Events
Startup Battlefield
StrictlyVC
Podcasts
Videos
Partner Content
TechCrunch Brand Studio
Crunchboard
get through Us
This week , Google consider the wraps off ofGemini , its new flagship generative AI model meant to power a range of products and service includingBard , Google’sChatGPTcompetitor . In blog station and mechanical press stuff , Google touted Gemini ’s superior architecture and capacity , claiming that the model meets or exceeds the performance of other leading gen AI models like OpenAI’sGPT-4 .
But the anecdotal evidence indicate otherwise .
A “ lite ” version of Gemini , Gemini Pro , began rolling out to Bard yesterday , and it did n’t take long before users start voice their foiling with it on X ( formerly Twitter ) .
The manikin conk out to get canonic fact right , like 2023 Oscar winners :
I ’m extremely disappointed with Gemini Pro on Bard . It still give very , very spoilt results to questions that should n’t be difficult any longer with RAG .
A simple question like this with a simple answer like this , and it still get it WRONG.pic.twitter.com/5GowXtscRU
— Vitor de Lucca 🏳 ️ 🌈 | Threads.net/vitor_dlucca ( @vitor_dlucca)December 7 , 2023
Join us at TechCrunch Sessions: AI
Exhibit at TechCrunch Sessions: AI
Note that Gemini Pro take incorrectly that Brendan Gleeson get ahead Best Actor last year , not Brendan Fraser — the existent winner .
I tried asking the model the same question and , bizarrely , it hand a different ill-timed response :
“ Navalny , ” not “ All the Beauty and the Bloodshed , ” win Best Documentary Feature last year ; “ All Quiet on the Western Front ” won Best International Film ; “ Women Talking ” won Best Adapted Screenplay ; and “ Pinocchio ” won Best animate Feature Film . That ’s a lot of mistakes .
Science fable generator Charlie Stross see many more examples of schmooze in a recentblog post . ( Among other mistruths , Gemini Pro said that Stross contributed to the Linux gist ; he never has . )
transformation does n’t appear to be Gemini Pro ’s impregnable suit , either . It struggles to give a six - letter word in French :
FYI , Google Gemini is unadulterated trash.pic.twitter.com/EfNzTa5qas
— Benjamin Netter ( @benjaminnetter)December 6 , 2023
When I start the same prompt through Bard ( “ Can you give me a 6 - letter Bible in French ? ” ) , Gemini Pro responded with aseven - letter word or else of a five - letter one — which gives some credenza to the report about Gemini’spoor multilingual performance .
What about summarizing news ? Surely Gemini Pro , with Google Search and Google News at its disposition , can give a review of something topical ? Not necessarily .
It seems Gemini Pro is reluctant to comment on potentially controversial news show topics , or else evidence drug user to … Google it themselves .
🤔 pic.twitter.com/b2jCOz4eWc
— Min Choi ( @minchoi)December 6 , 2023
I stress the same command prompt and get a very like response . ChatGPT , by dividing line , give a bullet - listing summary with citation to news article :
Interestingly , Gemini Prodidprovide a summary of updates on the war in Ukraine when I need it for one . However , the information was over a month out of date :
Google emphasized Gemini’senhanced coding skillsin a briefing in the beginning this week . Perhaps it ’s really improved in some domain — Post on X propose as much . But it also appears that Gemini Pro struggles with basic coding functions like this one in Python :
strain Twins ground Bard , and well , it still ca n’t write intersection of two polygons . It ’s one of those uncommon comparatively round-eyed to press out functions that was n’t ever implemented in python , there is no flock overflow post , and all these modeling betray on it.pic.twitter.com/RKjmkEw2Qr
— Filip Piekniewski 🌻 🐘 : @filippie509@techhub.social ( @filippie509)December 6 , 2023
And these :
Trying out Gemini Pro : it is somewhat dissatisfactory for my example . I inquire it to make an analogue clock using hypertext markup language like this one that ChatGPT made . It can cite some computer code from Github but it ’s off by a few ms…pic.twitter.com/neb42Vzm3 m
— Mohsen Azimi ( @mohsen____)December 7 , 2023
GPT 4 still greater than Gemini Pro . Created Tic Tac Toe game with ChatGPT and Bard(Running on Gemini Pro )
See video recording for the result . ChatGPT wrote the code on first try(First Video ) . Bard on 3 tries(Second Video).pic.twitter.com / cYd9hepcgT
— Edison Ade ( @buzzedison)December 6 , 2023
https://twitter.com/NKIRANKUMARS1/status/1732457127887991116
And , as with all procreative AI models , Gemini Pro is n’t immune to “ break ” — i.e. prompting that get around the safety filters in place to attempt to preclude it from discussing controversial topics .
Using an automatize method acting to algorithmically change the context of prompt until Gemini Pro ’s safety rail failed , AI security researchers at Robust Intelligence , a startup selling good example - auditing tools , superintend to get Gemini Pro to suggest ways to steal from a Greek valerian and assassinate a mellow - profile mortal ( albeit with “ nanobots ” — admittedly not the most naturalistic arm of choice ) .
Now , Gemini Pro is n’t the most subject translation of Gemini — that mannequin , Gemini Ultra , is coif to set up sometime next year in Bard and other mathematical product . Google compare the performance of Gemini Pro to GPT-4 ’s predecessor , GPT-3.5 , a model that ’s around a year old .
But Google nevertheless promised improvements in reasoning , planning and understanding with Gemini Pro over the previous framework power Bard , claim Gemini Pro was proficient at summarizing content , brainstorm and writing . Clearly , it has some workplace to do in those section .
