Claude 3 AI KILLS OpenAI GPT 4 & Google Gemini (What Will Sam Altman Do?) - Anthropic AI 🤖🤯

Today anthropic just launched claw 3 which beats out Sam Alman open AI chat GPT as well as Google Gemini which just beat out chat GPT in recent Benchmark tests this beats out both of those let's dive right into [Music] it so this is some crazy news you'll see here today we're announcing Claud 3 Model family which sets new industry B benchmarks across a wide range of cognitive tasks the family includes three state-of-the-art models in ascending order of capability Claude 3 Hau

Clae 3 Sonet and Claude 3 Opus each successive model offers increasingly more powerful performance allowing users to select the optimal balance of intelligence speed and cost for their specific application Opus and Sonet are now available to use in cloud. a and the CLA API which is now generally available in 159 countries haou will be available soon so here are the Claude 3 Model families and you can see here we have the intelligence bench scores on the left-and side and we have

The cost price per million tokens log scale on the bottom side and it is you know gradually increasing as we get from Haiku to Sonet to Opus and we'll dive into each specific one of those models in just a second so a new standard for intelligence Opus our most intelligent model outperforms its peers on most of the common evaluation benchmarks for AI systems including undergraduate level expert knowledge MML graduate level expert knowled or expert reasoning GP QA basic mathematics GSM 8K

And more it exhibits near human level comprehension and fluency on complex tasks leading the frontier of general intelligence all claw 3 models show increased capabilities in analysis and forecasting Nuance content creation code generation and conversing in non-english languages like Spanish Japanese and French below is a comparison of the CLA 3 models to those of our peers on multiple benchmarks of capability all right so you'll see here Claude Opus Claude Sonet and Claude Hau and then they're compared two models that we

Know very well such as GPT 4 GPT 3.5 Gemini 1.0 Ultra and Gemini 1.0 Pro all right so one thing I am noticing here is it's not uh compared to Gemini 1.5 Pro but you can see here we have Claude Opus on the left hand side and it's you know getting compared through different Benchmark tests like undergraduate level knowledge graduate level reasoning grade school math math problem solving M multilingual math code reasoning over text mixed evaluations knowledge Q&A and common knowledge

And you'll see here like we we're getting close to that 100% with some of these we're like 95% point4 and a 10 shot test 96.4 and a uh 25% shot uh or a 25 shot text uh test um 86% uh 83% 84% 90% 60% 95% 50% 86% and if we look over here it's beating out gp4 and every single category all right so what this tells me I mean we saw when Gemini ended up beating out GPT 4 um you know

For example different times when gemini or Google has these big launches within the AI news space such as you know Gemini coming out launching beating GPT 4 or gp4 turbo and one thing it's not showing gp4 turbo as well um but this is still really good of course that but when we do see those announcements we've always seen open AI or Sam ultman relaunch something to kind of out Shadow it we've seen this with Sora we've seen this with many different

Things um that open AI has done so who knows maybe this will spark Sam ultman to kind of retaliate and launch something um amazing all right or maybe who knows maybe anthropic isn't as you know much of a competitor um in his mind as Google let me know what your thoughts are in the comments down below um of course uh anthropic is has big Partnerships with you know Amazon and whatnot and they're doing some really cool stuff here like we see

Here this is pretty crazy so I think this will definitely um just really make things more competitive in this space I mean with Sam mman with open AI with Google everyone is really just pushing each other here so um it is starting to really heat up now a few different things with this new launch so near instant results the claw 3 models can power Live customer chats Auto completions and data extraction tasks where responses must be immediate and in real time

Hau is the fastest and most cost-effective model on the market for its intelligence category it can read and uh read an information and data dense research paper on archive 10,000 tokens with charts and graphs in less than 3 seconds following launch we expect to improve performance even further for the vast majority of workloads Sonic is two times faster than Claude 2 and 2.1 with higher levels of intelligence it excels at tasks demanding rapid responses like knowledge retrieval or sales automation Opus

Delivers similar speeds to CLA 2 and 2.1 but with much higher levels of intelligence so what this makes me think this is really good because we need basically like instant responses like that's where things are going especially when you're incorporating it into different applications as well as incorporating to different businesses like I give this example when I was doing my video on Gro which is the new chip which basically is faster than any um you know other chips faster than Nvidia

Chips for these AI models and it would be interesting too to see some of these new claw models run on the grock chips right now they're mainly on open source models and whatnot but imagine some of these fast really fast AI models using the grock chip um that would make things really really powerful like I mentioned in the grock video for things like cold calling centers or call centers support centers these are things that we've already started to see AI uh

Software that's able to do these AI sales a or call center agents or sales agents or whatever and when you're talking to an AI you can't you can't ask it a question saying how's your day going today and then it has 5sec weights and then it says my day is going good you know you need it right away um to enhance that customer experience right so this is on the right track with Claude and then strong Vision capabilities the Claude 3

Models have sophisticated Vision capabilities on par with other leading models they can Pro process a wide range of visual formats including photos charts graphs and Technical diagrams we're particularly excited to provide this new modality to our Enterprise customers some of whom have up to 50% of their knowledge bases encoded in various formats such as PDFs work uh flowcharts and presentation slides so this is also really good of course so now it actually had it's more multi it has multi modalities not

Just text it has you know a visualization and here is their Vision so and this is the benchmarks right here we can see so claw 3 Opus CL 3 Sonic CLA 3 Hau versus gp4 Vision Gemini 1.4 or 1.0 Ultra and pro and we'll see here the ones in the green are the ones that are the highest so it's tied for Math and reasoning with Gemini Ultra and then it's uh pretty high with document visualize uh visual uh Q&A um 89%

Uh math it is just uh at 50% the highest is 53% science diagrams 88% the highest is 88.7 it's actually 8 8.1 so it's the second highest and then chart Q&A 80.8% the highest is 81.7 so really good benchmarks comparative to these models and now I'm going to play the actual CLA 3 Opus video so you can get an idea of how it actually works in specific situations in this video we're going to see if Claude and a couple of friends

Can help us analyze the world economy in a matter of minutes okay I've asked Claude 3 Opus which is the largest model in anthropics new Claude 3 family to look at the GDP trends for the US and write down a markdown table of what it sees we've given Opus and all the other models in the CLA 3 family extensive training on tool use and one of the major tools it's using is this web view tool it goes to a URL looks

At what's on the page and because it's multimodal it can use the information on that page to solve complex problems so here's the markdown and it's important to note that cloud doesn't have direct access to these numbers it's literally looking at the same browser you and I are seeing looking at the trend line and trying to estimate what the exact numbers are let's see how accurate it was we've asked the model to create a plot of the data and it's used

This second tool this python interpreter to write out the code and then render the image for us to check and here's the image look it's actually added helpful little tool tip animations to explain some of the major Peaks and troughs in the last decade or two of the US economy and we can compare that graph with the actual data and it turns out it's pretty close it's actually within 5% accuracy and by the way cla's transcription here isn't just coming from

Its pre-existing knowledge of US GDP we tried it with a large sample of madeup GDP graphs and its transcription accuracy was within 11% on average next we asked the model to do some statistical analysis projecting out into the future performing simulations to see where the GDP of the US might head and we can see that it's run this analysis using Python and it's able to perform these Monti Carlo simulations to see what the range of GDP possibilities might look like for

The next decade or so but I wonder if we can go further we're going to get the model to analyze a more complicated question that is is how GDP might change across all of the biggest world economies and then to help it do that we're going to give it one more tool called dispatch sub agents this basically allows the model to break down the problem into lots of sub problems and then write prompts for other versions of itself to help pick

Out the slack the models can then complete a more complex task by all working together here you can see it's written this prompt and given very precise instructions that it wants the other models to follow including a format for the data that it's hoping to return it's dispatched a version of this prompt to one model that's going to look at the US one for China One for Germany Japan and so on we can see in these progress bars that the sub

Agent models are now completing the set task for each of the individual economies they're going to the relevant web pages they're getting the information they're running the code to analyze it just like we saw in the previous US example but all in parallel let's just skip forward to see what the model produced you can see it's run the analysis it's produced a pre- and post pie chart of how it expects the world economy to look in 2030 versus 2020 and it's

Given us a written analysis too where it makes variable predictions that relate to the statistical analysis that it ran it's telling us that it thinks the GDP share of particular economies will change and which ones will be larger or smaller by 2030 so there we have it complex multi-step multimodel analysis run by a model that can create sub agents to get even more tasks running in parallel we're excited to see what you our customers can do with these Advanced Claude 3

Capabilities all right so that video is pretty crazy and uh just to continue on we have also fewer refusals so this is a new uh update as well for these models so prev Cloud models often made unnecessary refusals that suggested a lack of contextual understanding we've made meaningful progress in this area Opus Sonet and Haiku are significantly less likely to refuse answers to prompts that border on the systems guard guard rails than previous generations of models as shown below the Claude

Uh 3 models show a more Nuance understanding of requests recognize real harm and refuse to answer uh and refuse to answer are harmless prompts much less often all right so we can see here incorrect refusals refused on harmless prompts on the left-and side for a percentage so 10% 20% 30% and then we have claw 3 Opus right here just over 10% we're looking at about 12% Sonet we're looking about 11% and then uh Hau we're looking about 7% and then if

You look at Claude 2.1 we were just over uh like around 26% so so we're looking at about around half uh I would say bit more than half that you're actually um going to get for you know less refused prompts for different harmless things and you would notice this too like if you were talking to it it was really annoying sometimes because you would ask it something and um it wouldn't be able to answer so I'm going to also play another

CLA 3 video for the high cou so you can get an understanding of how that model Works Claude Haiku is one of the fastest and most affordable Vision capable models in the world to demonstrate this we're going to read through thousands of scan documents in a matter of minutes the Library of Congress Federal writers project is a collection of thousands of scanned transcripts from interviews during the Great Depression this is a gold mine of incredible narratives and real life Heroes but

It's locked away in hard to access scans of transcripts imagine you're a documentary filmmaker or journalist how can you dig through these thousands of messy documents to find the best source material for your research without reading them all yourself since these documents are scanned images we can't feed them into a Texton llm and these scans are messy enough that they would be a challenge for most dedicated OCR software but luckily Haiku is natively Vision capable and can use surrounding text to

Transcribe these images and really understand what's going on we can also go beyond simple transcription for each interview and ask Haiku to generate structured Json output with metadata like title date keywords but also use some creativity and judgment to assess how compelling a documentary the story and characters would be we can process each document in parallel for performance and with claud's high availability API do that at massive scale for hundreds or thousands of documents let's take a look at some of

That structured output Haiku is able to not just transcribe but pull out creative things like keywords we've transformed this collection of many many scans uh into Rich keyword structure data imagine what any organization with a knowledge base of scann documents like a traditional publisher healthc care provider or Law Firm can do Haiku can parse their extensive archives and bodies of work we'd love for you to try it out and see what you build okay and another thing with these models is

Improved accuracy so business of all sizes rely on our models uh to serve their customers making it imperative for our models outputs to maintain High accy at scale to assess this we use a large set of complex factual questions that Target known weaknesses in current models we categorize the responses into correct answers incorrect answers or hallucinations and admissions of uncertainty where the model says it does know doesn't know the answer instead of providing incorrect information compared to Claude 2.1 Opus demonstrates

A twofold Improvement in accuracy or correct answers on these challenging open-ended questions while also exhibiting reduced levels of incorrect answers in addition to approving more TR uh producing more trustworthy responses we will soon enable citations in our claw 3 models so they can point to precise sentences in reference material to verify their answers so that's something I really like when using um AI models is having those references or the citations so you can actually check and see where it's getting that

Information if you don't know I mean you're just taking the AI model off of its word and it's a lot harder to fact check when it doesn't cite it because then if you actually were to fact check which most people a lot of people don't do um you would have to go type that information into something like Google and then find that article and it could be amongst you know 10 different articles or websites or wherever on the web um when

You have it plugged in directly into the AI model with the citations you can easily just fact check and it makes things a lot easier when doing your research so here we see the graph of hard questions percent of questions set we can see the percentage here um so Claude 3 Opus versus Claude 2.1 so correct answers on the uh right left hand right here we have o uh Claude Opus getting just over 45% I would say uh and then Claude

2.1 just over 20% uh correct and then incorrect questions we're getting under 30 35% for Opus and about 42 for Claude 2.1 so it's getting a lot more correct that need to be correct than quad almost double and then it's getting a bit less incorrect that it shouldn't uh you know get corre that it shouldn't be um that are incorrect right so and then I don't know or unsure it's getting a lot less of those so it's just under 20% about

19 for Opus and then about in the 35s I would say or 30 just actually about 30% for 2.1 Claude 2.1 so and then here is the Claude sonnet video this is just a simple prompt to turn sonnet into a dialogue agent that will talk with you in a language that you're trying to learn so I chose Spanish and I wanted it to basically take my imperfect Spanish and and help me improve it um so I decided I wanted it to

Do a few things I wanted it to take my m message which will be in kind of imperfect Spanish um and write out what it thinks I intended in English I then ask it to write back the ideal learner message which is just my message as it kind of should have been written in Spanish so I can see the kind of Ideal form of this uh then I ask it to write a teacher response which just a response to me in

Spanish uh so that I can continue the conversation great so this is basically just Sonet saying that we're ready to start so I'll just start with a simple first message and here it's following the format that I asked for so it's repeating the message that I tried to send it back to me in English it's telling me how I should have said it so it's corrected some of the grammar issues in my request um and then it's responded to me in

Spanish and then it's asked me uh where I'm from so okay now imagine I don't know a certain word in Spanish but I still want to say it I'm going to just include that word in English in square brackets and hopefully it will just translate it back to me and suppose I hit a roadblock because I just don't understand the message that it's sent to me I can just ask it to translate that message to me into English and then I

Can read that and I can respond to it again in Spanish continuing the dialogue and as a final step you could ask son it to create a little quiz for you based on the things that you've been discussing so hopefully that is a useful prompt if you're interested in using Sonet as a language learning partner I hope you try another thing that is added is a long context and near perfect recall all right so Claude also was known for having pretty

Long contacts before G pt4 turbo came and then Gemini came um it was known for having the longest contacts but now it's even having a longer contact so the claw 3 family of models will initially offer a 200,000 uh context window upon launch however all three models are capable of accepting inputs exceeding 1 million tokens and we may make this available to select customers who need enhanced processing power to process long context uh prompts effectively models require robust recall capabilities the

Needle in a hay stack nah evaluation measures a model of ability to accurately recall information from a vast Corpus of data we enhance the robustness of this Benchmark by using one of uh one of 30 random needles / question pairs per prompt and testing on a diverse crowdsource Corpus of documents claw 3 Opus not only achieved near perfect recall surpassing 99% accuracy but in some cases it ident even identified the limitations of the V evaluations Itself by recognizing that the needle

Sentence appeared to be artificially inserted into the original text by a human so this is really powerful if you haven't heard of the needle and Hast stack um evaluation this is something that that uh Google Gemini did as well for their models it's a very good evaluation when you know testing these large context models and being able to see if it's actually able to recall that information and like not just somewhat skim over it but it's like looking through big chunks

Of context window like say videos or text whatever the case may is for tokens and it's able to actually like notice things like a needle and a hay stack so it's crazy that these mods are getting so powerful 200,000 Co Contex window a year ago or would have been like unheard very unheard of or just crazy gp4 turbo announced their 140,000 context window um that was like a huge huge deal so um and then Gemini came and they even have more

Higher context models up to like a million or even gbt or Gemini 1.5 to 10 million potentially for probably like Enterprise clian clients once they get there it's it's getting pretty crazy all right as we see these context mod those increase as well as the recall increase think about how these models are improving to be able to do that and it's it's it I mean it's I could go on for a while for this I'm just going to move on and

Get to the next part but I'm just planning that idea in your head let me know what you guys thought thoughts are on how these models are increasing like that it's pretty crazy in my opinion so CLA 3 Opus recall accuracy over 2 200,000 tokens average over many diverse doc dou sourced and needle uh sentences so this is like the quote unquote almost perfect percentage and we can see here needle uh position percent on the left hand side and then context

Length you know depend you know from 12,000 context to 200,000 context it's getting those um finding those needles all right and getting it with almost basically 100% accuracy so pretty crazy stuff also these are easier to use so the CLA 3 models are better at falling complex multi-step instructions they are particularly Adept at adhering to Brand voice and response guidelines and developing customer facing experiences our users can uh trust in addition the cloth through models are better at producing popular structured

Outputs in formats like Json making it simpler to instruct Claud for use cases like natural language classification and sentiment analysis all right so pretty pretty crazy stuff guys and we'll get into kind of what the difference is between some of these three models if you couldn't really tell which you probably couldn't fully tell from those videos and one thing I just want to point out too before this is that in the first the first video with Opus you'll see how it

Mentions about using different models to achieve kind of the task kind of like somewhat getting closer to more like kind of like a agent type thing so that's really cool because obviously that's the direction we've been moving in with the AI space and uh as more companies start to move in this direction it's going to push other companies to do this because that's one thing that we are speculating that someone like open AI we kind of I mean it's pretty much

Starting to look like I mean I personally believe and pretty much a lot of people believe that they obviously have a lot more and they've said this people at their company have literally said this that they have so much more basically you know advancements within their labs and obviously is released to the public and I mean that's kind of you would kind of expect that but with that being being said they may already have something like agents already they've definitely been

Working on them they've stated that they've been working towards them so they may already have something like that or they may be really close so as more companies start putting these things out like agents or other advancements then it may push open AI to put some of their uh big advancements out all right because they're heavily leading the space right now and uh so the three clawed models just to sum it up opus highest intelligence that's their best model Sonet more

Affordable than other models with similar intelligence and it's good for scale and then Haiku is very fast and low cost all right and Opus um a little bit more detailed breakdown on these so Claud Opus is our most intelligent model with best in market performance on highly complex tasks it can navigate open-ended prompts and site unseen scenarios with remarkably fluen with remarkable fluency and humanlike understanding Opus shows us the outer limits of what's possible with generative AI so the cost so

This is cost for the first is input per million tokens and then the second is output per million tokens so $15 per input of a million tokens $75 per output of a million tokens kind of pricey and then the context window is 200k potentially well at the bottom you'll see 1 million tokens available to specific use cases please inquire and then potential use is Task automation plan and execute comp complex actions across apis and databases interacting Cod interactive coding R&D research

Review brainstorming and generation drug Discovery and then strategy Advanced analysis of charts grph financi financials and market trends and then differentiator is higher intelligence than any other model available next is Claude Sonet so strikes the ideal balance between intelligence and speed particularly for Enterprise workloads it delivers strong performance at a lower cost compared to its peers and is engineered for high endurance in large scale AI deployments cost per Mill is $3 per input and then $15 per output of tokens and

Then context window 200k potential uses data processing rag or search and retrieval over vast amounts of knowledge sales product uh recommendations forecasting targeted marketing and then time saving task code generation quality control parse text from images and then differentiat more affordable than other models with similar intelligence and better for scale next and lastly is clae 3 hu is our fastest model most compact model for near instant responsiveness it answers simple queries and requests with unmatched speed users will be able to

Build seamless AI experiences that mimic mimic human interactions cost uh per Mill of course 0.2 25 for uh input and then $125 per output and then context window 200k potential uses customer interactions quick and accurate support in live interactions translations content moderation catch risky Behavior or customer quests and then cost saving tasks optimize Logistics Inventory management extract Knowledge from unstructured data and then the differentiator smarter faster and more affordable than other models in its intelligence category all right so that is

That for the Claude launch and it's pretty crazy that this thing is you know beating gp4 Sam wolman's AI open AI as well as Google Gemini and uh in these benchmarks um this is showing that Claude is definitely something you know not don't definitely don't count them out of the race to AGI I mean they still got a ways to go I mean like I said everyone kind of has a bit of ways to go it looks like towards open AI

But maybe this will push Sam Alman to release some more features let me know what you guys think about that do you think is going to be you know pushed to release some new features maybe some new Sora stuff maybe he's going to push Sora faster or maybe he's going to maybe release GPT 5 who knows um but let me know what your thoughts are in the comments down below let me know what your thoughts are about Claude 3 I'll be

Definitely making more videos about clae 3 so make sure to smash that subscribe button smash that like button and other than that guys you will see that there is a bar scrolling right here um if you want to join our free YouTube Community it's a strive it's called stride Community it's a free AI Community where you can network with myself as well as like Mar like-minded individuals AI enthusiasts entrepreneurs business owners whatever the case may be there's a bunch of free

Resources there on AI prompts videos exclusive stuff it's a free Discord Channel and free Facebook group just go to stric community.com as well as uh our free 6-day accelerate your stride AI challenge a lot of value packed in there people charge like thousands of dollars for this stuff that people sell like courses for that or whatever so it's 100% free you can just go to accelerator stride. comom I'll be launching that live challenge very soon and then last couple things guys

Is I did just launch a big personal um milestone for me as I wrote this book took a really long time dedicated like the last few months of 2023 to actually writing this thing getting it published and uh put my Blood Sweat and Tears into this thing as well as my heart and my soul so it's called client acquisition code if you run a business with your coach marketing agency consultant anything online that you you need leads this book will dramatically

Help your business 600 pages of pure value 600 plus pages and will help you escape the guru Matrix all right so you can get the digital copy as well as a bunch of bonuses like I have a accelerator course as well as um free templates all this stuff go check it out there's eight eight uh eight bonuses you can get at clientacquisition code.com it's only 27 bucks guys and it's less than a cup of coffee day for a month and there's

A 30day guarantee so if you don't feel like you know you got value from it or whatever just shoot me an email and I'll refund you so there's literally nothing to lose I I also do go over the best CRM to use in that book and exactly how to set it up it's go high level it's the best CRM my whole business is built on it I literally swear by it I tell everyone to use it we have a a free

Snapshot right here it's it's ghost level.com so if you sign up through ghost High Lev through our affiliate link it doesn't cost you anything out uh else and you would would if you just signed up through the website I'll send you that free snapshot with custom websites custom forms custom all the all the automations and stuff you would need to get started as well as a free training with bunch of different bonuses definitely check that out there's a video on there

Explaining it all best CRM trust me um also the most affordable CRM I've tried all the ones in the industry like HubSpot close.io all the top crms go high level is the best other than that guys smash that subscribe button I will be posting a lot of content I'm going hard on the YouTube grind road to 100K appreciate all your support all your thoughts in the comments down below and I will see you in the next video guys keep hustling keep

Grinding and of course accelerate your stride take care guys