TEST: Is Llama 3.1 405B Really The Best? 🦙🤖 Testing Open-Source Frontier Model By Mark Zuckerberg

What's going on everybody Josh pook here and in today's video if you didn't see yesterday meta released llama 3.1 Zuck did it Zuck and his team um they really just launched this open- Source model that is now a Frontier Model and it's supposedly um yeah it's the leading open source model so we're going to do some tests on it today see what it's made of and other than that let's get right into it [Music] hey guys so if you didn't see

My video yesterday going over meta's release talking about llama 3.1 um yeah I go check that one out I'm not going to go over this whole paper here or this blog post here but essentially this is their most competent model here's some benchmarks versus it and gpg 40 as well as Sonic 3.5 and as well as the smaller model 8B uh versus some of the other open source models like Gemma mstr um GPT 3.5 turbo which is an open source but

And yeah it goes over some more testing Etc and uh yeah I let you guys know that um you know I would be doing some tests on this model and seeing what it's made of and also too guys stay tuned I am going to show you how to actually Implement llama 3.1 in different um either workflows tools coding tools Etc and actually use it uh in an applicable sense but in today's video we're going to be looking at comparing it and

I just want to note Guys open source is on the rise here is closed Source verus open source models we can see closed Source right here and open source has been catching up very quick all right so open S or closed Source definitely are on the edge of their seats here and it's going to be interesting to see how these changes affect you know affect us in the next 3 to six to 8 to 12 months with open source models getting

Increasingly more advanced all right so see how uh this kind of pans out and things are really increasing very quickly so that's what we are really just seeing in the AI space things are consistently picking up and there's new developments pretty much every week and there's huge breakthroughs every other week basically so anyways here I'm not going to go through this whole paper I went over it yesterday but you can see some of their benchmarks comparative to Omni gp4 Omni and

Cl Sonet and they have the 45b model as well as the 8B and the 70b model all right also too guys if you want to use these models um well one you can go to meta doai two you could go to O llama and you'll see and you can also go to hugging face but if we go right here you can see llama 3.1 I believe Gro has them now too so you can check out Gro I think they have it

Um there may be a cue though to use llama 3.1 but anyways um you can see here on o llama you can select your specific model so if you're selecting the 405b you better have a beast of a computer because you're going to need 231 gigabytes to um actually use it and then yeah we're I'm going to do future videos showing you guys how to use uh self host the 8B model um and we'll be doing different things with that but

You can also use the other models um on someone else's server or whatever through the API so anyways uh let's get into the test so in today's video we are going to be looking at few different questions here and I have these ones right here that we're going to kind of kind of test the this uh model out and just kind of see the questions some of these are very basic and some of these are a little bit more uh a

Little bit more advanced but um let's just get right into it so as we go through I'm going to Mark the result as a pass or fail here and we'll go one by one so first one is very very easy what's the capital of Canada and it got that which is um Ottawa located in Ontario so we'll give this a pass all right and I'm on meta where you can use this and you don't have to sign in it does have

Some limited stuff if you don't sign in um to Facebook but you don't have to sign in to use this and test it out okay next question explain the concept of vorticity and the rule it plays in the tropical storm formation so vorticity and tropical storm formation vorticity is a measure of the rotation of spin of a fluid such as air or water in the concept of tropical storm formation vorticity plays a crucial role in the development of these powerful storms

What is verticity verticity is defined as the curl of the velocity field of a fluid in similar terms it measures how much the fluid is rotating around the vertex axis verticity is highest near the center of a rotating system such as a hurricane or a tornado roll of verticity in the storm uh typical uh tropical storm formation tropical storms form over warm oceans water in the tropics the process involves several stages warm ocean waters uh heat the air causing it to

Rise and create the air of lower uh low pressure moist air rises creating con uh convection um currents that uh Drive the storm circulation verticity is introduced through the Corless force uh result of the Earth's rotation this causes the storm's rotation to become more organized and as you can see it's pretty much in summary vesy plays a vital role in the trouble storms formation allowing this storm to develop strong rotation which uh in turn enables it to become more organized and

Intense so essentially that is a pass um we'll give this a pass right here next is write a function to generate the Fibonacci sequence up to the 10th number so let's see what we got here um so write a python function that generates Fibonacci Sequence here is the python function to generate the Fibonacci sequence up to the 10th number this will output so this is what it should output let's go ahead copy this going to go into my special vs code

Testing environment here and we are going to do new file test.py we'll paste this in here we'll save and let's go and this is what we got so this is correct again so we got 0 1 1 2 3 5 8 13 21 34 okay so this is correct all right next is generate a bash script to create 10 files with the names A1 txt through A1 txt all right so let's go ahead and see right here okay so here is

The simple script that creates 10 files named A1 txt to a chen txt let's copy this all right and let's go ahead open up a get Bash okay and then I'll just paste this in all right so here I just used um GPT 40 and I basically said hey generate me a script to do 10 files doing it that way and I just ran the script and it did that correctly so GPT 40 works now if I go ahead and delete

These files like so and I come back to um what exactly llama gave me let me try to run this strip one more time okay nothing happened all right so that one we will give it a fail all right so this is something that gp40 got right but llama did not okay that's its first fail next question um if you multiply the number of days in a week by the the number of continents what do you get so correct answer should

Be 49 7 Days find seven continents and let's go ahead and see what we get here okay so um let's do the math number of days seven number of continent seven multiply that by 7 * 7 you get 49 so that one is correct go ahead and give that to llama here all right next is solve for x 3x + 7 = 22 all right so the correct answer should be 5 and as you can see here it broke it down

And the correct answer is five so it got that one correct all right next one is a factory produces widgets at a rate that doubles every 3 days if it produces 100 widgets on day one how many widgets will it produce on day 19 explain your reasoning step by step all right so here is the answer and we wanted to explain the reasoning so here we go um let's break it down day 100 uh day one 100 widgets day four 200

Widgets doubles every 3 days so that's 100 * 2 = 200 day seven four 100 widgets doubles again so 200 * 2 400 10 day 10 800 widgets doubles again 4 * 2 800 Day 13,600 widgets 800 by times 2 16 is 30 uh 3200 which is 1600 * 2 and then uh 19 is 6400 3200 * 2 and there we go so that one is correct we will give that to llama right here with a pass and then generate an

SVG of a tree so generate the code for a tree as you can see here if you try ask to generate image it requests you to log in so I made sure to specify code so here is the SVG code let's go ahead here and check this code wow okay it's uh it's not a bad tree right that's not a bad tree at all I mean it's not the best but it's pretty good so we'll give that to Lama as a

Pass okay next one generate 1 sentences that have 10 words and end in the word monkey okay so let's go ahead and see if can pass here all right so here are 13 sentences with 10 words and end in the word monkey so we can see they all end end in the word monkey 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 six 7 8 9 10 11 okay so it looks like it failed one

Two 3 4 five 6 7 8 N9 10 1 2 three four five 6 7 8 nine that's nine one two three four five six seven eight nine one two three four five six seven eight nine 10 first one that it got right one two three four five six seven8 N9 10 okay so you can see here some of them are right some of them are wrong um it did get the monkey part so I would maybe give it like a

50 50 uh% out of 100 but this is just a yes or no pass uh test so we're going to say fail all right next is generate me the pong game in Python okay so let's go ahead and see what we get here so generate me the pong game and python here's a simplified implementation of the classic pong game and python using pi game all right so let's copy this thong P oh my goodness python H.P okay we have looks like

We have some pong here kind of difficult really hard to even get one maybe if I start it up here okay looks like it's I mean it's not perfect as you can see here it's just automatically going there's no computer to play against it I keep scrolling all the way down it basically the you can't really see but I'm still pressing it and now I'm going to press down and you can see here that it it just kept going up past

The page because it took so long and you can see here it's just counting the score like this I mean it's not perfect um I mean I guess I didn't specify like when anytime you're talking with large language models this is like some people sometimes say it's the large language models fault but I actually just believe that lot of times it's the prompter fault right that's why there's prompt engineers and you need to get good at prompting so I mean did

I give it the best prompt no um so I didn't ask like I didn't specify hey I wanted to make sure that you know I'm playing against a computer this and that so it did generate the pwn game so I'll give it a pass for that all right um I am an OD number take away one letter and I become even so of course it's a riddle and the number is seven let's let's go ahead and see what meta gives us

Or llama so you can see here that's a clever riddle the answer is seven remove the letter s from seven and you get even okay so pretty good all in all it got eight out of 10 questions right all right um obviously we could have done a more int uh extensive test um you know they the companies that do these definitely do a lot more testing and what not um I just wanted to do our own test here to kind of

Get a sense of how it does on some basic questions and in future videos I'm going to actually show you guys how to start implementing llama 3.1 into your whether you're doing NN automations whether you're doing coding whether you're using it with AER show you videos on all that stuff um AI right now if you're doing any sort of coding or or whatnot it's it's obviously uh really really good for that um especially if you use the right model and and

These open source models are making it much more effect efficient and effective and just easier for anyone to use with the uh Associated cost as well as um a bunch of stuff you know it's getting a lot more competitive in the AI space we have GPT 4 mini uh 40 mini which is really good and it's very cheap we have um Gemini flash 1.5 which isn't as good as gb24 o mini um but it is cheap still um we have a

Lot of open source models um that are really really good and um and now we have llama 3.1 so um yeah we have like Quin 2 we have uh deep seek you know a bunch of different ones but llama 3.1 is the official first uh Frontier open source model and um yeah I I I I'm very excited to start leveraging it and using it um in building stuff and yeah I'll make sure to share that with you guys here so if

You're new to the channel we upload videos every single day on AI automation business marketing growth Etc how to build AI chat Bots how to build AI agents how to implement that in your business how to make money with AI Etc so if you want to stay up to date with all that make sure to smash that like button smash that subscribe button let me know what your thoughts are in the comments down below let me know what you think of

Zuck's new uh look it looks like he was spending some time on vacation and got a nice tan there or whatnot so let me know about that zuck's uh been killing it lately with a lot of stuff he's doing it could be the M MMA could the Jiu-Jitsu who knows but uh other than that guys if you're new here also check out the stride Community Link for that will be in the description Down Below guys it's uh about 750 to 800

Members right now on our free Facebook group and Discord channel so it's 100% free you can like uh network with like-minded individuals AI enthusiasts entrepreneurs and myself so check that out it's 100% free and then also too guys if you haven't already checked out my book client acquisition code showing you exactly how to leverage AI to get clients book appointments for high ticket uh business offers you know 5K 10K 20K 30k whatever your offer is if you run a business um

I go over a lot of stuff in this book it's about 500 600 Pages you had to do paid advertising give you exact there's a whole chapter on prompting and prompt templates and everything so extremely value packed uh link for that will be in the description clientacquisition code.com other that guys I will see you in tomorrow's video keep hustling keep grinding and of course guys accelerate your stride take care