NEW: Qwen 2.5 BEST Open Source LLM?!🤖 [TESTED] Beats Llama 3.1 405B + Deepseek vs OpenAI o1 + Claude

What's going on guys Josh pokok here and quen 2.5 just released 13 new models and based on some of the evals we're seeing these open source models may just be some of the best open- source models that we have right now encoding math and general purpose today we're going to take a deep dive look do some tests and see if they are catching up to some of our Frontier models let's dive right into it [Music] those of you who haven't heard

About quen I've done videos on quen 2 it's owned by Alibaba uh Chinese company and they posted this a few day uh maybe about last week and they essentially came out with 13 new models for quen 2.5 and there is quen 2.5 0.5b 1.5b 3B 7B 14b 32b and 72b in the general purpose then there's Co quen 2.5 coder of 1.5b 7 7B and 32b on the way and then quen 2.5 math uh at 1.5b 7B and 72b so there's really

Three different blog posts on all these different models one for kind of each of those categories coding math general purpose so we're not going to go through them all some of them are a bit longer but I will leave links for all them down below but here's a quick breakdown of all 13 models we can see the different contacts lengths so 32k contacts 4K contacts for the math ones 128k context for some of them so different size contacts you can see

The different specs here and here we can see quen 2.5 comparative to other La uh other models such as mraw llama 3.1 uh 70b as well as 405b and we can see that on a lot of different ones um different uh you know evals that quen 2.5 7 2 be instruct is very close on par if not sometimes outperforming uh llama 3.1 405b instruct so the EV valves are looking really really nice here so we have different benchmarks like MML Pro

Uh GP QA math human eval all the different ones here and here all right and this is even pretty interesting they benchmarked their API base model quen plus against leading provide proprietary and open source model models including GPT 40 CLA sonit 3.5 llama 3.1 405b and then deep seek version 2.5 and I did a video on deep seek version 2.5 as well so you can see quen plus here and GPT 40 here and Claud Sonic here and it is pretty damn

Close in many different evals and some it's even ahead like this one M mlu Redux it's actually higher on their eval test that um than CL Al on at 3.5 by uh by 8 based on the eval alone you know I'm really excited to test out this model um we can see quen 2.5 32b 14b and quen turbo comparative to Gemma quen 57b uh GPT 40 mini we can see those evals here all right we can see some evals for quen

2.5 coder as well as quen 2.5 math here and like I mentioned I'll leave links to all these different blog posts as as well as the uh quen 2.5 coder specific blog post and the math specific blog post if you want to take a deeper look at these evals um there's the GitHub here hugging face model scope demo and Discord so um you can go to model scope here see these different models here you want to use these models on your

Own um system you could use something like LM studio right here and you could search when 2.5 check out some of these models right here we're going to be using it today on hugging face so if I go to hugging face here we can see we can access quen 2.5 the different um sizes right here 72b 32b and then quen math here and quen coder okay so all right so we're going to be using these models today to do some tests

Right here now some of these tests that I did I've done videos on comparative to some of the other models and I also added some additional questions here because some people mentioned that you know we should include some even harder questions which I um completely agree with to test some of these models now that these models are getting very very um powerful and strong so we'll go through some of these uh ones just to see how it compares to some of

The other models and we'll also definitely test out these questions as well and we'll see how quen does against you know qu deep seek 2.5 01 mini 01 preview claw son at 3.5 so let's dive into it we're going to start off with some of the easier questions and then get into the harder ones okay so based on the type of question I will use the most powerful quen uh model so if I a math question I'll use math coding question

I'll use coder and just general purpose I'll use general purpose for things maybe like reasoning or whatnot this is a very easy question what the capital Canada is and it got that cor correct of course next is going to be write a python function to generate the Fibonacci sequence up to the 10th number okay so I wrote this code here I ran the python um file and we got that answer correct and next question is generate a b script to create

10 files with the file names a1. txt through a1. txt and it got that correct because all the other ones got that correct and it they all gave the exact same code that's three for three okay next question is if you multiply the number of days in a week by the number of continents what do you get the correct answer is 49 so I got that right all right next question is solve for x 3x + 7 = 22 correct answer

Is five so I got that right all right next question is a factory produces a widget widgets at a rate that doubles every 3 days if it produces 100 widgets on day one how many widgets will produce on day 19 explain your reasoning step by step and the correct answer is 6400 100 so I got that right all right next question is generate an SVG of a tree and it got that correct okay next question is generate 13 sentences that have

10 words and end in the word monkey so it already got the it wrong because it just keeps generating nine uh and some of them are different so none of these have got this question right and LMS tend to mess up like with kind of questions around like that which I understand all right next question is generate me the pong game using python okay and it looks like we got the Pawn game here and it's playing pretty good not going to

Lie the scoreboard is working really nicely uh this is actually one of the better generations of pong that I've seen from these models so good job with that all right I got this answer correct of course I'm an odd number when I take away one letter I become even the answer is seven okay we're getting a little bit closer to some of the more difficult ones so this question is a man stands on one side of the river uh his dog

On the other the man calls his dog who immediately crosses the river with without getting wet and without using a bridge or a boat how did the dog do it and it got this correct saying the dog likely crossed a frozen uh with a river that was frozen and it got this one correct which um some of the frontier models got correct too well all of them but deep seek 2.5 actually got this incorrect so that's probably one of its most

Uh competitive uh open source or like one of its you know on par open source competitors the new deep seek 2.5 versus quen 2.5 so so it's interesting to see let's see how it does in some of these other questions next question is create a landing page using CSS JS and HTML it should be a website for a beauty store that has a header Banner feature page testimonials and uh checkout section make it look professional and very modern okay and here

We got the website here so it has a features testimonials checkout section and uh it doesn't look the best but at the end of the day hey it worked it generated the code so that will be a pass all right next question is you are in a room that has three switches and a closed door the switches control three lights uh light bulbs on the other side of the door once you open the door you may never touch the switches again

How can you definitely tell which switches connected to each of the light bulbs so label the switches a BC turn on two of the switches uh turn on switch A and B leave switch C off wait for a while for about 15 minutes this will allow the bulbs controlled by switches A and B to heat up if they are on turn off one switch after 15 minutes turn off switch B uh switch a remains on and switch C remains off open

The door and observe the light light bulbs to identify so the light bulb that is on is controlled by switch a light bulb that is off but warm is controlled by switch B and the light bulb that is off and cool is controlled by uh switch C so I got that correct all right next question is I left my campsite and Hike 3 miles south then I turned East and Hike 3 miles then I turned North and Hike 3 miles at

Which when I came across a bear and said my tent eating my food what color was the bear so basically it's saying that the only place in the world where you can hike 3 miles south east and north is the North Pole which means the bear would be white which is correct all right so next question is how many words are in your answer here and it says to accurately count the words in my answer I would need to provide an

An you uh provide an answer first however if you're asking me about the number of words in my response it contains 14 words if you have a a specific question or topic you'd like me to address feel free to let me know so this got this one wrong which uh deep seek also got wrong uh the only models that so far I've got this one correct even how simple it is is 01 and 01 preview 01 mini and 01 preview all

Right now we're going to get into a little bit more challenging questions question so in a room are two men only one uh one always lies and one always tells the truth you don't know which is which you may ask one question to one of the men to determine which door leads to freedom and which one leads to death which question do you ask okay so I got this one correct as well if I were to ask the other man which

Door leads to Freedom what would he say here's how it works one if you ask the truth teller the truth teller will tell you what the liar would say since the liar always lies he would point to the door that leads to death therefore the truth teller will tell you that the liar would point to the door that leads to death and then two if you ask the liar the liar will lie about what the truth teller would say the truth

Teller would point to the door that leads to Freedom since the liar lies he would tell you that the true teller will point to the door that leads to death in both cases the answer will get you to the door that leads to death so you should choose the opposite door from to find Freedom all right next question is you have 12 balls all identical in appearance one of them is either heavier or lighter than the rest but you don't know

Which is which using a balance scale how can you identify the odd ball and determine which is heavier or lighter in just uh three weighings okay so it says step one first weighing divide the 12 balls into three groups of four balls each group a group b Group C weigh group a against Group B on the balance scale possible outc comes if group a balances with Group B the OD Oddball is in group C if group a does not balance with

Group B the Oddball is in either group A or B second Wang group uh case one group a balances with Group B take four balls from Group C and divide them into two groups of two balls each group C and group group C1 and group C2 we group C1 against group C2 okay guys so I'm not going to read through this whole entire output here because it's pretty long um and it's pretty systematic but I verified that this is the correct

Answer you want to like scroll pause it and like scroll through as I'm scrolling here if you really want to read the answer you can but it would just probably take me a couple minutes just to Simply read this answer so uh it did get it correct though which is pretty impressive all right next question is you have a bag of five red marbles and three blue marbles you draw two marbles without repl placement what's the probability that both marbles are

Red and I got this answer correct the probability that both marbles drawn are red is 5 over 14 all right next question is you have three boxes labeled apples oranges and apples and oranges each box is labeled incorrectly you can only open one box and look inside how can you label all boxes correctly by only opening one box all right so it says to solve this problem you can follow the following steps choose the Box labeled apples and oranges say all

Boxes are labeled incorrectly the Box labeled apples and oranges cannot not contain both apples and oranges it must contain either apples or oranges so once you open the box label apples and oranges if you find Apples inside then the Box must be in the if you find an Apple inside then this box must be the apples box if you find an oranges inside then this box must be the oranges box then relabel the box so suppose you find an Apple relabel

This box as apples if you found an orange relabel this box oranges and then determine the context of the other boxes by um so the Box originally labeled apples canot contain apples since all labels are incorrect therefore it must be oranges and then vice versa for oranges and then relabel the remaining boxes so it got this one correct all right so the next question is find the derivative of the function x uh equal x to^ 3 subtract 3 x^ 2 +

2x subtract 1 and it got this answer correct which is right here our right next question is solve the systems uh system of equations uh 2x + y = 5 and then X subtract y = 1 and it got this correct with X being 2 and Y being 1 and next question is a car travels 100 m in 5 seconds what is the average speed in me/ second and kilometers per hour and it got this correct again with 20 m/s and

72 km per hour all right next is the heights of a group of students follow a normal distribution with a mean of 170 cm and a standard deviation of 10 cm what percentage of students are taller than 180 cm okay so it broke it down and it got this correct as well 15.87% which is correct all right so pretty much got all the math questions correct um now next is write a python one liner that creates a list of all numbers

From 1 to 100 that are divisible by seven okay so it gave me this code right here and I just ran it and we get 7 14 21 28 34 or 35 42 49 56 63 70 77 84 91 and 98 so that is correct all right so lastly is create a responsive grid layout with HTML and CSS that displays four items per row on large screens two items per row on medium screens and one item per row on small screens

All right so here we have it um we're running this code right now and we have four um columns right here on a large screen on a medium screen we have two and then on a small screen we have one so it pretty much aced that test as well so just a summary here is what the results look like pretty much all the coding and math questions it got right it got this question wrong and it got uh this question wrong

Right here but um yeah this I mean it's a really good model it's a really strong and powerful model especially when it comes to coding and uh and math and even like logical and uh reasoning questions it did really really well and it honestly did better than I expected I I thought it was going to do really well but I just thought it may not get every you know all these rights so um all in all I think it's a pretty

Uh crazy model I did get one I did a test um video on deep seek and I know a lot of people were kind of like interested in seeing deep seek and just even seeing the evals that they put out and some people didn't even hear about deep seek re v2.5 and they're like okay well I'll definitely keep a lookout for this so with that being said um I would definitely you know do this same thing with quen uh 2.5 because

Because they've always had very impressive models um with even quen 2 but now quen 2.5 is maybe one of the best open-source models right now that maybe better than llama 3.1 and all these other models like just even looking at their benchmarks um not even including like the test we did today it is really really powerful um so as you can see here I'll also leave a link to the AMA qu 2.5 you can see right here we got uh some

Of the different models right here 72b 32b 14b 3B 1.5b and uh 0.5b um so yeah all the links I talked about in today's video will be down below but let me know what your thoughts are guys have you tried out quen 2.5 yet have you tried out deep seek 2.5 what's your favorite open source model do you think quen's better than than olama or then llama 3.1 or 3.2 or deep seek 2.5 and do you think that quen 2.5 or

Just really um any I mean open source in general is going to catch up to the frontier closed Source models like uh Claw on at 3.5 or the next Mo uh iteration of that or chat GPT o Open AI 01 or whatever the case may be so how long do you think it will be until open source uh you know genuinely catches up if not surpass is these models let me know comments Down Below guys also too guys we upload videos

Every single day so if you're new to the channel and you want to stay up to date with uh videos on AI business growth automation uh all that cool stuff anytime I'm learning something I'll be sharing with you guys and trying to provide as much value as possible when it comes to ai ai coding cursor whatever the case may be drop a like And subscribe to the channel to stay up to date for that and also too guys if you haven't

Checked out our free community our free Facebook group free Discord Channel go to stri community.com link for that will be in the description down below other than that guys let me know what your thoughts are on this I'll see you in tomorrow's video keep hustling keep grinding and of course guys accelerate your stride take care