NEW: Deepseek v2.5 Best Open-Source AI Coder vs OpenAI o1, o1-mini, & Claude Sonnet 3.5 [TESTED]

What's going on everybody Josh pook here and lately we've had some recent releases from open AI with 01 01 mini and we've even now had some releases from some of the open-source models and in today's video we're going to be looking at Deep seek version 2.5 seeing how some of its coding capabilities are with these new updates and seeing how it even compares to tools like 01 and Claw sonit 3.5 is open source catching up let's dive right into it [Music]

All right guys so if you're not familiar with deep seek I've done a video on deep seek V2 before but now there's version 2.5 and this is a Chinese company and uh they have some really it's one of the best open source uh you know AIS for coding there's quen 2.5 there's deeps 2.5 there's other ones too but but it's pretty cost effective and pretty good so as you can see here deep tweeted out exciting news we've officially launched deep seek

Version 2.5 a powerful combination of deep seek V2 and deep seek coder V2 so the previous models now with enhanced writing instruction following and human preference alignment it's available on web and API enjoy seamless call function calling fim Json output allinone note due to significant updates in this version if performance drops in certain cases we recommend adjusting the system prompt and temperature settings for best results you can see here little diagram showing the two base models uh instruct model and coder

Model right here and you see that deep seek version 2.5 is essentially a combination to get the best of that you can see they posted a graph right here showing that it outperforms deeps V2 and deeps V2 coder on most benchmarks and they also posted their internal Chinese evaluations deeps v2.5 shows a significant Improvement in win rates against GPT 40 mini and for o latest judged by GPT 40 compared to deeps version 2 so we can see the comparisons right here

Versus V2 versus GPT 40 and then v2.5 versus GPT 40 now interestingly enough AER uh leaderboard benchmarks show that deeps version 2.5 actually did a little bit worse than the coder for v2 in their benchmark test um but at the same time different people's benchmarks are different and I'm going to assume that deep seek probably did more extensive testing on their own models than just maybe AER did here in this test but I can't fully say we're going to see how

It actually stands up in our test today right here's someone tweeting out deep seek 2.5 versus GPT 4 for coding 21 times cheaper than Claw at 3.5 and 18 times cheaper than GPT 40 and we can see this human eval score got a 89 Claude got 92 and gpg 40 got uh 90 okay so the pricing is 14 cents per million input tokens and 28 cents per million output tokens so you can go to deep seek.com and you can get access

To their API you can add some credits and you can build within the API to use deep seek it's also open AI compatible for base URLs now you can see their benchmarks right here now I'm not going to go through them all right now cuz we're going to dive into this testing here but you want to pause this or go check them out yourself links will be down below and lastly before we dive in you can actually get access to deep

SE version 2.5 either on hugging face or you can also get access to it on olama I'll leave links to them below as well but today we're going to be accessing it in the chat interface so as you can see here in the chat interface and we can either use version 2.5 or the coder version 2.5 so we'll be testing these out now all right so here is our testing questions and I did a video not too long ago actually comparing

01 mini and 01 preview versus Claude Sonet so we're going to use these same questions and see how deep seek version 2.5 compares to these top Frontier models on these questions so let's get with the first one going all right so the first one was a simple one and I expected it to get it right it is what is the capital of Canada and it did with that being Ottawa next is write a python function to generate the Fibonacci sequence up

To the 10th number all right so it's generating this and it's pretty good for Speed Wise It's not super fast but you can see here they also got a new feature that is Sim ilar to something like Claude Sonet artifacts and um depending on the code you can actually get somewhat of an artifact when it's generating the code and actually get some sort of visual all right so I do like the output here one thing I like the the chat UI

And everything like that but it did a good explanation right here as well and it's telling us what the output should be so let's go ahead and test this code all right so I'm going to run Python f.p and boom it got it correct all right so that's two for two next prompt is generate a b script to create 10 files with the names a1. txt to a1. txt all right so I just ran it and it worked and you can

See that it's actually the exact same script essentially as the one that 01 and Claude Sonet 3.5 created they all actually created the same one so it's a fairly simple script so that makes sense so that's three for three now let's get into the fourth question so if you multiply the number of days in a week by the number of continents what do you get so very basic math question and it got it right right another math question solve for x

3x + 7 = 22 and the answer is five so I got that correct too so it's five for five next question is a factory produces widgets at a rate that doubles every 3 days if it produces 100 widgets on day one how many widgets will it produce on day 19 explain your reasoning step by step all right and I got that correct to at 6,400 widgets on day 19 next we're going to ask it to generate a SVG of a

Tree so as you can see here we have this run HTML button and if you run it you can actually see that this looks like a tree so I really like this new feature and it definitely did pass this okay next is a question that neither of 01 nor Claud Sonic 3.5 got correct so I'm don't have my hopes too high for this but generate 13 sentences that have the word that have 10 words and end in the word monkey okay

So it got it wrong too I even gave it a second chance and it got it wrong again so that question's a fail next is generate me the pong game using python right so we got the code here and I'm going to run it awesome so we got the game of pong here and now one thing I really like about this generation now now I'm not 100% sure if the other models did do this or not but I noticed with deep

Seek it actually explains to us how to play so left paddle is w and S to move up and down and the right paddle is up and down on the keyboards now I mentioned with the other test models that it's kind of annoying for pong since you can't only you can only move one but I don't if it did allow it to move two paddles now it looks like it's not bouncing off the pong oh okay it is sometimes okay now

It looks like it's working pretty well is actually better than some of the actual other models for generating the pawn game CU it has a score it's actually pretty good quality and this is the first time I've actually been able to play versus myself I don't know if I just missed those instructions on all the other times I've done tests but um or maybe it did have this this function but it just didn't explain it to me like deep seek I

Don't know but either way it definitely passes this test okay next question is pretty easy and it passed I an odd number take one uh take away one letter and I become even the number seven all right so the next question is a man stands on one side of the river his dog on the other the man calls his dog who immediately crosses the river without getting wet and without using a bridge or a boat how did the dog do it

Okay so I expected that actually may struggle with questions like these because mainly one I would say its strengths for being an open source model are really when it comes to the coding side of things and it's also built by a Chinese company so you know a lot of the data is it's trained on is probably Chinese language so you can see here it answered the dog crossed the river by swimming across the Riddle's warning is a little bit tricky as

It mentions dog crossing without getting wet but it doesn't spef ify that the dog has to stay dry to cross the dog could have swam across a swung across the river which would technically mean it got wet but it still crossed the river without using a bridge or a boat so that's wrong the the correct answer is the river was frozen which all these other models got correct but this is its second fail right so next question is create a landing

Page using CSS JS and HTML it should be a website for a beauty store that has a header Banner features testimonials and checkout section make it look professional and very modern all right and here is what we got so I mean pretty basic but technically a pass now the one thing I will say is that I did specify in the prompt use CSS JS and HTML so I did say JavaScript and it did give me JavaScript but it says add any

JS functionality here if needed but you know I guess it's technically not needed I think um like oh and preview gave me some js right here when it generated the code not nothing crazy but I will still give this a pass just because it did prettyy much do it and it didn't ignore the JavaScript it just really didn't give me anything so let say it's a pass all right so another question that it may not get right here you are in

A room that has three switches and a closed door the switches control three light bulbs on the other side of the door once you open the door you may never touch the switches again how can you definitely tell which switch is connected to each one of the light bulbs all right so it actually got it correct turn on the first switch and leave it on for a few minutes this will cause the corresponding light bulb to heat up turn off the

First switch and immediately turn on the second switch this will cause the corresponding light bulb to light up and then leave the third switch off open the door and observe the light bulbs the light bulb that is on is controlled by the second switch the light bulb that is off is hot uh off but this hot is controlled by the first switch and the light bu bulb that is off and cold is controlled by the third switch all right so another

One is I left my campsite and height South 3 miles then I turned East and height for 3 miles then I I turned North and hiked for 3 miles at which I came across a bear inside my tent eating my food what color was the bear and it got this question correct because the only place you can hike 3 miles south and east and north and end up in the same position is the North Pole meaning the bear would be white

And the last question that Claude actually got wrong in my test which was surprising but open AI models did get right is how many words are in your answer here and it says the answer to your question is just one word One incorrect okay so all in all it actually did pretty good it did better than I thought on some of these you know language or problem solving questions and it did good on these coding questions of course because that's where

Its strengths are so out of the 15 questions we tested it on today it got three of them wrong which comparative to 01 mini and 01 preview it only got one test wrong and for claudson it actually got two tests wrong so of course this isn't the most in-depth testing and I'll definitely continue trying to improve these tests over time for new models that come out but what I will say is that these open- Source models especially when it comes to

Coding are definitely getting uh really good improvements and it's going to be interesting to see as you know time goes on over the next year or so how these open source models will compare to Clos Source models is there going to be a Tipping Point where open source actually start surpassing close Source I don't think it's here yet but I will say that it is really useful sometimes to actually code using these open source models one because it's open source and

Two a lot of the time it's a way more cost effective than some of these Clos Source models and for certain things especially like autocomplete or some of these other coding tasks where you don't need such a extremely powerful model then it actually makes a lot more sense to use models like deep SE 2.5 but let me know what your thoughts are in the comments down below how do you feel about these open source models compared to these closed Source models

As well as if you are using open source models whether ATS in ama llm Studio what is your favorite open source model is it deep seek 2.5 is it quen 2.5 is it whatever the case may be let me know in the comments Down Below guys and if you're new here we upload videos every single day on AI automation business growth if you're into that type of content and you got some value from this video make sure to smash that like

Button and subscribe to stay up to date with the daily uploads also too guys check out our free stride Community Link for that will be in the description down below for our free Discord Channel free Facebook group you can join connect with myself as well as other like-minded individuals other than that guys I will see you in tomorrow's video keep hustling keep grinding and of course guys accelerate your stride take care