NEW: OpenAI Canvas Feature Beats Claude Artifacts?!?🤖 OpenAI Dev Day Realtime API Speech-To-Speech!

What's going on everybody Josh pook here and in today's video we are going to be talking about open ai's new releases one being the new canvas feature that they just literally launched today and two being the new devday announcements that they've been talking about they launched two days ago we got real time API Vision fine-tuning prompt caching and model distillation the team has been absolutely on a Ru let's dive right into it [Music] all right guys so we're going to start

Off by talking about the new canvas feature that just dropped today um also we will be going over like I said the dev day announcements so we have realtime API speech to speech uh Vision in the fine tuning API and then prompt caching and model installation now uh you can see here Sam's you know been tweeting out out about all this he said check out the canvas in GP chat GPT and then he said you'll get AGI too don't worry and

And now it's live for 100% of uh plus users all right so you check out the canvas blog post here I'm not going to read through it all but we can see here it's a new way of working with chat GPT we can see the image right here and it's essentially looking like sort of like a Google doc within chat GPT so we're introducing canvas a new interface we're working with chat GPT on writing and coding projects that go beyond simple

Chat canvas opens in a separate window allowing you to use chat GPT to collaborate on a project this early beta introduces a new way of working together not just through conversation but by creating and refining ideas side by side canvas was built with GPT 40 and can be manually selected in the model picker while in beta starting today we're rolling it out for chbt users blah blah blah blah blah and we also plan on making it available to free users when

It's out of beta all right so better collaboration you can do suggest edits adjust length change reading level add final polish add emoji all right it also has a coding canvas so um this canvas feature is somewhat similar to something like claw artifacts now there are some I guess pros and cons with each we'll go over those in just a second but uh coding is an iterative process can be hard to to follow all the revisions into your code in chat

Canvas makes it easier to track and understand chat gbt's changes and we plan to continue improving transparency into these kind of edits the coding shortcuts include review code add logs add comments fix bugs port to language all right so basically it's saying that the model knows when to open a canvas and a few following core behaviors so triggering the canvas for writing and coding generate diverse content types making targeted edits rewriting documents providing inline critique all right so one thing that's

Kind of cool to note is that we use novel synthetic data generation techniques such as distilling outputs from gp01 preview to post-train the model for its core behaviors this approach allowed us to rapidly address writing quality and new user interactions all without relying on human generated data the key challenge was defining when to trigger canvas we taught the model to open a canvas for prompts like write me a blog post on the about the history of coffee beans while avoiding overt

Triggering for General Q&A tasks like help me cook a new recipe for dinner reading test we PRI prioritized improving uh correct triggers at the expense of correct non-t triggers reaching 83% to Baseline zero shot GPT 40 with prompted instructions here we can see the canvas decision B boundary trigger writing and coding okay so without Ado let's dive into the canvas okay so if you don't know how to access the canvas well if you have a plus uh account on chat GPT

You simply just come up here and you can access it right here GPT 40 with canvas so um like I said before like it says in the blog this was basically trained with synthetic data um from 01 which is pretty cool um we're going to you start off with a prompt and I feel like this is a good prompt because um for those of you who you know maybe haven't been caught up with Dev day yet Dev day was 2 days

Ago and like I said at the start there were some pretty cool announcements but a post from me reading through each and and every blog post cuz there's about four different blogs they posted on from Dev day we're going to H get canvas to generate a um basically a summary of everything that happened on dev day so I'm going to go to each an individual blog and I'm basically just going to copy the text here okay so so my prompt here

Is I'm creating a YouTube video about open AI Dev day and need a one pager summary of all the key points here are the four different blog posts and then I pasted the four different blog posts okay so we're going to run this command here are you tired of pouring thousands of dollars into appointment Setters only to watch leads slip away imagine having a team of elite sales agents booking qualified appointments for you around the clock no more wasted time on

Training no more frustration with performance and no more draining your budget on inconsistent and expensive call centers introducing stride agents AI powered appointment centers that work 24/7 never get tired and book appointments while you sleep trained on thousands of successful conversations are AI agents outperform human teams at just one tenth of the cost join the ranks of businesses that doubled their appointments and booking rates in just a matter of weeks don't get left behind in the AI Revolution visit stride agents.com

Now and transform your entire sales process with Cutting Edge AI technology it's time to accelerate your with AI agents and boom as you can see the canvas has opened up and it's officially generating our response here so just to quickly go over what the UI looks like as you can see we got our chat bar on the left hand side so we can basically chat to it over here you'll see it's quite long right now here's a concise high quality summary

For your video on open AI Dev day announcements hope this summary works for your video let me know if you want some adjustments okay so then you can kind of click on it and uh if I close this out you'll see it's somewhat similar to like how a claw Sonet artifact looks like we can open it going to reopen the canvas here now on the bottom right here you'll see we have this little uh pencil icon that basically has a window

Here this window has different options such as ADD emojis add final polish reading level adjust the length and suggest edits so we'll do some of those in just a second here but first off let me kind of go through some of these um announcements here so we can kind of get a gist of what you know first off these what actually happened in Dev day so the realtime API for voice interaction open AI introduce real-time API enabling developers to create natural

Low latency multimodal voice applications this API supports six preset voices and allows seamless speech top speech experiences it eliminates the need for separate models for transcription reasoning and speech synthesis enhancing both speed and natural interaction the API will support advanced use cases like customer support and language learning with price based on text and audio tokens additionally an audio feature for the chat completions API will soon be available and the second thing is Vision find tuning in GPT 40 open AI now

Offers Vision find tuning allowing developers to improve GPT 40's image processing capabilities this upgrade supports applications like enhanced visual search object protection for smart cities and medical image analysis Vision fine tuning has shown real world success such as as improving mapping accuracy for grab and boosting automation success rates for automat fine tuning with both images and text enables developers to create tailored models with better accuracy and vision style consistency and then three prompt caching so prompt caching allows developers to reuse

Recent input across multiple API calls reducing latency and cost by 50% this feature helps developers save on API usage especially in scenarios where prompts are repeated such as in chatbot conversations or code edits prompt caching applies automatically for prompts longer than 1,200 or 1,24 tokens and is available on the latest version of gp2 40 and GPT 40 mini so we've seen this in other models too Claud or Claude um other models as well and then four model distillation so open AI

Introduced a new model distillation Suite to simplify the process of fine-tuning smaller cost efficient models using outputs of more advanced models like GPT 40 this Suite includes tools like stored completions for generating distillation data sets and EV vales for measuring model performance developers can now fine-tune smaller models to perform similarly to large models at a fraction of the cost model distillation is available now with free training tokens offered until October 31st to encourage developers to experiment so pretty cool stuff now

You will see if we hover over each point we kind of have like our paragraph and we can see that we could go ahead and click this plus button right here and then we could tell it to either edit or explain this so let's say if we wanted to say make this shorter more actually no let's say make this longer and more inde depth okay so oppos from it redoing the whole entire output and having to redo like the bottom too

Um we're actually just editing like how you would in like a Google doc one specific part of the document so we just actually edited only the first point as you can see the three four three other points are the exact same still um here it goes on to explain this more in depth all right so we can also click here and we could do suggest edits so now canvas is basically going through this reading this document and saying I've added comments

And suggestions to help improve the writing quality by breaking down long sentences providing additional context and making the uh text more concise and reader friendly let me know if you need further assistance so we can see these comments here consider breaking this sentence into two for clarity and make it easier to read we can go ahead and click apply and it's going to do this for us so it's kind of like cursor right now for um for like actual writing as

We're seeing right here I could go um here you might want to elaborate briefly on what a websocket connection is for readers who are not familiar with it so we can go ahead and click apply so really useful if you're a writer if you're doing blog posts really anything I mean if you're if you're a business owner you should be copyrighting copywriting is a very valuable asset so this is definitely very useful for anything in regards to that now if we

Come over here you can see here we have adjust length so we can scroll and make this like shorter for example all right as you can see it just made the whole thing a lot shorter now we could go here we could go reading level and say we want to make it a little bit easier to read so we have kindergarten Middle School keep the current reading level high school college graduate maybe we want to do graduate you know we want

To sound really intelligent okay so as you can see changed the languaging a bit here um we can go add final polish here it says I've added final polish including section titles and improve formatting for readability gramar mechanics have also been reviewed for consistency and Clarity let me know if there are any other adjustments you'd like and let's just say we want to have fun and we want to add some emojis and we can just click add emojis here and when

You look at that our whole entire document now is just popular with random emojis well not random emojis relevant emojis pretty much everywhere now also too I do want to note you can go ahead and start like actually using this as a Google Document and typing out your own stuff so let's say you wanted to use some AI you want to use your own writing you could go ahead and do that and actually create like a have a pretty good canvas

Or something like a blog post here now up in the top right you can do previous version you can do next version so you can undo your history which is good sort of like you can doing something like cursor you can copy the text here all right so that is our first canvas now let's say I wanted to do a second one more relating to coding so I said please build a snake game in Python so now it's building the snake

Game in Python here all right so you'll see just off the get-go we have a lot nicer of a UI for a code that's generated from chat GPT than the default one so like we can even you know expand and contract different uh you know elements right here so similar to how you would be able to in an actual IDE now for the code feature we actually have a few different things so we have ADD comments we have ADD logs we

Have fix bugs and we have port to language and code review let's say I want to do add comments okay so this is useful it's adding comments so let's say maybe even if you're learning to code or if you just really don't know uh or you for whatever reason I'd say it's good to add comments because even if you're you already understand the code um but you're coding with an L M and you it's actually good to have comments because it

Will actually help the llm know what the code is doing and be able to reference it a lot easier so it will help you it will help the llm so that feature is really good okay next we can add logs so let's say I want to add logs here okay so now it's added some print statements throughout the code to help with debugging and understanding the flow of the program okay we could go ahead and click on fix bugs I've reviewed

And fixed the bugs in the code including ensuring proper direction handling to prevent the snake from reversing into itself let me know if you need any further explanations all right just to test things out I'm going to go ahead and run this code so pythons snake. py okay and here we have our snake game okay so this is actually pretty good we're going to go see oops let me see if I can get this okay first of all I would want

To make the snake a better color hey not bad though so you could highlight you know different pieces of code in your uh code here and you could click on ask chat GPT all right so then you could either say edit or explain this code all right you can also click on code review like I just did here all right so now it's giving suggestions for our code so generating uh generating food coordinates could be extracted into separate functions to avoid

Code repetition and improve maintainability the game Clos Loop could be refactored into a separate function to improve code modularity and readability consider using more descriptive variables such as snake Direction X including instead of X1 change to improve readability consider renaming snake list to snake SE segments for better Clarity and length of snake to snake length all right we could also Port this to a language so on here we can see keep current we could see python C++ PHP typescript Java right

So we have a bunch of different options JavaScript let go ahead and try typescript okay so I converted the whole um snake game to typ typescript here as we can see and I just asked it okay how would I run this and it's giving us instructions to actually run this snake game in typescript make snake game directory npm in it install uh P5 install typescript all right so all all the different instructions on how to run it so you can interact

With this code and uh it's pretty useful I find I think so far just off using this off the GetGo um I think this is definitely a big step for open ai's coding capabilities I mean it's been decent at coding in terms of the models but just in terms of the interface uh definitely Claude one definitely Claude has really good models uh like Claude Sonet it's really good regardless but uh the Claude artifact if you're using the not through the if

You're not using curser not using through the API just in general with the claw artifact it's a lot nicer of a Dev experience so I think this is getting close to be on par or maybe even better it's definitely better in some ways than Claud Sonic the one thing Claud son it I'd say has is just the the capability of really being able to run it within the browser um other than that though uh canvas does have seems like it has

More features and I really am liking the experience of it when it comes to De uh coding as well as writing I really like how they differentiate between the writing features and the coding features I think this just makes this type of tool really really good and really useful for whatever you're doing so other than that guys let me know what your thoughts are on canvas you think it beats claw artifacts and and also too what do you think of opening

Eyes Dev day announcement and let me know if you've been using canvas what kind of cool things you've been building or using it for or different use cases you found with it and maybe if you found things you don't like about it let me know in the comments below why let me know what some of the improvements you would like to see on it are other than that guys that's pretty much it for this video I just wanted to give you

A quick overview of the new canvas feature as well as the new Dev day features that were announced so go check those out if you haven't already guys and if you're new to the channel we upload videos every single day on ai ai coding automation business growth Marketing sales Etc so if you like this type of content and you got some video here definitely hit that like button and smash that subscribe button to stay up to date with the daily uploads

Thank you guys so much for the recent 7K subscribers I'm extremely grateful and also too guys if you haven't already joined our free Facebook group and free Discord Channel a link to our stri community.com will be in the description down below as well as all my other social platforms so if you want to connect with me you can check those out and if you run a business and you need AI appointment sets booking your leads on autopilot go check out stride

Agents.com I'll leave a link down below for that as well and if it looks like something that you'd be interested in book a call and we can see if it' be a good fit or not other than that guys I will see you in tomorrow's video keep hustling keep grinding and of course guys accelerate your stride take care