FREE: AI Agent Controls Your Mouse & Computer! Open Interpreter OS Mode (Screenshots)🤖 Open Source

What's going on everybody Josh pokok here and in today's video we are going to be taking another look at open interpreter looking at some of the tests and best use cases you can actually use this tool for as well as we're going to be testing out open interpreter OS open interpreter operating system basically allowing it to completely control your computer with AI that's right you can do pretty much anything let's dive right into it [Music] all right guys so I did

A video yesterday I'll link it down below basically just an introduction to open interpreter and we'll go over some brief things talking about how to set it up very quickly very easy to do and then we're going to kind of test and push it to some of its limits and see how it actually does okay because yesterday I mentioned that you know there is an open interpreter OS mode and a lot of you were asking to actually do a video on

It so that's what we're going to do here today and yeah so yesterday just to kind of sum up for people who didn't see that video um it was we basically tested open interpreter and it worked fairly well um and better than a lot of the other agent models that I have personally uh used in the past so we're going to go ahead and use it as well so there is so this is kind of cool they are uh at least

Like very soon going to be releasing a desktop app um there is Early Access I don't have it at the moment I did apply for it but anyways so the link for the GitHub repo as well as the docs will be down below open interpreter has been around for quite a while now um and they do have let's see 52k stars on GitHub so pretty established project and very simple to install all you all you literally do is just run uh

The command pip install open interpreter in your terminal if I would go to my terminal here I would just run that command like so boom which I already I'm not going to run just cuz I already have it installed but once you run that it would install it okay once you do that all you do is just simply run interpreter all right yes literally just two commands and you can get up and running now within the doc section um I'll go

Over some brief things for you guys so you guys can kind of know where to look for what you want to do with open interpreter because there's a lot of stuff you can do with it so here um it talks about the ways to install it or set it up so if you want to do local mode um you can run this command we're going to run this command shortly OS mode um and then it explains what OS mode is essentially

Allowing you to I'll quickly read it OS mode is highly experimental mode that allows open interpreter to control uh the operating system visually through the mouse and keyboard it provides a multimodo llm like gp4 Vision with the necessary tools to capture screenshots of the display and interact with onscreen elements such as text and icons it will try to use the most direct method to achieve the goal like using Spotlight on Mac to open applications and using query parameters in the URL

To open websites with additional information OS mode is a work in progress if you have any suggestions or experience issues please reach out to us via Discord Etc so to enable it you just literally run the uh interpreter uh prompt with the OS flag here and we'll do that in just a moment and um also to yeah so it is experimental I think it's going to get a lot better over time and you know open interpreter probably won't definitely won't be

The only ones doing stuff like this eventually we'll probably just be able to talk in natural language to your computer and it will probably do your work for you right so that's where we're pretty much headed uh you can talk code out an application talk to it just do whatever so um you know set up a marketing campaign all that good stuff so that's definitely the direction we're going you won't have to probably use a mouse or use a keyboard then

Eventually it'll just read your mind right so anyways so I have a few different tasks I want to try with open interpreter as well before we try the OS mode and I personally haven't fully done multiple different tests yet using the OS mode so you guys will get to see my reaction and you'll get to see this all play out live now from what I believe my thesis before trying is that I think there's going to be certain tasks that would

Definitely be better without OS mode like it says in the docs it is an experimental experimental thing and I don't know if it's fully going to be there yet in terms of something that you would want to use on a day-to-day basis now we can go ahead and try this and see but um the reason yeah I don't know how it's going to actually perform so I'm going to try and for those of you who don't know of interpreter open interpreter

As well we're going to run some basic normal open interpreter um tasks right now just so you can kind of get the sense of the beauty of open interpreter and then we're going to run uh OS mode and see how it actually uh performs okay so I'm running the normal interpreter uh prompt first without the OS flag and then we're going to run the OS one afterwards okay so here is the first task in my blank or folder location there are

Images and random names rename them all uh JPEG and PNG files to follow the format photo um date where the date is the file creation date and XXX is the sequential is a sequential number ensured to maintain the original file extension all right so you'll see here if I go to photos right here this is the folder location as you can see they all have random names let's go ahead and run this okay so it always starts by creating a plan

Which I do like it's going to list all the files in the directory um so we're just going to say yes okay so it's listed all the files now it's going to uh and determined the their creation dates now we'll proceed with renaming the files accordingly to the specific format okay so it just renamed them now we're going to run this okay so now it just renamed them as you can see instantly it just renamed bulk all these files with photo

The data was completed or created and that's pretty cool okay so that was actually a lot quicker than I thought and a lot easier all right so the next prompt is I have a CSV file named dummy transactions data.csv in my folder right here continuing my bank transactions for the past year analyze this data to show my spending patterns by category IDE uh identify my highest expense categories and visualize my expense versus uh income versus expense over time create informative charts

And graphs to represent this data and save them all as image files also suggest areas where I could potentially uh uh save money based on the analysis let's go ahead and run this okay so it's going to run the code to actually get the data right now okay so now it's going to do the analysis we're going to say yes all right so it said the spending patterns by category chart has been successfully created and saved as spending patterns by category

Let's take a look wow nice so we got a nice beautiful chart right here uh we see our expenses on the left and our income on the right and category on the left hand side dining entertainment groceries miscellaneous rent salary transport utilities and then the total amount here so pretty cool stuff that it wrote this whole python script without me doing any code now it is going to visualize the income and expenses over time okay so now here is the income

Verse expenses over time wow so a nice line graph right here now it's going to suggest areas to save money all right so the summary of the highest expenses category right here suggestions for potential savings right here and then next steps um would you like me to create a summary report with these suggestions for further analysis or any other specific areas I'm just going to continue on to the next task all right now let's go ahead and get to the part

I'm going to run interpreter with the OS flag here okay we're going to start with something pretty simple like it says right here in this mode open interpal will not require approval before performing action so be ready to close this terminal so if you're running something that could potentially run you know commands that could do something to your system I wouldn't necessarily suggest that and definitely be careful regardless um because yeah it doesn't ask for your approval so we're going to

Start with something simple we're just going to say open Spotify app okay so it just opened up the Spotify app and we can see right here there was no error but the output seems empty let's verify whether Spotify app is open or not I'll take a look and confirm on the screen so you can see here computer display view it just took a screenshot and yeah so it will try a couple different ways as you can see it tried um a

First way initially and it didn't work and then it tried a second way and then it worked and now we can see the screenshot confirms that Spotify is successfully open and active your request to open the app is complete now you if you need any sense let me know so now we're going to try another task this time I'm going to say go to YouTube and play some relaxing music all right so this has a little bit more steps involved opposed

From just opening up an app let's see how it handles this okay so it just opened up Chrome here and it did a search on YouTube for relaxing music let's see if it's actually going to pick pick one okay so it seems like it's kind of get gone caught up in some sort of loop or glitch right now where the screenshot hasn't been working so it's been consistently opening relaxing music again and again in a new tab all right I'm having

A bit of issues with that one I think you need to be very specific with your prompts and you're going to probably encounter issues cuz this definitely isn't perfect but let's go ahead and kind of be a bit more specific go to Google and search the weather today in Tonto you know I may even want to say open it up in Google Chrome or Microsoft Edge or whatever the case may be but let's see how it does with this okay so

Computer browser search weather today in Toronto okay so this one's actually kind of cool it actually basically got the weather today uh I guess it may have done a browser search I so it didn't open it up in the computer in the Chrome um you know maybe I should have said open it up in Chrome but it did get the weather today for September 12th um to be mostly sunny with blah blah blah blah so let's go ahead and run that

Same task and let's say well we did say go to Google so but now I'm specifying search this in Google Chrome okay so now it's in Google Chrome at least and it did search the weather for uh Toronto and it should get it with the screenshot which is the same thing that it just said before okay so it looks like I've been encountering a bug in the last couple uh times here I don't know if this is okay now it's kind

Of annoying so at least for me you may encounter some bugs here it looks like I believe that it's having some issues with the screenshotting now and viewing it I don't know um we'll try one more but um yeah like I said guys this is definitely not a perfect tool like they said on their in their docks it's in beta if you're going to actually use this for you know day-to-day tasks I wouldn't probably as of now recommend the OS mode

I would definitely recommend open interpreter either within the chat mode uh using it in your like python projects or what not um but this is I think a glimpse into where we're headed with AI where it's going to be with AI agents where they can control your computer where you're going to be able to talk to them or they're going to be a companion to you you know they're going to uh you know you'll have Sarah you'll have Susie you'll have

Jim whatever the case may be this this friend this agent that's going to be able to help you with your daytoday your work your you know uh personal life whatever the case may be um you'll be able to tell them things and they're going to be able to do them on your behalf and right now even with this like it is cool what they're able to do on a small scale with like opening things controlling things on your computer um it

Is very slow at the moment uh with OS mode with normal mode it's it's pretty good but um yeah I think this is just a glimpse into where we're headed in the next 6 to 12 months I'd say 6 to 24 months tops all right here we're going to say create a new uh directory called test and open it and cursor IDE I don't know how it's going to do with cursor um maybe easier to say vs code since it's more

Maybe known for it but let's see how it does okay so first things first it did make the test folder so that is good wow okay so it did open it in cursor IDE that's awesome and it looks like the computer vision is actually working cuz it is taking the screenshots so that's good I guess maybe I didn't have a bug um but yeah I think you know you really I think if you play around with it for a bit longer

It actually is pretty useful that it's doing okay what's going on now so this is the thing sometimes it starts going like this and I just close it just cuz I don't when I start seeing it run a bunch of commands or do whatever you never really know right I know it's probably not doing anything harmful but um yeah so it's not perfect by any means but I think this like I said it's a step in the direction where we're headed

And if you want to stay up to date with everything along the lines of AI automation how you can leverage this in your business and your personal life in whatever way possible to benefit and grow every single day that's what we're all about on this channel we upload videos on AI automation business growth Marketing sales Etc so if you have any other ideas of tools or Frameworks or suggestions that you want me to do a video on and topics you want

Me to cover let me know in the comments Down Below guys let me know if you've tried open interpreter what type of use cases and things you've been able to get it to do have you tried OS mode have you had more success with OS mode than me maybe I'm doing something wrong or maybe you have some tips tricks whatever let me know in the comments Down Below guys other than that hope you enjoy this video check out our free community

Stri community.com our free Facebook group free Discord Channel Link in the description down below other than that guys I will see you in tomorrow's video keep hustling keep grinding and of course guys accelerate your stride take care