FREE: FireCrawl Scraper + OpenAI Realtime Voice API🤖🔥 Crawl4AI Alternative Open Source Cursor IDE

What's going on everybody Josh pokok here and in today's video we are going to be looking at how you can turn any website into llm ready data using fir crawl and we're also going to show you how you can connect this with opening eyes new realtime API voice tool so you can Leverage The Power of voice AI with fir crawl to scrape any single website you want let's dive right into it [Music] all right for those of you who saw my

Video a few days ago five days ago I did a video on crawl for AI with cursor and a lot of you guys really like that tool and I really like that tool as well um you can check it out check out the video I'll leave it link down below um it's a really good tool it's open source you can pretty much crawl any website you want and another really popular one is called fir crawl okay so this is fir crawl

Um Power your apps with clean data crawled from any website it's also open source so they do have an open it is kind of open like it's open source um but they do have a cloud version as well so you can try it out right here you can start for free with 500 credits if you're using Cloud version um SC crawl scrape clean trust by different companies integrate today node.js python um curl and then crawling Dynamic content to markdown reliability first

No caching built for AI different uh case studies and testimonials and then if you are going to pay for it here are the different options and a bit of FAQ here okay I'll leave a link to the docks as well so here's their docks it goes over their features which are crawl scrape map and llm extract there's different integration so you can integrate with Lang Chang llama index crew AI Diffy flow wise Lang flow and camel AI okay there's also contributing

Right here so open source vers Cloud running locally and self-hosting okay and I also made a document right here you guys can get access to this free document link for it will be down below and it basically gives you all the different resources around this tool fir crawl um so if you are planning on using it it's kind of like a central resource for all the different links you would probably want to see also a few different videos linked as well

As some information regarding self-hosting it now I personally was having some trouble self-hosting it um but I did link the docks as well as a few different things I tried so if you want to go ahead and do that you can go ahead and try it out and for the prerequisites for that you would need um there's different ways to self host it you could do it with Docker or um or not with Docker check out the self-hosting guide right here

If you want more information about that I also um leave some information regarding how far crawl compares to crawl for AI so if you want more information on that you can check that out some pros and cons of each um but yeah I'm not going to go over that we're just going to dive into setting crawl for AI up showing it in action and then actually using it with opening eyes real time API all right so far CW also has some

Sdks here python SDK node SDK rust and go they have a pretty cool blog with different examples such as scraping job boards using open AI reasoning models in your apps prompt caching bunch of different stuff definitely recommend checking out if you're going to be using this tool and then the API reference right here so it has different endpoints for scraping crawling uh getting the status cancelling the crawl and then mapping okay so if you sign up for the cloud version it's

Going to look like something like this there's a playground activity logs usage API keys and you can get your API key right here you would just copy this API key and we're going to be using it in just a second all right so now we're in cursor here and I have a basic fir crawl project and I'm going to show you how I actually got here and then we're going to show you this in action and how you can actually scrape

Content off a website all right now you can do a lot of different things of fire crawl you can crawl you can scrape you can set different U parameters up and whatnot like I said I'll reference the box down below so if you want to go more in depth you can definitely check that out um but here was my initial prompt okay and I did some iteration afterwards but please use fire crawl and I started off asking it to use crew

Aai now I ended up just not using crew AI because I was encountering some issues but anyways to create a scraper where I can give the input of a website and it will scrape it and save its contents to the file uh in a file that is named of the uh that is the name of the website create the. EnV the requirements file and the code read the QA uh creai file do and file craw documentation carefully to ensure you have

All the most recent date knowled uh upto-date knowledge as your current training set is out of date okay and if you've seen many of my other videos on cursor AI you'll know I always like to reference um documentation because lots of times llms have outdated models okay so this whole entire little project a small little project just very simple setup I'm going to leave a get Hub repo to this down below so you guys can get it 100% for free you

Can play around with it obviously improve upon it test it out whatever the case may be within that repo one there's going to be this fir crawl setup right here with the um the python files and everything but I'm also going to include um the crew I I'll leave the crew AI regardless just in case you want to try it out but I'm going to leave the fire crawl uh this is all their docks API um references features Etc so this

Will help you too if you're going to be using fir crawl within cursor so you can you see here how I just referenced all the different documentation that I thought would maybe be necessary or useful and yeah you can do so pretty much very simple um if you have any questions about that let me know in the comments down below but um that should help you out for sure to create more of a custom approach here from that point on it

Just gave me the output of what exactly I asked for told me in the instructions obviously just pip dashr install or- requirements.txt to install the requirements and then there were a few different issues that came about when you run it in the terminal sometimes you'll just get little errors like line whatever line this or line that so all I simply did was I just copied the terminal and just pasted it over into cursor composer and said here's an issue and then

Cursor would resolve it okay and this it took maybe a couple iterations just because um sometime I think it was referencing some outdated stuff in its training uh history even with referencing the docs but eventually it resolved everything and let's go ahead and give this a test all right so I'm going to run Python firor script.py and this is going to scrape a specific website URL you can use different uh API end points you can crawl different things you can do

Different stuff like I said check out the Docks but we're going to enter the URL of stride agents.com and it is going to start scraping it and if everything goes well we should see content save to stride agents. markdown right here so if we go to stride agents right here we can see this is literally the content of the website in markdown format all right 450 lines right here so this is good because um you know if you use cursor a

Lot or if you use any sort of llms markdown is one of the easiest things for it to read and if you're coding you're going to want to be feeding um you know markdown files to your llm to do things like feed it updated docks or do different stuff anything when it comes to training llms this is very very useful okay so that's kind of the purpose behind fir crawl so now you get a sense of how this works on a

Very simple scale in terms of scraping a uh simple uh URL let's see it integrated into the open aai realtime API one thing I also forgot to mention is you're going to of course want to enter in your open AI API key here as well as your fir craw API key okay so just to quickly go over one more time and like I said this is linked in the description down below for free you're going to get clone the repo right

Here this is the fir crawl repo we just went over that has all the docks and all the different um things to get you started and you're going to CD into that folder fir craw Das agent and then you're going to run code dot just to open up in vs code you're going to change the environment variables to include your open AI API key and Fir crawl Keys all right like so then you're going to run pip install d r requirements.txt

And then you're going to run python firor script.py you encounter any issues just rerun those issues into cursor and it should fix them or if you have any dependency errors just start up a uh a virtual environment all right then you're going to enter your website URL next is for the open AI realtime API times fir craw so you're going to get clone this repo right here okay once you do that you're going to a CD into this repo so you're

Literally just going to copy and paste these two commands then once you're uh in this repo what you probably want to do again is do code dot to open this in VSS code okay and once you have that you're going to run npmi okay this is going to initialize everything and install the dependencies then you're going to run npm start so this is actually going to start things up all right so this probably will look familiar to you if you've seeing

The news around realtime AI this is the realtime console from open AI um except this version has fir craw integrated into it okay so once you have that uh right here ran this will start up um the realtime console on Local Host 3000 okay then you're going to run npm run relay okay and just in case you're having any trouble all these instructions are right here too most of them in the repo as well so you can check that out if

You need help too but it's literally outlined right here you copy and paste it next is you're going to want to create a EnV in the repo B so you see I have the EnV right here and within the EnV you're going to want to add your open AI API key like so as well as just copy and paste this local relay server URL right here all right this is the one that we started up right here so you actually may

Want to run this after okay once you do that you're going to want to go to sour first Pages console do or console page. txx and you're going to want to search for fc- okay so all right so within the console page right here which I said is in Pages console page we're going to search for FC and then Dash okay I already I don't have that in here anymore because I removed mine but that's going to take you to the

Two in uh two references of the fir crawl API key and you really have two options you can either just hardcode your API key in there so just literally copy and paste your API key right where it says FC cuz that's the the just the placeholder or what you could do is just um you could uh basically make the whole line look like this so just paste this right over the whole entire line for the API key in those two instances

Make sure you do it in both and then you're just going to have to add react app fir craw API key into the environment variable location and then you could just put your actual API key there okay once you do that and I hope that makes sense if you have any questions about that let me know but it's literally outlined there so it's meant so you can literally just copy and paste those instructions and you should have this up and running

No problem once you do that let's go ahead and test this thing and see if it actually works okay so you can either use manual or vad right here we'll just use vad for this this is basically automatic and and we're going to connect are you tired of pouring thousands of dollars into appointment Setters only to watch leads slip away imagine having a team of elite sales agents booking qualified appointments for you around the clock no more wasted time on training

No more frustration with performance and no more draining your budget on inconsistent and expensive call centers introducing stride agents AI powered appointment centers that work 24/7 never get tired and book appointments while you sleep trained on thousands of successful conversations on our AI agents outperform human teams at just on10th of the cost join the ranks of businesses that doubled their appointments and booking rates in just a matter of weeks don't get left behind in the AI Revolution visit stride agents.com now

And transform your entire sales process with Cutting Edge AI technology it's time to accelerate your stride with AI agents hello there how can I assist you today hey please use the scrape data tool that you have to scrape info on Josh P I've got the information from josh.com it appears to be a no not the website use Google and use the scrape data for searching for Josh P I apologize for the confusion unfortunately I'm unable to use the scrape data tool

To search Google right now is there another way I can assist you not Josh pack Josh P C O K I see but I'm not able to directly search Google or scrape just use the scrape data like you just did to use Google and search for Josh pook I found some information on Josh peek he's the founder and CEO of executiv stride. comom a business coaching platform that has helped generate over 30,000 plus qualified appointments in the last 8 months you

Can find more about him on his LinkedIn Instagram and YouTube channel which have all the links in the description as well as his YouTube channel that's some info on Josh pook he's the founder and CEO of executive Str okay so as you can see it's I mean it works real time at least from what I'm using right here it's kind of annoying just in terms of I I guess for me because I guess my last name is a little complex for

It to spell so it was searching Josh peek and then um I think it's programmed or like almost like hardcoded or something to basically say that it can't do Google searches or it really can't I guess maybe but it's using the scrape data tool to do like a search like this so you see how I had to really sometimes tell it use the scrape data tool because that's what it's called the scrape data tool which is the fir crawl tool that's

Programmed into it so you can tell it to go to a website you can tell it to do you can kind of get around the whole Google thing um but yeah play around with it um this is real time console integrated with fir craw so I thought I'd add this in as well so you can check that out but that's pretty much it guys for this video there's the dock down below so check that out it literally has everything you need

For for getting this set up from what we went over in the video from from the basic starter repo for fir craw as well as the uh real time console setup here so you can try that out let me know what you think down below let me know what you guys think do you like fir crawl or do you like crawl for AI what's your favorite AI crawling tool me personally I mean I'm still trying them out more and we'll see

How it goes I do really like crawl for AI how it's just very easy to use for open source um and then fire crawl is good too they do have that cloud uh version though so you know really depends what you're what you like anyways guys so if you're new to the channel we upload videos every single day on AI automation AI coding Business Marketing sales Etc so if you got some value from this video and like that type of content

Make sure to smash that like button and smash subscribe to stay up to date with the daily uploads also too guys if you haven't already joined stri community.com our free Facebook group and Discord Channel link to that will be in the description down below and lastly if you do run a business and you need help booking appointments on autopilot let's say you run a call center you have appointment cters that you're actually paying humans like five 3 5 10K whatever it

Is a month to book appointments for you and you want to automate that at scale with AI appointment setting agents then make sure to check out stride agents.com link to that will be in the description down below other than that guys I will see you in tomorrow's video keep hustling keep grinding and of course guys accelerate your stride take care