NEW: Crawl4AI + Cursor AI Scrape Any Website🤖🕸 Best LLM Web Scraper - Scrape API Documents AI Coding

What's going on everybody Josh poock here in today's video we are going to be going over crawl 4 AI I'm going to show you exactly what it is exactly how to set it up and how you can leverage it with cursor AI to build Crazy cool applications if you're not familiar with crawl for AI it essentially simplifies asynchronous web crawling and data extraction making it accessible for large language models and AI applications let's dive right into it [Music] all right guys

So we recently had a a few updates lately with crawl 4 AI I actually haven't talked about this um tool on this channel yet it just came out with a few recent releases and this is the async version so we'll quickly go over what crawl 4 AI is talk about maybe some of the potential use cases and then I'm going to show you how you can set it up in uh a very very simple way so even if you've never coded

In Python before even if you've never done anything that involves web scraping before you'll be able to follow along and get it done they have documentation here all links will be of course down below few features here so it's great a great tool because it's completely free and open source blazing fast performance out performing many paid Services llm friendly outputs so Json cleaned HTML and markdown these are all very uh AI like he said llm friendly outputs if if you want

To know how llms and AI can read um text very well it's through this type of format and then supports crawling multiple URLs simultaneously extracts and returns all media tags images audio and video extracts all external links extracts metadata from page custom Hooks and authentification headers and Page modifications before crawling user uh agent customization take screenshots of the page execute most multiple custom javascripts before crawling generate structured outputs without l M using Json CSS extraction strategy various chunking strategies so topic

Based reject sentence and more advanced extraction strategies cosign clustering llm and more CSS selector support for precise data extraction and then passes instruction keywords to refine extraction proxy support for enhanced privacy and access session management for complex multi-page crawling scenarios and then asynchronous architecture for improved performance and scalability so now we're going to get into the installation so so it offers flexible installation options depending on your needs um you can install as a python package or you could use Docker we're

Going to be using python in this video and there is a couple different ways you could install it using python just letting you know so there's the basic installation so you can literally just run pip install crawl 4 Ai and if you didn't already clue in by now a prerequisite is to have python installed so go ahead and get that installed if you don't have it and then by default this will install the asynchronous version of crawl 4 AI using playright

For web crawling and here's basically just saying if you encounter any issues with it not installing playright then you can do this manually and then if you want the synchronous version you can do so this way and then there's a development installation too for this way so then here goes the docker version I'm not going to cover that but if you want to use doer you can check out this guide right here um and then here are some different ways you

Can go ahead and use it now I'm not going to cover all these um you can literally check these out read the docks um you have quick start right here so import async IO from crawl 4 AI import async WebCrawler and then you can go ahead and see this one's a very simple one a quick start you literally just would change the url here you're printing the result as a markdown and that's pretty much it um there's Advanced use cases so

Executing JavaScript and CSS uh using CSS selectors right here using a proxy right here uh extracting structured data without l M right here and then extracting structured data with open AI right here and then session management and dynamic crawling uh content crawling right here all right and they're giving a speed comparison here too comparative to fire crawl and crawl for AI um on their different versions here and obviously I mean it could be a little bit biased if you don't know

About fir crawl um it's also a um crawling tool they have open source version as well um maybe I'll do a video covering them too but uh I really do like crawl for AI here um also too in the GitHub repo they do give you a a Google co-pilot that you could go ahead and um try if you just want to um test things out first and if you have any questions or if you want to do specific things I would

Definitely refer to their documentation okay now I'm going to show you how you can pretty much do this like I could easily just go ahead and like copy these instructions and change it around a bit and maybe do a little bit of crawl but I'm actually just going to do something that's a little bit easier in my opinion and um like I said at the start of this video if you've never done anything with python if you've never done anything with

Web crawling you'll be able to follow along and get something up and running so all I'm doing is copying this whole entire um explanation and the whole entire thing right here are you tired of pouring thousands of dollars into appointment Setters only to watch leads slip away imagine having a team of elite sales agents booking qualified appointments for you around the clock no more Wasted Time on training no more frustration with performance and no more draining your budget on inconsistent and

Expensive call centers introducing stride agents AI powered appointment sets at work 24/7 never get tired and book appointments while you sleep trained on thousands of successful conversations our AI agents outperform human teams at just oneth of the cost join the ranks of businesses that doubled their appointments and booking rates in just a matter of weeks don't get left behind in the AI Revolution visit stride agents.com now and transform your entire sales process with Cutting Edge AI technology it's time to accelerate

Your stride with AI agents okay and then I'm going over to our good old handy friend cursor and I'm going to go to the composer here you'll see I have a new project set up with literally nothing in here and I'm simply going to paste in the entire repo um not the entire repo but the main the main parts where it's talking about showing the different examples and whatnot and then at the end I'm just going to outline what want it

To build right this is fairly simple I mean anyone can do this right so here I'm just keeping it really simple I'm just saying let's set up an app to use crawl for AI to scrape programming documents in markdown and structured data so we can feed it to llms all right now this is definitely useful if you've seen my videos on cursor if you seen my videos on AI coding one of the main things that you want especially when you're dealing

With different Tech stack is you want to refer to the docks and you're going to want to feed the docs to llms reference the docs and certain things so this is definitely useful because if you're building any sort of project you're going to need to be doing working with the docs a lot when it comes to AI coding so we're going to run this and really just sit back and take a cup of coffee and let cursor do its thing okay

So it did its thing and we can see here that it's outlining what it did it just created a python file called Dog scraper um right here it's kind of explaining what it does and what we may need to do so it's using the open a a i LM so you'll see here okay I'm just going to accept this okay and then you know you could definitely obviously improve upon this a few things I'm going to do is one I'm going

To update this just to the new one right here okay and then I'm just going to tell it just for the sake of this video we're saying create a EnV and Link it to file for open AI just so I can plug in my open AI key and not have to blur it out okay great so I pasted in my API key and the EnV um you can see here the different um you know stuff we're importing so Json OS async

Iio um crawl for AI I'm not going to go through the whole entire file here if you really want to know more about what it's doing like if you're trying to learn obviously you could reference the docs but you could also just go into the chat here and say okay so you could add saying please add comments to the entire file explaining uh everything so I can understand it as a beginner okay so if you wanted like comments to understand what's

Going on here say if like I said if you're not if you're new to using python or crawl for AI this is a good strategy that you can use and uh I usually use like a markdown file that has like a little bit more instructions for doing this but this works as well so accept so now we can see exactly what's going on here um you know look running for asynchronous code you see the different um stuff here different comments um

Defining data model for programming documentation all right so then if you want to make some changes to this um then you you know you can reference exactly where in the document and you know exactly what code is doing what so you can change it very easily all right so if you wanted to change the provider of open AI down here instead of using GPT 3.5 turbo you could do so and there's the instructions right here so extract the main documentation content

Including the title main text code Snippets and related topics or links right you could obviously change that if you want okay so you're seeing we're saving the markdown content in a em uh MD file um save structure data tojson file so we're scraping both and then you'll see our URLs right here we're just going to use um one URL just for this okay we'll use two we'll use two all right we'll just use the Mozilla one and the python uh one

Right here and the output directory is scra docks it looks everything looks good obviously we can customize this if we want make it look a bit nicer you know do whatever we want to but I'm just going to go ahead and run this okay so python docor scraper dopy and we are running it and if everything goes well well first off let's take a sip of coffee here all right you'll see warming up a sync webs crawler all right it's ready

To crawl now it's crawled this successfully craw this boom um block index logs right here here we go here to our scrape docks we don't have anything yet okay so we had some trouble saving to the output here so I just literally just said hey didn't save to the output for some reason so it went ahead add some logging um and I think it changed the path here let's go ahead and accept this and try this again okay so python dock

Scraper all right so we got the output now and make sure that you tell um cursor where how you want it to save the output so make sure you say you want it to save the output in a specific file maybe with the name of the you know doc or the website for example or however you want to save it make sure you specify that so you're actually going to get that and then you can see here we got the markdown

File right here for the python uh URL right as you can see here if we scroll all the way down it's uh 900 just under 900 lines of uh code here or just of markdown text so I'm just showing you this on a small scale of course but obviously you know this type of Technology the possibilities are endless feel free to build upon this expand upon it make it a lot more complex do whatever you want with this but this should

Give you kind of the building blocks to be able to scrape content from the web very very easily and you can reference the docks of course and I really do suggest always referencing the docks and learning things for yourself and getting a good hang of the tools you're using but of course leverage the AI Tech such as cursor we have here so even if you're new even if you don't know how to do really that much or anything with python or

Scraping you can get started you can start leveraging this Tech right now today that's pretty much it for this video guys if you have any questions let me know in the comments down below if you want me to do a video on a specific type of tool or topic let me know in the comments Down Below guys if you're new to the channel we upload videos every single day on AI automation business growth AI coding bunch of different stuff so if

You like this type of content you got some value here make sure to smash that like button and subscribe to stay up to date with the daily uploads thank you guys so much for the recent 7,000 subscribers road to 10K on the way also too guys if you haven't already joined our free community stri community.com link to our free Facebook group and Discord channel will be in the description down below and also to check out stride agents.com if you want real

AI agents in your business that leverage AI voice AI texting Etc to replace different appointment CS call center reps Etc which you can deploy for one tenth of the cost that's if interest to you go to stri agents.com and book a call so we can see if it's a good fit or not other than that guys I will see you in tomorrow's video keep hustling keep grinding and of course guys accelerate your stride take care