FREE GPT Crawler Scrape Any Website Into LLM Knowledge Base🤖 Custom AI Agents Cursor AI Coding Data

What's going on everybody Josh pokok here and in today's video we're going to be going over a tool called GPT crawler crawl a site to generate knowledge files to create your own custom GPT from one or multiple URLs now you could use this for a custom GPT or you could use this for knowledge files for something like cursor for really any sort of llm it's very important to scrape docks when you're doing coding or when you're building a project so we're

Going to show you how you can do that right now very simple let's Dive Right into [Music] all right guys so I came across GPT crawler and it is a pretty cool little tool and it's very easy to set up and get running now for those of you who aren't new to the channel we've covered uh different tools for when it comes to crawling such as fire crawl I covered crawl for AI and using that with swarm yesterday if you haven't

Seen that video check it out I'll leave a link down below um we could probably something well we could definitely set something like this up within some of those tools and if you want me to do a video regarding those tools showing you how to do something like this let me know and quite frankly I could probably make it a little bit more custom and maybe even a bit better for my specific use case but with that being said I still

Think this is a really great tool so we're going to Dive Right In and show you how to set it up now before I quickly do uh I will show you this blog right here I would definitely recommend checking it out if you're going to be using this tool and it's uh made by builder. I've like a few different stuff that has come from this company just in terms of like their content and their blog post and whatnot but uh here's

Just going over the importance of creating a custom GPT showing you how to set it up and just going a little bit more in detail and also talking about how you could integrate this to a custom open aai Assistant if you're using it within the uh open AI assistant and then you could have API access so you could use this to create a custom uh AI assistant using your API and then integrate that with your apps to have it trained on

Your data or different documents or whatever the case may be okay so first things first is we're going to clone the repo so you're going to run this Command right here get clone and then the URL okay and once you run this command you're going to CD into GPT crawler okay from that point on I personally run code dot which is going to open up the project in vs code right here as you can see I have it open and you're

Going to want to navigate to The Source directory right here and then to the the config.sys so you can go ahead and run that right here okay so to configure the config.sys any single URL on that page that has this in their uh the name so it's basically going to pull all the different docs and then here you can see that they got the dock uh the selector right here so if you went into inspect element and then got the specific

Selector um of the content that you want so it's not just pulling like the header or the footer or all that sort of stuff you would input that in the selector uh field right here and then you can select Max pages to crawl okay so for this example it's 50 and then you can change the output file name to whatever you want Json and it's going to Output that in Json format so there's also some additional options that you could potentially

Configure we're not going to do that right now but if you want to check that out you can check it out in the repo are you tired of pouring thousands of dollars into appointment Setters only to watch leads slip away imagine having a team of elite sales agents booking qualified appointments for you around the clock no more wasted time on training no more frustration with performance and no more draining your budget on inconsistent and expensive call centers introducing stride agents AI

Powered appointment centers that work 24/7 never get tired and book appointments while you sleep trained on thousands of successful conversations are AI agents outperform human teams at just oneth of the cost join the ranks of businesses that doubled their appointments and booking rates in just a matter of weeks don't get left behind in the AI Revolution visit stride agents.com now and transform your entire sales process with cutting letting Edge AI technology it's time to accelerate your stride with AI agents and

Then there's an alternative method to doing this with Docker we're not going to cover that and then here it talks about just uploading that data to open AI through the API or through the normal GPT so you could do either one or like I said before you could use this in something like cursor or really anything when it comes to training your llms on specific data okay so once you have that all set up so if we go to our config

Right here we're just using the example right here so you can see URLs to start the crawl if the URL is a s map it will crawl all the pages in The sitemap so we can see here we got the docks we got this all right so you could change this around of course we're just going to keep it basic and then you can see here that this is the pattern to match for all links on a page to subsequentially call

And you can see right here we got the docs one right here to crawl all the docs and then pattern to match against for links on a page to exclude from crawl crawling so you could go ahead and change this to and then the selector to grab the inner text from right here so we can see this example dot docs D Builder D- container of course you're going to want to input your own custom one and then don't crawl for more

Than 50 pages we'll just keep it like that and then we can see the output file uh is going to be output. Json okay so if we go ahead and we go to our terminal here and we just go npm start all right so you can see here playright crawler so the crawler has been started of course it's using playright as you can see here and we can see here crawling page 1 out of 50 builder. docs developer AB testing import

So we can see all the different docs following that same URL that we specified in the config file are being crawled for the maximum of 50 pages of course and cool thing about this of course is it's using playright you know we're not we don't have to put in any API key or it doesn't use up any tokens so it's essentially free to do this and then you can import it of course into uh your llm of choice cursor whatever the

Case may be and I also went over this deeply in our uh AI agent our first AI agent training course if you haven't seen that I'll leave a link down below it's a longer video but it's extremely valuable it took me 7 days to make that so I definitely check that out if I definitely recommend checking that out if you're into creating AI agents and all that good stuff but data is key when it comes to AI agents anything related to

Llms in general you know qual data is key so being able to extract data from different Tools in your Tex stack or different websites or information is crucial so I think this is a cool uh tool to do so okay so we can see that the crawling has been completed if we go to the output Json file that we specified we can see the different pages here so the title of the page the URL and then the HTML content this of

Course Scrolls to the to the right a lot it's not wrapping the text so it is a lot of text here I go ahead and paste this into a notepad you'll see right here that you know there is a bunch of different text from these these documents now of course if I was actually doing this and wanted to um you know make it a lot better I would um you know really specify the S uh the CSS selector and you could

Probably do stuff in regards to even cleaning this data or you could do this afterwards but um yeah you know I know you can do that with things like fire crawl or definitely with crawl for AI because I did that yesterday but I think this is a really quick lightweight tool and pretty cool um it really doesn't take much skill or you know much coding or anything like that to set up you can literally just run a few lines on the

Command line and then get this up and running and plug in any sort of website or document that you want and uh get a lot of good data for it so so just wanted to update you guys on this quick tool that I found and let me know what your thoughts are in the comments down below if you've used this tool let know if how you like it or not if you have any issues let me know and uh maybe use

A different tool maybe use fire uh fire crawl or crawl for AI or some other tool that I don't even haven't even talked about here let me know in the comments down below let me know which one is your favorite also too guys if you haven't seen the last two videos I did on AI agents and our new AI agent repo I'll leave those down below uh definitely recommend checking those out as well as checking out our stride AI agents repo

Um 100% for free literally giving you guys away a bunch of free value there so definitely recommend checking those out also too guys if you're new to the channel we upload videos every single day on AI automation business growth so you like that type of content make sure to smash this like button and subscribe to stay up to date with the daily uploads other than that guys if you haven't already joined our free community stri community.com our free Discord Channel and

Free Facebook group uh link for that will be in the description down below and then also too guys if you run a business and you need AI appointments that is booking you appointments in your calendar automatically while you sleep then check out stride agents.com other that guys I will see you in tomorrow's video keep hustling keep grinding and of course guys accelerate your stride take care