AI Twitter Scraping Robot - Scrape Unlimited X/Twitter Tweets With Apify & Make.com 🤖 (Airtable)

What's going on guys in today's video I'm going to show you how to set up a very very simple easy beginner friendly make automation but yet at the same time very powerful we're going to be using air table we're going to be using our friendly scraper appify if you're not familiar with appify I've done videos on it before it is a great great tool where you create actors that can basically do everything you can do for scraping with code without having

To be a coder and I'm going to show you how you can scrape Twitter data scrape Twitter posts videos threads Etc anything you want from any profile in the world any of your competition anyone that you want to stay up to date with and you can have it all organized nice and neatly in your air table system all right so without Ado let's dive right into it [Music] okay guys so in the next few videos that I do around make um

You know we're going to build on top of them and maybe get a bit more advanced or like add on to some of these features because I've been working very hard on developing some really cool systems around this but for those of you who maybe are new to air table or sorry new to make make.com or just automations in general or scraping I want to give you guys something as well as just anyone even if you're Advanced you're still going to

Get a lot of value from this because um it's it's it's ju you're going to learn exactly how I do how I plug it into air table and what exact data you can get and you may not be familiar with this actor or you may not be familiar with appify we to dive through it all in this video so I'm just going to walk you through the um EXE the whole thing right now so let me just go here pull this

Up so this is kind of what the end result is going to look like this is not actually the end result um as you can like this is actually fairly new in terms of like it's going to be improved upon there's going to be multiple views over here filtered right now I was just actually just testing it to ensure that it worked so there's only like I actually only use it on one one uh Instagram users um profile so far well

I was testing it with others but in in here what you can see now with the the final data is only one users's profile uh information and this is actually um just a bit I was testing as you can see making sure I could pull both images and videos so you see I was doing it here multiple times and I'll show you exactly what Fields I have and all that good stuff so basically the workflow is going to be here you

Have a table with an air table this table can be anything I just call it Twitter and it's basically the Twitter profiles I could call this x right it really depends it's up to you whatever um you know whatever ever you fancy and then here I just have tweets all right so the workflow is pretty simple all this is going to do is let's say I wanted to add someone's twitter let's just say for example I want to add mine and

If you don't follow me on Twitter you can go check that out and it is Josh F pook right you can follow me on other social platforms if you want um you Instagram LinkedIn Etc Facebook but anyways all I'm going to be able to do is this so it's a very on thee-o fly let's just say I find a really good Twitter profile maybe I want to model some of the tweets maybe and keep in mind guys once you get this

Data I I'm I'm just showing you the first step I'm going to be doing more videos and I've done videos on this already check out some of my other make tutorials about La last last week or so and um I've covered this a bit but I'm going to build upon it even more eventually I plan on doing like very long in-depth videos showing you a complete build right but I we want to work up to that so anyways once you get

The data into your table base then you can create scenarios and automations to then either analyze the data so you can be like have a bunch of data from let's say your top competitors who are creating content let's just say or maybe you want to stay up to date for news who knows right there's a lot of different use cases then you want to anal like you can't read through all that content right I mean you could but that's very tedious

So then you want to have something that analyzes every single piece of content and maybe categorizes it into different content types like hey this is it a uh social proof post this is a um a trans a viral transformation post where you're showcasing how you want from before state to after state or different categories right and these are things like I have all mapped out and I have a specific system within this whole thing and other systems on basically doing this

As well as chat TPT prompts documents training Etc we I've covered some of that already but we'll get into more of that uh later on and let me know if there's anything specific you guys want to see and then anyways it analyzes it and then from that point on you can like really look at the data you can analyze that data and you can make calculated decisions as well as you can use AI of course to generate you know scripts or

Generate your own style of content you can plug in your tone of voice and your your own background story and then like you can do a lot of stuff with this I I'm just want to give you the kind of long-term Vision we're just going to be doing this in this video to grab the data and get it nicely organized but you can really get creative with this that's what I'm trying to get around here so all I simply do is

I have a status field right here not started start scraping in progress complete um and then I have some basic Twitter stuff here I'll just show you just so you kind of know description uh this is a follow so for the f um for a lot of the profile details as you see any lookup field right here and a lookup field is when it has this little icon here all the lookup Fields I'm going to show you they're all structur the

Same way they're a lookup field that's connected to the tweets from a linked record so the link record you just go here you link a record you click on tweets it's going to link the two records so you can see here all the tweets for this user are populated here um and then all the lookup Fields right here basically they're set up in a way where it's limit I take I turn this on to limit the number of items shown just

To be one all right if I don't do that for example I'll just show you um and I actually only have because I was testing this isn't like final I actually only have this like linked to like two I believe so now you see two photos we obviously don't want that I'm just going to undo this there we go anyways from this point on let's just show let me just make sure I have okay let me turn this on now also

Too guys I covered this in other videos but I want to let you guys know if I go to automations what you're going to do to actually send the data from air table to make.com there's a couple ways you could do it you could use uh make.com airtable you know thing right here and you could watch records and there's a bunch of people talking on YouTube on how to do that and that that's it works um and this method isn't like

Something I created or anything like this this is something I don't see as many people talking about that I personally just always do because I like it a lot more and it just has more it's easier you'll see what I mean in a second so all I do is I create an automation within the air table automations tab the trigger is when the Twitter table aka the Twitter profiles status is changed to start scraping once that is changed I'm going to

Run a script all right I'll leave this script in the description even though I've left it in many different videos but it's very simple this is your go-to web hook script anytime you're connecting air table to make this is a script you're going to use and all you're going to do is just change out this um this URL right here so I I actually just noticed I don't even have the base URL here so I mean I don't even need that

You can have it it's really not a big deal um but let me just say discard because I mean it is what it is but you don't you could have extra variables if you want here um and I have a video showing you like exactly what this is it's very simple actually but all you're going to do is like you could add different variables like input variable and you could pass the variables over but it actually you don't need to do

That all you need to do is pass over the record ID because once you pass over the record ID what we're doing um or what you could do I actually don't even do it here what am I doing here oh okay so this is actually good because I I I need to set this I need to change this and improve this so it's good that you guys are going to get to see exactly how you do this so by the way

Okay so well we haven't even got to this yet but this is the tool we're using appify you want this tool if you're doing any sort of scraping this is the best tool I've found pretty much and if you know any other tools that are like on par or like maybe even better let me know because I'll actually use them or do videos on them but this is like as of now this is like my goto vper what this does is

It has a store right here where you can like there's pre-made scrapers Etc you can get really creative with some of them you can even develop your own scraper if you want using like Cheerio and using um different uh languages but anyways what we're going to use is actually one of their most popular scrapers right here by API Dojo called tweet scraper V2 paper result it's actually good because it's a paper result some of these you have to actually like rent

Them additionally um so this one's good cuz it's p result it's very cheap it's 30 cents every th000 tweets all right so not super cheap but it's like pretty good I I actually like their app do API dojos I like their uh actors so not going to go through appify um I've talked about it in other videos but long story short I'm paying I believe 50 bucks a month yeah 49 bucks a month right here um there is a free uh

You can create a free account and get five bucks and then you can even go up here and create another another organization and it will give you a free five bucks uh each organization and you can like I think create one maybe every seven days or something I don't know if you like there's a limit but you can see I got a couple here um and yeah it's yeah I would suggest like if you're going to be using a lot just

Pay the 49 bucks as a credit based system um mine renews in 14 days and I'm only at 26 bucks so I'm going to have to do a lot of scraping in the next couple weeks which I guarantee I'll I guarantee I'll use those credits anyways um so what you basically do and this applies to any actor um so there are filters here you could play around with these um let's just see so I never usually touch run options really or

Advanced options um different actors have different filters query wizard we're not even going to worry about that filter tweets you could do only verified users only Twitter blue only image only video only quote right um and then Max tweets per query max number of items on output so these those are numbers you definitely want to touch and you could add conversation IDs or Twitter handles or Search terms or URLs so let's just say let's just say we want to do like

URLs so what I usually do is I just go to Json so I usually will like edit it within the you know nice user interface area and then when I go to Json I can really just copy this and it's very simple to edit I'll show you what I mean in a second so we are going to go here and when you're okay so let me just show you this from scratch I actually messed I didn't mess this up but this

Actually incomplete so this is good you're going to get to see all it is is a web hook that gets the data so you're going to create the web hook it's going to give you this link from that point on you're going to copy that link and all you're going to do within the air table script and you're going to do this every time you need to do an automation you're just going to change this URL and then you're going to

Click save finish editing all right the reason why we're only uh passing over the record ID is because I always do an air table get record and as you can probably guess the record that I get which is viral red content o and then I am going to go to Twitter and we are going to just get that record ID what is that going to do well what's that going to allow us instead of having to manually like put all the

Different variables we want to pass over on the air table side we can just grab everything just from one record ID also too if you're passing like long blocks of text you can't do that just by passing it through the air table side because it just it's just slow but when you do just a record ID and then look up the record ID with the make air table module you can get literally everything doesn't matter how long it is so and

It's instant so it's great so so yeah that and we'll call this just a rename I always like to do this um magnifying and we just call this like look look up record or get record right and oops I didn't mean to explain we can just Auto align next is the scrape so what you're going to do is you're just going to add a appify module and you're going to run an actor all right when you run the actor you're going

To like need to make the connection of course and you're just going to select your actor which is uh tweet creper we're going to delete this input Json we're going to okay so I I like how the uh sometimes I'll do this uh where is it I just like um pasting it here or like in a notepad and then pasting it here because then it maintains that that Json look so I can easily examine it so what I usually do here

Is include Search terms we don't need that so the numbers you're going to want to like verif uh you know have as variables are this this only image false only quote false no we don't need that sort by top I like that and then URL so this is going to be the Twitter URL so as you can see what I'm doing is we got the record ID from the web hook then we're getting that record ID into the get record and

Air table which is pulling all the data and then I'm using the Twitter URL from that data and I'm just putting it here so that means any Twitter URL we put will be used in our appify scenario right now what I'll also do too um what I like to do is this you'll see in a second so we go here I like to sometimes do this number okay and I'm just going to go number uh or one three and like um

I don't even know what to call this what do I call this uh result um max item sure I'll just call it Max items for now Max items so you could do this as a variable or you could just type it in if it's if you're going to like I I sometimes do it as a variable like you'll see here because you'll see why in a second so let's just say I wanted only to pull like 10 for example that's fine

And also too just so I can get the record ID and test this I'm just going to go like this I'm going to go formula and I'm going to go record ID like this going to add this just so I can test it sometimes I do this and you'll see what I mean in a second then I don't have to run an actual like thing I can just go copy this and like this and we can just right click and run

This module only only get the record ID here so now we're actually pulling over all the info this is all the filled in info as you can see so what I can do now is instead of having a hardcoded number in here which you could have like I could put this as 100 or a thousand right but I'm just going to put this as um this right here Max items you could make like a Max tweets per quer uh what's the

Difference between these two you may ask if you just hover over here it kind of explains one of them is basically like uh maximum number of items you that you want as an output and then this is maximum number of tweets to return per query right so there uh I think if you put one of them basically TW P query I I think I can just remove this um could be wrong if so I can fix that but I'm just going

To do this for now and technically I could probably remove all this other stuff but I like top for sure as we're getting the top tweets and that's pretty much it so there is a like you could run s uh uh Synchro synchronously all right so from that point on what I'm going to do is I'm just going to click okay we're going to save this and let's just say do I want to test it so no not yet I'm just

Going to so what this is going to run the actor from that point on so this is just so we can know Twitter scraper one out of two we're gonna save that this is our Twitter scraper two out of two it's very simple just two scenarios and what this is going to do is it's going to so we're going to go like this and you're adding an amplify and you're watching for actor runs what and you're going to add a new

Web hook and you're going to name it right here and then you're going to select that same actor so when the Tweet scraper is done running it will trigger this then what we're going to do is we're going to get that juicy data so all you really do is you just go to appify you click on get data set items right very simple from that point on right now I was actually just testing it so by the way if you ever

Need to test your data variables um I can show you how to do that you just go to your runs to your history and I could just click on something for example like this and then I could go to storage and then there I have my data set ID if you ever lose it sometimes you can just get it within make but sometimes if you want to reference back or like a his one that you ran in the past and it's

Not loaded and make you can get the data set ID there if you ever need a test all right so let just go back and so yeah that's what I was doing I was testing here but what you're what we're actually going to want to do is we're going to want to put the variable right here default data set ID you can put the limit here so if I wanted the limit to a th000 you could do that all right I'm

Going to save this then we're going to take that juicy data and we're going to put it into our air table base put it on the table and this is like I guess the only it's not even complex it's very simple but this is like sometimes you just when you're mapping out the data you can uh I'll tell you this guys you may have um little issues and I'll show you what I mean in a second what I mean is this

When when you're doing scraping like whether it's Twitter or uh Facebook or Instagram or whatever you could get like say if you're scraping posts there could be posts say if I was scraping my profile one of my posts could be an image one of my posts could be a video one of my posts could just be text depending on the order of that so let's say if I was like running this through the scraper and I'm trying to map out the

Data the you with make you'll actually only see so say if the first thing is a video or sorry not a video it's like a text or an image um but the third thing is like a video you actually won't when you go and you're trying to map it out you won't see like it will show let's see if it shows it here I don't even know if it's showing here um so yeah media you see media right here it's actually

Like this array and it's called an array with it make it's blank it's empty because the first thing that I'm scraping doesn't have that the third thing does but the first doesn't um okay if you guys know a better workaround than this let me know like maybe this isn't the most professional or technical I I don't know but what I do what I did is all I did is I just found a person's profile that had a video at the top

Um I believe that's what I did either that or with this scraper you may be able to just take an actual video post URL put it in there and then run it um let me just double check for you guys URLs you want to retrieve data top oh this is Tik Tok um oh this is Tik Tok I went to the Tik Tok let me go back to Twitter good old friends ATX um hopefully elon's not watching this video um I

Don't think he would like it and um yeah he's he's already pretty pissed off about Apple intelligence so I don't need him pissed off at me um got too many billionaires pissed off at me I'm joking anyways oh yeah you can filter right here by like top latest media Etc filter tweets okay um what was I even looking here for oh yeah if you could run um yeah you could so like see for example I literally just put a video post

From Twitter here I ran it clicking start and I was um and I had this like running so it would catch the data when it was done running and you could also go into the data set and try and like what you could do is go like this uh you just go unlink and you drag this over here so it starts here and then you would just go in here and put in that data set number ID from the storage so

You could if you need a test multiple times all right but anyways yeah and then you'll be able to see those these hidden Fields cuz like you'll see and when I click here I can't see Media stuff but here I have like you'll see I'm going to show you the field so Twitter I just put the id- author username I'm not going to like show you exactly where to find these you can like do it yourself there's a lot but this

Will like you you can easily just search these up right you can search it up it's a very simple then for the media uh field file URL it's it's it's looks like this it's media so you need to get if you're basically what you're going to so like yeah like if it doesn't have a media URL it just won't put it right but if it does that's why you want to find a specific post with that and then test with it

So you can get those variables within make and then you can actually click them right I hope that makes sense um anyways so then yeah I clicked media uh or you could just like you could probably if you're using this exact um if you're using the exact thing then um you could type it out I'll show you like what it looks like like you could cuz like if I uh copy this for example and I put it in a plain notepad

It looks like this you'd have to change the number of whatever um wherever you're getting the data from as you can see this is module 2 for me but um yeah you could do that if you understand like how to write it out like I'll show you even this one just so you know and if you want to like pause the video and write it out instead of like testing it with a test video you can do that this is what

The variable looks like but you need to change out the you need to change out this to and if your mod if uh you actually may not because your module if it's exactly like this it would probably be module 2 for the G data set items but anyways few things to know so okay for I have two videos I have a video SD and video HD not going to explain the whole thing why I mean it's pretty simple to explain I

But I'm not going to go super in depth um so these are the same except the only difference is this one has a three in the middle and this one has a four when you're pulling data some of them like these this file stuff right here you can't see it in this one cuz this one doesn't isn't a video okay let me just show you what it looks like with a video real quick um just so just for the purpose of

This let's go to runs here was probably I think it was probably this one I go this as long as this is yep so I can go to storage I'll show you what I mean guys just for the sake of this video just so it's easier for you guys to do so now instead of using a nonv video we're going to drag this over here and we're going to paste this data set ID in um and we're just going to run

This so it worked right and you can see here this is a video because we got media here well this is the thumbnail and then we got the entities right here this is an array right here entities and is this the video this okay I don't know ex I can't oh it could be this one actually extended entities um it's one of these things but basically it's right here so for the videos all you really need to know is just copy

This like it's find the extended entities media uh video info variance this URL so you could literally type in exended and then we're in media then we are in I think it's very variance where is the [Music] variance let me just type in variant here we go so yeah you can literally just type in variance and let me just double check boom we go here URL yeah so that's what I did I just type in variant and then you're going to

Select that one right here the URL and then you're going to put a three in the M oh uh uh let me cancel that you're going to put a three in the middle of this number because there's going to it's like an array oh wait wait wait wait wait let me just make sure you guys can see this cuz my head's right there okay hopefully you guys can see all all the stuff I'm doing um yeah you're just going to put

Cuz yeah this is the most important part where you should see send so yeah you just type in variance you click that URL one and then you put a three here and this is the SD for like lower quality and then the four is the higher quality one the reason I have two Fields is because not every video has the HD one so the fourth one right here but if it does I want I it just pulls both good thing is

Since it automatically uploads into air table air table is a really good one for that you don't actually have to download it within make and like waste your data right anyways hope that explains everything if you if you have any issues just rewatch that or let me know in the comments next is just text next is URL next is Twitter URL actually I think I these are I don't know if these are the same or not may be able to delete

One ID retweet count reply count like count quote count literally just all the important Fields uh view count bookmark count created at Ty uh type uh location I just put Source you don't even need that one really uh username for author so this is username for author it's under author uh profile URL under author profile ID under author profile picture under author which is here and then file name cover image um same thing all under author description followers following uh media

Count status count and then name these those are all under author okay and that is literally it all right so I could just go like this and hook this back up and put this to default data and if everything goes well if everything's right and correct here then I should be able to simply just go like this oh this is okay let's just wait for new data and I can also go like this of course like once like you want to

Make sure you're turning them on and leaving them on but I'm just for the sake of this this isn't on I'm just running it um and I should be able to just go to here I should just go to here and then go click on this click on start scraping let's see if this works so if this works this should start going any second and about three two one boom Ran So you need to check out what happened we got the

Record ID we got the record ID and air table and then we just ran the actor next over here oh it's already done cuz we only did 10 Perfect 10 actor ran next we got the data and next it uh yeah it just added them the one other thing you could do um is one second so let's just double check here to make sure everything worked out well so yeah you can see here Josh pook we got 10 different tweets a

Lot of these are retweets should be a video right so WDC uh WWDC and yeah the only other thing you could do is and look we got the and we got all this so let me just check here we got twet this is hidden so we go here Twitter is there any other linked record Twitter Twitter Twitter let me just go here where's our Twitter field oh it's the first field let me just tweet we'll call this tweet okay this will

Be called Twitter and what we're going to do I'll show you one extra thing um so over in our second field what we could do if you want is you could okay so I just I think figured this out let's see if it works here I'm testing again with that same record ID we just ran here and I'll show you what it what I did in a second let's see so boom boom boom searching the records now there is an issue

That I found with this and if someone knows how to fix this um in a more efficient way let me know but this is just what I've been doing and let's see if it worked okay so looks like it works it gets the job done um I'm not saying it's oh another thing you probably well you definitely want to do you is actually no it it's actually good so yeah so yeah that's it actually works so basically what I did is

We wanted to have it so it automatically links these tweets to the Twitter right here in that linked record so all it is is basically we added a search record module so just make our magnifying glass here search Twitter profiles and lots of times when I do this I do something like this um like for example IG username equals and then right here this is like the VAR a variable from the data set um and I actually just asked chat GPD

Anytime I don't know something about make guys I literally just asked chat GPT or anything about ear table a formula whatever I asked chat GPT CU that's what chat GPT is for I said Hey how do instead of making it equals how do I make it contains because now that we have x.com and twitter.com sometimes when I grab a link it could be x.com or it could be twitter.com I don't want to just do um like uh equals because if I

Do Twitter and then it's X um then it won't work right so what I'm doing here is you're just doing a search record module so you boom put in your base we're searching in our Twitter profile section I select it all just cuz you you you don't you could select um as long as you're selecting Twitter URL it's pretty much all the only one you need I believe but I just like select all for for whatever reason formula is this it's

Search and then bracket um username because we're pulling the username right here and then as as long as the username is contained within this Twitter URL which is with in our air table base then it's going to pull it and then we're just pulling a limit of one um so yeah the only issue is like with this you're adding an extra 10 scenario runs or however many times you're using this for your make.com so you're kind of like doubling it per

Um per thing so there's definitely should be an easier way to do this if you know let me know I just haven't really spent the time to figure it out yet um but yeah as of now this does work so if you want to link the records together so you have a nice system there this is how you do it all right so I hope that helps guys if you got value from this uh and let me let me just add

This back here and then go like this and to finish it off we're going to just put it back to our default data set ID boom save it and there you have it we have our full scenario right here not there right yeah you saw it uh yeah I closed the first one but you got it right so if you need to watch that back watch that back hope that really helps guys if you got value from this video I'd definitely

Appreciate um you know some support with a Sub sub uh if you subscribe if you want to you don't have to but a lot of you guys that do watch this channel I was looking at the analytics today aren't subscribed subscribe so I'm really going to be putting a lot of value for you guys in the next coming weeks terms of make air table AI Etc if you want if you haven't joined our free community our stride Community it's a group

Of like-minded individuals entrepreneurs AI enthusiasts Etc um you can join that link in the description down below we have about 7 100 members right now on our Facebook group Discord Channel and uh we could be potentially in the future starting a maybe a paid group so as your uh yeah this is your time now to get in for free uh to the stride Community while you can and other than that guys if you haven't already checked out the client acquisition code

Book you can check that out it's an in-depth guide on everything client acquisition systems hiring setting up funnels using tools that go high level there's AI literally uh copy and paste AI prompts it's over 500 600 Pages um and yeah link for that will be in the description it's not just some AI written ebook or anything this is something I actually spent over two years getting ready I sat down every day for multiple hours and actually like organized it all uh

Wrote wrote Every Day put it all together and took me a very very long time so yeah super super valuable there's a money back guarantee on that too even though it's super cheap just just cuz I know you're going to love it if you do get it but other than that guys oh and if you do want to speak to our team and see if we can help you grow your business whether you're a coach consultant marketing agency if you're in

A B2B service based business whatever the case may be you need help with automations AI streamlining your processes uh packing your calendar full of clients running ads all that good stuff we do that here at our company executive.com you can go to executive.com apply book a call and just to speak with myself or someone from the team and it's it's not like a hard sales call or anything like that it's literally just a conversation to see if we can help so

If it if you don't think we can help then no worries no harm no foul other than that guys we upload videos every single day here so um yeah smash that subscribe button so you can stay up to date and I will see you in tomorrow's video guys keep hustling keep grinding and of course accelerate your stride take care