In recent years, Twitter has become one of the largest social communities on the internet. It hosts discussions from all over the globe on every imaginable topic — from politics to history and, most importantly for this post, honest and extensive discussions about every brand or product you could envision.
This makes Twitter an invaluable tool for data-driven market research, providing deep insights such as:
- Consumer perception of any brand:
- Their opinions
- What drives them to use the product
- What they like or dislike
- What they wish for
- How they compare it to other products
- And many more insights
In this blog post, we will be using Twitter to conduct in-depth research into customer sentiment, brand perception, customer feedback,and anything else you want to know about a brand or product using Notebook LM and Apify.
Our Goal: Conduct market research on Apify by analyzing up to 5,000 tweets about everything users say about it.
Outline
Part I: Get the data:
- Use Apify Twitter scraper
- Export results
Part II: Analyze it:
- Upload to Notebook LM
- Get AI insights
Why Twitter
If you’re wondering why Twitter is so successful, here are five key reasons:
- Real-Time Updates: The platform provides immediate, real-time feedback and discussions from users worldwide.
- Global Reach: Twitter’s global user base offers diverse perspectives and insights from different markets.
- Hashtag System: Twitter’s hashtag organization makes it easy to track specific topics and conversations.
- Direct Brand Engagement: Users frequently interact directly with brands, providing valuable feedback.
- Trending Topics: Twitter’s trending system helps identify emerging issues and conversations quickly.
This makes Twitter an invaluable source of information for market research on your brand or competitor analysis.
How to Conduct Market Research with Twitter (the Ordinary Way)
You could open Twitter, enter a search term, and go through the tweets while making notes. This will certainly provide you with numerous insights and ideas on how your competitor or your own brand is performing, how customers perceive it, what they miss, what they like or dislike, etc.
That’s great. However, it sounds like an awful amount of effort — especially if you want to get an aggregated view, the big picture based on hundreds or even thousands of tweets. This could become very tedious if done manually.
Hence, there are tools like Make.com and others that allow you to create pipelines that process large amounts of data and derive knowledge from this data. However, there are many caveats:
- Complex Configuration: You need to configure quite a lot. For example, if you want to access the Twitter API, you need to be familiar with Twitter, the API, and, of course, platforms like Make.com or whichever you use. The same applies to any other platform you use as a source of information for your market research, e.g., Reddit, etc.
- Data Preprocessing: You need to preprocess the data to be suitable for AI processing.
- AI Configuration: You need to configure the AI to leverage this data effectively.
In essence, it requires a lot of knowledge about the platform, the API, and how to integrate all the moving parts to get a good picture of your brand or the brand you are trying to analyze.
The AI-way of conducting market research
Fortunately, yes. A solution that addresses all the above problems and makes AI-powered market research a breeze.
Here’s how it works:
- Leverage a no-code Scraping Platform: We use a scraping platform called Apify and a Twitter scraper (also known as a Twitter actor) that does all the heavy lifting — setting keys, limits, proper data formats, everything. Essentially, all we need to do is provide the search keyword we want to analyze, and that’s it. It’s just magic.
- Use Notebook LM: We take the output of this scraper and feed it into Notebook LM, another powerful tool provided by Google for analyzing (and even generating podcasts from) massive amounts of data.
With these two tools, we combine the power of massive data with the power of AI to provide you with insights into any brand, product, company, or whatever you can imagine.
If you have never worked with these tools before, I recommend you check out my blog posts (just use the search, you will find plenty of articles).
What We Will Need
- An Account on Apify.com: If you don’t already have an account, you can sign up here.
- A Google Account: To use Notebook LM.
Sounds good? Let’s try it.
Part I : Collecting the data
1. Open the Twitter Scraper
Go to the store and open the Twitter scraper actor:
If you prefer another tool, you can use it as well, but this one worked well for me, so I’ll use it here.
And with a free plan, you can test it for free.
2. Configure the Scraper
In the configuration, you need to set basically only two things: the keyword and the limit of tweets you want to scrape.
You can set the configuration in the UI, but you can also enter a JSON with all the configuration to save time copying and pasting.
Configuration Snippets:
searches: apify
maximum number of tweets:1.000,
You can copy and paste this JSON code below to automatically configure all necessary fields for this tutorial:
{
"excludeImages": false,
"excludeLinks": false,
"excludeMedia": false,
"excludeNativeRetweets": false,
"excludeNativeVideo": false,
"excludeNews": false,
"excludeProVideo": false,
"excludeQuote": false,
"excludeReplies": false,
"excludeSafe": false,
"excludeVerified": false,
"excludeVideos": false,
"images": false,
"includeUserInfo": true,
"language": "any",
"links": false,
"media": false,
"nativeRetweets": false,
"nativeVideo": false,
"news": false,
"proVideo": false,
"proxyConfig": {
"useApifyProxy": true,
"apifyProxyGroups": [
"RESIDENTIAL"
]
},
"quote": false,
"replies": false,
"safe": false,
"searchQueries": [
"apify"
],
"tweetsDesired": 1000,
"verified": false,
"videos": false
}
Then hit Save.
Save Configuration
3. Start the Scraping Job
Now go to the list of runs (the scraping jobs) to see the job you have just started.
After a few minutes, you should start getting some results.
4. Export the Results
Now let’s export those results.
Make sure to select the field “full_text”; this includes all the data we want to analyze without all the metadata like usernames, dates, etc. We leave them out for this use case, but they could certainly be valuable for other purposes.
After exporting, you should have a JSON file with a format like this:
JSON File
Rename the file to .txt to be able to import it into Notebook LM.
Tip for Coders: You can convert it to markdown with a command similar to this:
cat dataset\_twitter-scraper\_2024–12–12\_10–08–25–793.json | jq ‘.\[\].full\_text | gsub(“n”; “ n”)’But simply renaming the file works well for analysis with Notebook LM.
Part II: Evaluation with Notebook LM
1. Upload to Notebook LM
Now head to Notebook LM and upload the file we just renamed.
And watch the magic happen.
After a few minutes, you can see on the left side the sources you just uploaded and, in the chat window, a summary. From there, you can drill down — the sky is the limit for the types of questions you can ask. I find it just mind-blowing.
2. Get Insights
To get a first insight, we can request a high-level overview of how users leverage Apify for lead generation.
Request Overview
And here is the result:
Overview Result
The Notebook LM answer:
Leveraging Apify for Lead Generation
The sources provide a few examples of how users are leveraging Apify for lead generation:
Building custom scrapers for lead generation. One user is recording a course for systems agency owners that includes a section on “Apify Master-Class” for lead generation automation. The course will teach users how to connect scrapers to Make.com for functions and scraping, and how to train large language models (LLMs) for optimal output. Another user is building workflows to scrape data about people running ads, specifically mentioning Apify as a tool for this.
Scraping LinkedIn profiles. A user suggests building a lead generation platform that scrapes LinkedIn profiles for corporate and personal emails. This platform would be built using Bolt and Supabase on top of Apify and would implement a credit usage system to avoid monthly subscriptions.
Using Apify’s pre-built scrapers. Apify offers a variety of pre-built scrapers that can be used for lead generation. For example, the eBay Store Scraper can extract product information from eBay stores, which can be used to identify potential leads. The Indeed Global Job Scraper can be used to collect data about job postings, which can be helpful for recruiters and marketers looking for leads.
The sources highlight that Apify can be a powerful tool for lead generation. Its flexibility allows for both custom scraper development and utilization of existing solutions, catering to a range of user needs and technical expertise.
These overarching themes offer valuable insights into the nature and potential of Apify. The platform emerges as a dynamic space for writers and readers to connect, engage with niche content, and navigate the evolving world of online publishing.
Quite impressive for what we get with just three clicks!
You can take this to the next level and, with a fourth click, generate a podcast. If you haven’t tried it before, I highly recommend you do. You will be amazed.
Wrap-up
In this tutorial, I have demonstrated how you can take your market research to the next level by conducting a deep analysis of Twitter tweets and leveraging AI to gain profound insights into any brand.
With these two tools, you can analyze not just Twitter but also Reddit, TikTok, Facebook, and hundreds of other rich information sources, providing you with unparalleled insights.
Top comments (0)