This weekend I stumbled upon this article:
A bot that tweets new DEV articles about Vue
Daniel Elkington ・ Mar 3 '19 ・ 6 min read
And I immediately thought: "Hey, this is an awesome idea. Let's make one about React as well, cause why not?"
So I did. I created a bot that checks dev.to/t/react/latest every minute and tweets any new article. Here it is: @TheReactDev
TypeScript? Nah.
I choose not to do TypeScript on this one. It is a super small function and I felt like JavaScript would get the job done faster.
Serverless
You have many options when you do serverless. You can go AWS, Azure, GCloud, etc. The article explains in detail how to setup the Azure function and because I had never used Azure before, I followed along. The integration of Azure with VSCode is pretty cool, can't deny it, but I might switch up later on for the serverless framework to be sort of platform agnostic. Not that it matters for a project like this though. 😄
UPDATE: I had some headaches with azure. Probably because I don't have much experience with. So after a couple of hours I switched to AWS with the serverless framework for the tooling.
Scraping
Because dev.to does not have an api yet, I had to scrape away the data from the url. I had used puppeteer before, but I absolutely loved the x-ray
lib. The API is so clean. I'll definitely use more of it.
Problems I had
At first, when I fetched the recent tweets from the @TheReactDev, the twitter api didn't always returned the posted urls on the entities.
[
//...
[ { url: 'https://t.co/AA3sKrLsDZ',
expanded_url:
'https://dev.to/peterj/run-a-react-app-in-a-docker-container-kjn',
display_url: 'dev.to/peterj/run-a-r…',
indices: [Array] } ],
[ { url: 'https://t.co/hptME9oAgf',
expanded_url: 'https://twitter.com/i/web/status/1102686466436681731',
display_url: 'twitter.com/i/web/status/1…',
indices: [Array] } ],
// ..
]
Because of this, I started checking for the article title and author on the tweet body. The problem is that the twitter API sometimes truncate the body. The solution to this was to use the tweet_mode: "extended"
on the request for the tweets.
But, just as I was writing this post, I observed that this also solves the problem of the links. With this mode, the twitter API not only returns the full_text
but also returns the full extended_url
as well. 😅
Checking for the URL is way more reliable so I'm rewriting my function to use this as well.
Telegram
I also added a telegram bot that sends a message to a channel in case things go wrong. I choose telegram because it'll probably be the fastest way that the notification reaches me.
The repo
frontendwizard / TheReactDev
The function that feeds @TheReactDev twitter.
TheReactDev
The function that feeds @TheReactDev twitter.
This project was inspired by this article.
This function depends on a few environment variables:
-
dev_tag
: the dev.to tag to which the crawler will get the latest articles. -
twitter_bot_screen_name
: the name of the account that will tweet the articles. - Twitter keys
twitter_consumer_key
twitter_consumer_secret
twitter_access_token
twitter_access_token_secret
-
telegram_bot_token
: the telegram bot who's going to log messages for maintenance/debug purposes -
telegram_chat_id
: the channel where the bot will send messages
You just need to create a env.yml
file and put those values in, like this:
dev_tag: "react"
twitter_bot_screen_name: "NameOfTheTwitterAccount"
twitter_consumer_key: "..."
twitter_consumer_secret: "..."
twitter_access_token: "..."
twitter_access_token_secret: "..."
telegram_bot_token: "..."
telegram_chat_id: "..."
It's worth noting that I used telegram here because it was simple and it probably is the…
Like it?
Well, it was super fun doing this. The feeling that you get when you see the function publishing new articles on twitter is amazing. 😄
If you like it, be sure to follow the @TheReactDev on twitter!
Thank you for your time and until the next one!
Top comments (4)
😄
Let’s all work together to make these accounts as useful as possible!
Awesome job! Reading up on
tweet_mode: extended
, I think this relates to Twitter's change in 2017 to go from 140 characters to 280 - so that there were no breaking changes the API by default returns 140 characters with a link to view the full tweet. You have to 'opt-in' to enabling more than 140 characters withtweet_mode: extended
. I suspect I haven't had this problem with @TheVueDev yet because I'm not (yet) tweeting tags and the article titles haven't been too long, so all tweets have so far been under 140 characters - but I'll definitely make the change to include this parameter! Thanks for finding this future bug for me 😃FYI, there is actually an API for articles etc for dev.to though it isn't really documented anywhere as far as I have seen. I found out about it from a comment Ben made on another post.
dev.to/api/articles?tag=react
There are a few other endpoints too if you want to get articles for a specific person etc.
For a specific article, you can use this if you know the ID: dev.to/api/articles/88679
That can eliminate the scraping :)
Will look into it today.