TL;DR
In this article I am going to show how to build a simple service which monitors our HackerNews account and sends a notification to our Discord channel when there are new comments to our submissions.
We are going to use th following packages/technolgies:
- Node.js / express
- discord-notification
- node-cron
Before we start… I have a favour to ask. 🤗
I am building an open source feature flags platform and if you could star the project on github it would really help me to grow the project and would allow me keep writing articles like this!
https://github.com/switchfeat-com/switchfeat
Setup the Express server
Let’s start creating a server
folder and install express
and some other depedencies we are going use:
cd server & npm init -y
npm install express cors typescript
npm install @types/node @types/express @types/cors dotenv
npx tsc --init
Let’s create the index.ts
file, and push the esieast Express configuration you can have in tyescript:
import express, { Express } from 'express';
import cors from "cors";
const app: Express = express();
const port = 4000;
app.use(express.urlencoded({ extended: false }));
app.use(express.json());
app.use(cors());
app.get("/api", (req, res) => {
res.json({
message: "(HackerNews -> Discord) notifier",
});
});
app.listen(port, () => {
console.log(`⚡️[server]: Server is running on port 4000`);
});
In package.json
of the sever, we need to add a script to run it:
"scripts": {
"start": "node index.js"
}
To finish our setup, let’s compile typescript and run the server:
tsc & npm run start
Cool! Let’s go!
Installing server dependancies
In the server folder let’s install the dependancies we need to interact with both HackerNews API and Discord webhook.
npm i discord-notification node-cron @types/node-cron
Building the HackerNews API connection
The HackerNews API exposes a bunch of public endpoints, there is no authentication, so I would assume there is some sort of rate limits in place, so be careful not to overload it with requests. More information here.
Let’s create a simple abstraction layer which would hold the connection to HackerNews and run simple GET
requests to their API. Having these calls in separate file enforces separation of concerns and allows to use other libraries in the future with minimal effort.
The hackerNews.ts
file would look like this:
export const getUser = async (userName: string) => {
const resp = await fetch(getUserURL(userName), {
method: "GET",
headers: {
Accept: "application/json"
}
});
return await resp.json();
};
export const getSubmission = async (submissionId: string) => {
const resp = await fetch(getItemURL(submissionId), {
method: "GET",
headers: {
Accept: "application/json"
}
});
return await resp.json();
};
Let’s start building the scheduling logic, in a way that our app will automatically fetch the latest data from HN on a schedule. For this, we are going to use the node-cron
library which is a very solid and mature library which allows to run different sorts of scheduling patterns.
We are going to setup the scheduler to check every 15min if there are new comments to our latest submission.
node-cron
uses cron-tab based cron expressions, you can see how to write more complex once here.
Let’s create a scheduler.ts
file as follows:
import * as cronJob from "node-cron";
export const initScheduledJobs = () => {
const scheduledJobFunction = cronJob.schedule("*/15 * * * *", () => {
// - intaract with HN API
// - send nofitication to Discord
});
scheduledJobFunction.start();
}
Finally let’s instruct our express server to start the scheduler at startup in the index.ts
:
import * as scheduler from "./scheduler";
scheduler.initScheduledJobs();
Great! Now that we have a scheduler in place, let’s add the HackerNews fetching logic to the initScheduledJobs
function:
const userData = await hackerNews.getUser(userName);
console.log(userData);
Running this code, we are going to see the response from HN for my user data dev-bre
on the console.
{
created: 1587898826,
id: 'dev-bre',
karma: 23,
submitted: [
36846077, 36712635, 36712606, 36548356,
36547441, 36147979, 36131968, 36131217,
36131215, 36128924, 36022807, 34781854,
34109805, 34109230, 34109200, 34105053,
33909549, 33909537, 33887733, 33388763
]
}
The submitted
field in the response contains an array of itemIds each one referring to a single submission I made, sorted by date. For the purpose of this article we are going only to monitor the latest submission, which means we are only interested in the first item in that array.
In order to get the list of comments for this post, we need to run an additional API call as follows:
const submissionData = await hackerNews.getSubmission(selectedPost);
The HN API returns the data in the form of a Tree datastructure, which means this API response would contain a field: kids
in the form of an array of numbers which are the direct children of the current item, i.e. direct comments to the requested comment. In order to traverse the entire set of comments, on each level we need to use recursion.
A recursive approach to get HN data
In order to get all comments to a post on any level, we need to go with a recursive approach!
Without going too much into the details of how recursion works, in a nutshell a recursive function is a function which calls itself progressively on a smaller data sets, until it reaches the base case which allows the function to return.
In our scenario, the base case is when we find a comment with no children (no one replied to that comment). Here it is how our function would look like:
const callRecursively =
async (node: ResultType, results: ResultType[]): Promise<void> => {
// call HN for the current node
let curr = await hackerNews.getSubmission(node.id);
const kids = curr.kids as number[];
results.push({ id: node.id, text: node.text, time: node.time });
// check kids and go recursive
if (kids && kids.length > 0) {
for (let x = 0; x < kids.length; x++) {
const childNode = {
id: kids[x],
text: curr.text,
time: curr.time
};
await callRecursively(childNode, results);
}
}
};
At the end of the processing, the results
array is going to contain all the comments for the requested post.
Next step is to keep track of the previous execution and compare the current execution with the previous one. If the two arrays are different, then we have updates and we need to send a discord notification.
Here it is how all this is going to be integrated with the rest of the app we are builduing:
console.log("Starting check HackerNews -> Discord");
const currentResults = await grabContentFromHN(
userName,
selectedPostId);
console.log(JSON.stringify(currentResults));
// Compare the latest results with the previous run,
// if there is a change, then there are updates to this post.
const updateAvailable = compareWithLastRun(currentResults);
if (updateAvailable) {
console.log("New updates available!");
// notify Discord!
}
// update the global state
globalResults = [...currentResults];
The super simple comparing logic simply converts the 2 arrays in JSON strings and use the string comparator to do the work. This not the best solution, simply because it's not the fastest, but does the job:
const compareWithLastRun =
(currentResults: ResultType[]) : boolean => {
if (globalResults.length < currentResults.length) {
return true;
}
if (JSON.stringify(globalResults) !== JSON.stringify(currentResults)) {
return true;
}
return false;
}
For the purpose of this article we are going to save the state of the previous run locally, which means everytime we restart our app, we are going to receive an intial notification, the first execution sets the baseline. A more elegant approach would save the previous run in a storage instead.
Discord integration
In order to send Discord notifications, we are going to use the discord-notification
library, this is not a library which covers every single feature that the Discord API offers, but it is good enough for our usecase.
Before using the library we need to create a new Webhook on Discord. I am not going into the details on how to create a webhook, the Discord documentation is really well done. Here it is the link.
Now that we have our webhook, let’s save it in the .env
file of our project. The .env
should have this structure:
HN_USERNAME=
HN_POST_ID=
DISCORD_WEBHOOK=
Let’s create a new file discordNotifier.ts
and let’s push this code in it.
const discordWebHook = process.env.DISCORD_WEBHOOK as string;
export const discordNotification =
new DiscordNotification('HN-Notifier', discordWebHook);
export const notifyDiscord = (message: string,
postId: string,
postText: string,
comments: number) : Promise<void> => {
return discordNotification
.sucessfulMessage()
.addTitle(message)
.addDescription(`[Link to the post](https://news.ycombinator.com/item?id=${postId})`)
.addContent(postText)
.addField({name: 'postId', value: postId, inline: true })
.addField({name: 'comments', value: comments.toString() })
.sendMessage();
};
The code above receives in input the notification data and uses the Webhook to send the notification. Couldn’t be easier than this!
Finally let’s integrate this logic in our scheduler, simply using this function like this:
if (updateAvailable) {
console.log("New updates available!");
// notify Discord!
await discordNotifier.notifyDiscord(
"New comments(s) available!",
selectedPostId,
currentResults[0].text,
currentResults.length - 1);
}
Here it is how our new shiny Discord notification will look like!
Well done!
We now have fully working Hacker News monitoring system and we won’t lose comments anymore!
We could expand on this putting additional data into the notification, like, what was the text of the latest comment, the time when the latest comment has been published, etc. This system we just built is flexible enough to allow these chanegs with minimal effort.
So.. Can you help? 😉
I hope this article was somehow interesting. If you could give a star to my repo would really make my day!
Top comments (2)
Great post!
Finally no need to read their ugly feed! :)
Thanks Nevo! I am glad you found it useful!