Progress
Until now, my first PR for adding route \feeds\invalid
to Posts service merged, and I also sent another PR for the issue #2413. So everything is going well, and I have not had any significant challenges working on these issues.
Finding
The hardest part for implementing the route \feeds\delayed
to Posts service is testing my code. When microservices finished downloading feeds and processing them, I checked the Redis server to find the delayed one. And I found nothing. After asking a question on Telescope's Slack channel, I got an answer from prof. David that this case is not usual. The reason is a feed is marked delayed only when the processor receives an HTTP 429
response ("the user has sent too many requests in a given amount of time ("rate limiting")"). This issue made it hard to test my new functions against the real-time database.
The workaround suggested by prof. David is implementing an integration test to create delayed feeds on the fly. Surprisingly, those feeds did not have the time to live when I used ttl
command. I found that the setDelayedFeeds()
function did not set the expire time for the key.
// old code
setDelayedFeed: (id, seconds) => redis.set(createDelayedFeedKey(id), seconds, 1),
// new code
setDelayedFeed: (id, seconds) => redis.set(createDelayedFeedKey(id), seconds, 'EX', seconds),
Take-away
The biggest take-away for me this time is learning about Redis commands and Readable stream in NodeJs.
Streams is one of the core concepts in NodeJs, and it is often used when we need to work with a large amount of data.
When calling the scanStream()
method on a Redis client, I actually created an object of readable stream.
Scanstream : Convenient class to convert the process of scaning keys to a readable stream
My code:
const getFeedKeysUsingScanStream = (matchPattern) => {
const keys = new Set();
const stream = redis.scanStream({
match: matchPattern,
}); //create a readable stream object
return new Promise((resolve, reject) => {
stream.on('data', (data = []) => {
data.forEach((k) => keys.add(k));
});
stream.on('error', (err) => {
logger.error({ err }, 'Error while scanning redis keys');
reject(new Error('Error while scanning redis keys'));
});
stream.on('end', () => {
resolve([...keys]);
});
A stream object has many events: 'closed', 'data', 'end', 'error', etc. Because the stream processes data piece by piece, we need to wrap it inside a Promise and only resolve when there is no more data to consume _ the end event.
In the end, I was able to make the route work, and I am happy with the progress.
Top comments (5)
Heyyo Emily this is a great blog post :D
Good progress! How do you like contributing to open source?
Thank you. I love that there is always something new to learn from open source project :)
Wow learning streams, that very cool! I've been a developer for 9+ years and have yet to work with streams. Keep up the great work! :)
Thank you for your kind words <3