Human Murmuration

Credulous at best, your desire to believe in
Angels in the hearts of men

​ The week started off not with a bang but with a whimper, as I opened an issue that Davedalf the White noticed during our weekly triage meeting. I've been trying to reproduce the bug, but like Pedro I just couldn't no matter how hard I tried. Software development can be strange like this sometime, but unfortunately if this can't be reproduceable fixing the invisible will post quite the challenge. (Edit: As of this Friday, during our weekly deployment meeting we're currently under the impression that this is no longer an issue.)

This was followed up by a quick microservice meeting to discuss the latest status and the massive changes required to fully land the Users Microservice. I'll talk more about that later but the clock is ticking, t-minus 2 weeks and counting from today's date.

Josue and I had a quick meeting with Pedro to discuss some typing issues related to his latest PR which updated the Dynamic Image component we use on Telescope's home page. The meeting was concluded by the suggestion to implement MUI Types. I kind of wish I had learned more about typescript and NextJS but I think I'm okay focusing mostly on the backend for now.

After the meeting, Josue and I reviewed a PR from Davedalf, following some excellent advice to not test this locally (as it's not really something that's testable locally), but instead read the code and the tests. It's hard to understate the significance of brief (yet detailed) code explanations.

I also reviewed Yuan's latest PR which adds a really that hides the navbar on the initial render of our website, but shows it as the user scrolls down. I pointed out an issue that (presumably) caused this PR to clash with another newly landed PR from Duke which adds (awesome) scroll snapping. Unfortunately I don't know enough to help beyond pointing out flaws. Living the dream.

On a similar note, I also left my thoughts (such as they are) on Huy's PR which touches up the author section of Telescope. I also reviewed another of Dave's PRs, yet again joined by Doc Josue, which is helpful given that I can barely read a lot of the code he's pushing out lately. Having someone to dumb things down for you is also helpful.

I also had a brief meeting with Dave regarding some issues he was running into when attempting to POST users to the Users microservice in some tests. Yet again Doc Josue was able to save the day, and a fix PR arrived shortly after. It turned out our backend was mocking our node-fetch requests, resulting in data not being sent to the microservice.

I finished off the night by leaving some thoughts about a weird code escaping bug, which was conveniently being caught by another blog post by yours truly. I've worked on our backend's sanitizer previously so fortunately I have a small bit of "insight" into what may or may not be causing the problem.

​ Friday, the week is over but we still have to deploy and land all our PRs before hand. Here's a flurry of PR's that I approved:

• A PR which (finally) allows user authentication on our Vercel deployments. Something I've been waiting for for a while now.

• and also ... a followup PR to fix something that I noticed just broke with the Vercel fix!

• I also left some notes on Illya's search microservice.

• Finally, I also approved a last minute fix from Doc Josue which fixed our latest prod deployment.

Users Microservice

​ We finally landed a fix for our paginated GET route (something I tried my best to review) which, admittedly, had me nervous for a few days. The problem we had previously was that the paginated GET route that I created only worked for users who had IDs starting from 0. Since we hash our user IDs, this obviously is not a solution. The solution is actually really clever: it relies on keeping track and embedding where we left off in the response's header so the subsequent request has all the information where to continue from in the response header:

const query = await db
    .collection('users')
    .orderBy('id')
    .startAt(userToStartAt)
    .limit(perPage)
    .get();

vs.

let query = db.orderBy(documentId()).limit(perPage);

// If we were given a user ID to start after, use that document path to add .startAfter()
if (startAfter) {
    query = query.startAfter(startAfter);
}

const snapshot = await query.get();
const users = snapshot.docs.map((doc) => doc.data());

// Add paging link header if necessary, so caller can request next page
addNextLinkHeader(res, users, perPage);

module.exports.addNextLinkHeader = function (res, users, perPage) {
    // If there aren't any results, there's no "next" page to get
    if (!users.length) {
        return;
    }

    // Similarly, if the number of users is less than the perPage size,
    // don't bother adding a next link, since there aren't going to be more.
    if (users.length < perPage) {
        return;
    }

    // Get the id of the last user in this page of results
    const lastUser = users[users.length - 1];
    const lastId = lastUser.id;

    // Construct the body of the header, giving the URI to use for the next page:
    // '<https://api.telescope.cdot.systems/v1/users?start_after=6Xoj0UXOW3&per_page=100>; rel="next"'
    const link = new LinkHeader();
    link.refs.push({ uri: `${USERS_URL}?start_after=${lastId}&per_page=${perPage}`, rel: 'next' });

    res.set('Link', link.toString());
};

Clever fixes like these are amazing.

Aside from going over this PR a few times, Josue and I wrote a tool to help export users from the Planet CDOT Feed List (a list of Telescope users and their blog information.) It... actually didn't turn out bad at all! The code is easy to read, maintainable, and best of all, short. It went through a few rounds of reviews (a new personal best for me.)

I also started initial discussions and research into proper e2e testing for the Users microservice, and implementing our own Redis cache. More on that to come by next Friday.

Overall a good week which I spent a lot of time reviewing, commenting on stuff, and having meetings.

In other more personal news:

• Been playing a bit of Assassin's Creed Valhalla, it's a solid 6/10 which (in the few hours I've played) has been otherwise enjoyable.
• Been thinking about getting into Rust a lot lately. All this JavaScript lately has been making me itch to get back into a "lower level" language.
  • I really wish I had pushed myself more in OSD600 to try something new. I don't regret my time using Python at all (as it was also new to me), but I guess the grass is, in fact, always brighter on the other side. Or maybe it's always rustier? Who knows.
• Still enjoying The Way of Kings a lot. Finally on part 3 after starting this book nearly 4 months ago. Highly recommend the graphic audio version if anyone else is an audio book fan (they also have a preview on their website which is awesome)
• Wish me luck with my finals start next week.