Regardless of where you are in the industry, there is a good chance that Serverless functions are a topic you've seen brought up on Reddit, Hackernews, a local Meetup, or where ever you find two programmers talking. If you have never worked with them before, they can sound intimidating; yet another stack to learn or some deep Dev Ops magic. The beauty of Serverless functions is they make things easier by introducing a layer of abstraction between the code you want to run and the servers running the code. That's right, despite them being called Serverless, they most definitely do run on servers, just not servers you explicitly know or care about.
Having one less thing to worry about always sounds great to me. Granted, it is not all sunshine and roses. Like any engineering tool, there are costs and trade-offs involved, which I am not going to cover in this post. Generally, I'd say this feels like an abstraction that is needed in the industry at large, and I wouldn't be surprised to see more and more engineers apply a serverless approach to an increasingly varied set of domains.
If you find yourself in a position to start to use them, you may be at a loss as to where to begin. How do you structure your code? How do you ensure your code is high quality? The same questions you ask yourself when engaging with any new technology. The answer is, as it is to any interesting question, it depends.
In this post, I am going to lay at a reasonable starting approach that can be adjusted to your specific needs as you become more familiar with the technology. Using a TDD approach and a familiar backend architecture, I am going to guide you through a simple scenario.
Firebase
There are many different platforms out there that support Serverless Functions, but lately, I've been using Firebase's Cloud Functions on client projects. The Firebase ecosystem is exciting, and the more I learn about it, the faster, I feel like I can deliver software to my clients, so it's all worth learning about, but for today, let's focus on Firebase's functions.
Here is what a basic Hello World function looks like:
import * as functions from "firebase-functions";
export const helloWorld = functions.https.onRequest((request, response) => {
response.send("Hello from Firebase!");
});
Pretty straight forward, the firebase-functions API works by registering functions to handle a request, something that should not be too surprising for those who have used Express or Hapi in the past.
One of the cool things about Firebase is the ease at which you can integrate with other technologies on the platform. Firebase supports Google's Cloud Storage. Here is how you would write a function to listen to a bucket and do something when a file is uploaded:
export const helloStorage = functions.storage
.object()
.onFinalize((objectMetadata: ObjectMetadata, context: EventContext) =>
console.log("Hello from Firebase storage!")
);
I'm using Typescript to get a better view of what the Firebase API is going to send my function. Both ObjectMetadata and EventContext are rather large, so I am not going to post the full types here. The TLDR is that ObjectMetadata will give you details about the object that just finalized (or uploaded) and EventContext, gives you auth context, a timestamp, etc. This simple API can be used as a launching point to solve some interesting problems. Coupled with Firebase's rather amazing client-side API, we can kick off whatever behavior we need from a simple file upload.
Diving In
Let's work together on solving a problem using the tools Firebase is giving us. If you are interested in following along, I highly recommend following the getting started docs for Firebase Functions. You ultimately want to spin up a blank project with Functions, Firestore, and Storage configured. Following along, you can add code and libraries as I mention them below.
Problem
Currently, Video Game review aggregators, like OpenCritic and MetaCritic, are in the midst of introspecting about the best titles over the last decade. Let's write some code to help them out.
Let's say I have a CSV file that looks like this,
game,publisher,year,rating
Slay The Spire,Mega Crit Games,2019,89
Luigi's Mansion,Nintendo,2019,86
Super Mario Maker 2,Nintendo,2019,88
Red Dead Redemption 2,Rockstar Games,2018,96
God of War,Sony Interactive Entertainment,2018,95
Super Mario Odyssey,Nintendo,2017,97
Persona 5,Atlus,2017,94
and I want to create a screen of all high ranking games grouped by years,
Let's not worry about the UI side of things. Let's assume that if we can get into the database, we can revisit it later and write a UI on top of it. Let's use Firebase's own Cloud Firestore as our database since it is going to plug nicely into the Firebase infrastructure. Firestore is a NoSQL database that has a few unique qualities that are out of scope for this blog post, but it's pretty great and worth reading up on. We are also going to use Typescript for more transparent code samples and because it is awesome.
To store our Games Of The Year data, let's plan on creating a single document in Firestore. The UI will then only need to retrieve this document to be able to display a nice retrospective of the last 10 years of Video Games to the user.
At a high level, we need to do the following:
- Read in a CSV file
- Transform it into a JSON document
- Save it to our database
- Test all of that to the best of our abilities
Testing Approach
For this example, we may be tempted to write a test that tests all the way through the functions, feeding in a CSV, and then checking the database for the expected result.
While I think that sounds reasonable, I know from experience that Firebase Integration Tests are not super easy to write with the isolation needed to run them in an automated fashion. While Google has a function emulator, it has its own nuances and challenges. Another approach is to set up a separate Firebase environment just for tests, which also has some significant downsides, especially if you are working on a team.
For the purposes of this blog, let's scope everything down and focus on Unit Tests. The Unit Tests we write are going to run ultra-fast, be small and simple, require no external dependencies, and be excellent at covering error conditions and corner cases. Let's save talking about Integration Tests for another post.
Thinking it through
When we start thinking about Unit Tests, we have to start thinking about how our application is architectured and how our responsibilities are layered. We need to divide these responsibilities into pieces of code that are essentially our units to test.
Let's take an approach that should be familiar to most people who have prior experience with backend code. Let's divide our architecture into 3 tiers:
- The Controller Layer - Responsible for adapting incoming messages from our platform (Firebase) into our domain
- The Service Layer - Where our business logic lives.
- The Repository Layer - Takes messages from our domain and saves them to our database.
In addition to this, our index.ts file is going to be responsible for registering our logic with the Firebase function API, creating each of these layers, and composing them correctly.
This 3 tier approach isn't the only way we can approach Serverless Functions, but it does represent a reasonable starting point and a smooth transition from other backend frameworks.
While I typically drive interfaces out as I develop my tests, for clarity, let's lay out the interface for each of our layers in advance:
export interface GamesOfTheYearController {
create(objectMetadata: ObjectMetadata): Promise<void>;
}
export interface GamesOfTheYearService {
add(games: Game[]): Promise<void>;
}
export interface GamesOfTheYearRepository {
add(gamesOfTheYear: GamesOfTheYear): Promise<void>;
}
Typescript is useful here as we can focus on the messages that each layer is responsible for. First of all, each layer is returning a void, but it is a void wrapped in a Promise. This means that though we aren't returning anything, there are two signals that our layers are passing up.
- The task is complete.
- The task was successful or unsuccessful.
So even though we are dealing with void, these messages have a lot of information to convey. Additionally, keep in mind that each layer is calling the one underneath it, so they are also implicitly responsible for sending off that message as well.
Now let's turn our attention to the message that each interface receives:
Controller
Our controller is receiving a message with ObjectMetadata, which is the type defined by Firebase that we talked about previously.
Service
Our service takes a Game array message. What is this Game type? We haven't defined it yet, but it may be reasonable to define it so it matches our CSV file.
So:
game,publisher,year,rating
Slay The Spire,Mega Crit Games,2019,89
Luigi's Mansion,Nintendo,2019,86
Becomes:
interface Game {
game: string;
publisher: string;
rating: number;
year: number;
}
Repository
Moving on to our repository, which takes another new type, GamesOfTheYear. This layer is where we need to start thinking about what we want to store in our database. We want a set of years, which each year having multiple games attached to it:
export interface GameForYear {
game: string;
publisher: string;
rating: number;
}
export interface GamesOfTheYear {
years: {
[key: number]: GameForYear[];
};
}
Above I am using a plain old Javascript object, which Firestore refers to as a map as our set. This set allows us to maintain an association between a year and the games that came out during it.
So why not just store our Game type directly into the database? Keep in mind; we want to be able to return a single document to our UI that has all the information we need. So while we could store an array of Game's into our Firestore document, our UI is now going to be responsible for parsing all those rows into an object that makes sense to display. Let's save ourselves (or someone else) some work and have our Serverless Function store this data in a format that is going to be seamless for our UI to work with.
Testing and implementing
Controller
Let's start at the top and think of what our controller is responsible for,
- Taking in a CSV
- Telling our service layer to add those rows in that CSV
Let's drive that functionality with a test:
// For the purposes of these examples, I am using Mocha Chai for testing and SafeMock for mocking.
it("adds a GamesOfTheYear", async function() {
// GameOfTheYearService is my interface we defined above
const mockGamesOfTheYearService = SafeMock.build<GamesOfTheYearService>();
when(mockGamesOfTheYearService.add).resolve();
// CloudStorageGamesOfTheYearController is my implementation of that interface
const controller = new CloudStorageGamesOfTheYearController(
mockGamesOfTheYearService
);
// since we are returning a void there is nothing to capture here
// but we do want to ensure the method to be done before asserting, hence await
await controller.create(
createMockStorageObject({
bucket: "bucket",
name: "filename"
})
);
const expectedGames = [
{
game: "Slay The Spire",
publisher: "Mega Crit Games",
year: 2019,
rating: 89
},
{
game: "God Of War",
publisher: "Sony Interactive Entertainment",
year: 2018,
rating: 95
}
];
verify(mockGamesOfTheYearService.add).calledWith(expectedGames);
});
// helper to create the message coming from the Firebase API
const createMockStorageObject = (
overrides: Partial<ObjectMetadata> = {}
): ObjectMetadata =>
({
bucket: "test-bucket",
contentType: "text/csv",
name: "test-file",
metadata: {
templateName: "test-template"
},
...overrides
} as ObjectMetadata);
This result is not too dissimilar from how the tests for an HTTP based controller would look in a traditional backend.
There is a wrinkle here; the Firebase API is only going to give us a filename. It is up to us to download the file to whatever instances this function is running on. We could say this unit of code is also responsible for downloading as well. However, due to a dependency on the Firebase APIs, we would then need to bring in an external dependency whenever we ran this test. How can we avoid this?
What if we followed the approach we have taken so far and put that functionality behind an interface? Essentially, we segment the difficult to test behavior away from the current unit we are testing.
export interface FileService {
// download takes in where it needs to download from and returns a string containing the downloaded file path
download(location: string, filePath: string): Promise<string>;
}
Now we can adjust our tests and add in one more mock to our setup, right before we create our controller:
const mockFileService = SafeMock.build<FileService>();
when(mockFileService.download("test-bucket", "test-filename")).resolve(
"./tests/GamesOfTheYear/fixtures/two-games.csv"
);
const controller = new CloudStorageGamesOfTheYearController(
mockFileService,
mockGamesOfTheYearService
);
This mock looks a little different from the mockGamesOfTheYearService we did previously. In this case, we are taking in two parameters and return our filename if, and only if, these two parameters match. To make sure that happens, let's change the ObjectMetadata that we are invoking our controller with to one that matches our mock:
await controller.create(
createMockStorageObject({
bucket: "test-bucket", // previously "bucket"
name: "test-filename" // previously "filename"
})
);
We also need to add a local file fixture, two-games.csv:
game,publisher,year,rating
Slay The Spire,Mega Crit Games,2019,89
God Of War,Sony Interactive Entertainment,2018,95
This file is loaded from our local filesystem and used to ensure that our function can load a file and read it.
Our tests are now in an excellent place to drive our implementation. Let's run our tests first to make sure we haven't messed something up and that we have a red test.
FirebaseGamesOfTheYearController 1) adds a GamesOfTheYear
0 passing (15ms)
1 failing
FirebaseGamesOfTheYearController
adds a GamesOfTheYear:Error: add was not called with: (Array[{"game":"Slay The Spire","publisher":"Mega Crit Games","year":2019,> "rating":89},{"game":"God Of War","publisher":"Sony Interactive Entertainment","year":2018,"rating":95}])
- expected - actual -[] +[ - [ - { - "game": "Slay The Spire" - "publisher": "Mega Crit Games" - "rating": 89 - "year": 2019 - } - { - "game": "God Of War" - "publisher": "Sony Interactive Entertainment" - "rating": 95 - "year": 2018 - } - ] +]
Awesome! Our test is failing exactly like we want.
Let's implement our controller:
// 'cvstojson' is an NPM module to load our CSV
import * as csvtojson from "csvtojson";
import { FileService } from "./FileService";
import { GamesOfTheYearService } from "./GamesOfTheYearService";
import { ObjectMetadata } from "firebase-functions/lib/providers/storage";
export class CloudStorageGamesOfTheYearController
implements GamesOfTheYearController {
constructor(
private readonly fileService: FileService,
private readonly gamesOfTheYearService: GamesOfTheYearService
) {}
async create(objectMetadata: ObjectMetadata): Promise<void> {
const filePath = await this.fileService.download(
objectMetadata.bucket,
objectMetadata.name!
);
const rows = await csvtojson({ checkType: true }).fromFile(filePath);
await this.gamesOfTheYearService.add(rows);
}
}
If we re-run our tests they should now pass!
You'll notice I needed to refer to objectMetadata.name with a !. This is due to the type being possibly undefined. To me, this suggests we have another case to handle with a test. Going higher level, I want to handle the remaining cases in my tests:
- If the
objectMetadatadoesn't contain a name, I should throw a clear error - If the file given to my controller is CSV, I should throw another error
These are both error cases. So far, we have only covered our "Happy Path," which is when the function gets everything it needs, and everything goes according to plan. What makes programming difficult is usually handling error cases, and as I mentioned before, Unit Tests are great at that. If we go ahead and test those two scenarios and implement them, we end up with the following:
// tests
describe("FirebaseGamesOfTheYearController", function() {
before(function() {
// using the library 'chai-as-promised' to help with better assertions
chai.use(chaiAsPromised);
});
it("adds a GamesOfTheYear", async function() {
// snip, this is the same as above
});
it("throws an exception if the file is not a csv", function() {
const notCsvObject = createMockStorageObject({
contentType: "not-csv"
});
const controller = new CloudStorageGamesOfTheYearController(
SafeMock.build<FileService>(),
SafeMock.build<GamesOfTheYearService>()
);
return expect(controller.create(notCsvObject)).to.be.rejectedWith(
"File type is not csv"
);
});
it("throws an exception if there no filename", function() {
const noNameFileObject = createMockStorageObject({
name: undefined
});
const controller = new CloudStorageGamesOfTheYearController(
SafeMock.build<FileService>(),
SafeMock.build<GamesOfTheYearService>()
);
return expect(controller.create(noNameFileObject)).to.be.rejectedWith(
"Can not resolve filename"
);
});
});
// full implementation
export class CloudStorageGamesOfTheYearController
implements GamesOfTheYearController {
constructor(
private readonly fileService: FileService,
private readonly gamesOfTheYearService: GamesOfTheYearService
) {}
async create(objectMetadata: ObjectMetadata): Promise<void> {
if (!objectMetadata.name) {
throw new Error("Can not resolve filename");
}
if (objectMetadata.contentType !== "text/csv") {
throw new Error("File type is not csv");
}
const filePath = await this.fileService.download(
objectMetadata.bucket,
objectMetadata.name
);
const rows = await csvtojson({ checkType: true }).fromFile(filePath);
await this.gamesOfTheYearService.add(rows);
}
}
All our tests should now be green. At this point, we have a complete controller. There are two ways we can go now, either implement FileService or GameOfTheYearService. Let's do the latter first.
Service
We would test-drive our service exactly like we did our controller and end up with the following:
Tests:
describe("GamesOfTheYearServiceImpl", function() {
it("add saves games of the year", async function() {
const mockRepository = SafeMock.build<GamesOfTheYearRepository>();
when(mockRepository.add).resolve();
const gamesOfTheYearService = new GamesOfTheYearServiceImpl(mockRepository);
await gamesOfTheYearService.add([
{
game: "Luigi's Mansion",
publisher: "Nintendo",
rating: 86,
year: 2019
},
{
game: "Red Dead Redemption 2",
publisher: "Rockstar Games",
rating: 96,
year: 2018
}
]);
const expectedGamesOfTheYear = {
years: {
2019: [
{
game: "Luigi's Mansion",
publisher: "Nintendo",
rating: 86
}
],
2018: [
{
game: "Red Dead Redemption 2",
publisher: "Rockstar Games",
rating: 96
}
]
}
};
verify(mockRepository.add).calledWith(expectedGamesOfTheYear);
});
it("sorts games of the year by score", async function() {
const mockRepository = SafeMock.build<GamesOfTheYearRepository>();
when(mockRepository.add).resolve();
const gamesOfTheYearService = new GamesOfTheYearServiceImpl(mockRepository);
await gamesOfTheYearService.add([
{
game: "Luigi's Mansion",
publisher: "Nintendo",
rating: 86,
year: 2019
},
{
game: "Resident Evil 2",
publisher: "Capcom",
rating: 92,
year: 2019
}
]);
const expectedGamesOfTheYear = {
years: {
2019: [
{
game: "Resident Evil 2",
publisher: "Capcom",
rating: 92
},
{
game: "Luigi's Mansion",
publisher: "Nintendo",
rating: 86
}
]
}
};
verify(mockRepository.add).calledWith(expectedGamesOfTheYear);
});
});
Implementation:
import _ = require("lodash");
export class GamesOfTheYearServiceImpl implements GamesOfTheYearService {
constructor(private readonly repository: GamesOfTheYearRepository) {}
private static reduceYears(result: GamesOfTheYear["years"], game: Game) {
const key = game.year;
const years = result[key] || (result[key] = [] as GameForYear[]);
years.push({
game: game.game,
publisher: game.publisher,
rating: game.rating
});
return result;
}
async add(games: Game[]): Promise<void> {
await this.repository.add({
years: _(games)
.orderBy("rating", "desc")
.reduce<GamesOfTheYear["years"]>(
GamesOfTheYearServiceImpl.reduceYears,
{}
)
});
}
}
Our business logic right now is a simple data transformation. Unlike our controller, there are no other responsibilities to segment out in our business logic, so our tests are self-contained.
The bits that are harder to test
FileService
Now let's go back up a level to our FileService that we need for our controller. This is where we get into parts of our Serverless Function that become difficult to test in isolation. Since we don't currently have a good way to test drive this implementation with a Unit Test, let's make a concession and say FileService won't be tested:
import * as admin from "firebase-admin";
// tmp is the 'tmp' npm library
import * as tmp from "tmp";
export class CloudStorageFileService implements FileService {
public async download(location: string, filePath: string): Promise<string> {
const bucket = admin.storage().bucket(location);
const tempLocalFile = tmp.tmpNameSync();
await bucket.file(filePath).download({ destination: tempLocalFile });
return tempLocalFile;
}
}
It is kind of a bummer that we can't test this file due to the entanglement with Firebase.
We could mock out that admin import, but at that point, we are diving pretty heavily into the anti-pattern of Mocking What You Don't Own. Put succinctly; we are going to make guesses of how this code we don't own, admin, is implemented to test it. If we make the wrong guess, our test is useless, and if Google ever changes that code to make our previous guess wrong, our test is not going to tell us we have something wrong.
This is a situation where we want to integrate test this piece of code, but let's leave that for another blog post and another day.
One thing that makes me feel pretty good about not testing here is the lack of any real logic. It has no branching paths or complex behavior. There are two ways this code can behave,
- It works.
- Or it blows up due to an environmental issue.
Since we kept the interface verbs super simple, the resulting implementation is simple as well. If we have code that we can't test, this is the approach we want to take. We want to move complexity and branching logic out of untested code and into a testable layer.
Once we deploy this code, this function will either not work or work, and if it works, it will continue to work due to lack of logic. That one successful manual run we perform with cover all the logic paths.
Repository and index.ts
To implement our remaining functionality, we once again hit a situation where we can't Unit Test. Let's try and follow the example we set with our FileService and keep branching logic to a minimum.
export class FirestoreGamesOfTheYearRepository
implements GamesOfTheYearRepository {
async add(gamesOfTheYear: GamesOfTheYear): Promise<void> {
await admin
.firestore()
.collection("gamesOfTheYear")
.add(gamesOfTheYear);
}
}
Mission accomplished, there is even less logic in our repository than in our FileService.
Our index is equally as simple.
import * as functions from "firebase-functions";
import { FirestoreGamesOfTheYearRepository } from "./GamesOfTheYear/GamesOfTheYearRepository";
import { GamesOfTheYearServiceImpl } from "./GamesOfTheYear/GamesOfTheYearService";
import { CloudStorageGamesOfTheYearController } from "./GamesOfTheYear/GamesOfTheYearController";
import { CloudStorageFileService } from "./GamesOfTheYear/FileService";
import { ObjectMetadata } from "firebase-functions/lib/providers/storage";
import * as admin from "firebase-admin";
// Initialize our Firebase config
admin.initializeApp();
// Construct our dependencies
const gamesOfTheYearRepository = new FirestoreGamesOfTheYearRepository();
const gamesOfTheYearService = new GamesOfTheYearServiceImpl(
gamesOfTheYearRepository
);
const fileService = new CloudStorageFileService();
const gamesOfTheYearController = new CloudStorageGamesOfTheYearController(
fileService,
gamesOfTheYearService
);
// Register our function with the Firebase APIs
export const importGamesOfTheYear = functions.storage
.object()
.onFinalize((objectMetadata: ObjectMetadata) =>
gamesOfTheYearController.create(objectMetadata)
);
You can go and deploy our function to Firebase using npm deploy. After deploying, upload a CSV file to your project's default storage bucket, and you should then see a new entry in your database representing the transformed file.
And there we have it; a simple but tested Serverless Function. We've kept all our logic isolated from the bits we can't test yet, and there is an excellent chance that this works the first time we deploy it.
As you develop code in the future using this approach, I think you'll find that if something were to go wrong, it would most likely be a misusing of an external API in untested code. I'd love to have some quicker feedback on that as well, so stay tuned for a future blog post exploring Integration Testing strategies with Firebase.
Top comments (0)