Get Mail Nyow! 📩
Since our original launch of inboxkitten.com, a free opensource serverless project for creating a disposable email service...
The 🐈 kittens have multiplied and gotten out of hand...
- 3rd on product hunt
- February bill shock of: $143
- 221 GB of static website data (this is from cloudflare cache miss)
- 215 GB of email data
- 77 Million API requests
- 20k+ Unique Visitors (according to cloudflare)
- Depletion of my steam summer sale funds 😭 (that $143 got to come from somewhere)
Ok cat team, we need a plan to protect the summer sale! 💳
One of the key limitations of existing AWS lambda or GCP/Firebase cloud functions is that it limits 1 request to an application instance at any point of time.
While that is awesome, if you are compressing images, or doing complex stuff.
Inboxkitten API only role is to make HTTP requests to mailgun, which holds the actual email, with the API key, and echo the result back.
As a result our resource consumption is well below 1% of a CPU, or <10MB of ram per request. Well, below the minimum of 128MB ram and dedicated CPU per request.
So our original plan was that once these kittens grow in very constant load, would be to toss it onto a $5/month Linode Nanode 1GB instance.
However, the downside is that it wouldn't "scale" automatically at peak workload or if it ever goes to a gazillion request a month, which would be beyond one instance. (Not that we would ever need to support such workload)
So in keeping to the spirit of serverless fun (which made this project popular), we would put that option aside and ask...
What if, there was a serverless platform, with an alternative billing model. What if it bills only per request, or only for the amount of CPU and ram used. What if we ...
Use cat Cloudflare workers ☁️
Cloudflare worker, which is part of the growing trend of "edge" serverless computing. Also happens to simplify serverless billing down to a single metric.
Turning what was previously convoluted, and hard to decipher bills from GCP...
| Billable Item | Usage | Unit | Total Cost | Avg price (per million invocations) |
|---|---|---|---|---|
| Invocation | 77,418,914 | invocation | $30.17 | $0.390 |
| CPU Time | 4,677,797 | GHZ-second | $44.78 | $0.578 |
| Outgress | 215.12 | Gibibyte | $25.21 | $0.326 |
| Memory Time | 2,923,623 | Gibibyte second | $6.31 | $0.082 |
| Log Volume | 61.16 | Gibibyte | $5.58 | $0.072 |
| Total: | $112.05 | $1.447 |
Into something much easily understood, and cheaper overall per request ...
| Billable Item | Usage | Unit | Total Cost | Avg price (per million invocations) |
|---|---|---|---|---|
| Invocation | 77,418,914 | invocation | $39 | $0.5 |
| Total: | $39 | $0.5 |
😸 That's enough net savings for 7 more $9.99 games during summer sale!
And has a bonus benefit of lower latency edge computing!
But, there are catches ...
1) <5ms CPU time limit
Each request is limited to < 5ms of CPU time, this differs greatly for the request time (also known as wall clock time), in that it only counts the amount of time the CPU spent on our function itself and ignoring all its sleep/wait time.
This is as opposed to how GCP or AWS measure the time taken from start to end of the function, including all sleep/wait time.
In such a setup, serverless functions which fold DNA strands or cat images, with large CPU or RAM consumption. Will find Cloudflare unusable.
However, our serverless functions spend 99.99% of the time waiting for mailgun API to respond. Making the <5ms limit purrfect.
Also if we ever need to upgrade this to <10ms, it is an optional addon.
2) Incompatibility with express.js (as it uses web workers pattern)
Another thing to note is that Cloudflare workers are based on the web workers model, where it hooks onto the "fetch" event in cloudflare like an interceptor function.
So instead of the following in express JS (as we currently do for firebase)
const express = require('express')
const app = express()
app.get('/', (req, res) => res.send('Hello World!'))
const port = 3000
app.listen(port, () => console.log(`Example app listening on port ${port}!`))
In Cloudflare, it would be the following
addEventListener('fetch', event => {
event.respondWith(fetchAndApply(event.request))
})
async function fetchAndApply(request) {
return new Response('hello world')
}
As this is a huge fundamental change in the code structure. It can be a major no go for larger existing projects, due to the sheer amount of rewrite work involved. Despite the extremely close similarity.
Perhaps one day someone will come up with a Cloudflare to express.js adapter?
But for us, due to the simplicity of the project, it would be just a simple rewrite. You can compare the difference in the code between express js version here and the Cloudflare version here on github.
And while workers is an open standard, with Cloudflare being the only major provider, it is currently also a form of vendor lock-in.
3) one script limitation per domain (unless you are on enterprise)
While not a deal breaker for inboxkitten, this can be one for many commercial / production workload.
Because Cloudflare serverless packages cannot be broken down into smaller packages for individual subdomains and/or URI routes.
This greatly complicates things, making it impossible to have testing and production code separated on a single domain, among many more complicated setups.
However in our use case, as this is a hobby project.... it doesn't matter...
The scratches in summary...
- <5ms CPU time limit
- Incompatible with express.js
- one script limitation per domain
And all it took was a quick one day rewrite, by @jmtiong, and we are done.
That's cool, but Why do I even need Inboxkitten disposable email for again?
One of the key use cases currently, and why we built this project, is to perform email validations as part of our automated test scripts. Such as the following...
// Lets goto inbox kitten
I.goTo("https://inboxkitten.com");
I.see("Open-Source Disposable Email");
// Go to a random inbox inbox
I.fill("email", SAMPLE.id(22));
I.click("Get Mail Nyow!");
// Check that its empty
I.see("There for no messages for this kitten :(");
// Testing for regular email
// (sent using a jenkins perodic build)
I.goTo("https://inboxkitten.com");
I.see("Open-Source Disposable Email");
I.fill("email", "ik-reciever-f7s1g28");
I.click("Get Mail Nyow!");
// See an email we expect, nyow
I.see("Testing inboxkitten subject");
With sharable test results such as
Plus it's simple, cool, and fun to tinker around with.
Hey, how about the $31 firebase static file hosting?
The easiest free solution would be to throw the entire website onto github pages
However in the spirit of opensource...
We are gonna do a shout out for our friends from 🐑 commonshost.com, an open source static site hosting platform, being built out of Singapore 🇸🇬
And help them push their network with a real production work load test across their 22+ servers on its global network.
As to why commons host instead of GitHub... cause its cool, and I want to support the underdog of the CDN world, being an underdog of the testing world.
Oops, did I say underdog? I mean undercat 😼
What's next?
Due to the project rather unique simplicity (quick to rewrite), and its heavy production load. I am thinking of potentially expending its deployment option to as many serverless options as possible, or docker based deployments even.
Exploring the various trade-offs with actual 24/7 production load.
Done
- GCP/Firebase: function and hosting
- Cloudflare workers
- Commonshost hosting
Todo
- Docker container deployment
- ECS Fargate
- Digital Ocean Kubernetes
- AWS lambda
- Others??
Let's see where this cat ship will sail next... Till then we will be taking a cat nap
Top comments (7)
Nice writeup and great usage of CF workers, hot tip - you can use CF Key-value database to store the static files!
Thanks, @tmikaeld we did actually consider using KeyValue or alternatively using the Cache API - though it would very likely double our bill.
For a more commercial production use case, having such edge like performance could be well worth the cost. For a hobby project, not as much haha 😅
Currently, we even limit the workers to strictly the required endpoints required, to cut out the random API scans, from bots, etc. As from the firebase logs, we realized it can be quite sizable, due to the site's popularity in Russia, China, and the USA.
That being said - for email response body, that is something we definitely want to do next! Less so for performance or cost... But to do so for fun!
Yeah, I realized after i wrote it that you'd still hit the Workers and incur a cost for it, so normal hosting would be cheaper.
Though if you get creative. There might be some possible Rube-Goldberg level of cost-cutting that might be possible. Something one should never do in production for code maintainability (maybe?)
In general, our API functions in 2 major step - listing of emails, and reading of the email content, with separate API calls. And due to its workflow, its always in that sequence.
From the documentation of Cloudflare VCL and Cache API
What we can do is split the API routing into
So that during the initial "listing of emails" - what we can do is to fill in the Cloudflare cache for the subsequent "reading of email" API call. For the top 10 emails in the list (if any).
And when the subsequent "reading of email" is called on the user click, it would hopefully be reading from cache, saving a single call hit... And if it misses, it would simply fall back to firebase.
But like I said, it's a crazy rube-Goldberg. And would only technically work cause there is a reasonable chance of hitting the same Cloudflare node due to the magic of http2.
And that's assuming it works lol - would be amused to see someone experiment with such setups.
You mention DigitalOcean Managed Kubernetes: have you tried it?
Nope, though I am interested in doing so. As using it with autoscale (like google or aws kubernetes), is kinda "close to serverless".
Plus they have very bandwidth friendly pricing 😼
Please Fix inboxkitten that if we put space on username it will show all email, it serious problem bro