Until recently, like many of us, the first thing I used to do in the morning would be to check my mobile phone. I realised that it was taking about 20 mins to half an hour of my morning time. So I stopped doing it and replaced it with keeping my eyes closed and meditating instead for 15-20 mins. But the incident I'm going to tell you about happened when I still used to look up for my mobile phone first thing in the morning, in half asleep state, with one eye open and the other closed, hid behind the pillow contour.
On 3rd September, 2020, I woke up to an automated email from Google Cloud Platform. First response – shock. It was a bill of $377. That amount meant a lot because I didn't earn much in INR, and it was a significant portion of my salary. Without even opening my Google Cloud console, I started a chat with Google Cloud support. In about a couple of hours, they told me in detail what I was billed for.
I was billed for a Google Cloud project which had used Google Maps JavaScript APIs for over 2 months. I was so sure that I had done nothing wrong that I started telling this to the customer support over emails and chat messages. But they kept reiterating that my project had infact used Google Maps JavaScript APIs, for which I was charged.
I worked for Goibibo at the time. Goibibo is one of the largest online travel agencies in India. Two months before this happened, I was working on the hotels details page's Google Maps project — you know — to show the location of the hotel in the map, and all the points of interest around that hotel, and the ability to search for any location in reference to the hotel.
I read a partiular line in one of the Google Cloud support communication emails:
Just a quick disclaimer, we cannot reveal domain information if the usage is from a domain outside of your authorized ones due to our privacy and security policy.
And then it hit me.
Goibibo's google cloud account didn't have localhost added to their whitelist of allowed domains. So it was impossible to for me to develop locally. To avoid the round trip of having to find the required person, and request them to add localhost to the whitelist, and then start developing, I created a google cloud project from my personal account and started developing locally. I created an API key and enabled Google Maps JavaScript APIs against that key in my project. Here's how I kept both of the Goibibo's and my API keys in the code while developing:
/* Only use this key for production, comment the one below */
// export const googleMapsApiKey = `<goibibo account's api key>`;
/*
* WARNING: Do not push the key to production. Is only meant
* for development purposes only
*/
export const googleMapsApiKey = `<my personal account's api key>`;
I frantically checked the code in the main branch to see if I had accidetally forgotten to follow my own advice. That was in fact the case, and I did a big: 🤦🏽♂️
I git blamed the file to find the pull request which had enabled this blunder. Then I sent its screenshot to both the google cloud support, apologizing for an honest mistaken, and to my team's slack channel to blare loudly at them for not having done a decent code review.
I had also forgotten to restrict my personal account's API key to allow requests to Google Maps APIs from only localhost domain.
That meant that
2 months worth of Goibibo's Details Page traffic was charged against my personal account's project. Good thing that we had lazy loaded map, which meant that all the JavaScript and other files necessary to render google maps was loaded only when the user had scrolled down to the location section. It'd have cost a lot more, if the resources loaded as soon as the details page loaded.
Had someone extracted that API key by watching the API request from browser developer tools, they could have potentially bombarded requests to Google Maps APIs billed against my personal account from any domain.
I didn't detect this blunder for a month, because for the first month, google had used the $100 free credit that I had gotten from somewhere. So in the first month's billed it showed $0. It was only a month later that that month's $277 showed as amount charged on my credit card.
On multiple requests the Google Cloud team refunded that amount after taking confirmations from me that I had read and now understood all the ways to restrict access of api keys, and to set billing alerts on google cloud projects. After giving multiple assurances and acknowledgements that I had indeed read up all the docs that taught how to do all these things, I found peace. 🕊✌🏼☮️
Top comments (2)
Nice learning experience and breakdown of your process/approach!
Thank you! :)