DEV Community

I broke login for 80k users

rhymes on September 10, 2018

Let me start by saying that I'm not happy about this :-D The following is a post mortem of carelessness, naivety and bad changelogs and upgrade gu...
Collapse
 
realshadow profile image
Lukáš Homza • Edited

Let me tell you a story about breaking the login page. About 10 years ago, two of my colleagues at that time were supposed to delete around 10 test users from database, we had no clue about soft deletes (nor foreign keys for that matter) at that time so they decided to do a serious delete... I am sure you know where this is going. Both of them stared at the screen and confirmed multiple times that the delete clause is bullet proof, hit execute. And deleted 50k users from database!

Now here comes the fun part, the ONLY backup we had at that time was 3 days old on my laptop. So in the end they still partially saved the day and we lost 3 days worth of new users

Collapse
 
rhymes profile image
rhymes

ahahahah two human beings being sure that the change is safe. That's definitely a sort of an "echo chamber" right there :D

Collapse
 
buinauskas profile image
Evaldas Buinauskas

And that's exact reason why you run select first with the same condition to see what will be deleted. 🤔

Collapse
 
sethusenthil profile image
Sethu Senthil

Yeah, this is why I stick with firebase!

Collapse
 
rhymes profile image
rhymes • Edited

Firebase wasn't even around when we implemented that :-D

At the time there were also a series of issues about ownership and privacy of user's data.

Collapse
 
yorodm profile image
Yoandy Rodriguez Martinez

sights this youngsters and their services in "the cloud"....

Thread Thread
 
rhymes profile image
rhymes

what do you mean?

Thread Thread
 
yorodm profile image
Yoandy Rodriguez Martinez • Edited

Just remembering an anecdote not unlike yours, when a friend deleted our whole user data. I was talking to some junior developers about it and one of them looked at me and said: "Why didn't you just kept your users in our PaaS service like we do now?"

Collapse
 
kyslik profile image
Martin Kiesel

Got me cracking right here:

Note: this is many weeks after the first complaints. August is a dead month in Italy. The mobile team was on holiday, the system integrator was on holiday, my client's client was on holiday. Probably also the poor users were on holiday and trying to use the app from there.

Thanks, what a day maker.

Collapse
 
rhymes profile image
rhymes

:-D

Collapse
 
vkvikaskmr profile image
vkvikaskmr

Hi Rhymes! I just read your post here : github.com/doorkeeper-gem/doorkeep...
I have some followup questions though. You mentioned that the android app is not sending the client secret. In what situations is that possible? Are you using implicit grant type in this case?

Collapse
 
rhymes profile image
rhymes

I'm sorry, I'm not familiar with the app internals anymore. I don't remember how that happened :)