When it comes to access, a recent trend observed is having a self-serve tooling platforms for engineering and non-tech teams. This is to facilitate debugging during production issues, run background jobs or sometimes take a database backup by the said teams themselves and/or compliment the SRE teams.
Now, IAM - Identity Access Management was spawned with similar problem statements in mind and apparently proves useful. But while managing its infrastructure at scale (think of orgs with 250+ people across development, product, data-science, design etc..), it can be a constant hassle. This not only slows down genuine feature building but can cause friction among teams if there's too little or too much process around it.
While most people are careful with their privilege when it comes to production but it's only a matter of time before one right command on the wrong environment (unintentional or not) can cause all sorts of panic, ad-hoc debugging and restoration to fix arising outages.
There are concepts like least privilege, single sign-ons, SSH keys, and RBAC that are battle tested and newer ones like Zerotrust that might end up creating more problems than they solve, if used incorrectly. On the other hand, denying access proves to be counter productive since teams need to rely on others to get vital information they could've obtained themselves.
Therefore the million dollar question that comes up is:
How do you think Dev/Design/Product Teams should access the Infrastructure?
Top comments (4)
For AWS, I think teams should create multiple AWS accounts for each environment (i.e dev, prod, uat) and use AWS Organizations to put all of that into one managed umbrella. After that, dev teams should only have access to dev and uat so they can debug the application. The ops team (or the more senior devs for a DevOps-centric company) are the only ones with access to the prod AWS account.
This way, access is self-contained to the AWS account level. Even at the AWS account level, a lot of things can be done. So for this, we can apply the principle of least priv so we only give access to what AWS services the developer needs to access (nothing more)
Thanks for the valuable inputs Raphael, very helpful.
Separating access to dev/UAT & production based on seniority sounds a bit off-track to me as it'd leave junior colleagues with little understanding of prod architecture & issues + become a blocker if none of the seniors are available.
It can also lead to teams starting up resources in dev/UAT & miss on cleaning them up later, creating high bills in AWS/GCP.
What do you think is a good approach to solve this?
Hi Vinay, I do think this multi-account structure can be improved upon by adding tags on resources and giving devs write permissions for those tagged resources.
PRODUCTION
DEV / QA
Resource: docs.aws.amazon.com/IAM/latest/Use...
The suggestion of using tags & the shared documentation certainly helps, thank you Raphael!