DEV Community

Cover image for Low Cost "Overkill" AWS Infrastructure for a Newborn Startup

Low Cost "Overkill" AWS Infrastructure for a Newborn Startup

Nicolas El Khoury on March 28, 2023

Introduction On a cold and dark evening in December 2022, a good friend of mine calls me and says: "Nicolas, I am creating a product tha...
Collapse
 
cassioscofield profile image
Cássio Scofield

Using EKS on a app with only 2 components and 2 people working on it definitely seams like overkill to me. Any reason why you didn't opt to run it on something like Beanstalk or Heroku?

Collapse
 
devopsbeyondlimitslb profile image
Nicolas El Khoury

Definitely, for many reasons actually:

  • The 2 components Will become 20 very rapidly.
  • Kubernetes deployment files are highly portable, across any cloud provider or on premise.
  • Kubernetes is much better to use than EB and Heroku.
  • Team expertise taken into account.
Collapse
 
bugb profile image
bugb

ECS seems better

Collapse
 
nicklarsennz profile image
Nick

ECS still doesn't support ConfigMaps (well, the equivalent) which quickly becomes a nuisance.

Thread Thread
 
bugb profile image
bugb

@nick well you have many choices: AWS secret manager or use env file from s3: docs.aws.amazon.com/AmazonECS/late...

Collapse
 
dnitsch profile image
dnitsch

I'm surprised client VM (+ VPN) is still a thing, with session manager allowing RDP as well. Much more secure and simpler thing would be to have proper IAM controls around session manager with some "client VM" like AMIs.

Collapse
 
devopsbeyondlimitslb profile image
Nicolas El Khoury

How do you provide direct access to private resources for developers?

Collapse
 
dnitsch profile image
dnitsch

Via the AWS console. you can have a Linux or Windows VM (whatever the prefer) with desktop clients like workbench/pgadmin/kubectl. The Dev eks doesn't really need to have a private endpoint unless it's a specific requirement, and provided you haven't networked Dev and prod VPCs, as Kubectl is easier to have locally as well as using docker for MySQL/postgresql, thus guiding developers to develop everything as code.
Give fleet manager/session manager a go. The "client VMs" don't need to be in the public subnet either, see how you could use it for your use case, it does make like easier.

Collapse
 
brianmcbride profile image
Brian McBride

I bet if you used Google Cloud. Cloud Run for the service. Something like Firestone for the DB. You can get crazy low bills, like sun $50/m and have just as much power.

GCP has the best free tier. If cost is a major factor.

Collapse
 
augustosamame profile image
Augusto Samamé • Edited

Each time I add codebuild and codepipeline into the mix my deploy times go through the roof. I always think I may be doing something wrong. I come from a RoR background and my Capistrano deploys take all of 30 seconds. Can you share how long it takes from the time code is commited in repo, to server with new code being up and running? With codepipeline and ecs this always takes about 10 minutes for me, even with dead simple apps which IMO is unacceptable.

Collapse
 
devopsbeyondlimitslb profile image
Nicolas El Khoury

Hi Augusto,

Do you suffer from a delay before the pipeline starts? or the build and deployment times take 10 minutes from the moments the pipeline starts execution?

Collapse
 
augustosamame profile image
Augusto Samamé

Delay is on the pipeline / codebuild side. Basically it takes a LOONG time for codepipeline to spin up the new code version Fargate ECS containers into the cluster, have them as available, and then remove the previous code versions.
I realize you're not using Fargate but EKS. However, I'd like to know in your opinion, what would an "acceptable" deploy time for containerized deploys be, as compared to more traditional deploy methods.

Thread Thread
 
devopsbeyondlimitslb profile image
Nicolas El Khoury • Edited

@augustosamame I think I know what the problem is, if the ECS tasks are part of a Target Group, navigate to this Target Group and modify the de-registration delay to 10 seconds or an even smaller number.

This will reduce the time.

Collapse
 
amitkad profile image
Amitkad • Edited

I like this post a lot!!
Would be happy to see a tutorial of all that.
It can be gold for any learner or tech lead, like your friend, who wants to create a
Reliable infrastructure to his team and product.

Collapse
 
devopsbeyondlimitslb profile image
Nicolas El Khoury

Thank you for your kind words @amitkad. I already have a free Udemy course (Introductory course). Please check it out and leave a feedback: udemy.com/course/intro-fullstack-d...

I hope to create more tutorials in the future.

Collapse
 
christiankaseburg profile image
christiankaseburg • Edited

Great write up. You said they have a mono repository using nx for the build system. Do you mind sharing some details of how you structured the IAAC with the AWS CDK for the mono repo? I have a project that uses a mono repo and NX as well and I have been struggling to figure out what a good approach is for separating my CDK code from my application code. My current approach now is having an infra/ directory at the root of the repository with multiple nested folders for my different infra to keep my build and deployments separated. E.g., infra/frontend and infra/backend.

Collapse
 
devopsbeyondlimitslb profile image
Nicolas El Khoury

Hi @christiankaseburg,

I don't think there is one solution that fits all. I started out similar to your method, by including a folder in the root directory containing subfolders and files.

But eventually, you would want to separate this from developers to avoid unwanted changes or errors.

In my opinion à quick solution would be to either create a separate repository for it, or store it in some file storage such as S3.

Have the pipeline download it before executing commands.

I hope this answers your question.

Collapse
 
dbones profile image
Dave R. • Edited

Interesting read, if I read correctly, scale and possible future complexity are the main drivers.

Did you use the serverless RDS? And fargate eks? Possible reduced cost and complexity

Could ECS be used, or do they need an exit strategy off Aws?

Could scaleway (same tech, postgres and k8s) be used to reduce costs?

(Just wondering)

Collapse
 
devopsbeyondlimitslb profile image
Nicolas El Khoury

Hi Dave! How are you?

Yeap, you're right, the main driver behind this infra is to minimize headaches in the future.

Yes I used Aurora. No I did not use Fargate Eks, but definitely worth investing in it.

I avoided ECS for many reasons, especially lock in, and lack of portability. Once the application is stable on kubernetes, I can easily redeploy it anywhere (such as on premise)

Collapse
 
_lost_in_code profile image
Lost Stranger • Edited

Do you have cost breakdown of this solution? 500$/month seems reasonable but looking at the picture, it looks quite expensive. I am wondering how much cost comes from each component? Are there usage charges like egress bandwidth etc.?

Collapse
 
theswaminator profile image
Nikhil Swaminathan

Did you try to Amplify Hosting?

Collapse
 
devopsbeyondlimitslb profile image
Nicolas El Khoury

Unfortunately, Amplify will not meet all the requirements stated in the article.

Collapse
 
adeelamin15 profile image
Adeel Amin

What did you end up using for observability? Essentially the tracing part.

Also did you authored any IaC for infrastructure orchestration?

Collapse
 
devopsbeyondlimitslb profile image
Nicolas El Khoury

Hi Adeel,

Cloudwatch for Metrics. Tracing was not implemented yet, as it needs development effort. But in my opinion, either use X-Ray, or a custom built mechanism that integrates with Cloudwatch logs.

Collapse
 
irwansyahwii profile image
Irwansyah

Thank you! Any pointer on how to learn Kubernetes in the fastest way?

Collapse
 
devopsbeyondlimitslb profile image
Nicolas El Khoury • Edited

Hi,

There are multiple courses and tutorials. However, in my opinion, Kubernetes is an advanced and complicated tool. The only way to learn is to actually be involved in a large project that uses Kubernetes heavily and endure all the headaches and head-banging until everything makes sense :))) That's how I learned it. Painful, but worth it

Collapse
 
nisam360 profile image
nisam360

Nicely written.
How front end communicating with back end? Are you using API gateway?

Collapse
 
devopsbeyondlimitslb profile image
Nicolas El Khoury

Hi Nisam,

The backend deployed on Kubernetes is fronted by an internet-facing ALB. The frontend application communicates with this ALB, which in turn redirects the traffic to the backend service.