DEV Community

Cover image for ‴️ Top 6 projects for building scalable cloud-based applications πŸš€
Eden Federman for Odigos

Posted on

‴️ Top 6 projects for building scalable cloud-based applications πŸš€

TL;DR

This article lists 6 top projects for simplifying the deployment and building scalable cloud-based applications for developers. πŸŽ‰

Feel free to explore these projects, contribute to your favorites, and support them by starring their repositories.🌟

mylist


Odigos - Open-source Distributed Tracing

Monitor all your apps simultaneously without writing a single line of code!
Simplify OpenTelemetry complexity with the only platform that can generate distributed tracing across all your applications.

We are really just starting out.
Can you help us with a star? Plz? 😽

https://github.com/keyval-dev/odigos

GiveStar


1. OpenTelemetry

πŸ’‘ Collect telemetry data and forward it to analysis tools.

OpenTelemetry

OpenTelemetry is an observability toolkit that helps us collect data like traces, metrics, and logs. It is vendor and tool-neutral πŸ› οΈ meaning it can work with different Observability Backends.

What OpenTelemetry is not? 🫣

OpenTelemetry is not a monitoring backend. Many people confuse themselves between OpenTelemetry and Observability backends like Jaegar, Datadog, etc. OpenTelemetry only generates, collects, and exports telemetry data. Storage and visualization are left to other tools. πŸ“Š

OpenTelemetry sits between the code and the observability tools we use. This provides us with flexibility and makes it easier to switch tools. πŸ”„

To better understand how OTel (short for OpenTelemetry) works, take a look at the below image. πŸ‘‡πŸ»

Working of OpenTelemetry

🌟 OpenTelemetry on GitHub


2. Kubernetes

πŸ’‘ Container orchestration platform for managing applications.

Kubernetes

Kubernetes is the most important repo on the list when we are talking about cloud-native applications. How, can we skip Kubernetes right? 😜 It is the most famous orchestration platform for container-based applications and a foundation for many cloud-native tools to work.

What is Container Orchestration, and why do we even need it? πŸ€”

A container engine is sufficient for managing a few containers, but for hundreds or thousands of containers and hundreds of services running, the complexity grows huge πŸ“ˆ. Trust me, it is something I have had experience with. πŸ₯²

This is where the orchestration engine like Kubernetes comes into play.

The best way I think of Kubernetes is as a puppet master 🎭 for containers, manipulating their actions, coordinating their tasks, and ensuring they work in harmony across a cluster of servers.

Refer to the image below to get a better understanding of what I meant by puppet master.

Working of Kubernetes

🌟 Kubernetes on GitHub


3. Backstage

πŸ’‘ Unifies tools for a streamlined development environment.

Backstage Dev Portal

Backstage is an open-source project that serves as a developer portal, offering a unified view of infrastructure resources, CI/CD pipelines, and observability metrics. πŸ“Š It gives us all the building blocks to create a platform ourselves that manages our complex software development ecosystem.

It provides us with easy access to essential information without requiring in-depth tool knowledge 🧰, acting as a central hub πŸ›οΈ for company-wide development needs.

The core functionalities of Backstage are:

  • Service Catalog πŸ“‹: This is the core Backstage foundation. It allows us to track and manage all our services, apps, and pipelines in one place.
  • Software Templates πŸ“„: With the Backstage UI interface, we can easily create new services with just a few clicks or use a YAML template to add necessary resources.
  • Tech Documentations πŸ“ƒ: It adds seamless technical documentation management, ensuring teams can effortlessly keep content up-to-date, all from their code repositories.

In short, this abstracts away a lot of CI/CD and provides us with information on the company's services. πŸš€ I would say this is a must-have tool for any company that is planning to upscale and streamline their development operations.

🌟 Backstage on GitHub


4. Prometheus

πŸ’‘ System monitoring tool with time-series data.

Prometheus Monitoring Tool

Prometheus is a free open-source software application used for event monitoring and alerting. 🧐 It records real-time metrics in a time series database (allows querying the data using its own query language called PromQL).

It also provides an alerting system that allows us to define thresholds and conditions for alerts.

When certain metrics exceed or fall below defined thresholds, Prometheus can trigger notifications, helping in successfully addressing potential issues before they impact the system's health.

In short, Prometheus helps keep a close watch πŸ‘€ on our infrastructure's health and performance.

🌟 Prometheus on GitHub


πŸ”΄ GOOD TO KNOW: Notice that I've mentioned two similar projects, OpenTelemetry and Prometheus, which serve similar purposes. πŸ”„

Prometheus is primarily a system-level monitoring and alerting tool, while OpenTelemetry is focused on collecting and forwarding application-level telemetry data.

Prometheus didn't initially support OpenTelemetry metrics, but now they can work together.

For more details, you can refer to this article.


5. Cilium

πŸ’‘ Efficient eBPF-based networking for containers.

Cilium

Cilium is a free and open-source software solution that provides networking and security πŸ›‘οΈ for containerized applications.

Cilium leverages eBPF to provide high-performance networking and security and has been designed to integrate with container orchestration platforms like Kubernetes. ☸️

One important feature of Cilium is its ability to apply security rules at the network level, also allows us to encrypt and authenticate network traffic, making sure that data is transmitted securely between containers. πŸ‹

In short, Cilium secures container workloads in Kubernetes with eBPF, offering network security, encryption, and authentication.

To better understand how Cilium works at its architectural level, take a look at the below image. πŸ‘‡πŸ»

Cilium Architecture

🌟 Cilium on GitHub


6. Rook

πŸ’‘ Simplifies cloud-native persistent storage management.

Rook.io persistent storage management

This tool really is a lifesaver for anyone who is working with the Ceph storage. 😍

To understand and implement Rook, you must have a basic understanding of Ceph software-defined storage. If not, let me briefly introduce Ceph.

What exactly is Ceph? 🧐

Ceph is an open-source software storage platform that provides object storage, block storage, and file system storage in a single unified storage cluster. πŸ“¦

Rook simplifies the deployment and management of Ceph storage systems on Kubernetes.

By deploying Rook on Kubernetes, we can make Ceph storage highly available, automated πŸ€–, and simple to operate at scale.

In short, Rook turns Ceph into a fully managed service for Kubernetes and abstracts away the complexity of Ceph, making it easy to set up and manage cloud-native storage for our containers.

🌟 Rook on GitHub


Introducing Odigos

πŸ’‘Distributed tracing without code changes.

keyval-dev banner

If you are working with OpenTelemetry then this is the tool you need. ✨

Odigos automatically generates telemetry data in OpenTelemetry format to any Observability backend, without any code change. 😲

It achieves this all by the following:

  • Automated Instrumentation πŸ€–: Odigos enables automated instrumentation of applications with OpenTelemetry and eBPF, eliminating the need for code modifications.
  • Universal Observability Tool Compatibility 🀝: It smoothly integrates with various observability tools, providing comprehensive support and efficient collector management.
  • Ease of installation ✨: Odigos excels in user-friendliness, seamlessly integrating OpenTelemetry, ensuring easy installation and adaptability to specific requirements with minimal hurdles.

We've recently published an article on implementing Odigos. Read here.


If you think of any other helpful projects that I haven't covered in this article, please share them in the comments section below. πŸ‘‡πŸ»

So, that is it for this article.

Thank you so much for reading! πŸŽ‰πŸ«‘

Top comments (9)

Collapse
 
shricodev profile image
Shrijal Acharya

Love this article! πŸ’“πŸ’“

Collapse
 
kumarkalyan profile image
Kumar Kalyan

Its a great article but it actually it contains 6 tools instead of 5

Collapse
 
makwasi1 profile image
Makwasi

Great Article.

Collapse
 
annaredbond profile image
annaredbond

Fantastic cat graphic, great list!

Collapse
 
shubhadip_bhowmik profile image
Shubhadip Bhowmik

Just Amazing

Collapse
 
pinky057 profile image
ishrat

Great Article !

Collapse
 
itsdevvick profile image
Derrick

Great read. Took notes and ready to try at least one from the list.

Collapse
 
fernandezbaptiste profile image
Bap

Great piece!

Collapse
 
nora13 profile image
Nora

amazing article