What is Memphis?

#opensource #devops #github #contributorswanted

To be respectful to the title and your time, the short answer is -

Memphis is an open-sourced dev-first event processing platform,

oh, and also the city of Elvis.

Now, let’s understand more :)

The source

A quick baseline — From time to time, I will use the term data-driven/real-time/event-driven applications, which are applications that heavily interact with data like:

Collecting data
Change and maintain schema (The shape of the ingested data)
Analyze and enrich data
Understand the required next event based on the ingested data
Integrations with 3rd party applications
Performing an action based on an event and more

The struggles

As a developer who was tasked with building data-driven apps, I had some really painful struggles like:

Too many data sources to handle (Each has its requirements and API).
Different and complex schemas (For each data source or service).
Transform the schema and analyze streamed data per source.
With multiple services that produce and consume data, after a certain point, it's hard to understand who does what.
The event’s journey is hard to debug and troubleshoot.
Kafka, RabbitMQ, NATS, and other MQs are HARD to deploy, manage, secure, update, onboard, and tune.
Turning batch processes into real-time can be complicated and time-consuming.
Data pipelines can travel between different teams and applications, hard to collaborate in complicated pipelines.

Please Welcome Memphis

Memphis delivers the most easy-to-use, deploy, and manage, k8s-native messaging queue, forked from NATS.IO, with amazing UI, CLI, and with built-in observability that allows you, the developer, to actually understand the full journey of your events.

RabbitMQ has Queues, Kafka has Topics, and Memphis has Stations.

The strength of Memphis reveals on day 2. By providing a framework for developers where they can offload most of the business logic out of the producers and consumers and embed it inside Memphis’ station (queue) like:

Inline schema changes and schema registry.
Enrich and analyze ingested data on-the-fly.
Ready-to-use connectors and components (Similar to “Kafka” sinks) that will pull and push data autonomously.

Helping developers develop data-driven apps Faster, Better, and happier!

Dev-First? What does it mean?
Thanks for reaching out to this point :)

We do our best to create any feature, button, and capability in a way it will make sense for the developer and DevOps on the other side, and really help them with their day-2-day tasks. A few examples —

Installation. Single command via Docker or K8S.
UI and CLI. Management tools to better understand and debug your data-driven app.
Troubleshooting and Debug. The bread and butter of every developer. Enabling developers to follow (via GUI) the full path of a single event.
Monitoring. Out-of-the-box ability to track and understand who are your producers and consumers, what do you do, and how much.
Interact with Memphis in any programming language. More to come!

Thank you for reading, hope you find it useful.

Please let me know in the comments If something is missing or vague.

Memphis documentation: https://docs.memphis.dev/
Discord - https://discord.gg/jGkWe3wqFw
Github -https://bit.ly/3MInIMZ