Prometheus - How it's work

Prometheus itself is monitoring software. It runs as a long-running service that scrapes endpoints (which you configure as scrape targets) and stores the metrics in a time-series database.

Those endpoints are expected to provide metrics in the format you have mentioned, which has become known as the 'OpenMetrics' format.

The OpenMetrics format started as just a Prometheus thing, but it's become popular enough that many other monitoring tools will happily also scrape or 'understand' OpenMetrics formatted metrics.

Prometheus can run on pretty much anything because it's a Go program it just runs as a single binary.

That means you can run Prometheus as a Windows executable, a Linux container, (or) Kubernetes as pods if you want, among others.

But it's part of the *CNCF * ecosystem, so it's commonly associated with Kubernetes and it is very Kubernetes-friendly.

You can easily deploy Prometheus to run on K8s, and it also has automatic service discovery that allows it to easily discover and monitor K8s workloads for you. But it doesn't need to run on K8s (or) monitor it, it's just well suited to that.

I'm not familiar with New Relic, but from some quick googling, it mentions it can integrate with Prometheus' remote_write feature.

Remote write was a feature that was developed for Prometheus that allows it to scrape its endpoints, and then forward the data onto another service.

This lets you use Prometheus for what it's good at - scraping endpoints - and then ship the data off to something like New Relic to act as a data warehouse or for visualizing data.

Prometheus can also run in what's called 'agent mode', which is where it doesn't store any metrics but is used exclusively to scrape and then forward metrics (using remote_write) to something like New Relic. So it acts as an 'agent' rather than the traditional heavyweight server mode where it stores data.

It looks like New Relic has also developed a newer way to monitor Kubernetes clusters with what they are calling the 'New Relic Prometheus Agent'.

From reading their docs, this is simply a thin wrapper around Prometheus in agent mode that is configured to forward metrics to New Relic out of the box.