Prometheus blackbox_exporter; Unconventional Way

Many of us have different requirements and different complicated setups.
Prometheus has provided us the true power of monitoring and observability, Thus, I'm still learning and figuring things out every single day.

Case

One of the most recent requirements I had, that we have to monitor a certain route of accessibility to maintain applications performing correctly. With out-of-the-box tools' functionality, the monitoring system is always the source, where our case is different that we need to make sure that System-A is reachable from every network in the datacenter.

To have a simple and less complicated example, let's take north-ntw, south-ntw and west-ntw to reach the internet by probing https://wikipedia.com
by this I mean, we are monitoring egress traffic in each network whether it could reach the internet through different routes with reasonable latency.

Strategy

We're going to use Prometheus as a server and Blackbox Exporter
The blackbox exporter allows blackbox probing of endpoints over HTTP, HTTPS, DNS, TCP and ICMP. It's often used to be installed alongside to prometheus server and this makes the prometheus server is always the source.
Here is the trick, we're going to deploy blackbox exporter on every monitored network and instruct prometheus to scrape those exporters as a source of probes to assure that System-A (aka Wikipedia) is reachable from those networks.

Let's go

You can follow as many as guides and tutorial on how to install Prometheus and Prometheus exporters, however, here's a quick win with ansible and playbooks for ansible-roles from the great team at cloudalchemy:
ansible-prometheus and ansible-blackbox-exporter
I will leave the installation part for you to be done in your preferred way

scraping configs

This is the most important part of this journey where we instruct prometheus where to find it's exporters

## /etc/prometheus/prometheus.yml
  - job_name: blackbox_metadata
    params:
      module: [http_2xx]
      target:
        - https://wikipedia.com
    metrics_path: /probe
    scrape_interval: 30s
    scrape_timeout: 10s
    static_configs:
      - targets:
        - south.rootsami.dev:9115
        - north.rootsami.dev:9115
        - east.rootsami.dev:9115
    relabel_configs:
      - source_labels: [__param_target]
        target_label: target
      - source_labels: [__address__]
        separator:     ';'
        regex:         '(.*):.*'
        target_label: instance
        replacement:   '${1}:9115'

target under the params section defines the destination that you want to reach which is https://wikipedia.com, whereas the static_configs targets are the little blackbox-exporter that we are probing from.

As shown above, target endpoints are the deployed exporters which are being scraped by prometheus server and showing that Wikipedia is reachable from that network, as well as latency can be measured from each source.

Conclusion

Monitoring and observability have no limits, tools are there! All you have to do is to find the blind spots to maintain systems up and running at all times.