HowTo: Find egress traffic destination in Istio service mesh

Before we enable the REGISTRY_ONLY options for Istio, we want to capture all the existing outbound traffic and add it as a ServiceEntry.

There are two metrics we can monitor:

• istio_requests_total{destination_service_name="PassthroughCluster"}
• istio_tcp_connections_closed_total{destination_service_name="PassthroughCluster"}

We can query these two metrics in prometheus to see if there’s any outbound traffic that was not registered yet.

The first one istio_requests_total is easy. It captures all the http requests. And because it’s a http request, the domain will be recorded in the destination_service field.

The second one istio_tcp_connections_closed_total is more complicated. It may be an HTTPS or TCP request. And for these requests the destination domain was not recorded. The only thing we can know is which workload generated these requests.

Envoy Debug Log

To find out where the request is going, we must first turn on the debug log of the sidecar proxy. There are two ways we can do that.

The first one is to add an annotation to the workload: sidecar.istio.io/logLevel: debug

If you don’t want to restart the pod, you can also use istioctl to enable debug log during runtime:

istioctl pc log <pod_name>.<namespace> --level debug

After the debug log was enabled, we then needed to trigger the application to make it send those requests. And then we can try to find PassthroughCluster in the logs.

k logs -n <namespace> <pod_name> -c istio-proxy | grep "Creating connection to cluster PassthroughCluster" -B2 -A2

After you enabled the REGISTRY_ONLY mode of the Istio service mesh, there will be no PassthroughCluster, instead, you should be monitoring BlackholeCluster.

Envoy Access Log

Another tool we can use is the access log of envoy proxy. We can enable the access log for specified namespace and workload. Here’s an example of enabling envoy access log for monitoring namespace.

apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: envoy-access-log
  namespace: monitoring
spec:
  accessLogging:
    - providers:
      - name: envoy

After that, we can tail the log and pipe to engarde to view the logs.

k logs -n <namespace> <pod_name> -c istio-proxy -f | engarde --use-istio | jq 'select(.upstream_cluster=="BlackHoleCluster")'