A Comprehensive Tutorial on Service Mesh, Istio, Envoy, Access Log, and Log Filtering

First things first, I confess: I haven’t used photos that I took as featured images for a while, and I truly miss that.

In this article, we will briefly introduce Envoy, enable Envoy access log in Istio, play with Envoy's access log filters, and figure out ways to configure Envoy access log filters with Istio.

Some basic knowledge of Istio is expected, but even if you have none, you can follow this tutorial to have a successful local setup.

Without further adieu, let's get started.

1 A Very Short Introduction to Envoy

Envoy is an L7 high-performance proxy and communication bus developed in C++ and designed for large modern service-oriented architectures.

• It is a self-contained process designed to run alongside every application server.
• At its very core, Envoy is, in fact, an L3/L4 network proxy. A pluggable filter chain mechanism allows filters to be written to perform different TCP/UDP proxy tasks and inserted into the main server.
• Envoy supports an additional HTTP L7 filter layer; HTTP filters can be plugged into the HTTP connection management subsystem that performs different tasks such as buffering, rate limiting, routing/forwarding, etc.
• When operating in HTTP mode, Envoy supports a routing subsystem capable of routing and redirecting requests based on path, authority, content type, runtime values, etc. This functionality is most useful when using Envoy as a front/edge proxy but is also leveraged when building a service-to-service mesh.

Envoy has many other high-level features, but the ones mentioned above make it perfect to be used as a sidecar proxy in a service mesh.

Two core concepts about Envoy are related to this tutorial:

1.1 HTTP Connection Management

HTTP is such a critical component of modern service-oriented architectures that Envoy implements a large amount of HTTP-specific functionality.

Envoy has a built-in network-level filter called the HTTP connection manager, which translates raw bytes into HTTP level messages and events (e.g., headers received, body data received, trailers received, etc.). It also handles functionality common to all HTTP connections and requests, such as access logging, request ID generation and tracing, request/response header manipulation, route table management, and statistics.

1.2 Access Logging

When used in a service-mesh scenario, for example, in Istio, the simplest kind of logging is Envoy's access logging.

Envoy proxies print access information to their standard output. The kubectl logs command can print Envoy's containers' standard output.

2 A Very Introduction to Istio (And Service Mesh)

2.1 Service Mesh

A service mesh is a dedicated infrastructure layer added to distributed microservices. It allows us to add capabilities transparently like:

• observability
• traffic management
• security

And these are achieved without changing the application code.

As the deployment of distributed services grows in size and complexity, it becomes harder to understand and manage all the services, the requirements of which include discovery, load balancing, failure recovery, metrics, monitoring, etc. And inter-service communication is what makes a distributed application possible. Routing this communication within and across application clusters becomes increasingly complex as the number of services grows.

A service mesh helps reduce the aforementioned while easing the strain on development teams.

It can often handle more complex operational things like A/B testing, canary deployments, rate limiting, access control, encryption, and end-to-end authentication.

2.2 Istio

Istio is an open-source service mesh that layers transparently onto existing distributed applications. Istio's robust features provide a uniform and more efficient way to secure, connect, and monitor services. Istio is the path to load balancing, service-to-service authentication, and monitoring – with few or no service code changes. Its powerful control plane brings vital features, including:

• Secure service-to-service communication in a cluster with TLS encryption, strong identity-based authentication, and authorization
• Automatic load balancing for HTTP, gRPC, WebSocket, and TCP traffic
• Fine-grained control of traffic behavior with rich routing rules, retries, failovers, and fault injection
• A pluggable policy layer and configuration API supporting access controls, rate limits, and quotas
• Automatic metrics, logs, and traces for all traffic within a cluster, including cluster ingress and egress

Istio is designed for extensibility and can handle various deployment needs. Istio's control plane runs on Kubernetes. You can add applications deployed in that cluster to your mesh, extend the mesh to other clusters, or even connect VMs or other endpoints outside Kubernetes.

A large ecosystem of contributors, partners, integrations and distributors extend and leverage Istio for a wide variety of scenarios. You can install Istio yourself, or some vendors have products that integrate Istio and manage it for you.

2.3 Istio and Envoy

Istio uses Envoy as proxies, deployed as sidecars. These proxies mediate and control all network communication between microservices. They also collect and report telemetry on all mesh traffic.

It's worth mentioning that, for the sake of unbias, there are other choices of service mesh besides Istio, some of which are even better from certain standpoints. But since Envoy and Istio are closely related, today we will use Istio to demo Envoy's access log settings.

OK, now that we cleared the nomenclature of service mesh and Istio out of the way, let's play with Envoy's access log. We will do this with Istio.

3 Prepare a Local Kubernetes Cluster with Istio

3.1 Prepare a Local Kubernetes Cluster

One of the easiest ways to start a local Kubernetes cluster is to use minikube. Follow the instructions in the official installation documentation, then run the following:

minikube start

3.2 Install Istio

Download Istio, install, then enable istio-injection in the default namespace by running the following commands:

curl -L https://istio.io/downloadIstio | sh -
# your version might differ
cd istio-1.16.1  
# for easier usage
export PATH=$PWD/bin:$PATH 
# here, we use the minimal profile so that the access log isn't enabled by default, which we want to do ourselves
istioctl install --set profile=minimal -y
# enable injection in the default namespace
kubectl label namespace default istio-injection=enabled

3.3 Deploy the Testing Apps

First, let's deploy some sample apps for testing:

kubectl apply -f samples/sleep/sleep.yaml
export SOURCE_POD=$(kubectl get pod -l app=sleep -o jsonpath={.items..metadata.name})
kubectl apply -f samples/httpbin/httpbin.yaml

This set of commands will deploy two applications: one is the "curl" command in a pod, which we will use to send out HTTP requests, and the other is an HTTP server that receives requests, which we will use to demonstrate the access logs of Envoy.

4 Envoy Access Logs in Istio

4.1 Enable Access Logs

Then, let's enable access logs.

Istio offers a few ways to enable access logs. Use of the Telemetry API is recommended:

First, we create a file telemetry.yaml with the following content:

apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: mesh-default
  namespace: istio-system
spec:
  accessLogging:
    - providers:
      - name: Envoy

Then apply it:

kubectl apply -f telemetry.yaml

4.2 Test

First, let's send a request from sleep to httpbin:

kubectl exec "$SOURCE_POD" -c sleep -- curl -sS -v httpbin:8000/status/418

If we check the httpbin's log, we can see something similar to the following:

kubectl logs -l app=httpbin -c istio-proxy
[2020-11-25T21:26:18.409Z] "GET /status/418 HTTP/1.1" 418 - via_upstream - "-" 0 135 3 1 "-" "curl/7.73.0-DEV" "84961386-6d84-929d-98bd-c5aee93b5c88" "httpbin:8000" "127.0.0.1:80" inbound|8000|| 127.0.0.1:41854 10.44.1.27:80 10.44.1.23:37652 outbound_.8000_._.httpbin.foo.svc.cluster.local default

5 Envoy Access Log Filter

Now that we have enabled access logs for Envoy, let's play with it.

5.1 The Task

Imagine the following situation: your application has some endpoints, for example, /status, /liveness, and /readiness, which you don't want logs because there might be multiple requests per minute. These status check logs could not be a good use of logging resources.

Or, you may want to customize your access logs to allow different requests and responses to be written to separate log files.

Can we achieve that?

Yes.

Envoy supports several built-in access logs and extension filters registered at runtime. Now let's try to demo the log filter functions:

5.2 Download Envoy And Prepare a Config File

The easiest way to play with Envoy's configuration is by running it locally instead of as a sidecar as part of Istio. So let's do this:

Install Envoy:

brew update
brew install envoy

Download the demo config from here

Then let's start it:

envoy -c envoy-demo.yaml

Then, open another tab, and verify that Envoy is running on port 10000:

curl -v localhost:10000

If we go back to the tab where Envoy is running, we can see the access log from Envoy, which will look similar to the following:

[2023-01-19T03:10:59.085Z] "GET / HTTP/1.1" 200 - 0 17304 747 467 "-" "curl/7.85.0" "4b35db72-baf8-4730-885e-3e628757f0e3" "www.envoyproxy.io" "34.143.223.220:443"

5.3 Envoy Log Filter

Envoy provides a bunch of log filters, for example, status code filter (as the name suggests, filter by status code), header filter, etc. Read the official doc here for more details.

The documentation could be more detailed; it doesn't provide examples of the usage of each filter. But that won't block a senior DevOps engineer. Based on some simple searches on Google (and GitHub), let's try this piece of config:

...
          access_log:
          - name: Envoy.access_loggers.stdout
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
            filter:
              header_filter:
                header:
                  name: :Path
                  string_match:
                    exact: /status
...

It's not hard to understand: it works on the header, and if the path matches something, the filter catches it.

If we add this piece of code into the file envoy-demo.yaml, and if we access the /status URL:

curl -v localhost:10000/status

There will be some logs like:

[2023-01-19T03:23:08.054Z] "GET /status HTTP/1.1" 404 - 0 3413 776 690 "-" "curl/7.85.0" "e507c86f-004f-4cba-b622-2004c7cf97c7" "www.envoyproxy.io" "34.126.184.144:443"

But if we access URLs other than /status, such as /:

curl -v localhost:10000

We get no logs.

OK, so the log filter is working now. Except that we want to filter out logs like /status.

5.4 More on Envoy Log Filters

OK, let's go back to the official doc and some google searches.

It's not hard to see that some filters like "and_filter" are helpful. Let's try that:

...
            filter:
              and_filter:
                filters:
                - header_filter:
                    header:
                      name: :Path
                      string_match:
                        exact: /status
                      invert_match: true
                - header_filter:
                    header:
                      name: :Path
                      string_match:
                        exact: /liveness
                      invert_match: true
                - header_filter:
                    header:
                      name: :Path
                      string_match:
                        exact: /readiness
                      invert_match: true
...

We can use and_filter to combine a bunch of filters, and we can negate the matching result by using invert_match.

This config updated in the envoy-demo.yaml will only show logs if the URL doesn't match /status, /liveness, or /readiness.

OK, we have achieved our goal: we can use filters to control Envoy's access logging. In the example above, we used and_filter and header_filter to filter out specific unwanted log entries. If you mix and match all Envoy's filters, you can quickly achieve things like "direct certain logs to a specific file", "only log 4xx requests", etc.

But (and this is a big but), the above config is only for local usage or running Envoy as a standalone process.

How to put those filter settings into a sidecar Envoy?

6 Telemetry API

As previously mentioned, Istio offers a few ways to enable access logs. We used Telemetry API in section 4.1, so let's see if we can customize Telemetry API to achieve the log filtering goal.

Telemetry defines how the telemetry is generated for workloads within a mesh. It has a feature AccessLogging.Filter where you can specify an expression in a string to achieve goals like response.code >= 400 or connection.mtls && request.url_path.contains('v1beta3').

According to the official documentation (link above), the expression should be a CEL expression (read more details on the language definition here.

So, if we convert the filters we wrote above into a CEL expression, it would be like this:

expression: "request.url_path != '/status' && request.url_path != '/liveness' && request.url_path != '/readiness'"

If we update the telemetry.yaml with the following content:

apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: mesh-default
  namespace: istio-system
spec:
  accessLogging:
    - providers:
        - name: Envoy
      filter:
        expression: "request.url_path != '/status' && request.url_path != '/liveness' && request.url_path != '/readiness'"

And apply it:

kubectl apply -f telemetry.yaml

Then the istio-proxy sidecar (Envoy) will only log entries where the request URL path isn't /status, /liveness, or /readiness.

7 EnvoyFilter

EnvoyFilter provides a mechanism to customize the Envoy configuration. Use EnvoyFilter to modify values for certain fields, add specific filters, or even add entirely new listeners, clusters, etc. This feature must be used carefully, as incorrect configurations could destabilize the entire mesh. Unlike other Istio networking objects, EnvoyFilters are additively applied. Any number of EnvoyFilters can exist for a given workload in a specific namespace. The order of application of these EnvoyFilters is as follows: all EnvoyFilters in the config root namespace, followed by all matching EnvoyFilters in the workload's namespace.

We can use EnvoyFilter to configure Envoy access logs, although officially, "Istio Telemetry API provides a first-class way to configure access logs and traces. It is recommended to use that method."

Nevertheless, let's look at EnvoyFilter and how to use that.

First, let's delete the Telemetry we created before to disable the access log:

kubectl delete -f telemetry.yaml

Then, following the format in this doc, we can create a file envoyfilter.yaml with the following content:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: access-log
spec:
  configPatches:
  - applyTo: NETWORK_FILTER
    match:
      context: ANY
      listener:
        filterChain:
          filter:
            name: "Envoy.filters.network.http_connection_manager"
    patch:
      operation: MERGE
      value:
        typed_config:
          "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager"
          access_log:
          - name: Envoy.file_access_log
            typed_config:
              "@type": "type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog"
              path: /dev/stdout
              format: "[%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_CODE_DETAILS% \"%RESP(GRPC-STATUS)% %RESP(GRPC-MESSAGE)%\" %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\"\n"
            filter:
              and_filter:
                filters:
                - header_filter:
                    header:
                      name: :Path
                      string_match:
                        exact: /status
                      invert_match: true
                - header_filter:
                    header:
                      name: :Path
                      string_match:
                        exact: /liveness
                      invert_match: true
                - header_filter:
                    header:
                      name: :Path
                      string_match:
                        exact: /readiness
                      invert_match: true

We create an object of the "EnvoyFilter" kind, and the content looks the same as Envoy's configuration file.

If we apply it:

kubectl apply -f envoyfilter.yaml

Then the istio-proxy sidecar (Envoy) will only log entries where the request URL path isn't /status, /liveness, or /readiness, just like the Telemetry way.

8 Teardown

minikube delete

Summary

In this tutorial:

• We introduced Envoy, service mesh, and Istio.
• Then, we created a local Kubernetes cluster and installed Istio inside it.
• We enabled Envoy access logs in Istio via Telemetry and played with Envoy's configurations to achieve log filtering.
• We also looked at two ways of setting log filtering for the Envoy sidecar in Istio (Telemetry and EnvoyFilter).

I know not all people have the need to fine-tune Envoy’s access logging, but if you do, you will find it hard to get some examples off the internet. That’s one of the reasons I decided to publish my experience as a blog: if you are searching hard to figure out how to write filters, and_filter, EnvoyFilter, or Telemetry API AccessLogging, CEL expression syntax, you are not alone, and I hope this article helps.

For people who are not so crazy about Envoy access logging, this tutorial still serves as a very good introduction to service mesh/Istio/Envoy.

If you like this article, please like, comment, subscribe. See you in the next one!