yq : A command line tool that will help you handle your YAML resources better

When you want to update the fields in your YAML files, the standard practice is to use a templating tool that can replace values dynamically. However, Kubernetes does not provide any templating mechanism of its own as the deployed manifests are supposed to be static YAML files.

Enter yq.

yq is a great command-line tool for templating YAML files, and it’s a lighter weight option to popular tools such as Helm and Kustomize.

What is yq?

yq is a command-line tool designed to transform YAML. It is similar to jq which focuses on transforming JSON instead of YAML.

What can it do?

yq can take a YAML file as an input and:

1. Read values from the file.
2. Add new values.
3. Update Existing values.
4. Generate new YAML files.
5. Convert YAML into JSON.
6. Merge two or more YAML files.

Installation

You can install yq on Mac OS with:

$ brew install yq

On Linux with:

$ sudo add-apt-repository ppa:rmescandon/yq
sudo apt-get install yq

In case you don't have the add-apt-repository command
installed, you can install it with apt-get install software-
properties-common.

If you're on Windows, you can download the executable from Github.

Use case scenarios

1. Reading YAML values

Say you are working with the following Pod:

apiVersion: v1
kind: Pod
metadata:
  name: test-pod
spec:
  containers:
  - name: test-container
    image: k8s.gcr.io/busybox
    env:
    - name: DB_URL
      value: postgres://db_url:5432

You could read the value for the environment variable ENV with the following command:

$ yq r pod.yaml "spec.containers[0].env[0].value"
postgres://db_url:5432

$ 

The command works as follows:

• yq r is the command to read a value from the YAML file.
• pod.yaml is the file path of the YAML that you want to read.
• spec.containers[0].env[0].value is the query path.

2. Changing YAML values

Let’s take the previous example here. Say you wish to deploy the app in a production environment and change the URL to the production database.

Here’s how you can do the same using just a single command:

$ yq w pod.yaml "spec.containers[0].env[0].value" "postgres://prod:5432"
apiVersion: v1
kind: Pod
metadata:
  name: test-pod
spec:
  containers:
  - name: test-container
    image: k8s.gcr.io/busybox
    env:
    - name: DB_URL
      value: postgres://prod:5432

$

If you notice here, yq printed the result on the standard output. If you prefer to edit the YAML in place, you should add the -i flag.

The advantage of using yq over say sed (with bash) is that unlike sed, yq understands the YAML format, and can navigate and mangle the structured markup.

Note: sed treats files as strings and it doesn't mind if the
file isn't a valid YAML.

3. Merging YAML files

Let's assume that you want to inject an extra container to all the Pods submitted to the cluster.

But instead of using an Admission Webhook, you decide to add an extra command in your deployment script.

You could save the YAML configuration for the extra container as a YAML file:

apiVersion: v1
kind: Pod
metadata:
  name: envoy-pod
spec:
  containers:
  - name: proxy-container
    image: envoyproxy/envoy:v1.12.2
    ports:
      - containerPort: 80

Assuming you have a Pod like this:

$ apiVersion: v1
kind: Pod
metadata:
  name: test-pod
spec:
  containers:
  - name: test-container
    image: k8s.gcr.io/busybox
    env:
    - name: DB_URL
      value: postgres://db_url:5432

$

You can execute the following command and merge the two YAMLs:

$ yq m --append pod.yaml envoy-pod.yaml

Please notice the --append flag that is necessary to append
values to an array. You can find more details in the official
documentation.

The output should have a proxy named Envoy as an additional container:

$ apiVersion: v1
kind: Pod
metadata:
  name: test-pod
spec:
  containers:
  - name: test-container
    image: k8s.gcr.io/busybox
    env:
    - name: DB_URL
      value: postgres://db_url:5432
  - name: proxy-container
    image: envoyproxy/envoy:v1.12.2
    ports:
      - containerPort: 80

Please note that yq sorts the YAML fields in the output alphabetically, so the order of fields in your output could be different from the > above listing.

In other words, the two YAML files are merged into one.

Limitations of using yq?

While yq works great with transforming YAML, it does have a few issues of its own:

1. The two YAML files are merged at the top level. For instance, you cannot add a chunk of YAML file under .spec.containers[].

2. The order of the files matters. If you invert the order, yq keeps envoy-pod for the Pod's name in metadata.name.

3. You have to tell yq explicitly when to append and overwrite values. Since those are flags that apply to the whole document, it's hard to get the granularity right.

Even with its limitations, yq has a lot of potential for use, especially if you are working on smaller projects.

Alternative(s)

If you wish to apply more complex transformations – Kustomize is a better option than yq. Kustomize uses a YAML file called kustomization.yaml to decide how to template the YAML – thus making all the changes traceable.

You can learn more about Kustomize and other interesting alternatives here: Templating YAML with real code