DEV Community

Cover image for JQ Recipes for HashiCorp Vault
🦄N B🛡
🦄N B🛡

Posted on • Edited on

JQ Recipes for HashiCorp Vault

#vault #jq #api #bash

This post will show some recipes for working with the jq utility in Bash to get data from HashiCorp Vault Enterprise via its ReST API.

You can install jq from here:

https://stedolan.github.io/jq

And if you don't have a "Bash" environment, you can get one from here: https://git-scm.com (make sure to select the "Git Bash" option when given the choice in the installation process)

If you're on Windows you might have some troubles or need to use some different name for jq-64.exe.

You also need to download and unzip Vault from https://vaultproject.io, and run it in a Git Bash like this:

$ cd ~/Downloads
$ echo 'alias vault="~/Downloads/vault.exe"' >> ~/.bash_profile ##(Only do this if you're on Windows)
$ source ~/.bash_profile ## Again, needed only if you're on Windows
$ ./vault server -dev -dev-root-token-id=s.JzcMzvFCAVHvxE4xusglvoHF

==> Vault server configuration:

             Api Address: http://127.0.0.1:8200
                     Cgo: disabled
         Cluster Address: https://127.0.0.1:8201
              Go Version: go1.14.7
              Listener 1: tcp (addr: "127.0.0.1:8200", cluster address: "127.0.0.1:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls
: "disabled")
               Log Level: info
                   Mlock: supported: false, enabled: false
           Recovery Mode: false
                 Storage: inmem
                 Version: Vault v1.5.3
             Version Sha: 9fcd81405feb320390b9d71e15a691c3bc1daeef

WARNING! dev mode is enabled! In this mode, Vault runs entirely in-memory
and starts unsealed with a single unseal key. The root token is already
authenticated to the CLI, so you can immediately begin using Vault.

You may need to set the following environment variable:

PowerShell:
    $env:VAULT_ADDR="http://127.0.0.1:8200"
cmd.exe:
    set VAULT_ADDR=http://127.0.0.1:8200

The unseal key and root token are displayed below in case you want to
seal/unseal the Vault or re-authenticate.

Unseal Key: t/5BTQzZ5LbD0JIYkpQ9dY3fLEqZEMyena3aCh4+sWw=
Root Token: s.JzcMzvFCAVHvxE4xusglvoHF

Development mode should NOT be used in production installations!

==> Vault server started! Log data will stream in below:

2020-10-07T15:14:47.286-0700 [INFO]  proxy environment: http_proxy= https_proxy= no_proxy=
2020-10-07T15:14:47.289-0700 [WARN]  no `api_addr` value specified in config or in VAULT_API_ADDR; falling back to detection if possible, but this value s
hould be manually set
2020-10-07T15:14:47.327-0700 [INFO]  core: security barrier not initialized
2020-10-07T15:14:47.327-0700 [INFO]  core: security barrier initialized: stored=1 shares=1 threshold=1
2020-10-07T15:14:47.332-0700 [INFO]  core: post-unseal setup starting
2020-10-07T15:14:47.349-0700 [INFO]  core: loaded wrapping token key
2020-10-07T15:14:47.349-0700 [INFO]  core: successfully setup plugin catalog: plugin-directory=
2020-10-07T15:14:47.356-0700 [INFO]  core: no mounts; adding default mount table
2020-10-07T15:14:47.360-0700 [INFO]  core: successfully mounted backend: type=cubbyhole path=cubbyhole/
2020-10-07T15:14:47.363-0700 [INFO]  core: successfully mounted backend: type=system path=sys/
2020-10-07T15:14:47.365-0700 [INFO]  core: successfully mounted backend: type=identity path=identity/
2020-10-07T15:14:47.370-0700 [INFO]  core: successfully enabled credential backend: type=token path=token/
2020-10-07T15:14:47.370-0700 [INFO]  rollback: starting rollback manager
2020-10-07T15:14:47.370-0700 [INFO]  core: restoring leases
2020-10-07T15:14:47.376-0700 [INFO]  expiration: lease restore complete
2020-10-07T15:14:47.376-0700 [INFO]  identity: entities restored
2020-10-07T15:14:47.376-0700 [INFO]  identity: groups restored
2020-10-07T15:14:47.377-0700 [INFO]  core: post-unseal setup complete
2020-10-07T15:14:47.380-0700 [INFO]  core: root token generated
2020-10-07T15:14:47.380-0700 [INFO]  core: pre-seal teardown starting
2020-10-07T15:14:47.380-0700 [INFO]  rollback: stopping rollback manager
2020-10-07T15:14:47.380-0700 [INFO]  core: pre-seal teardown complete
2020-10-07T15:14:47.381-0700 [INFO]  core.cluster-listener.tcp: starting listener: listener_address=127.0.0.1:8201
2020-10-07T15:14:47.381-0700 [INFO]  core.cluster-listener: serving cluster requests: cluster_listen_address=127.0.0.1:8201
2020-10-07T15:14:47.381-0700 [INFO]  core: post-unseal setup starting
2020-10-07T15:14:47.381-0700 [INFO]  core: loaded wrapping token key
2020-10-07T15:14:47.381-0700 [INFO]  core: successfully setup plugin catalog: plugin-directory=
2020-10-07T15:14:47.382-0700 [INFO]  core: successfully mounted backend: type=system path=sys/
2020-10-07T15:14:47.383-0700 [INFO]  core: successfully mounted backend: type=identity path=identity/
2020-10-07T15:14:47.383-0700 [INFO]  core: successfully mounted backend: type=cubbyhole path=cubbyhole/
2020-10-07T15:14:47.384-0700 [INFO]  core: successfully enabled credential backend: type=token path=token/
2020-10-07T15:14:47.384-0700 [INFO]  rollback: starting rollback manager
2020-10-07T15:14:47.384-0700 [INFO]  core: restoring leases
2020-10-07T15:14:47.384-0700 [INFO]  identity: entities restored
2020-10-07T15:14:47.384-0700 [INFO]  expiration: lease restore complete
2020-10-07T15:14:47.384-0700 [INFO]  identity: groups restored
2020-10-07T15:14:47.384-0700 [INFO]  core: post-unseal setup complete
2020-10-07T15:14:47.384-0700 [INFO]  core: vault is unsealed
2020-10-07T15:14:47.401-0700 [INFO]  core: successful mount: namespace= path=secret/ type=kv
2020-10-07T15:14:47.411-0700 [INFO]  secrets.kv.kv_8bdaeac5: collecting keys to upgrade
2020-10-07T15:14:47.411-0700 [INFO]  secrets.kv.kv_8bdaeac5: done collecting keys: num_keys=1
2020-10-07T15:14:47.411-0700 [INFO]  secrets.kv.kv_8bdaeac5: upgrading keys finished
Enter fullscreen mode Exit fullscreen mode

Leave Vault running in that terminal window, and open another one to run the following stuff in:

export VAULT_ADDR=http://127.0.0.1:8200
export VAULT_TOKEN=s.JzcMzvFCAVHvxE4xusglvoHF

Now, to prepare further, create some auth methods:

./vault auth enable kubernetes
./vault auth enable userpass
./vault auth enable okta
./vault auth enable aws

Example 1: Get a list of all of the Auth Methods that are of type okta

Shameless theft from this SO dude: https://stackoverflow.com/questions/18592173/select-objects-based-on-value-of-variable-in-object-using-jq/31911811#comment112920282_18608100

First, look at the response without jq:

curl --request GET --header "X-Vault-Token: $VAULT_TOKEN" --header "X-Vault-Namespace: $VAULT_NAMESPACE" $VAULT_ADDR/v1/sys/auth

Here's the output from that command:

$ curl --request GET --header "X-Vault-Token: $VAULT_TOKEN" --header "X-Vault-Namespace: $VAULT_NAMESPACE" $VAULT_ADDR/v1/sys/auth
{"kubernetes/":{"accessor":"auth_kubernetes_9bc9df5a","config":{"default_lease_t
tl":0,"force_no_cache":false,"max_lease_ttl":0,"token_type":"default-service"},"
description":"","external_entropy_access":false,"local":false,"options":null,"se
al_wrap":false,"type":"kubernetes","uuid":"72df2827-2ac1-1fb1-7d5a-c6262a4f5300"
},"token/":{"accessor":"auth_token_332e9370","config":{"default_lease_ttl":0,"fo
rce_no_cache":false,"max_lease_ttl":0,"token_type":"default-service"},"descripti
on":"token based credentials","external_entropy_access":false,"local":false,"opt
ions":null,"seal_wrap":false,"type":"token","uuid":"ba81c552-e8aa-9ba9-5243-0580
59545ccc"},"userpass/":{"accessor":"auth_userpass_1dd44d0b","config":{"default_l
ease_ttl":0,"force_no_cache":false,"max_lease_ttl":0,"token_type":"default-servi
ce"},"description":"","external_entropy_access":false,"local":false,"options":nu
ll,"seal_wrap":false,"type":"userpass","uuid":"e02a55d0-bd07-efeb-f5de-37fb5dcd5
84a"},"okta/":{"accessor":"auth_okta_fb869c57","config":{"default_lease_ttl":0,"
force_no_cache":false,"max_lease_ttl":0,"token_type":"default-service"},"descrip
tion":"","external_entropy_access":false,"local":false,"options":null,"seal_wrap
":false,"type":"okta","uuid":"1a445f3a-d302-ac6b-4508-def7a289894a"},"request_id
":"dc3dbab5-e001-498b-514c-05665baf09d2","lease_id":"","renewable":false,"lease_
duration":0,"data":{"kubernetes/":{"accessor":"auth_kubernetes_9bc9df5a","config
":{"default_lease_ttl":0,"force_no_cache":false,"max_lease_ttl":0,"token_type":"
default-service"},"description":"","external_entropy_access":false,"local":false
,"options":null,"seal_wrap":false,"type":"kubernetes","uuid":"72df2827-2ac1-1fb1
-7d5a-c6262a4f5300"},"okta/":{"accessor":"auth_okta_fb869c57","config":{"default
_lease_ttl":0,"force_no_cache":false,"max_lease_ttl":0,"token_type":"default-ser
vice"},"description":"","external_entropy_access":false,"local":false,"options":
null,"seal_wrap":false,"type":"okta","uuid":"1a445f3a-d302-ac6b-4508-def7a289894
a"},"token/":{"accessor":"auth_token_332e9370","config":{"default_lease_ttl":0,"
force_no_cache":false,"max_lease_ttl":0,"token_type":"default-service"},"descrip
tion":"token based credentials","external_entropy_access":false,"local":false,"o
ptions":null,"seal_wrap":false,"type":"token","uuid":"ba81c552-e8aa-9ba9-5243-05
8059545ccc"},"userpass/":{"accessor":"auth_userpass_1dd44d0b","config":{"default
_lease_ttl":0,"force_no_cache":false,"max_lease_ttl":0,"token_type":"default-ser
vice"},"description":"","external_entropy_access":false,"local":false,"options":
null,"seal_wrap":false,"type":"userpass","uuid":"e02a55d0-bd07-efeb-f5de-37fb5dc
d584a"}},"wrap_info":null,"warnings":null,"auth":null}
Enter fullscreen mode Exit fullscreen mode

Pretty messy, right? Let's clean it up a bit:

curl --request GET --header "X-Vault-Token $VAULT_TOKEN" --header "X-Vault-Namespace $VAULT_NAMESPACE" $VAULT_ADDR/v1/sys/auth | jq

The above command will show the output in more of a structured format with indentation to indicate deeper levels of the object.

Here's the output from the above command:

$ curl --request GET --header "X-Vault-Token: $VAULT_TOKEN" --header "X-Vault-Namespace: $VAULT_NAMESPACE" $VAULT_ADDR/v1/sys/auth | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2695    0  2695    0     0   105k      0 --:--:-- --:--:-- --:--:--  105k
{
  "token/": {
    "accessor": "auth_token_332e9370",
    "config": {
      "default_lease_ttl": 0,
      "force_no_cache": false,
      "max_lease_ttl": 0,
      "token_type": "default-service"
    },
    "description": "token based credentials",
    "external_entropy_access": false,
    "local": false,
    "options": null,
    "seal_wrap": false,
    "type": "token",
    "uuid": "ba81c552-e8aa-9ba9-5243-058059545ccc"
  },
  "userpass/": {
    "accessor": "auth_userpass_1dd44d0b",
    "config": {
      "default_lease_ttl": 0,
      "force_no_cache": false,
      "max_lease_ttl": 0,
      "token_type": "default-service"
    },
    "description": "",
    "external_entropy_access": false,
    "local": false,
    "options": null,
    "seal_wrap": false,
    "type": "userpass",
    "uuid": "e02a55d0-bd07-efeb-f5de-37fb5dcd584a"
  },
  "okta/": {
    "accessor": "auth_okta_fb869c57",
    "config": {
      "default_lease_ttl": 0,
      "force_no_cache": false,
      "max_lease_ttl": 0,
      "token_type": "default-service"
    },
    "description": "",
    "external_entropy_access": false,
    "local": false,
    "options": null,
    "seal_wrap": false,
    "type": "okta",
    "uuid": "1a445f3a-d302-ac6b-4508-def7a289894a"
  },
  "kubernetes/": {
    "accessor": "auth_kubernetes_9bc9df5a",
    "config": {
      "default_lease_ttl": 0,
      "force_no_cache": false,
      "max_lease_ttl": 0,
      "token_type": "default-service"
    },
    "description": "",
    "external_entropy_access": false,
    "local": false,
    "options": null,
    "seal_wrap": false,
    "type": "kubernetes",
    "uuid": "72df2827-2ac1-1fb1-7d5a-c6262a4f5300"
  },
  "request_id": "633adc4e-bd04-e579-58d4-34bde3dc91ee",
  "lease_id": "",
  "renewable": false,
  "lease_duration": 0,
  "data": {
    "kubernetes/": {
      "accessor": "auth_kubernetes_9bc9df5a",
      "config": {
        "default_lease_ttl": 0,
        "force_no_cache": false,
        "max_lease_ttl": 0,
        "token_type": "default-service"
      },
      "description": "",
      "external_entropy_access": false,
      "local": false,
      "options": null,
      "seal_wrap": false,
      "type": "kubernetes",
      "uuid": "72df2827-2ac1-1fb1-7d5a-c6262a4f5300"
    },
    "okta/": {
      "accessor": "auth_okta_fb869c57",
      "config": {
        "default_lease_ttl": 0,
        "force_no_cache": false,
        "max_lease_ttl": 0,
        "token_type": "default-service"
      },
      "description": "",
      "external_entropy_access": false,
      "local": false,
      "options": null,
      "seal_wrap": false,
      "type": "okta",
      "uuid": "1a445f3a-d302-ac6b-4508-def7a289894a"
    },
    "token/": {
      "accessor": "auth_token_332e9370",
      "config": {
        "default_lease_ttl": 0,
        "force_no_cache": false,
        "max_lease_ttl": 0,
        "token_type": "default-service"
      },
      "description": "token based credentials",
      "external_entropy_access": false,
      "local": false,
      "options": null,
      "seal_wrap": false,
      "type": "token",
      "uuid": "ba81c552-e8aa-9ba9-5243-058059545ccc"
    },
    "userpass/": {
      "accessor": "auth_userpass_1dd44d0b",
      "config": {
        "default_lease_ttl": 0,
        "force_no_cache": false,
        "max_lease_ttl": 0,
        "token_type": "default-service"
      },
      "description": "",
      "external_entropy_access": false,
      "local": false,
      "options": null,
      "seal_wrap": false,
      "type": "userpass",
      "uuid": "e02a55d0-bd07-efeb-f5de-37fb5dcd584a"
    }
  },
  "wrap_info": null,
  "warnings": null,
  "auth": null
}
Enter fullscreen mode Exit fullscreen mode

Now, let's get all of the Auth Methods that are of type okta:

curl --request GET --header "X-Vault-Token: $VAULT_TOKEN" --header "X-Vault-Namespace: $VAULT_NAMESPACE" $VAULT_ADDR/v1/sys/auth | jq .data | jq 'to_entries[] | select(.value.type=="okta") | .key'

The first part gets you the contents of the "data" part of the response. The second part, with the 'to_entries[], gets you the data in a format where you can read out the key names separately. The | select(.value.type=="okta") part gets you the auth methods that have the value "okta" under the key type. The | .key' part gets you the key of the auth method, usually ending in a /.

Here's the output for my example (should just be "okta/"):

$ curl --request GET --header "X-Vault-Token: $VAULT_TOKEN" --header "X-Vault-Namespace: $VAULT_NAMESPACE" $VAULT_ADDR/v1/sys/auth | jq .data | jq 'to_entries[] | select(.value.type=="okta") | .key'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2695    0  2695    0     0   125k      0 --:--:-- --:--:-- --:--:--  125k
"okta/"
Enter fullscreen mode Exit fullscreen mode

Example 2: How to get an accessor based on a Bash variable:

export OKTAAUTH='okta/'

cat << EOF > jqparams.txt
."data"."${OKTAAUTH}/"."accessor"
EOF
Enter fullscreen mode Exit fullscreen mode

curl --request GET --header "X-Vault-Token: $VAULT_TOKEN" --header "X-Vault-Namespace: $VAULT_NAMESPACE" $VAULT_ADDR/v1/sys/auth | jq -f jqparams.txt

Here's an example:

$ export OKTAAUTH='okta/'

$ cat << EOF > jqparams.txt
> ."data"."${OKTAAUTH}"."accessor"
> EOF

$ curl --request GET --header "X-Vault-Token: $VAULT_TOKEN" --header "X-Vault-Namespace: $VAULT_NAMESPACE" $VAULT_ADDR/v1/sys/auth | jq -f jqparams.txt
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2695    0  2695    0     0   125k      0 --:--:-- --:--:-- --:--:--  125k
"auth_okta_fb869c57"

$
Enter fullscreen mode Exit fullscreen mode

Example 3: How to get an Accessor Without a Bash Variable

In this case, it's the same as Example 2, except that you needn't bother with a jqparams.txt file.

."data"."okta/"."accessor"

Further reading for the nerds:

https://remysharp.com/drafts/jq-recipes

https://stedolan.github.io/jq/tutorial/

https://stedolan.github.io/jq/manual/

Top comments (0)

Read next

nausaf profile image

An Attempt to return meaningful Problem Details responses for model binding errors in an ASP.NET Core Minimal API

nausaf -

frederickollinger profile image

Using a Self-Signed Certificate with Git Clone Https

Frederick Ollinger -

ramadhan002 profile image

Axios interceptor + React JS

ramadhan.dev -

leandroveiga profile image

Mastering Error Handling in .NET 8: Strategies for Minimal APIs and Controllers

Leandro Veiga -