JSONs versatility makes it a great choice for many things beyond web communications. Configuration files and logging
files are two things that often are written in, or contain JSON. Working with these files using traditional tools like
awk
and sed
is cumbersome and frustrating. This is where jq
comes in.
What is jq
?
jq
is a JSON processing tool that uses a very high level, purely functional language to query, mutate, and create
JSON. Utilizing the familiar concepts of pipes, filters, and streams; jq
accepts expressions written in the JSON-like
syntax unique to jq
.
Examples
jq
can accept either a file that contains the JSON, or JSON piped through stdin. This allows you to do pretty cool
things like pipe the body of a curl
request into jq
, apply some business logic to derive a new JSON body, and pipe
that body back into another curl
request to send to a remote server. The following examples are going to be ran
against some made-up status style JSON and fed in as a program argument. You can use jqplay
to try out jq
in the browser and to help build complex commands.
# config.json
{
"version": "22.04",
"dimensions": {
"width": 1920,
"height": 1080
},
"devices": [
{
"id": 22,
"state": "on",
"group": 0,
"status": "OK"
},
{
"id": 422,
"state": "off",
"group": 0,
"status": "OK"
},
{
"id": 702,
"state": "off",
"group": 99,
"status": "FAILED"
}
]
}
Read Property
Expressions are passed into jq
as a string. Because we are eventually going to be using double quotes to work with
nested strings, we can wrap everything in single quotes to avoid having to escape every quotation.
The dot operator is used to access the top-level entity within the scope. This gives us access the version property.
jq '.version' config.json
# output:
"20.04"
The truncate tool (tr
) can be used to clean up the quotes
jq '.version' config.json | tr -d /"
# output:
20.04
Nested properties can be accessed like expected.
jq '.dimensions.width' config.json
# output:
1920
Arrays
Arrays can be iterated over by calling the array directly.
jq '.devices[]' config.json
# output:
{
"id": 22,
"state": "on",
"group": 0,
"status": "OK"
}
{
"id": 422,
"state": "off",
"group": 0,
"status": "OK"
}
{
"id": 702,
"state": "off",
"group": 99,
"status": "FAILED"
}
Arrays can also be indexed like expected.
jq '.devices[0]' config.json
# output:
{
"id": 22,
"state": "on",
"group": 0,
"status": "OK"
}
jq
also supports slicing of both arrays and strings.
jq '.devices[:-1]' config.json
# output:
{
"id": 22,
"state": "on",
"group": 0,
"status": "OK"
}
{
"id": 422,
"state": "off",
"group": 0,
"status": "OK"
}
Pipes
The real power of jq
starts with the pipe operator. Properties can be selected and then piped into the next expression.
The above example of iterating over an array can also be written using the pipe operator.
jq '.devices | .[]' config.json
# output:
{
"id": 22,
"state": "on",
"group": 0,
"status": "OK"
}
{
"id": 422,
"state": "off",
"group": 0,
"status": "OK"
}
{
"id": 702,
"state": "off",
"group": 99,
"status": "FAILED"
}
Accessing the state
property of each object.
jq '.devices | .[] | .state' config.json
# output:
"on"
"on"
"off"
Filtering
jq
has a number of built-in functions, one of which is select
.
select
accepts a boolean expression and returns if the condition evaluates to true.
Selecting all devices that are disabled and do not have a status
of OK.
jq '.devices | .[] | select(.state=="off") | select(.status != "OK")' config.json
# output:
{
"id": 702,
"state": "off",
"group": 99,
"status": "FAILED"
}
Regex
Instead of comparing strings directly, jq
has several regex functions with the simplest being test
, which returns a
boolean if the input string matches the regular expression. The above filter can be rewritten using test
.
jq '.devices | .[] | select(.state|test("off")) | select(.status|test("^((?!OK).)*$"))' config.json
# output:
{
"id": 702,
"state": "off",
"group": 99,
"status": "FAILED"
}
Building JSON
jq
also lets you mutate and create JSON. Maybe you want to build a new JSON object using only two fields from the
source JSON. You can assign the parent property to a variable so that it is still accessible in the nested scope.
jq '.version as $version | .devices | .[] | {device_id: .id, version: $version}' config.json
# output:
{
"device_id": 22,
"version": "22.04"
}
{
"device_id": 422,
"version": "22.04"
}
{
"device_id": 702,
"version": "22.04"
}
You can wrap the expression in square brackets to collect the objects into an array.
jq '[.version as $version | .devices | .[] | {device_id: .id, version: $version}]' config.json
# output:
[
{
"device_id": 22,
"version": "22.04"
},
{
"device_id": 422,
"version": "22.04"
},
{
"device_id": 702,
"version": "22.04"
}
]
Lets take the dimensions
object and squash it into a new string to add to our objects. This time instead of
assigning a property to a variable, an entire JSON object is created to be referenced later. Parenthesis must be used to
group expressions together. The full expression is starting to get quite long so it's now broken up into multiple lines
and indented to be more readable.
jq '
[
{
version: .version,
dimensions: ((.dimensions.width|tostring) + "x" + (.dimensions.height|tostring))
} as $extras
| .devices
| .[]
| {
device_id: .id,
version: ($extras|.version),
dimensions: ($extras|.dimensions)
}
]' config.json
# output:
[
{
"device_id": 22,
"version": "22.04",
"dimensions": "1920x1080"
},
{
"device_id": 422,
"version": "22.04",
"dimensions": "1920x1080"
},
{
"device_id": 702,
"version": "22.04",
"dimensions": "1920x1080"
}
]
Conclusion
This guide doesn't cover the full depth of jq
but should be more than enough to get started doing some pretty cool
things. I've only just discovered jq
but I can already see so many ways that it can benefit my workflow. There are
more features included in jq
such as function definitions, conditionals, comparisons, and modification assignment
operators. You can find overviews of all these features and more in the jq
manual.
Top comments (0)