A Beginner's Guide to Curl: Part 4 - Json APIs

Now let's start using curl to interact with APIs. A popular data format that get's passed around the internet is Json (Javascript Object Notation).

Json is structured by key-value pairs inside curly brackets and lists of key-value pairs inside square brackets. It uses double quotes and colons.

{
  "name": "Jesse",
  "articles": [
    {
      "name": "A Beginner's Guide to Curl: Part 4",
      "langauge": "English"
    },
    {
      "name": "A Beginner's Guide to Curl: Part 5",
      "length": {
        "words": 27321
      }
    }
  ]
}

Let's grab a Json response from a free Pokemon API service found at https://pokeapi.co. Notice it's .co not .com.

Webservices that offer API endpoints typically namespace them with /api and the API version, in this case /v2.

Here's the command.

$ curl https://pokeapi.co/api/v2/pokemon/charizard

You should have received a large block of Json printed to your STDOUT. It's pretty hard to read isn't it?

There are several tools you could use to pipe the HTTP Json response into a human-readable format. We'll use something called "JQ". You can read about it and download it at https://stedolan.github.io/jq/.

If you're on MacOS and use Homebrew as your package manager, you can type this in your terminal.

$ brew install jq

Now curl the same endpoint but this time pipe the response into JQ.

$ curl https://pokeapi.co/api/v2/pokemon/charizard | jq

Now, the Json should be indented in a way you can read it easily. It may even have color!

Currently, our data is only printed to our STDOUT, it's not persisted anywhere. We want to keep this around for future use. Do you remember the curl option to do this? Scroll down after you take a guess.

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

If you guessed --output or -o, you're correct!

Let's save our Json to a file called charizard.json

$ curl -o charizard.json https://pokeapi.co/api/v2/pokemon/charizard 

Now, let's grab another one. Pick another Pokemon and substitute the API endpoint with its name. Also change the file name to match. For instance:

$ curl -o dragonite.json https://pokeapi.co/api/v2/pokemon/dragonite

I don't know about you, but these Json payloads contain more information then I need. The JQ tool has a way to scope the data by specific fields. There is a learning curve, but if you follow along, you'll at least get a basic understanding.

The Json payload has three keys that we're interested in: "name", "id", and "types". Type this command, and you'll get the response scoped to only these attributes and printed to your console.

$ curl https://pokeapi.co/api/v2/pokemon/charizard | jq '{name, id, types}'

Now, to refine this slightly, let's only grab each type name and put it in an array. Here's what that would look like.

$ curl https://pokeapi.co/api/v2/pokemon/charizard | jq '{name, id, types: [.types[].type.name]}'

And the output should be the following:

{
  "name": "charizard",
  "id": 6,
  "types": [
    "fire",
    "flying"
  ]
}

Final modifications

To finish up, let's explore two more helpful tips.

1. Reformatting the .json files we've already made
2. An alternate way to pipe output to a file

Use Vim to modify the .json files.

Don't worry, I'll help you get out. Or, you can just purchase a book like this...

Enter the following command to start editing the charizard.json file.

$ vim charizard.json

You need to enter the command mode. You can do so by typing the colon key :.

Now your cursor should be at the bottom. Enter the following command.

Note, you can only use this if you have JQ installed.

:%!jq '.'

Hit return and the file should instantly reformat itself!

Now type colon again and the following command to save and exit.

:wq

Hit return, and you're successfully exited vim. Congratulations!

Now if you cat the charizard.json file, it should be formatted correctly. You can also trim down the file with a vim command like the following. This will remove all excess data from the file.

:%!jq '{name, id, types: [.types[].type.name]}'

Now, if you don't want reformat or trim down file in the first place, instead we'll trim and format the data before writing to a file.

In order to do this, we can't use curl's -o option. We need to first pipe the response to JQ for formatting, then pipe that output to a file.

Here's how it's done.

Notice, we're using the \ to put our command on multiple lines

$ curl https://pokeapi.co/api/v2/pokemon/lapras | \
    jq {name, id, types: [.types[].type.name]} | \
    > lapras.json

Now, the lapras.json file will have the formatted and scoped content sent to it by JQ. To do that, we just use the > key followed by the name of a file.