A Beginner's Guide to Curl: Part 5 - Setting Headers

HTTP requests are sent with metadata contained in headers. This includes data such as the request method, authentication credentials, and cookies.

Headers consist of a "field name" and "field value" separated by a colon. Multi-word field names are separated by dashes and are case-insensitive.

Field-Name: Some Value or field-name: Some Value

You can see exactly which headers are being passed by adding the -v or --verbose flag to your curl requests. There is also a free service that has an API endpoint that echos your request headers back to you as a JSON response.

Type the following command to see the default headers sent by curl.

$ curl https://httpbin.org/headers

I got this JSON response in return.

{
  "headers": {
    "Accept": "*/*",
    "Host": "httpbin.org",
    "User-Agent": "curl/7.85.0"
  }
}

By default curl accepts any content type in return, "*/*". Normally the "Accept" header would specify the MIME type the request is expecting in return, such as "text/html" or "application/json".

The "Host" header is set because of the URL supplied to curl.

The "User-Agent" in this case is the curl application, but let's switch that now.

Send this instead.

$ curl -H 'User-Agent: Mozilla/5.0' https://httpbin.org/headers

Now, as you can see, the response shows your agent header set to 'Mozilla/5.0'. All headers can be set in Curl with the -H or --header. Headers can be placed before or after the URL.
If you want to set multiple headers at once, you'll need us use multiple flags. Here's a way you can construct a complex curl request on multiple lines using the backlash to escape newlines.

(By the way, you can make up your own headers, and you don't have to prefix them with an 'X'.)

$ curl https://httpbin.org/headers \
    -H 'Accept: application/json' \
    --header 'Referer: https://dev.to' \
    -H 'My-Cool-Custom-Header: foo'

Authorization Header

One of the main usages of HTTP headers is authentication. Interestingly the header for this is called 'Authorization'. I guess these two concepts are similar enough.

Authorization headers work by specifying a scheme followed by a space and the value. Here's an example.

Authorization: basic username:password

Here's an API to test out basic auth. You construct the endpoint using basic-auth as the first segment followed by the username of your choice, and finally the password. In our case, we will attempt to authenticate via the endpoint basic-auth/username/password.

The API will either return a 200 or 401 response. Supplying the -i or --include flags will reveal the response headers and status code.

$ curl -i https://httpbin.org/basic-auth/username/password \
    -H 'Authorization: basic username:password`

Huh? Why is it a 401?

HTTP/2 401

This is because the internet specifies that basic HTTP auth credentials be encoded using base64.

Base 64 is not encryption. Nothing about it is secure, it's just a different encoding. This video offers a great explanation.

You can easily encode anything you want in Base64 without leaving the command line. Just echo an expression and pipe it to the base64 cli.

$ echo -n username:password | base64

You should see the result as dXNlcm5hbWU6cGFzc3dvcmQ=.

Ok, now that we have the string 'username:password' encoded in Base64, let's try the API again, and see if we can successfully authenticate.

$ curl -i https://httpbin.org/basic-auth/username/password \
   -H 'Authorization: basic dXNlcm5hbWU6cGFzc3dvcmQ='

And here's the response!

HTTP/2 200
{
  "authenticated": true,
  "user": "username"
}

Hooray!

But that seems pretty tedious. Thankfully, the developers of Curl already thought of that. That's why Curl provides a special option for the authorization header. Just provide the -u or --user flag followed by the username:password in plain text. You can specify the the scheme with the --basic; however, it's already basic by default. Curl will then encode your username and password for you.

$ curl -i https://httpbin.org/basic-auth/apple/banana
    --user 'apple:banana'

To verify that Curl really did encode the credentials, just supply the -v or --verbose flag to see all request headers. In my case with apple and banana, my header looked like this.

authorization: Basic YXBwbGU6YmFuYW5h

Another type of HTTP authorization is called bearer. Instead of supplying a username and password, you would use the word 'bearer' followed by a space and some sort of token. Oftentimes these tokens are randomly generated keys or actual data that's gone through a hashing algorithm, such as a JSON Web Token (JWT) which contains JSON data.

Once you're confident supplying header data in your Curl requests, you'll find it easy and fast to test out APIs straight from the command line.