Demystifying the HTTP Host header

The HTTP host header is a small, important part of our modern web. It is used more than you might think in Fastly. In this post, we’ll dig into the history of the header and show how crucial it is to serving content stored in an object store through Fastly.

How it started: single domain hosting

When the web was growing up, some early assumptions worked well. As we’ll see, some of our assumptions had to change.

Let’s work through an example. It’s 1996. Gina G’s “Ooh Aah... Just a Little Bit” is playing on the radio. Real physical computers, which were probably beige, ran web servers on port 80. Thus, each server could only serve content for a single domain. You open up Netscape Navigator and browse to www.example.com.

The browser resolves www.example.com, connects to port 80 of the server, and sends the few characters:

GET / HTTP/1.0

The web server listens on port 80, receives the request and sends back an HTML response. Success!

The web took off: virtual hosting

As the global hypertext dream came true in practice, there was a lot of success. So much success that the limitation of only one domain per server became a problem. The Host request header is introduced.

Let’s work through an example. It’s 1997. The Cardigan’s “Lovefool” is playing on the radio. A real computer, which is now a sleek black, runs a pretend computer which in turn runs a web server. You open up Netscape Communicator and browse to www.example.com.

The browser resolves www.example.com, connects to port 80 of the server and sends the few characters:

GET / HTTP/1.1
Host: www.example.com

The web server listens on port 80, receives the request, pays attention to the Host request header and sends back an appropriate HTML response.

The effect of this is that the web server can use the Host header to direct your request to one of many websites hosted on the same machine. It's no longer necessary to have one IP address per website. And thank goodness, because we didn't have anywhere near enough IP addresses for that!

TLS

As the web grew, ecommerce started taking off. Sending credit card numbers and other sensitive data in plain text across the network became a problem. We needed to encrypt content in transit across the web. The Secure Sockets Layer (SSL) provided secure communications between web browsers and web servers and led to the Transport Layer Security (TLS) protocol.

It’s 1999. Britney Spears’ “...Baby One More Time” is playing on the radio and our web server is now run by something of indeterminate colour by a mass hosting company. You open up Internet Explorer 5 and browse to www.example.com.

Before the browser and server get to speak HTTP, they first participate in a TLS conversation. The browser connects to port 443. As part of the TLS Server Name Indication extension, the browser indicates the name of the server it is contacting. The secure web server directs your request to one of many web servers.

After TLS is negotiated, the HTTP conversation continues over the secure connection. Sensitive information is no longer carried in plain text across the network and the Dot-com boom happens.

Today

It’s today. As we speak, hypertext spans the globe and we now listen to radio using HTTP. Every request sends a Host request header (for HTTP/1.1 requests) or an :authority pseudo-header (for HTTP/2 and HTTP/3 requests).

Fastly’s powerful edge cloud platform enables developers to build exceptional websites and apps. We sit in between our customers’ customers and our customers' servers (origins). It’s common to modify the request Host header in the Fastly layer as your origins might have a different naming convention to your public domains.

For example, www.example.com might be the public domain, whereas the real service might run on production.example.com.

This simple case is easy to configure. For a Fastly Delivery or Compute service, you can specify an override host on the origin. For a Fastly Compute service, use override_host in fastly.toml for development. For production, the Fastly CLI sets the override by default when adding a backend.

Object stores

While running physical servers and software used to be the only way to go, many of our customers use on-demand central cloud computing platforms. In particular, object stores such as Amazon S3 and Google Cloud Storage are great for storing images, assets and even static websites. Having Fastly in front of these provides supreme global performance.

These object stores live on different domains so we need to modify the Host header as the request travels through Fastly.

For Amazon S3, the simplest way to access a bucket is to use Virtual-hosted–style access. The override host name should be in the format:

<BUCKET NAME>.s3.<REGION-CODE>.amazonaws.com

For Google Cloud Storage, the simplest way to access a bucket is similar. The override host name should be in the format:

<BUCKET NAME>.storage.googleapis.com

We have a lot of useful information about these settings on Overriding the Host header.

As before, for a Fastly Delivery service, you can specify an override host on the origin. For a Fastly Compute service, use override_host in fastly.toml for development. For production, the Fastly CLI sets the override by default when adding a backend.

Don’t forget to secure access to your buckets using a signed AWS authorization header or a Google Cloud HMAC authentication. Enable shielding to reduce latency and the number of requests made to the object store.

Conclusion

We’ve traveled through time and demystified the HTTP Host header, a small part of the large web.