Visualize your Fastly traffic on a real time globe using Glitch

On the Fastly developer hub, we recently added a visualization of the traffic flowing across the Fastly edge network. It was a big hit and customers ask me how they can visualize their own traffic like this.

Fear not. I've repackaged the React component we made to do this as a Glitch app, so you can create your very own globe. Not only that, but a lot of the feedback I was getting was stuff like "This would look great on the giant screen in our lobby" so we made a non-interactive version designed to fill a 16:9 screen. Let's dive in.

Make a log processor app

First, you need somewhere for Fastly to send your log data. A Glitch app is ideal for this (Glitch is a collaborative and free code playground which is now part of Fastly).

1. Remix the fastly-globeviz-data Glitch app
2. Note the URL that Glitch assigns to your new app, eg monthly-ferret-taxi.glitch.me or glitch.com/edit/#!/monthly-ferret-taxi.

What does this app do? It's a one-file ExpressJS app that has a few route handlers:

• POST requests to / are unpacked and each line is treated as an event to emit on a Server sent events stream.
• GET requests to / receive a streaming response that includes any events as they are created.
• A GET request to /.well-known/fastly/logging/challenge is answered as required by Fastly's log challenge rules for HTTP endpoints.

Now you have your own log collector, it's time to send some data to it!

Instrument your Fastly service

Configure your Fastly service to emit traffic data to the Glitch app you just created.

1. Create a Fastly service if you don't have one already. If you do, create a new draft version of the service.
2. Decide on a three character code to identify this traffic, e.g. if you are the New York Times, you could use NYT. This is so that if you want to you can send multiple different website traffic streams to the same globe visualization, you can identify them and colour code them.
3. Decide on a sample rate. A good rate to sample is 20-50 requests per second. If that represents roughly a thousandth of your traffic, the sample rate would be 1000. A sample rate of 1 is 100% of traffic. Err on the side of sampling too little and dial it up, otherwise you may trigger rate limiting on Glitch.

4. Add the following VCL as an INIT VCL snippet:
   

   table globeviz_config {
     "service_name": "ABC", // Change this value
     "sample_rate": "1000"  // Change this value
   }
   
   // https://deviceatlas.com/device-data/explorer/#defined_property_values/7/2705619
   table globeviz_known_browsers {
     "Chrome": "C",
     "Chrome Mobile": "C",
     "Firefox": "F",
     "Firefox for Mobile": "F",
     "Edge": "E",
     "Kindle Browser": "K",
     "Safari": "S",
     "Samsung Browser": "A"
   }
   
   // Record the following data (space separated)
   //    PIO             Name of source service (3 char)
   //    LCY             Name of Fastly POP (3 char)
   //    51.43,-0.23     Client location (var length)
   //    17234           Duration (var length)
   //    EN              Normalized Accept-Language (2 char)
   //    M               Cache state (1 char: H, M, P)
   //    2               HTTP version (numeric, variable length)
   //    1.2             TLS version (numeric, variable length)
   //    C               Browser (1 char: C, F, E, K, S, A)
   //
   sub vcl_log {
     // Record only non-shielding, non-VPN requests at the specified sample frequency
     if (fastly.ff.visits_this_service == 0 && client.geo.proxy_type == "?" && client.geo.latitude != 0 && randombool(1,std.atoi(table.lookup(globeviz_config, "sample_rate", "100000")))) {
       log "syslog " req.service_id " fastly-globeviz :: "
         table.lookup(globeviz_config, "service_name", "-") " "
         server.datacenter " "
         client.geo.latitude "," client.geo.longitude " "
         time.elapsed.usec " "
         std.toupper(
           accept.language_lookup(
             "en:de:fr:nl:jp:es:ar:zh:gu:he:hi:id:it:ko:ms:pl:pt:ru:th:uk",
             "en",
             req.http.Accept-Language
           )
         ) " "
         substr(fastly_info.state, 0, 1) " "
         regsuball(req.proto, "[^\d.]", "") " "
         regsuball(tls.client.protocol, "[^\d.]", "") " "
         table.lookup(globeviz_known_browsers, client.browser.name, "Z")
       ;
     }
   }
   

5. Create an HTTPS log endpoint called fastly-globeviz pointing to your Glitch URL (choose a content type of "text/plain" and a placement of "none"). You can do that in the UI or with this cURL command:
   

   curl -i  -X POST "https://api.fastly.com/service/$SERVICE_ID/version/$VERSION/logging/https" -H "Fastly-Key: $YOUR_FASTLY_TOKEN" -H "Content-Type: application/x-www-form-urlencoded" -H "Accept: application/json" -d "name=fastly-globeviz&placement=none&content_type=text/plain&url=https://$GLITCH_APP_ID.glitch.me/"
   

6. Activate the new version of your service

Test the log receiver

Open the URL of your glitch app in a browser and you should start to see traffic log data streaming in.

This is the data your Fastly service is emitting. This is a good time to check that you're happy with the type of data you're sharing with the world, since the Glitch app you're sending it to is exposing it publicly.

View it on a map!

Now you can visualize your log data on a globe. You don't actually need your own version of the globe, so you don't need to remix an app for this, you can just use mine directly - and point it at your data source.

1. Go to https://fastly-globeviz-ui.glitch.me/ in your browser. You should see traffic animating, but that's not your traffic right now, it's the default traffic source.
2. Add a query parameter ?SOURCE_URL=<your log collector> to the URL, e.g. ?SOURCE_URL=https://monthly-ferret-taxi.glitch.me. You should now see your data!
3. Go look at the src/config.js file in the UI Glitch app. This is where you can see all the variables that are configurable. SOURCE_URL was one of them, but they are all overridable by adding more query parameters to your URL. If you prefer, you can remix the app and edit config.js directly.

Give feedback and contribute

Both the Glitch apps are remixable, so go ahead and make changes, and let us know what you come up with in the Fastly community forum!