Custom Gatsby.js Source Plugin

One of the greatest superpowers of Gatsby.js is its ability to use data and content from a variety of sources, such as Wordpress, Markdown files, or APIs. During the build process, Gatsby will query for the data and turn it into content for a static site. There are a wide variety of plugins, called "source plugins", that will help you pull data from different places. Today we'll be looking into how we can make our own source plugin and how they work under the hood.

Gatsby Custom Source Plugin ( view source )

The source we want is https://rawg.io/apidocs, which is an API for video games. Before we use the API, let's take a look at how data and sources work in Gatsby. We'll use the Gatsby default starter to kick things off.

# Download and install the starter using the Gatsby CLI
gatsby new games-source https://github.com/gatsbyjs/gatsby-starter-default
cd games-source
gatsby develop

In Gatsby, there is a GraphQL layer which helps you pull in data. You can build sites without the GraphQL layer, but the GraphQL helps to structure the data and makes it a lot easier to work with.

With the development server running, open up the GraphiQL playground at http://localhost:8000/___graphql. On the lefthand side, you should see a variety of dropdowns and checkboxes under the Explorer sidebar. This is the current data that's available to you via the GraphQL data layer, built into the Gatsby Default Starter. We can make a query for the site's metadata, which is coming from the gatsby-config.js file, using the following query:

query {
  site {
    siteMetadata {
      author
      description
      title
    }
  }
}

If you paste that into the box and hit the "Play" button, you'll see the siteMetadata show up in the right box.

Whenever we add a source plugin, new sources will appear in the Explorer sidebar and we can run queries on those sources. To see this in action, we can add our own custom test source into Gatsby.

Open up the file called gatsby-node.js. This file allows us to have access to Gatsby's APIs, the one we'll focus on today is the sourceNodes API. We can now make data "nodes" for the GraphQL data layer.

// In gatsby-node.js
exports.sourceNodes = async ({ actions, createNodeId, createContentDigest }) => {
  // We'll make the newNode object here for clarity
  const newNode = {
    title: 'Test Node!',
    description: 'This is a test node with static data',
    id: createNodeId('TestNode-testid'), // required by Gatsby
    internal: {
      type: 'TestNode', // required by Gatsby
      contentDigest: createContentDigest('testnode') // required by Gatsby, must be unique
    }
  };
  // This is where we actually create the data node, by passing in the newNode object.
  actions.createNode(newNode);
};

We create the actual node using actions.createNode(), which takes in an object. There are certain properties that are required by Gatsby in order for a node to work. We use createNodeId and createContentDigest, which are helper functions provided by Gatsby, to help use fill in those required properties.

To see this new node, we have to restart our development server. In the terminal, terminate the server using control + c and restart it using gatsby develop.

Open up the GraphiQL playground again and paste in this query:

query MyQuery {
  testNode {
    title
    description
    internal {
      contentDigest
      type
    }
  }
}

As you can see, we now have the test data node showing up!

We now know how to create data nodes, so the next step is to use our video game API to get real data and create nodes. All the functionality in gatsby-node.js runs using Node.js, and because of that we'll need to install an additional npm package that allows us to fetch data using Node.js.

Shut down the development server and install the package node-fetch

yarn add node-fetch

You can use either yarn add or npm install, but since the Gatsby CLI uses yarn, it's probably the better choice right now.

Open up gatsby-node.js again with the following

const fetch = require('node-fetch');

exports.sourceNodes = async ({ actions, createNodeId, createContentDigest }) => {
  try {
    // Fetch the data
    const res = await fetch(`https://api.rawg.io/api/games`);

    // Transform the data into json
    const data = await res.json();

    // Map over the results array, calling action.createNode on each item in the array
    data.results.forEach(game => {
      const node = {
        ...game, // We copy all of the properties from the game object
        id: createNodeId(`RAWG-game-${game.id}`), // Needs to be unique
        internal: {
          type: 'RAWGGame',
          contentDigest: createContentDigest(game) // We pass in the game object to make sure it's unique
        }
      };

      // Create the actual data node
      actions.createNode(node);
    });
  } catch (error) {
    console.log(error);
  }
};

Be sure to require the node-fetch package at the top of the file. We do a GET request on the endpoint https://api.rawg.io/api/games. I like this as an example because there are no other parameters needed to use this endpoint. Once we get the data, we loop over data.results and using each game object in the array, we make our data nodes.

Restart the development server and head over to http://localhost:8000/___graphql once again. You can now run queries on the games like so:

query MyQuery {
  allRawgGame {
    edges {
      node {
        id
        name
        background_image
      }
    }
  }
}

We can access all the data under allRawgGame > edges > node. Using the amazing GraphiQl explorer, we can toggle all the fields available to us with ease! So now we have this data, let's use it in our Gatsby site. Open up the main src/pages/index.js page.

import { graphql } from 'gatsby';
import React from 'react';
import Layout from '../components/layout';
import SEO from '../components/seo';

// The GraphQL data is passed into the component via props.data
const IndexPage = props => {
  return (
    <Layout>
      <SEO title="Home" />
      {props.data.allRawgGame.edges.map(({ node }) => {
        return (
          <div key={node.id} style={{ marginBottom: 20 }}>
            <p>{node.name}</p>
            <p>
              <img src={node.background_image} alt={node.name} />
            </p>
          </div>
        );
      })}
    </Layout>
  );
};

// We use the GraphiQL query here
export const query = graphql`
  query MyQuery {
    allRawgGame {
      edges {
        node {
          id
          name
          background_image
        }
      }
    }
  }
`;

export default IndexPage;

Now if you go to the normal development site at http://localhost:8000/ you'll see our games! During the gatsby-build process, this data will be injected into our static site and we can just serve it up, blazingly fast!

Now if you wanted to make this into a stand-alone source plugin, you can create a new directory called plugins in the root folder. You'll copy over the gatsby-node.js file into that folder and create a package.json that describes the source plugin and also the dependency node-fetch.

To use the plugin, you would add it into the gatsby-config.js file like the other plugins. If your source API needs an API key or other options, you can pass those options in when adding it into the gatsby-config.js file

plugins: [
  {
    resolve: `gatsby-source-rawg`,
    options: {
      apiKey: 'API_KEY_HERE'
    }
  }
];

In the gatsby-node.js file, you can access these new options in the sourceNodes like so

exports.sourceNodes = async ({ actions, createNodeId, createContentDigest }, options) => {
  console.log(options); // The options are passed as another argument
  try {
    const res = await fetch(`https://api.rawg.io/api/games`);
    const data = await res.json();
    data.results.forEach(game => {
      const node = {
        ...game,
        id: createNodeId(`RAWG-game-${game.id}`),
        internal: {
          type: 'RAWGGame',
          contentDigest: createContentDigest(game)
        }
      };
      actions.createNode(node);
    });
  } catch (error) {
    console.log(error);
  }
};

Also, it's worth noting that currently the images are NOT static, they will be loaded from whatever url we get back from the endpoint. We'll make these static using the Gatsby Image plugin in a future tutorial. Hope this was helpful!

Gatsby Custom Source Plugin ( view source )