e2e testing Netlify deploy previews

When working on my side projects I find it very important to get a straight testing process; since the projects can be untouched for months I want to be sure everything is ok when I push new code. Usually I use Netlify to deploy my sites; it's free (open source plan), it's easy to integrate with GitHub and it gives me deploy previews for each pull request.

Since e2e testing is all about being as close as possible to "real" environments I want my tests to run against Netlifys deploy previews instead of serve or some other local server setup (actually I've found bugs which was only produced in Netlifys production).

Netlify is creating deploy previews by listening for new PR's. I thought it was possible to deploy previews from CLI instead, and then wait for the response to run my e2e tests. Something like:

yarn netlify deploy && yarn e2e

But it seems there's no way to deploy previews from Netlifys REST API neither is it possible with their incoming webhooks, so I had to figure out an alternative way.

Bridge between Netlify and CI environment with AWS Lambda

To solve my problem I added an outgoing webhook who calls an URL upon deploy succeeded. I pointed the webhook to a custom AWS Lambda function I called netlify-travis-integration. netlify-travis-integration reads the deploy preview URL from the payload and triggers a Travis job to run e2e testing against.

Travis configuration

Since I couldn't find a way to run a specific Travis job I had to distinguish my non-Netlify required jobs from my Netlify required jobs by checking if the build was triggered by API or not:

jobs:
  include:
    - stage: lint
      if: type != api
      script: yarn lint
    - stage: test
      if: type != api
      script: yarn test:ci && codecov
    - stage: packtracker
      if: type != api
      script: yarn build
    - stage: lighthouse
      if: type = api
      script: yarn lh -- --perf=96 --bp=96 --no-comment $SITE_URL
    - stage: e2e
      if: type = api
      script: yarn e2e

This is a bit ugly, but I couldn't find a better way to solve it when using Travis. I think this is easier to solve with CircleCI, so I'll prefer to choose that for my next project.

GitHub statuses

To keep track of my e2e tests I added a webhook to my Travis config to let it call another AWS Lambda function. This function checks if the Travis job was triggered by an API and then creates a pull request status.

Conclusion

After one year and 100 pull requests I'm satisfied with the solution and it has caught a lot of errors I've missed. Especially it's comfortable when using Renovate who creates PR's for dependency updates, then it's easy to find out whether something breaks or not.

Way forward

I created netlify-travis-integration just to solve a problem I had, but in case there's others who'd like to solve the same problem it's not super easy to do the setup; you need to create an AWS account, setup AWS permissions, configure Travis and Netlify and also create a GitHub API token. It should be considered as a proof of concept. Maybe I'll take the next step and make it easier to integrate with new projects.

How are you solving this problem? Are you just running your e2e tests against a local web server?

netlify-travis-integration