Introduction
Playwright is a modern cross-browser testing framework developed by microsoft itself.
GitHub Actions is the out-of-the-box solution for anything related to CI/CD pipelines on GitHub.
Last but not least GitHub Pages is a GitHub feature which allows to deploy static websites.
And all of this comes completely free for public repositories!
In this blog post I will show you how to setup a basic a Playwright project, integrate it into GitHub Actions and finally deploy an HTML report of the test results to GitHub Pages.
ℹ️ INFO
The complete source code can be found in this repo: ysfaran/playwright-gh-actions-gh-pages
This post is not focusing on explaining all concepts of Playwright, but rather how to connect Playwright, GitHub Actions and GitHub Pages in a easy, non-cost way.
Setup Playwright
💡TIP
For an up-to-date installation guide please always refer to https://playwright.dev/docs/intro. At the point of writing this posts Playwright's latest version was1.25.1
.
Fortunately Playwright makes it really easy to setup a new project using the CLI:
yarn create playwright
This will start an interactive session. Make sure to at least enable GitHub Action workflow generation. Following is the configuration I used:
✔ Do you want to use TypeScript or JavaScript? · TypeScript
✔ Where to put your end-to-end tests? · tests
✔ Add a GitHub Actions workflow? (y/N) · true
Generate HTML Report
One of the most important generated files is playwright.config.ts
, so let's have a look at it:
ℹ️ INFO
I simplified some of the generated files. For a full list of these changes seeaab0b9c2
.
const config: PlaywrightTestConfig = {
testDir: "./tests",
// Run all tests within a file in parallel to speed up test execution
fullyParallel: true,
// Helpful for uncontrollable flaky tests, which are tests, occasionally failing for various reasons
retries: 3,
// Generates a HTML report to ./playwright-report/
reporter: "html",
use: {
// Tests will be run against this page
baseURL: "https://playwright.dev/",
// Collect trace when retrying the failed test. See https://playwright.dev/docs/trace-viewer
trace: "on-first-retry",
},
// Cross-browser testing setup
projects: [
{
name: "chromium",
use: {
...devices["Desktop Chrome"],
},
},
{
name: "firefox",
use: {
...devices["Desktop Firefox"],
},
},
{
name: "webkit",
use: {
...devices["Desktop Safari"],
},
},
],
};
Since the html
reporter is already enabled by default, there are no other adaption necessary.
To get a nice HTML report covering all test result cases (sucess
, fail
and flaky
) following tests have been implemented:
import { test, expect } from "@playwright/test";
test.beforeEach(({ page }) => page.goto("https://playwright.dev/"));
test("should succeed", async ({ page }) => {
await expect(page).toHaveTitle(/Playwright/);
});
test("should fail", async ({ page }) => {
await expect(page).not.toHaveTitle(/Playwright/);
});
test("should be flaky", async ({ page }) => {
if (Math.random() > 0.5) {
await expect(page).toHaveTitle(/Playwright/);
} else {
await expect(page).not.toHaveTitle(/Playwright/);
}
});
To run the tests execute:
npx playwright test
After running all tests Playwright prints a summary of test restuls and publishes the HTML report to a server at a localhost address. Playwright also emits all sources for the HTML report to playwright-report/
.
Integrate GitHub Actions
Luckily Playwright already generated a basic .github/workflows/playwright.yml
during Setup Playwright to integrate Playwright tests into a GitHub Actions worklow:
name: Playwright Tests
on:
push:
branches: [main, master]
jobs:
test:
timeout-minutes: 60
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/setup-node@v2
with:
node-version: "14.x"
- name: Install dependencies
run: yarn
- name: Install Playwright Browsers
run: npx playwright install --with-deps
- name: Run Playwright tests
run: yarn playwright test
- uses: actions/upload-artifact@v2
if: always()
with:
name: playwright-report
path: playwright-report/
retention-days: 30
Essentially the workflow is triggered on every push
and executes the all tests, just like you would do it locally.
It's important to note that it also uploads the Playwright HTML report as workflow artifact.
This artifact will be available to download in the Actions
tab of your GitHub repository in each workflow run.
Please refer to the official Playwright docs on how to download and view HTML report files of a specific workflow run.
Publish HTML Report to GitHub Pages
Being able to download, view and debug HTML reports generate by a CI/CD pipeline locally is already convenient, but there is an easier and faster way to check test results: automatically publish the HTML reports to GitHub Pages and view them directly in the browser.
Firstly a new orphan branch, which means the branch has no parent commit, for the GitHub Pages' static content needs to be created:
# Create a new branch without any commit on it
git checkout --orphan gh-pages
# Source files get autoamtically staged so remove them from git
git rm -rf .
# Optional: Check if there are really no files in the git staging area anymore
git status
# Create an initial, empty commit
git commit --allow-empty -m "setup empty branch for GitHub Pages"
# Push the branch to make it available online
git push --set-upstream origin gh-pages
Secondly the GitHub Pages feature needs to be enabled for your repo. Usually GitHub is enabling GitHub Pages automatically when you name a branch gh-pages
.
If it doesn't, make sure to enable GitHub Pages manually in your repository settings:
Any change that is pushed to gh-pages
will now automatically update the GitHub Pages website, which is publicly available at https://<user>.github.io/<repo>/
.
Because there is no real commit on the gh-pages
branch yet a 404 error page will be shown.
So the next logical step is to add a new job to the GitHub Actions workflow in order to push the HTML reports to gh-pages
branch:
publish_report:
name: Publish HTML Report
# using always() is not ideal here, because it would also run if the workflow was cancelled
if: "success() || needs.test.result == 'failure'"
needs: [test]
runs-on: ubuntu-latest
continue-on-error: true
env:
# Unique URL path for each workflow run attempt
HTML_REPORT_URL_PATH: reports/${{ github.ref_name }}/${{ github.run_id }}/${{ github.run_attempt }}
steps:
- name: Checkout GitHub Pages Branch
uses: actions/checkout@v2
with:
ref: gh-pages
- name: Set Git User
# see: https://github.com/actions/checkout/issues/13#issuecomment-724415212
run: |
git config --global user.name "github-actions[bot]"
git config --global user.email "41898282+github-actions[bot]@users.noreply.github.com"
- name: Download zipped HTML report
uses: actions/download-artifact@v2
with:
name: playwright-report
path: ${{ env.HTML_REPORT_URL_PATH }}
- name: Push HTML Report
timeout-minutes: 3
# commit report, then try push-rebase-loop until it's able to merge the HTML report to the gh-pages branch
# this is necessary when this job is running at least twice at the same time (e.g. through two pushes at the same time)
run: |
git add .
git commit -m "workflow: add HTML report for run-id ${{ github.run_id }} (attempt: ${{ github.run_attempt }})"
while true; do
git pull --rebase
if [ $? -ne 0 ]; then
echo "Failed to rebase. Please review manually."
exit 1
fi
git push
if [ $? -eq 0 ]; then
echo "Successfully pushed HTML report to repo."
exit 0
fi
done
- name: Output Report URL as Worfklow Annotation
run: |
FULL_HTML_REPORT_URL=https://ysfaran.github.io/playwright-gh-actions-gh-pages/$HTML_REPORT_URL_PATH
echo "::notice title=📋 Published Playwright Test Report::$FULL_HTML_REPORT_URL"
Basically this job commits the HTML report files to the gh-pages
branch, which will then automatically redploy the website.
It does so by specifying a new unique path for each test result:
HTML_REPORT_URL_PATH: reports/${{ github.ref_name }}/${{ github.run_id }}/${{ github.run_attempt }}
The job then tries to push these changes and rebases if necessary until a push was succssful.
Rebasing is safe here, because - as already mentioned - each workflow run gets its own uniquie path, so there can't be any file conflicts ever.
Last but not least the publicly available URL is printed as GitHub Worfklow Annotation:
💡 TIP
It might take some while for GitHub Pages to update its content after a new push, so the printed URL will point to a 404 page initially.
In theActions
tab of your repo you can checkpages-build-deployment
workflow runs, which is automatically triggered for GitHub Pages, to see if the GitHub Pages deployment was successful.
Add Multi-Branch Support
Only in rare cases you work on a single branch. Also the pipeline is currently not integrated in any pull request process, meaning that there are no restriction to push or even force push to your main
/master
.
Its time to change that!
First of all we need to enable branch protections rules for the default branch:
- Go to the settings of your repo
- Select
Branches
underCode and automation
in the menu - Hit the
Add branch protection rule
button - Add your default branchs name for
Branch name pattern
- Configure your branch protecting rules as desired, most importantly being:
Note that I only put Test
under Status checks that are required.
because you should still be allowed to merge to main
branch if some HTML report couldn't be published.
A green Test
job for a branch represents a valid state anyway in this case.
Secondly the workflow needs to be adapted:
on:
push:
- branches: [ main, master ]
- pull_request:
- branches: [ main, master ]
+ branches-ignore: [ main, gh-pages ]
So the workflow is triggered after each push on every branch except main
and gh-pages
.
I purposely removed pull_request
here, because it's really hard (at this point in time) to make push
and pull_requests
work together with GitHub Actions.
Feel free to check my answer on StackOverflow answer regarding this topic.
⚠️ CAUTION
If you want to work withpull_request
every occurence of${{ github.ref_name }}
should be replaces with${{ github.head_ref || github.ref_name }}
.
Check the related StackOverflow answer for details: https://stackoverflow.com/a/71158878/6489012.
I also highly recommend to enable concurrency groups in your workflow file to make sure only one workflow is running at the same time for each branch:
concurrency:
group: ${{ github.ref_name }}
# optional
cancel-in-progress: true
Delete Reports from GitHub Pages
After a branch was successfully merged it makes sense to delete all corresponding HTML reports from GitHub Pages. This requires the implementation of a new workflow file:
name: Delete
on:
delete:
branches-ignore: [main, gh-pages]
# ensures that currently running Playwright workflow of deleted branch gets cancelled
concurrency:
group: ${{ github.event.ref }}
cancel-in-progress: true
jobs:
delete_reports:
name: Delete Reports
runs-on: ubuntu-latest
env:
# Contains all reports for deleted branch
BRANCH_REPORTS_DIR: reports/${{ github.event.ref }}
steps:
- name: Checkout GitHub Pages Branch
uses: actions/checkout@v2
with:
ref: gh-pages
- name: Set Git User
# see: https://github.com/actions/checkout/issues/13#issuecomment-724415212
run: |
git config --global user.name "github-actions[bot]"
git config --global user.email "41898282+github-actions[bot]@users.noreply.github.com"
- name: Check for workflow reports
run: |
if [ -z "$(ls -A $BRANCH_REPORTS_DIR)" ]; then
echo "BRANCH_REPORTS_EXIST="false"" >> $GITHUB_ENV
else
echo "BRANCH_REPORTS_EXIST="true"" >> $GITHUB_ENV
fi
- name: Delete reports from repo for branch
if: ${{ env.BRANCH_REPORTS_EXIST == 'true' }}
timeout-minutes: 3
run: |
cd $BRANCH_REPORTS_DIR/..
rm -rf ${{ github.event.ref }}
git add .
git commit -m "workflow: remove all reports for branch ${{ github.event.ref }}"
while true; do
git pull --rebase
if [ $? -ne 0 ]; then
echo "Failed to rebase. Please review manually."
exit 1
fi
git push
if [ $? -eq 0 ]; then
echo "Successfully pushed HTML reports to repo."
exit 0
fi
done
The job of this workflow is pretty similiar to the one publishing the HTML report. This time it's just the other way around: gh-pages
branch is checked out and the folder reports/<branch-name>
is deleted.
Then the same push-rebase loop is initated to trigger a GitHub Pages deployment.
Important to note here is the following part:
ℹ️ INFO
This time the name of the (deleted) branch is neither retreived bygithub.head_ref
andgithub.ref_name
, butgithub.event.ref
.
In case the workflow was triggered by adelete
the value ofgithub.ref_name
will be the one of the default branch, heremain
.
concurrency:
group: ${{ github.event.ref }}
cancel-in-progress: true
This statement makes sure that any running workflow within the same concurrency group will be cancelled.
In our case this means if tests for a deleted branch are still running, they will be cancelled.
Conclusion
We created a Playwright project from scratch, integrated tests in GitHub Actions, deployed test results to GitHub Pages and finally established branch protections rules with a simple but important pull request strategy.
All of this, without any costs.
All in all it's all about automation and process optimization. Keeping a team busy with daunting manual tasks can decrease effincy and motivation quite a bit.
This can go from manual deployment plans to smaller aspects like hard to debug test results. In the long run saving some minutes of daily, manual work can have a huge impact.
Questions and feedback are always welcome. 🙂
Top comments (2)
This is exactly what I wanted and needed!
Thanks for sharing the specifics of exactly how to execute this idea well.
You are welcome, happy to hear that it helped someone :)