A/B Testing Next.js Applications

When you build a website, your goal is potentially to attract visitors. It is especially true if your website is a software as a service (SaaS) that you want to show to as many potential customers as possible. An obvious choice is to run some marketing campaigns and see how the website (and the software) works with real users. Some folks even promote their landing pages well before the software is built to know the market's reaction before jumping into a new project that could last several months. If there are no registrations or downloads during the marketing phase, they go to the next project and don't waste time on a useless idea.

But how can you determine whether a landing page is good without having something to compare it to? What if you only have to change the motto or the main hero image, and those small changes would result in 5 times as many registrations? How can you test these alternatives with a single website using a single domain? This concept is called A/B Testing.

What is A/B Testing?

A/B Testing (or split testing) is when you have two versions of the same website, version A and version B (hence the name), and you want to test which one converts visitors into users better. The change between the versions can be anything:

• The background color of the primary call to action button,
• The hero image,
• The motto,
• The alignment of the text,
• Or anything you can think of.

When two versions of the same website are available to the public, you gain invaluable information. You can not only calculate how many visitors out of 1000 buy or sign up for your product but also test whether an extremely simple change results in a significantly better (or worse) performance. You can experiment with any combination you can think of.

Large companies often do this and have an easier time than us indie makers since they have regular site visitors. It is very easy for them to test a new feature or website motto on 1% of the users because even that one percent is probably tens of thousands or even millions of people. When an individual launches a new product, they have fewer visitors, meaning less information. Still, running an A/B test on two user groups with 50-50% can have surprising results.

How to A/B Test a Website?

No need to worry because A/B testing is well-established, and multiple tools are made exactly for this purpose. Here is a non-exhaustive list of them:

• VWO
• Optimizely
• AB Tasty
• Convert
• SiteSpect

These tools are A/B Testing beasts with in-depth analysis, extensive options, and much more to offer than most of you probably need. For this reason, most of them are rather pricy. Some are entirely out of your budget range, at least initially.

But running just a slight A/B test on your site still makes sense. Even if it is as simple as changing the main call-to-action button from "I'm interested" to "I need this". You don't know which will work, but if you never test alternatives, you never will.

How to A/B Test for Free?

– Mom, can we have A/B testing?
– We have A/B testing at home.
– A/B testing at home:

It's hard to find a tool with a small feature set suitable for individuals who want to test simple websites with a limited number of regular visitors. As a developer with multiple online tools, I also faced this problem. And just like a stereotypical developer would do, I developed my solution because I didn't want to pay for these expensive tools. I will probably need them later, but not at the start. Here is how mine works and how you can also use it for free on your website. Note that I am using Next.js, so this guide will work best if you have the same setup.

Install the nexperiment Node module with your favorite package manager as a first step. Feel free to check out the source code on GitHub and the module on NPM.

The first version of my A/B testing tool was less than 100 lines of code, and it served an extremely simple feature: to change one phrase somewhere on the site to a different one. As I mentioned, this can be a call to action button, a motto, or anything else you need. My goal was to test different button texts and mottos. The tool is as simple as a React Hook and a Context.

Once you successfully installed the module, wrap your application in the ABTestProvider context in one of your Layout files. Then, pass the items you want to test in the items parameter. The prefix parameter is optional, but ab-test- is its default value. It is the prefix used for the generated IDs to differentiate them from the rest of the application in a later step. It should look something like this:

import { TestItems, ABTestProvider } from "nexperiment";

export default function Layout({ children }: { children: React.ReactNode }) {
  return (
    <ABTestProvider items={items} prefix="ab-test-">
      {children}
    </ABTestProvider>
  );
}

The items parameter has the following interface:

const items: TestItems = {
  motto: {
    A: "Your words, now available in audio. Simple as that.",
    B: "Converting your blog into audio has never been easier.",
  },
  cta: {
    distribution: 0.7,
    A: "Let's hear it",
    B: "I need this",
  },
};

If you don't set a distribution, the default value will be 0.5, meaning that each version has a 50% probability of becoming visible to any user. In the example above, the motto will be A or B 50% of the time, while in the case of the call-to-action button text, A will be rendered 70% of the time and B in the rest.

You can now use the useABTest hook and its getItem method to retrieve a value in your code. Here is an example based on the above:

export default function HeroSection() {
  const abStore = useABTest();
  const cta = abStore.getItem("cta");

  ...

  return (
    <div>
      <button id={cta.id}>{cta.text}</button>
    </div>
  );
}

This code will select one of the call-to-action texts and the correspondent ID. When a user opens the website, the hook will choose one of the texts from the options for the button and store it in the local storage. It will also return a unique ID, which will be ab-test-cta-A or ab-test-cta-B, in case you didn't change the prefix to something else earlier. You will need these IDs in the next step because, so far, you only have a random website but no analytics about user behavior. The point of saving the chosen configuration to local storage is that once users see a version, they are stuck with it and won't experience the site changing on each refresh. They will permanently become user A or user B.

Tracking User Behavior

As I mentioned, using the module is completely free. There are no servers behind, no blocking JavaScript, nothing. You must rely on a separate analytics tool to collect user information. I chose Google Analytics (GA) and Google Tag Manager (GTM) to keep things simple, but you can use any other tool you are familiar with. From now on, the post will be about how to set up tracking with GA and GTM, so feel free to skip to the last part if you are going with an alternative.

The point of generating unique IDs in the previous step was to have the ability to easily track what consequences a simple change on the landing page has. If you don't use the generated ID, you will have no information about user behavior because you cannot decide which version a user saw before clicking the button.

With these IDs, tracking button clicks is easy. You only have two steps left:

• You have to configure a Trigger in GTM that collects every click on elements with IDs that start with ab-test- (or your custom prefix),
• You must configure a Tag that uses this Trigger to send events to Google Analytics.

If you are unfamiliar with how to configure the above, you can find plenty of useful information about setting up Tag Manager on a site using the Next.js App Router in one of my previous posts. I only cover the basics there, but it is more than enough for this simple A/B testing use case.

Once you have everything set up, you should slowly see the events appearing in Google Analytics as the users visit both versions of your website.

Tips and Caveats

1. If you need an extensive A/B testing solution with in-depth analysis and dozens of features, this module is not for you. The potential users are indie makers on a small budget testing a simple and new landing page with different setups.

2. This module will only work with Next.js and React applications. I don't intend to extend it to other frameworks because this is the stack I am currently using. But I figured it might also be helpful for others using the same stack.

3. In its current version, you cannot configure boolean, image, or composite items where multiple texts belong to the same configuration. It only supports single strings. However, these use cases look simple and common enough that I plan to add them in later versions.

4. You can configure different interactions using the unique ID, not just a single button text and its click. For example, it is possible to configure the hero text as a variable and then use the corresponding ID for the login button in the navigation bar. This way, you can track which motto attracted more users. Use the same name when calling getItem in both locations. The hook will return the same information for a given item name regardless of how many times you call it.

If you are wondering about this now, you are completely fine without A/B testing. It's not a requirement for a successful business. You probably don't need it if you write amazing copy on each website on the first try. But if you like experimenting like me, you now have a simple tool in your arsenal to get started.