Implementing A/B Testing on your Shopify App with Mixpanel and Split.io

One of the keys to any successful app is validation of the idea. When you create an app you want to know if that app is making a difference to your users. You can then use that information to research and discover new ways that your app is not only being interacted with, but also changes and adjustments you can make (small or large) to improve your usability, performance and enhance the experience for the brands using your app. This is where A/B testing (split testing) comes in. It's more focussed on the side of validation of ideas, but is equally important in discovering how your app can improve based on the way it's being used currently alongside demonstrating the proof of your app’s value to your customers

In this post we'll be talking about how we've implemented A/B testing, how we use it to track conversion rates with our application, and how you can do the same or utilise the same tools to track different metrics for your own applications. Conversion rates are important in the case of any app utilising App Blocks or Embeds, as we need to ensure we are not decreasing the number of sales that a customer store makes, or even demonstrate that we are improving that number of sales as well. We'll also be briefly touching upon how we can then action these metrics to make data driven decisions in changing an application to improve the user experience.

The Toolbox

The title of this post mentions two tools: Mixpanel and Split.io. Let's go over the roles that they will play in our split testing setup:

• Mixpanel - This allows us to collect "events" on the frontend of our application as well as to pass properties to those events to fine tune the data we collect and the state of our app as users interact with it. These events could occur on the click of an element (i.e. a button or link) and the data supplied could be the current product or variant ID, or even the items they have in their basket, etc. We feed these into a dashboard that we then use as a reference to see how / how often different parts of the application are interacted with.
• Split.io - This SDK allows us to direct traffic selectively. In this case we may want to show 50% of users the page with the application and 50% of users the page without the application present. We can then infer metrics from the way that these users interact with the page in combination with Mixpanel to draw conclusions on how effectively our app is increasing, decreasing or maintaining certain interactions with a page. We can show that our apps decrease, increase or maintain conversion rates on variants for instance.

By routing traffic selectively to pages with Split.io and then tracking how that traffic interacts with the pages they see, we can gain full insight into how a user is leveraging our app or how our app is influencing the user flow. Therefore we are capable of understanding the full user journey and the role our app plays in that flow to either positive or negative effect, thus allowing us to make data driven adjustments to improve a specific target metric or improve the overall experience for our users.

Let's code!

For our implementation, we should follow the user journey to simplify the implementation for ourselves. So let's start with Split.io. As we mentioned, Split.io is the first thing that loads in our app to check which group a user belongs to (group A or group B, where one sees the application and one doesn't), so let's look at how to load Split.io within our app.

We first need to download the split.min.js file from the Split.io website and put it in our assets directory. We can then use a bit of JS magic to load the Split library in asynchronously as to not affect load times and only when it is specifically needed. We do this using a loadSplit function shown below:

async function loadSplit() {
  return new Promise(resolve => {
    const flareContent = document.getElementById("flare-add-to-cart");
    const script = document.createElement("script");
    script.src = "{{ "split.min.js" | asset_url }}";
    script.onload = () => {
      resolve(splitio);
    }
    flareContent.appendChild(script);
  })
}

We can call this function inline with a .then() following to run code once the library is loaded

loadSplit().then(() => {
// Split Testing Code
})

// Alternatively use the async/await syntax

async function myFunc() {
  const splitio = await loadSplit();
}

We then initialise a Factory inside of Split.io and retrieve a client instance

loadSplit().then((splitio) => {
  var factory = splitio.SplitFactory({
    authorizationKey: "<YOUR_AUTH_KEY>",
    key: "A Random Key for Each User"
  });

  var client = factory.client();
});

Then we can implement the split. We need to create feature flags for our stores in Split.io to match with our store name so we can track traffic dependent on the store that our app is currently tracking on. We use a bit of regex to grab the shop name from the URL and pass it to our client. The first function we need is the extractShopName function:

function extractShopName(url) {
// Regular expression to match the shop name at the start of the URL, before '.myshopify.com'
    const pattern = /^(.+?)\.myshopify\.com$/;
    const match = url.match(pattern);

    if (match) {
      return match[1];// The first capturing group contains the shop name
    } else {
      return null;// Return null if the URL does not match the expected format
    }
}

This should sit above our Split.io code so we can call it within the loadSplit function. We then grab our client treatment via the SDK:

loadSplit().then((splitio) => {
  var factory = splitio.SplitFactory({
    authorizationKey: "<YOUR_AUTH_KEY>",
    key: "A Random Key for Each User"
  });

  var client = factory.client();

///////////////////////////////////////////////

  client.on(client.Event.SDK_READY, function() {
    var treatment = client.getTreatment(extractShopName(product.shop));
    if (treatment == "app_hidden") {
// The code here runs for users who aren't shown the app
    } else if (treatment == "app_shown") {
// The code here runs for users who are shown the app
    } else {
// Something funky is happening
    }
  });
});

Which completes our Split.io portion of code for the app.

Now let's look at Mixpanel. We have a split of users, some looking at our app and some not. We want to see what they do. The beauty of Mixpanel is that we can track the same events for each of these users, but separate the users at this Split stage to ensure we're collecting the same data for users who see the app and users that don't. In any case we need to start with enabling the Mixpanel SDK. We can do this using the Mixpanel CDN as Mixpanel has to be loaded on page load (unlike Split.io). We can do this with the following in our app block liquid code:

<script type="text/javascript">
// Optional custom proxy route to stop ad blockers
// const MIXPANEL_PROXY_DOMAIN = '<YOUR_CUSTOM_PROXY_URL>';
  const MIXPANEL_CUSTOM_LIB_URL = MIXPANEL_PROXY_DOMAIN ? MIXPANEL_PROXY_DOMAIN + "/lib.min.js" : undefined;
  (function (f, b) {
    if (!b.__SV) {
      var e, g, i, h;
      window.mixpanel = b;
      b._i = [];
      b.init = function (e, f, c) {
        function g(a, d) {
          var b = d.split('.');
          2 == b.length && ((a = a[b[0]]), (d = b[1]));
          a[d] = function () {
            a.push([d].concat(Array.prototype.slice.call(arguments, 0)));
          };
        }
        var a = b;
        'undefined' !== typeof c ? (a = b[c] = []) : (c = 'mixpanel');
        a.people = a.people || [];
        a.toString = function (a) {
          var d = 'mixpanel';
          'mixpanel' !== c && (d += '.' + c);
          a || (d += ' (stub)');
          return d;
        };
        a.people.toString = function () {
          return a.toString(1) + '.people (stub)';
        };
        i =
          'disable time_event track track_pageview track_links track_forms track_with_groups add_group set_group remove_group register register_once alias unregister identify name_tag set_config reset opt_in_tracking opt_out_tracking has_opted_in_tracking has_opted_out_tracking clear_opt_in_out_tracking start_batch_senders people.set people.set_once people.unset people.increment people.append people.union people.track_charge people.clear_charges people.delete_user people.remove'.split(
            ' '
          );
        for (h = 0; h < i.length; h++) g(a, i[h]);
        var j = 'set set_once union unset remove delete'.split(' ');
        a.get_group = function () {
          function b(c) {
            d[c] = function () {
              call2_args = arguments;
              call2 = [c].concat(Array.prototype.slice.call(call2_args, 0));
              a.push([e, call2]);
            };
          }
          for (var d = {}, e = ['get_group'].concat(Array.prototype.slice.call(arguments, 0)), c = 0; c < j.length; c++)
            b(j[c]);
          return d;
        };
        b._i.push([e, f, c]);
      };
      b.__SV = 1.2;
      e = f.createElement('script');
      e.type = 'text/javascript';
      e.async = !0;
      e.src =
        'undefined' !== typeof MIXPANEL_CUSTOM_LIB_URL
          ? MIXPANEL_CUSTOM_LIB_URL
          : 'file:' === f.location.protocol && '//cdn.mxpnl.com/libs/mixpanel-2-latest.min.js'.match(/^\/\//)
          ? 'https://cdn.mxpnl.com/libs/mixpanel-2-latest.min.js'
          : '//cdn.mxpnl.com/libs/mixpanel-2-latest.min.js';
      g = f.getElementsByTagName('script')[0];
      g.parentNode.insertBefore(e, g);
    }
  })(document, window.mixpanel || []);
</script>

We won't touch on custom proxies for Mixpanel but by setting one up and uncommenting the first line in the script tag, you can leverage a proxy to gather data that might be blocked by an Ad blocker (lots of Mixpanel URLs are).

We now have access to the Mixpanel library in our scripts so long as the above is added in the <head> tag. We can then implement it in the Split.io loader with the following:

loadSplit().then((splitio) => {
  var factory = splitio.SplitFactory({
    authorizationKey: "<YOUR_SPLITIO_AUTH_KEY>",
    key: "A Random Key for Each User"
  });

  var client = factory.client();

///////////////////////////////////////////////

  mixpanel.init("<YOUR_MIXPANEL_AUTH_KEY>", { api_host: "<YOUR_MIXPANEL_PROXY_DOMAIN> or <OTHER>", debug: true, persistence: "localStorage" });

///////////////////////////////////////////////

  client.on(client.Event.SDK_READY, function() {
    var treatment = client.getTreatment(extractShopName(product.shop));
    if (treatment == "app_hidden") {
// The code here runs for users who aren't shown the app
    } else if (treatment == "app_shown") {
// The code here runs for users who are shown the app
    } else {
// Something funky is happening
    }
  });
});

We've initialised Mixpanel, so now let's add our first event:

loadSplit().then((splitio) => {
  var factory = splitio.SplitFactory({
    authorizationKey: "<YOUR_SPLITIO_AUTH_KEY>",
    key: "A Random Key for Each User"
  });

  var client = factory.client();

  mixpanel.init("<YOUR_MIXPANEL_AUTH_KEY>", { api_host: "<YOUR_MIXPANEL_PROXY_DOMAIN> or <OTHER>", debug: true, persistence: "localStorage" });

  client.on(client.Event.SDK_READY, function() {
    var treatment = client.getTreatment(extractShopName(product.shop));
    if (treatment == "app_hidden") {
// The code here runs for users who aren't shown the app
      mixpanel.track(
        "App Page Viewed",
        {
          app_shown: false
        }
      )
    } else if (treatment == "app_shown") {
// The code here runs for users who are shown the app
      mixpanel.track(
        "App Page Viewed",
        {
          app_shown: true
        }
      )
    } else {
// Something funky is happening
    }
  });
});

We should now see events appear in the Mixpanel interface. We are officially tracking and split testing! We can also conditionally render the app based on these if statements with a simple document.getElementById("my-app-container").remove() in the treatment == "app_hidden" check result.

From here onwards we can use the Mixpanel SDK to track other events for users interacting with the page. Let's say we have a button in our App Block that we want to track:

<button onclick="runEventAndTrack()">My App Button</button>

The function would look something like:

function runEventAndTrack() {
  console.log("Button Clicked!");
  mixpanel.track("My App Button Clicked");
}

Which will tie in with the initial event we tracked within our loadSplit callback to understand whether or not a user who is or isn't seeing the app has clicked the app button (a user who isn't seeing the app shouldn't be able to click the button, great for debugging!). This also can be tied into event listeners on existing elements on the page:

// Get the Add to Cart Button from the Page
const addToCartButton = document.getElementById("add-to-cart-button");
addToCartButton.addEventListener("click", () => {
  mixpanel.track("Add to Cart Button Clicked", {
    variant_id: "<VARIANT_ID>"
  });
});

Which will allow us to check whether or not users who have or haven't seen the app have added the current variant to the cart. As such we can deduce whether or not our app is increasing or decreasing conversion.

Metrics are Cool

Hopefully this gives you a good starting point to implement split testing on your own applications. data driven decisions are important because you can't make assumptions about what your customers want, the likelihood is you won't meet the largest market you can if you don't rely on your customers for feedback. By leveraging data and tuning your app appropriately based on that data you can ensure that your customers get the highest value out of your application and you will continue to expand your list of them.

Happy coding,

~ Josh