Explaining A/B testing algorithm

We are developing a crypto news aggregator Bitesapp.co. A few days ago, we implemented A/B testing tool to test different scenarios for delivering news to users.

In this article, I will explain the A/B testing algorithm in the simplest possible way.

Main concepts

Inputs

• User attributes (such as user_id, email, device_id,...)
• Experiments
• Weight for each experiment

Requirements

• The experiment variant must always be the same with a set of input attributes
• The user's probability of participating in the experiment is relative to the weight of the experiments (the higher the weight, the more users will be assigned to this experiment).

Idea

This algorithm shares some similarities with the Weighted Random algorithm I explained in the previous article (Link).

To better understand this algorithm, let's imagine it as a space where each region represents the weight of an element. This space is analogous to a line, and the length of each segment on this line corresponds to the weight of an Experiment. If you throw a stone into this space, the probability of the stone landing in a specific area is directly proportional to the weight of that segment, regardless of the order of areas

In the Weighted Random algorithm, we likened the process to randomly throwing the stone. However, in the A/B testing algorithm, we need to find a way to consistently land in the same area for the same user with specific attributes.

Implemenation

We will implement the above idea by the following steps:

• Scale the weights to a specific range, corresponding to fitting the areas into a specific space. The sum of the output weights will be equal to the maxScale.
• Hash the input attributes to a number between 1 and maxScale.
• Determine the area which this hashing value belongs.

1/ Scale the weights to a specific range

// golang
func scaleWeights(weights []float32, maxScale int) []float32 {
    var total float32 = 0.0
    for _, weight := range weights {
        total += weight
    }
    for i, weight := range weights {
        weights[i] = weight / total * float32(maxScale)
    }
    return weights
}

2/ Hash the input attributes to a number between 1 and maxScale

func hash(userID string, maxScale int) (int, error) {
    h := fnv.New32a()
    if _, err := h.Write([]byte(userID)); err != nil {
        return 0, err
    }
    hashValue := int(h.Sum32())
    return hashValue % maxScale + 1 , nil
}

To return the value between 1 and maxScale, we take a modulus for maxScale.

This is why we scale the weights into maxScale.

3/ Determine the area which this hashing value belongs

func getExperimentIndex(weights []float32, maxScale int, hashValue int) int {
    weights = scaleWeights(weights, maxScale)
    var cursor float32 = 0
    for i, weight := range weights {
        cursor += weight
        if cursor >= float32(hashValue) {
            return i
        }
    }
    return -1
}