Digital products are never done, market evolves, your customer needs change and technology keeps improving; a good product must adapt. For a product to adapt in a successful way, instrumenting your digital product is paramount so that you are able to learn what works and what doesn't work by measuring performance.

A/B testing is one of many tools that you can tap into in order to learn more about your customers, validate or disprove hypotheses and be certain that the new features that you want to develop impact positively the users and the business.

Individually you can measure the success of your tests, but the reality is that this needs to be part of the routine of running a digital product. It is like brushing your teeth in the morning, you need to do it everyday in order to make sure that you learn from your users and continually build a product they love, use and solve for the problems they have.

The case for an Experimentation Program

The success of your testing approach will be determined by the sum of the number of tests run (volume) and the percentage of tests that generate a win. If you still wish to go further in trying to measure the success of your approach you can - think about the relative conversion increase achieved, the volume that you could impact over a period of time and the revenue that you generate by conversion and you will have the impact of your testing efforts.

This is why it is super important to think about this as a Program, The Experimentation Program. A/B testing is not, and shouldn't, be considered an isolated activity. It is not a one-time activity that will fix everything under the sun. Conversely it can be a mistake to throw everything under the A/B umbrella.

With this approach you will be able to optimize feedback loops, create the opportunity not only to improve the current unhappy paths that you have in your customer journeys, but also to create the opportunity to quickly prove or disprove hypothesis that you have and are directly correlated to potential future innovations on your product.

There are a couple of tenets that you need to consider in order the set up a good experimentation program:

• Needs to be scalable (used across the board)
• Needs to be collaborative (success in the agency of others)
• Needs to be performative (results)

The Experimentation Program

In order to be scalable and collaborative, and needless to say repeatable, there is the need to establish a process - in short a program that will build our pipeline of tests:

• Hypothesis log
• Hypothesis assessment
• Plan and schedule
• Experimentation in progress
• Post-mortem
• Winners
• Losers

Hypothesis log

For the program to be scalable you need to find a way to get everyone to participate, as it will be totally impossible to generate all potential hypothesis from a unique team. It is therefore critical that we enable anyone to participate in the program.

The hypothesis log will be the repository of all ideas that the team will have and we need to find a way to make this process as easy as possible so that we can have a good amount of ideas and hypothesis. Team members will be required to add details about what they are trying to experiment in a template.

Generally speaking hypothesis will lead to experimentation's that in turn should be aimed at increasing conversions. When thinking about conversions, we can focus on 2 things:

• Keeping users in the flow;
• Bringing users back into the flow;

Hypothesis assessment

The team that is managing the program (normally the product growth team) will be responsible to clarify the hypothesis, better understand the potential impact as well as assessing the potential effort together with the eng team.

Depending on the existing number of hypothesis the team will need to prioritize the experiments that will need to be developed. Ideally the Hypothesis log template should have that in mind so that the team can make the assessment and capture that data point.

Plan and Schedule

Planning the test is going to be very important because you will likely need help from different teams. Surely you will need a new design or a couple of tweaks, in turn, that could impact your frontend and backend, so it will be crucial that talk to the engineering team.

One of the things that will also be important, in case you are considering A/B testing, is to plan and negotiate volume and traffic with business. We discuss this on our A/B testing post.

Tracking progress

After the change you need to carefully track results to better understand the impact of the change you made. Here you will likely need to have statistical significance, whenever possible, in order to make sure that you can prove or disprove your hypotheses.

Measuring the impact of your program is super important because you want to take stock of all the learnings, the losing and winning experiments and evolve the product as you go along.