Experimental Validation | Heurística Lab

Decision-makers want to know whether implementing their ideas or possible solutions will actually influence user behavior.

In other words, to determine the best decision, they need to establish a causal relationship between their actions and the results.

For example: if I change variable X (the cause), will outcome Y (the effect) change?

To answer this question, some organizations mistakenly just implement the idea and then measure changes in the results. The big mistake in this approach is that the observed effect could be due to a range of external circumstances, not the variable of interest.

Imagine your company wants to boost sales of a product, so it launches an ambitious advertising campaign. After two months, you evaluate the results and see 20% sales growth. Can you safely claim the campaign caused that increase?

No.

Because other factors could have influenced the outcome (for example, changes in a competitor's marketing mix, shifts in customers' purchasing power, or seasonal demand variation), you can't reliably pinpoint the true source of the sales change.

The only way to reliably determine whether an idea or solution works is through an experiment: a procedure following the scientific method to support or reject a hypothesis. There are different types of experiments, but the most effective for establishing a cause-effect relationship is known as a Randomized Controlled Trial (RCT).

The Randomized Controlled Trial

A Randomized Controlled Trial precisely reveals the effect of a stimulus or intervention on the outcome you care about, by isolating it from other variables that could also be influencing the result. It works by taking a sample of your user population and randomly dividing them into different groups. One of these groups is exposed to a standard stimulus or status quo (the control group), while the other groups are exposed to the different variants or treatments we want to test.

We then estimate the average outcome in each group. The differences between the control group's average and the treatment groups' averages give us the effect of the stimulus on user behavior.

However, it's still not possible to rule out that the observed effects are simply due to chance.

To do that, we determine the p-value through statistical analysis. When that value is at or below a conventional threshold (usually below 5%), we can say the effect isn't a coincidence, and the intervention worked.

Finally, with that statistical confidence, you can implement your idea or solution across the organization's entire user base.

Diagram: a sample of users is randomly split into a control group, treatment 1, and treatment 2. Treatment 2 beats the control (green check); treatment 1 performs worse than the control (red X).

Figure 1

Experimenting to test ideas

Simulating a project you could run with Heurística Lab's researchers and statisticians.

Imagine you're responsible for a digital product. Your team recently shipped a feature with strong potential. So far, though, the results haven't been as good as you expected: visitors to the site explore the feature, but most don't use it.

After researching your users, two possible paths forward emerge. Both options look promising, but you're not sure which to implement. On top of that, you're not sure it's even worth changing anything, since this kind of change can be costly, both in resources and in customer satisfaction.

The best way to resolve this dilemma is through a Randomized Controlled Trial. Under this procedure, the first step is to take a sample of your user population and randomly assign these people to one of three groups (Figure 1).

People in the control group interact with the current version of the site, as if nothing had changed. The treatment groups, meanwhile, are each exposed to one of the possible solution ideas.

In our example, after statistically analyzing the results, we find that people exposed to idea 2 use the new feature more than those who see the current version of the platform (the control group). So implementing that change is a good move.

Idea 1, on the other hand, performed worse than the control group, so implementing it before testing would have been a mistake, with potentially serious consequences for our business.

Figure 2. The same experiment, seen person by person. Each dot is a user. Illustrative data.

Make better decisions by putting your ideas to the test

Test hypotheses, ideas, or products, and determine their odds of success before launching them to market.

Experiment design (specific methodology, controlling for confounding variables, sample size calculation, and randomization).

Co-designing solution ideas (experimental stimuli).

Data processing and statistical analysis of the experiment.

Interpretation and presentation of results, focused on your business decision.

Additional recommendations based on inferential (correlational) analysis.

Want to know if your idea really works?

Let's design the experiment that gives you certainty before you decide.