Skip to main content

Statistical significance

What is statistical significance ?​

Hypothesis testing​

When you launch an experiment in Kameleoon, you want to know if the variation you are proofing is actually improving a certain metric compared to your original site. When you frame this in statistical terms, you are making a hypothesis which you will then try to refute by observing data.

To help you decide, Kameleoon uses the frequentist statistical framework, which helps assess how likely the data observed is if the “null” hypothesis is true. This probability is computed by the “p-value”..

Statistical significance and reliability​

An experiment is statistically significant if the sample data Kameleoon collects is sufficiently inconsistent with the data Kameleoon should have observed under the null hypothesis.

To help you assess if an experiment result is statistically significant, Kameleoon provides the experiment's reliability, which is calculated as:

"reliability" = 1 - "p-value"

Example​

Let's take the example below:

  • Reference variation: 142,000 unique visitors tested, 3.52% conversion rate (5,000 sales)
  • Variation 1: 216,000 unique visitors tested, 3.64% conversion rate (1,850 sales)

In this example, we are testing if variation 1 is more efficient than the reference variation. We are trying to reject the null hypothesis, which is that the reference is equally or less efficient than the variation. We feel confident in incorrectly rejecting the null hypothesis only 5% of the time, so we set our significance level at 0.05.

We notice that variation 1 is more efficient than the reference variation, and the reliability is 96%: the outcome of the experiment is statistically significant at our significance level 5%.

Checking statistical significance​

If you are using Google Analytics or Kameleoon as a reporting tool, Kameleoon will automatically calculate your experiment’s statistical significance, allowing you to check if one variation of your experiment is more or less effective than the reference.

Reliability​

The Results page lets you check your campaign’s performance. There are multiple ways to check reliability.

Primary goal​

Your variation’s reliability rate relative to the primary goal is listed in the Primary goal section of the Overview menu.

Here, the reliability rate for both variations is >99%.

Charts​

You can view a visual representation of reliability rate over time in the Charts section of the Overview menu.

You can select which variations are displayed using the dropdown menu (all variations are selected by default). You can also choose to view data by Day, Week, or Month. Hover over the chart to view more reliability rate data.

Metrics​

Reliability rates are displayed in the Metrics menu. Click Goals or Variations to view reliability rate by goal or variation.

What can you do if sufficient reliability is not achieved?​

If your reliability rate is stable but insufficient, there may be a couple of issues. Among them:

  • The traffic on the page is not sufficient.
  • The difference between the original's performance and the variation's is too small to draw conclusions (for example, the modification you have made has a very small impact on the behavior of your visitors).

However, you can draw conclusions for your website from a reliability rate stabilized at 75%. If the traffic on the page is not sufficient, your reliability rate will probably not reach 95%. But the Kameleoon results page offers a wide variety of data and indicators that will help you understand your audience.

note

You can change the minimum reliability rate required for Kameleoon to consider that a variation is winning. To do so, click Admin > Sites in the Kameleoon App. In the tab dedicated to Experiments, you can modify this minimum reliability rate.

Further reading​

If you would like more details on how Kameleoon's statistic engine works, you can read our Statistical paper.