Your browser is out of date. The site might not be displayed correctly. Please update your browser.

Knowledge Base
Specialty Products
SplitSignal
How to Read SplitSignal Test Results?

How to Read SplitSignal Test Results?

Question

Effect Size Percentage of increase/decrease in sessions during the lifetime of the test.

Absolute Effect - Absolute number of increase/decrease in sessions during the lifetime of the test.

Confidence Level This metric indicates the level of statistical significance of your test results. 

How significant is the result:

  • 95% or Higher: the change is large enough to be statistically significant. We are confident that the test change caused the increase/decrease in sessions. 
  • 94% or less:  the change is not large enough to consider it statistically significant

Test Duration - shows how long the test ran.

Result Type: 

  • Positive - The test is positive and statistically significant.
  • No Change - The test didn’t reach statistical significance.
  • Negative - The test is negative and statistically significant.

How to Read SplitSignal Test Results? image 1

“Good” Prediction Interval (Real Traffic and Predicticted Traffic very closely match. This makes for the most conclusive tests) 

How to Read SplitSignal Test Results? image 2

“Mediocre” Prediction Interval (Real Traffic and Predicticted Traffic mostly match. This makes for an often conclusive tests)

How to Read SplitSignal Test Results? image 3

“Poor” Prediction Interval (Real Traffic and Predicticted Traffic do not closely match. This makes for inconclusive tests)

How to Read SplitSignal Test Results? image 4

 

NOTE On Predicted Traffic:

We don’t necessarily ‘predict’ traffic of the control group but instead build a counterfactual prediction based on the control traffic and its relationship with variant historical traffic. We understand the historical relationship between control and variant and then use it to understand if something has changed during the test. We’re not predicting the future. If the relationship remained stable through the test - the results are insignificant. If the relationship has changed through the test - the results become significant.

Frequently asked questions
Show more