How to Read SplitSignal Test Results?
Effect Size - Percentage of increase/decrease in sessions during the lifetime of the test.
Absolute Effect - Absolute number of increase/decrease in sessions during the lifetime of the test.
Confidence Level - This metric indicates the level of statistical significance of your test results.
How significant is the result:
- 95% or Higher: the change is large enough to be statistically significant. We are confident that the test change caused the increase/decrease in sessions.
- 94% or less: the change is not large enough to consider it statistically significant
Test Duration - shows how long the test ran.
- Positive - The test is positive and statistically significant.
- No Change - The test didn’t reach statistical significance.
- Negative - The test is negative and statistically significant.
“Good” Prediction Interval (Real Traffic and Predicticted Traffic very closely match. This makes for the most conclusive tests)
“Mediocre” Prediction Interval (Real Traffic and Predicticted Traffic mostly match. This makes for an often conclusive tests)
“Poor” Prediction Interval (Real Traffic and Predicticted Traffic do not closely match. This makes for inconclusive tests)
NOTE On Predicted Traffic:
We don’t necessarily ‘predict’ traffic of the control group but instead build a counterfactual prediction based on the control traffic and its relationship with variant historical traffic. We understand the historical relationship between control and variant and then use it to understand if something has changed during the test. We’re not predicting the future. If the relationship remained stable through the test - the results are insignificant. If the relationship has changed through the test - the results become significant.