← Blog · July 29, 2026
A push notification A/B test with 50 recipients per variant and a 2-percentage-point difference in open rates is not a meaningful result — it is noise. Most notification platforms call a winner based on raw counts or a simple percentage comparison, which means they surface false positives at high rates whenever sample sizes are small or variance is high. The practical consequence: you optimize for copy that happened to hit during a favorable delivery window, not copy that actually converts better.
Push notification open rates are binary outcomes (opened or not), so the right significance test is a two-proportion z-test. It computes a z-score from the observed open rates of both variants and their sample sizes, then compares that score to a threshold that sets your acceptable false-positive rate. At a z-threshold of 1.645 you are targeting a one-tailed 95% confidence level — meaning if you ran this experiment 100 times when there was truly no difference, you would expect to declare a false winner about 5 times.
z = (p1 - p2) / sqrt(p_pooled * (1 - p_pooled) * (1/n1 + 1/n2)) where: p1, p2 = observed open rates for each variant n1, n2 = sample sizes p_pooled = (opens1 + opens2) / (n1 + n2)
At typical push open rates (5–15%), you need at least 300–500 recipients per variant to reliably detect a 3-percentage-point difference at 95% confidence. Under that floor, declare no winner and keep collecting data. Declaring "inconclusive" after 10× the per-variant minimum is a reasonable stopping rule — if a winner has not emerged at that scale, the real difference is probably too small to care about.
Pushbrain runs a z-test on each decision tick, stays in "testing" status until the z-score clears the threshold, and marks experiments "inconclusive" rather than forcing a winner when the ceiling is hit without significance. Variant open-rate counts and current status are visible on each test in the dashboard.