{"id":116079,"date":"2023-10-24T13:38:36","date_gmt":"2023-10-24T13:38:36","guid":{"rendered":"https:\/\/staging.wingify.com\/blog\/sequential-testing\/"},"modified":"2026-09-15T13:35:24","modified_gmt":"2026-09-15T08:05:24","slug":"sequential-testing","status":"publish","type":"post","link":"https:\/\/wingify.com\/blog\/sequential-testing\/","title":{"rendered":"How to Effectively Use Sequential Testing"},"content":{"rendered":"\n

In A\/B tests where you can see the data coming in a continuous stream, it\u2019s tempting to stop the experiment before the planned end. It\u2019s so tempting that in fact a lot of practitioners don\u2019t even really know why one has to define a testing period beforehand. <\/span><\/p>\n\n\n\n

Some platforms have even changed their statistical tools to take this into account and have switched to sequential testing which is designed to handle tests this way. <\/span><\/p>\n\n\n\n

Sequential testing enables you to evaluate data as it\u2019s collected to determine if an early decision can be made, helping you cut down on A\/B test duration as you can ‘peak’ at set points.<\/strong><\/p>\n\n\n\n

But, is this an efficient and beneficial type of testing? Spoiler: yes and no, depending on the way you use it.<\/span><\/p>\n\n\n\n

Why do we need to wait for the predetermined end of the experiment? <\/span><\/h2>\n\n\n\n

Planning and respecting the data collection period of an experiment is crucial. Historical techniques use “fixed horizon testing” that establishes these guidelines for all to follow. If you do not respect this condition, then you don\u2019t have the guarantee provided by the statistical framework. This statistical framework guarantees that you only have a 5% error risk when using the common decision thresholds.<\/span><\/p>\n\n\n\n

Sequential testing<\/span><\/a> promises that when using the proper statistical formulas, you can stop an experiment as soon as the decision threshold is crossed and still have the 5% error risk guarantee. The test user here is the sequential Z-test, which is based on the <\/span>classical Z-test<\/span><\/a> with an added correction to take the sequential usage into account. <\/span><\/p>\n\n\n\n

In the following sections, we will look at two objections that are often raised when it comes to sequential testing that may put it at odds with CRO practices. <\/span><\/p>\n\n\n\n

Sequential testing objection 1: \u201c<\/span>Each day has to be sampled the same<\/span><\/i>\u201d<\/span><\/h2>\n\n\n\n

The first objection is that one should sample each day of the week the same way. This is basically to have a sampling that represents reality. This is the case in a classic A\/B test. However, this rule may be broken if you use sequential testing since you can stop the test mid-week but this is not always applicable. Since in reality there are seven different days, your sampling unit should be by week and not by day to account for behavioral differences over the course of a week. <\/span><\/p>\n\n\n\n

As experiments typically last 2-3 weeks, then the promise of sequential testing saving days isn\u2019t necessarily correct unless a winner appears very early in the process. However, it\u2019s more likely that the statistical test yielded significance during the last week. In this case, it\u2019s best to complete the data collection until each day is sampled evenly so that <\/span>the full period is covered<\/b>.<\/span><\/p>\n\n\n\n

Let’s consider the following simulation setting:<\/span><\/p>\n\n\n\n