The truth is, any website can be optimized<\/strong> \u2013 you just need to tailor your optimization strategy to suit your unique situation.<\/p>\n In this article, we will cover:<\/p>\n In order to make this article easier to understand, let\u2019s start with an analogy. Imagine that instead of measuring two variants and picking a winner, we are measuring the performance of two boxers and placing bets on who will win the next 10 rounds.<\/p>\n So, how will we place our bet on who will win<\/strong>?<\/p>\n Imagine that boxer A and boxer B are both newbies<\/strong> that no one knows. After the first round, you have to make your choice. In the end, you will most likely place your bet on the boxer who won the first round. It might be risky if the winning margin is small, but in the end, you have no other way to base your decision<\/strong>.<\/p>\n Imagine now that boxer A is known to be a champion, and boxer B is a challenger that you don\u2019t know. Your knowledge about boxer A is what we would call a prior \u2013 information you have before that influences your decision<\/strong>.<\/p>\n Based on the prior, you will be more likely to bet on boxer A as the champion for the next few rounds, even if boxer B wins the first round with a very small margin.<\/p>\n Furthermore, you will only choose boxer B as your predicted champion if they win the first round by a large margin. The stronger your prior, the larger the margin needs to be in order to convince you to change your bet.<\/p>\n Are you following? If so, the following paragraphs will be easy to grasp and you will understand where this \u201c95% threshold\u201d comes from.<\/p>\n Now, let\u2019s move on to tips for optimizing your website with low traffic.<\/p>\n This is the most common complaint about CRO for websites with lower traffic and for lower traffic pages on bigger websites.<\/p>\n Before we dig into this most common problem, let\u2019s start by answering the question, where does this 95% \u201cgolden rule\u201d come from?<\/em><\/p>\n Let\u2019s start our explanation with a very simple idea: What if optimization strategies were applied from day one? If two variants with no previous history were created at the same time, there would be no \u201coriginal\u201d version challenged by a newcomer.<\/p>\n This would force you to choose the best one from the beginning.<\/p>\n In this setting, any small difference in performance could be measured for decision-making. After a short test, you will choose the variant with the higher performance. It would not be good practice to pick the variant that had lower performance and furthermore, it would be foolish to wait for a 95% threshold to pick a winner.<\/p>\n But in practice, optimization is done well after the launch of a business<\/strong>.<\/p>\n So, in most real-life situations, there is a version A that already exists and a new challenger (version B) that is created.<\/p>\n If the new challenger, version B, comes along and the performance difference between the two variants is not significant, you will have no issues declaring version B \u201cnot a winner.\u201d<\/p>\n Statistical tests are symmetric<\/strong>. So if we reverse the roles, swapping A and B in the statistical test will tell you that the original is not significantly better than the challenger. The \u201cinconclusiveness\u201d of the test is symmetric.<\/p>\n So, why do you set 100% of traffic toward the original at the end of an inconclusive test, implicitly declaring A as a winner? Because you have three priors:<\/p>\n Points 1 & 2 are the bases of a CRO strategy, so you will need to go beyond these two priors<\/strong>. Point 3 explains that version A has more data to back its performance. This explains why you trust version A more than version B<\/strong>.\u00a0Version A has data<\/strong>.<\/p>\n Now you understand that this 95% confidence rule is a way of explaining a strong prior. And this prior mostly comes from historical data<\/strong>.<\/p>\n Therefore, when optimizing a page with low traffic, your decision threshold should be below 95% because your prior on A is weaker due to its traffic and seniority.<\/p>\n The threshold should be set according to the volume of traffic that went through the original from day one. However, the problem with this approach is that we know that the conversion rates are not stable and can change over time. Think of seasonality \u2013 i.e. black Friday rush, vacation days, Christmas time increases in activity, etc.<\/em> Because of the seasonal changes, you can\u2019t compare performances in different periods.<\/p>\n This is why practitioners only take into account data for version A and version B taken at the same period of time and set a high threshold (95%) to accept the challenger as a winner in order to formalize a strong prior toward version A.<\/p>\n It\u2019s hard to suggest an exact number to focus on because it depends on your risk acceptance<\/strong>.<\/p>\n According to the hypothesis protocol, you should structure a time frame for the data collection<\/strong> period in advance<\/strong>.<\/p>\n This means that the \u201cstop\u201d criteria of a test are not a statistical measure or based on a certain number. The \u201cstop\u201d criteria should be a timeframe coming to an end. Once the period is over, then you should look at the stats to make an appropriate decision.<\/p>\n Wingify, our customer experience optimization and feature management software, uses the Bayesian framework<\/a> which produces a \u201cchances to win\u201d index which encourages a direct interpretation instead of a p-value, which has a very complex meaning.<\/p>\n In other words, the \u201cchances to win index\u201d is the probability for a given variation to be better than the original.<\/p>\n Therefore, a 95% \u201cchance to win\u201d means that there is a 95% probability that the given variation will be the winner. This is assuming that we don\u2019t have any prior knowledge or specific trust for the original.<\/em><\/p>\n The 95% threshold itself is also a default compromise between the prior you have on the original and a given level of risk acceptance (it could have even been a 98% threshold)<\/em>.<\/p>\n Although it is hard to give an exact number, let\u2019s make a rough scale for your threshold<\/strong>:<\/p>\n The original 95% threshold is far too high if your business has low traffic because there\u2019s little chance that you will reach it. Consequently, your CRO strategy will have no effect and data-driven decision-making<\/a> becomes impossible.<\/p>\n By using Wingify as your experimentation platform, you will be given a report that includes the \u201cchance to win\u201d along with other statistical information regarding your web experiments<\/a>. A report from Wingify would also include the confidence interval on the estimated gain as an important indicator. The boundaries around the estimated gain are also computed in a Bayesian way, which means it can be interpreted as the best and the worst scenario.<\/p>\n Now you understand the exact meaning of the well-known 95% \u201csignificance\u201d level and are able to select appropriate thresholds corresponding to your particular case.<\/p>\n It\u2019s important to remember that this approach only works with Bayesian statistics since frequentist approaches give statistical indices (such as p-Values and confidence intervals that have a totally different meaning and are not suited to the explained logic).<\/p>\n Yes, they are valid as long as you do not stop the test depending on the result.<\/p>\n Remember the testing protocol says once you decide on a testing period, the only reason to stop a test is when the timeframe has ended<\/strong>. In this case, the stat indices (\u201cchances to win\u201d & confidence interval) are true and usable.<\/p>\n You may be thinking: \u201cOkay, but then I rarely reach the 95% significance level\u2026\u201d<\/em><\/p>\n Remember that the 95% threshold doesn\u2019t need to be the magic number for all cases<\/strong>. If you have low traffic, chances are that your website is not old. If you refer back to the previous point, you can take a look at our suggested scale for different scenarios.<\/p>\n If you\u2019re dealing with lower traffic as a newer business, you can certainly switch to a lower threshold (like 90%). The threshold is still higher because it\u2019s typical to have more trust in an original rather than a variant because it\u2019s used for a longer time.<\/p>\n If you\u2019re dealing with two completely new variants, at the end of your testing period, it will be easier to pick the variant with the higher conversions (without using a stat rest) since there is no prior knowledge of the performance of A or B.<\/p>\n Sometimes the traffic problem is not due to a low-traffic website, but rather the webpage in question. Typically, pages with lower traffic are at the end of the funnel.<\/p>\n In this case, a great strategy is to work on optimizing the funnel closer to the user\u2019s point of entry. There may be more to uncover with optimization in the digital customer journey<\/a> before reaching the bottom of the funnel.<\/p>\n Controlled Experiment Using Pre-Experiment Data<\/strong> is a newer buzzword in the experimentation world. CUPED is a technique that claims to produce up to 50% faster results. Clearly, this is very appealing to small-traffic websites.<\/p>\n Not exactly, for two reasons: one is organizational and the other is applicability.<\/p>\n What\u2019s often forgotten is that CUPED<\/span><\/strong> means Controlled experiment Using Pre-Experiment Data<\/strong>.<\/p>\n In practice, the ideal period of \u201cpre-experiment data\u201d is two weeks<\/strong> in order to hope for a 50% time reduction.<\/p>\n So, for a 2-week classic test, CUPED<\/a> claims that you can end the test in only 1 week.<\/p>\n However, in order to properly see your results, you will need two weeks of pre-experiment<\/strong> data<\/strong>. So in fact, you must have three weeks to implement CUPED in order to have the same accuracy as a classic 2-week test.<\/p>\n Yes, you are reading correctly. In the end, you will need three weeks time to run the experiment<\/strong>.<\/p>\n This means that it is only useful if you already have two weeks of traffic data that is unexposed to any experiment. Even if you can schedule two weeks of no experimentations into your experimentation planning to collect data, this will be blocking traffic for other experiments.<\/p>\n In addition to the organizational\/2-week time constraint, there are two other prerequisites in order for CUPED to be effective:<\/p>\n You will see in the following paragraph that these two constraints make CUPED virtually impossible for e-commerce websites and only applicable to platforms.<\/p>\n Let\u2019s go back to our experiment settings example:<\/p>\n Constraint number 1 states that we need to have the same visitors in pre-experiment & experiment<\/strong>, but the visitor\u2019s journey in e-commerce is usually one week.<\/p>\n In other words, there is very little chance that you see visitors in both periods. In this context, only a very limited effect of CUPED is to be expected (up to the portion of visitors that are seen in both periods).<\/p>\n Constraint number 2 states that the visitors must have the same behavior regarding the conversion (the KPI under optimization). Frankly, that constraint is simply never met in e-commerce.<\/p>\n The e-commerce conversion occurs either during the pre-experiment or during the experiment but not in both (unless your customer frequently purchases several times during the experiment time<\/em>).<\/p>\n This means that there is no chance that the visitors\u2019 conversions are correlated between the periods<\/strong>.<\/p>\n In summary: CUPED is simply not applicable for e-commerce websites to optimize transactions<\/strong>.<\/p>\n It is clearly stated in the original scientific paper, but for the sake of popularity, this buzzword technique is being misrepresented in the testing industry.<\/p>\n In fact, and it is clearly stated in scientific literature, CUPED works only on multiple conversions for platforms that have recurring visitors performing the same actions.<\/p>\n Great platforms for CUPED would be search engines<\/strong> (like Bing<\/a>, where it has been invented) or streaming platforms<\/strong> where users come daily and do the same repeated actions (playing a video, clicking on a link in a search result page, etc).<\/p>\n Even if you try to find an application of CUPED for e-commerce, you\u2019ll find out that it\u2019s not possible.<\/p>\n CUPED does not work for e-commerce websites where a transaction is the main optimization goal<\/strong>. Unless you are Bing, Google, or Netflix \u2014 CUPED won\u2019t be your secret ingredient to help you to optimize your business.<\/p>\n This technique is surely a buzzword spiking interest fast, however, it\u2019s important to see the full picture before wanting to add CUPED into your roadmap. E-commerce brands will want to take into account that this testing technique is not suited for their business.<\/p>\n Brands with lower traffic are still prime candidates for website optimization, even though they might need to adapt to a less-than-traditional different approach.<\/p>\n Whether optimizing your web pages means choosing a page that\u2019s higher up in the funnel or adopting a slightly lower threshold, continuous optimization is crucial.<\/p>\n Want to start optimizing your website?<\/em> Wingify<\/a> is the best-in-class experience optimization platform that empowers you to create a richer digital experience \u2013 fast. From experimentation to personalization, this solution can help you activate and engage your audience to boost your conversions.<\/p>\n\n
CRO analogy<\/h2>\n
1. Solving the problem: \u201cI never reach the 95% significance\u201d<\/h2>\n
The origin of the 95% threshold<\/h3>\n
\n
What is the appropriate threshold for low traffic?<\/h3>\n
\n
The importance of Bayesian statistics<\/h3>\n
2. Are the stats valid with small numbers?<\/h2>\n
3. Go \u201cupstream\u201d<\/h2>\n
4. Is the CUPED technique real?<\/h2>\n
What is CUPED?<\/h3>\n
Does CUPED really work that well?<\/h3>\n
The organizational constraint<\/h4>\n
The applicability constraint<\/h4>\n
\n
\n
\n
Conclusion about CUPED<\/h3>\n
Optimization for low-traffic websites<\/h2>\n
![\"\"](\"https:\/\/static.wingify.com\/gcp\/uploads\/abtasty\/2024\/05\/CTA-Banner.png\")
<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"