{"id":116054,"date":"2022-10-17T16:00:45","date_gmt":"2022-10-17T16:00:45","guid":{"rendered":"https:\/\/staging.wingify.com\/blog\/type-1-and-type-2-errors\/"},"modified":"2026-09-15T14:15:31","modified_gmt":"2026-09-15T08:45:31","slug":"type-1-and-type-2-errors","status":"publish","type":"post","link":"https:\/\/wingify.com\/blog\/type-1-and-type-2-errors\/","title":{"rendered":"Statistics: What are Type 1 and Type 2 Errors?"},"content":{"rendered":"\n

Statistical hypothesis testing implies that no test is ever 100% certain: that\u2019s because we rely on probabilities to experiment<\/strong>.<\/p>\n\n\n\n

When online marketers and scientists run hypothesis tests, they\u2019re both looking for statistically relevant results<\/a><\/strong>. This means that the results of their tests have to be true within a range of probabilities (typically 95%).<\/p>\n\n\n\n

Even though hypothesis tests are meant to be reliable, there are two types of errors that can still occur.<\/p>\n\n\n\n

These errors are known as type 1 and type 2 errors<\/a><\/strong> (or type i and type ii errors).<\/p>\n\n\n\n

Let\u2019s dive in and understand what type 1 and type 2 errors are and the difference between the two.<\/p>\n\n\n

\n
\"Type<\/figure>\n<\/div>\n\n\n

<\/p>\n\n\n\n

Understanding Type I Errors<\/h2>\n\n\n\n

Type 1 errors<\/strong> \u2013 often assimilated with false positives \u2013 happen in hypothesis testing when the null hypothesis is true but rejected<\/strong>. The null hypothesis is a general statement or default position that there is no relationship between two measured phenomena<\/em>.<\/p>\n\n\n\n

Simply put, type 1 errors are \u201cfalse positives\u201d \u2013 they happen when the tester validates a statistically significant difference even though there isn\u2019t one.<\/p>\n\n\n

\n
\"\"
Source<\/a><\/figcaption><\/figure>\n<\/div>\n\n\n

Type 1 errors have a probability of  \u201c\u03b1\u201d correlated to the level of confidence that you set. A test with a 95% confidence level<\/strong> means that there is a 5% chance of getting a type 1 error.<\/p>\n\n\n\n

Consequences of a Type 1 Error<\/h2>\n\n\n\n

Why do type 1 errors occur<\/strong>? Type 1 errors can happen due to bad luck (the 5% chance has played against you) or because you didn\u2019t respect the test duration and sample size initially set for your experiment.<\/p>\n\n\n\n

Consequently, a type 1 error will bring in a false positive<\/strong>. This means that you will wrongfully assume that your hypothesis testing has worked even though it hasn\u2019t.<\/p>\n\n\n\n

In real-life situations, this could potentially mean losing possible sales due to a faulty assumption caused by the test.<\/p>\n\n\n\n

Related: Sample Size Calculator for A\/B Testing<\/a><\/strong><\/p>\n\n\n\n

A Real-Life Example of a Type 1 Error<\/h2>\n\n\n\n

Let\u2019s say that you want to increase conversions on a banner<\/strong> displayed on your website. For that to work out, you\u2019ve planned on adding an image to see if it increases conversions or not.<\/p>\n\n\n\n

You start your A\/B test by running a control version (A) against your variation (B)<\/strong> that contains the image. After 5 days, variation (B) outperforms the control version by a staggering 25% increase in conversions with an 85% level of confidence.<\/p>\n\n\n\n

You stop the test and implement the image in your banner. However, after a month, you noticed that your month-to-month conversions have actually decreased.<\/p>\n\n\n\n

That\u2019s because you\u2019ve encountered a type 1 error: your variation didn\u2019t actually beat your control version in the long run<\/strong>.<\/p>\n\n\n\n

Related: Frequentist vs Bayesian<\/a><\/strong> Methods in A\/B Testing<\/p>\n\n\n\n

Want to avoid these types of errors during your digital experiments?<\/p>\n\n\n\n

Wingify<\/strong><\/a> is an a\/b testing tool embedded with AI and automation that allows you to quickly set up experiments, track insights via our dashboard, and determine which route will increase your revenue.<\/p>\n\n\n\n

Understanding Type II Errors<\/h2>\n\n\n\n

In the same way that type 1 errors are commonly referred to as \u201cfalse positives\u201d, type 2 errors are referred to as \u201cfalse negatives\u201d<\/strong>.<\/p>\n\n\n\n

Type 2 errors happen when you inaccurately assume that no winner has been declared between a control version and a variation although there actually is a winner.<\/p>\n\n\n\n

In more statistically accurate terms, type 2 errors happen when the null hypothesis is false and you subsequently fail to reject it<\/strong>.<\/p>\n\n\n\n

If the probability of making a type 1 error is determined by \u201c\u03b1\u201d, the probability of a type 2 error is \u201c\u03b2\u201d. Beta depends on the power of the test (i.e the probability of not committing a type 2 error, which is equal to 1-\u03b2).<\/p>\n\n\n\n

There are 3 parameters that can affect the power of a test:<\/strong><\/p>\n\n\n\n