Humans vs AI

The test was run by the brand’s agency partner: Optimized-Marketing.com

All three AI copies won. Variation 2 resulted in the maximum uplift (15.77%) whereas Variation 1 and Variation 3 resulted in an uplift of 9.13% & 7.13% respectively with a statistical significance of >90%. 

Human copy 1 won this test. We have shown blurred screenshots to mask the identity of the hotel property.

The uplift between human copy and AI copy was less than 5% (and statistically insignificant too). Hence for practical purposes, we conclude the test to be a tie.

The uplift between human copy and AI copy was less than 5% (and statistically insignificant too). Hence for practical purposes, we conclude the test to be a tie.

The uplift between human copy and AI copy was less than 5% (and statistically insignificant too). Hence for practical purposes, we conclude the test to be a tie.

The test has been deemed inconclusive since the result was not statistically significant i.e. the statistical significance threshold of 90%.

The test has been deemed inconclusive since the result was not statistically significant i.e. the statistical significance threshold of 90%.

The test has been deemed inconclusive since it was paused by the participating website before achieving the statistical significance threshold of 90%.

The test has been deemed inconclusive due to an error in the configuration of the test which led to erroneous results.

The test has been deemed inconclusive since it was paused by the participating website before achieving the statistical significance threshold of 90%.

The test has been deemed inconclusive since it was paused by the participating website before achieving the statistical significance threshold of 90%.

The participating website used the Multi-Armed Bandit (MAB) approach instead of A/B testing and hence we have considered it as inconclusive. However, the winning variation in the MAB test was generated by an AI.

The test has been deemed inconclusive since the website had very low traffic to conclude the test on.

The test has been deemed inconclusive since it was paused by the participating website before achieving the statistical significance threshold of 90%.

A bug was reported in the test where the copy was not appearing on pages as expected, which means that the AI-generated text wasn’t displayed to users who visited the test page. Hence, we have termed this test as inconclusive.