New “Evolution of Experimentation” study from the world’s largest digital laboratory reveals the number of companies experimenting is up in the last five years.
NEW YORK, Nov. 27, 2023 /PRNewswire/ — Optimizely, the leading digital experience platform (DXP) provider, today unveiled a new research report, “The Evolution of Experimentation” that delivers an analysis of the over 127,000 true experiments conducted on Optimizely Web Experimentation and Optimizely Feature Experimentation between 2018 and 2023. With this report, Optimizely hopes to advance the practice of experimentation around the world by sharing valuable insights and guidance for digital practitioners.
The report reveals that since 2018, Optimizely has observed significant growth in the number of experiments its customers conducted, an uplift in the volume of companies experimenting, and a 227% increase in the usage of feature experimentation as a percentage of all experiments. While data shows broader adaptation of experimentation in recent years, substantial differences remain in the quality of experimentation occurring across the landscape.
“This data confirms what many at Optimizely already know — the best in the game embrace experimentation to deliver more exceptional digital experiences to their users,” said Eric Lang, Senior Consultant of Strategy & Value. “There is no doubt that every test uncovers valuable information on customer behavior, no matter how small an experiment may seem. Those who experiment gain key insights into behavior in order to deliver exceptional digital experiences and develop long-lasting, personal relationships with their customers.”
The results of the report underscore the value of a robust experimentation operation for any digital brand. The data shows initial ideas implemented by digital marketers are very rarely successful. According to the report, 88% of the ideas implemented by digital teams are not winners, meaning they do not result in a positive significant change in user behavior. However, even if a test is not a winner, it provides valuable insights into customer behavior and allows teams to eliminate guesswork and pivot quickly.
“Experimentation allows digital businesses to achieve the highest level of customer-centricity by checking all of their ideas with the only audience that matters – their customers,” said Pete Gray, SVP of Customer Growth Strategy at Optimizely. “The emotional cost of experimentation is to learn how wrong we can be. The prize is the ability to never make a mistake again and to unlock real value by increasing the velocity of improvement.”
Additional key takeaways from Optimizely’s new study include:
Contrary to industry advice that good experiments should be minute in scope, the highest-performing experiments have very different characteristics.
Experiments that test multiple treatments are at least 1.5x more successful than simple A/B tests
Experiments that make significant changes to the user experience are 25% more successful
Experiments that are personalized create 41% higher impact
To be in the top 10% of experiment velocity, companies need to run over 200 tests annually.
The median company runs 3 experiments per month
Reaching the Top 10% of velocity requires scaling to over 16 tests per month
Only 3% of companies have joined the elusive 500 tests per year club
High and low performers exist in all industries, and all companies have the potential to improve their performance. The average experiment win rates by sector are:
Food and Beverage: 17%
Automotive: 16.2%
Hospitality and Travel: 15.7%
Media and Entertainment: 15.5%
Financial Services: 15%
Real Estate: 14.4%
Healthcare and Medical: 13.3%
Sporting and Recreation: 13.2%
Retail and Distribution: 13.1%
Telecommunications: 12%
To read the full report and learn how Optimizely can help your team run experiments, generate insights, and continuously optimize experiences across the board, visit https://www.optimizely.com/the-evolution-of-experimentation/
About OptimizelyAt Optimizely, we’re on a mission to help people unlock their digital potential. We do that by reinventing how marketing and product teams work to create and optimize digital experiences across all channels. With Optimizely One, our industry-first operating system for marketers, we offer teams flexibility and choice to build their stack their way with our fully SaaS, fully decoupled, and highly composable solution. We help companies around the world orchestrate their entire content lifecycle, monetize every digital experience and experiment across all customer touchpoints – all through Optimizely One, the leading digital experience platform that powers every phase of the marketing lifecycle through a single, AI-accelerated workflow.
Optimizely has nearly 1500 employees across our 21 global offices and has 700+ partners. We are proud to help more than 10,000 businesses, including H&M, PayPal, Zoom, and Toyota enrich their customer lifetime value, increase revenue and grow their brands. At Optimizely, we live each day with a simple philosophy: large enough to serve, small enough to care. Learn more at optimizely.com.
SOURCE Optimizely
