Shiva Manjunath shares how debunking best practices, embracing failure, and fostering a culture of learning can elevate experimentation to new heights.
In this episode of The 1000 Experiments Club, guest host and Wingify’s Head of Marketing, John Hughes, sat down with Shiva Manjunath, Senior Web Product Manager of CRO at Motive and Host of the podcast From A to B. Shiva’s journey through roles at Gartner, Norwegian Cruise Line, Speero, Edible, and now Motive, has made him a passionate advocate for the transformative power of experimentation.
During their conversation, Shiva discussed the pitfalls of following “best practices” blindly, the importance of creating an environment where failure is seen as a step toward success, and how companies can truly build a culture of experimentation.
Here are some of the key takeaways.
The myth of ‘Best Practices’
Too often, the so-called experimentation best practices become a checkbox exercise, rather than a thoughtful strategy.
“If you’re focused on best practices, you’re likely missing the point of true optimization,” Shiva notes.
He recounted a situation at Gartner where simplifying a form—typically hailed as a best practice—actually led to a sharp drop in conversions. His point? Understanding user motivation and context is far more important than relying on one-size-fits-all rules. It’s this deeper, more nuanced approach to experimentation that drives real results.
“If what you believe is this best practice checklist nonsense, all CRO is just a checklist of tasks to do on your site. And that’s so incorrect,” Shiva emphasized, urging practitioners to move beyond surface-level tactics and truly understand their audience.
Embracing failure in experimentation
A major theme of the discussion was the pivotal role failure plays in the journey to success. Shiva was candid about his early experiments, admitting that many didn’t go as planned. But these “failures” were crucial stepping stones in his development.
“My first ten tests were all terrible. They all sucked,” Shiva admitted, underscoring that even the most seasoned experts start with mistakes. He stressed that organizations must create an environment where employees can experiment freely, learn from their mistakes, and continue to improve.
“If you’re penalized for running a losing test, you’re not in a culture of experimentation,” Shiva insists.
Organizations that punish failure are stifling innovation. Instead, Shiva advocates for an environment where employees can test, learn, and iterate without fear. “The idea that you have the flexibility to discuss failures and focus on, ‘Well, I ran this test. It lost. Now, what do we do next?’—that’s a culture of experimentation.”
Scaling experimentation maturity
Shiva also explored the varying levels of experimentation maturity within organizations. Many companies claim to have a “culture of experimentation,” but few truly practice it at scale. Shiva emphasized the importance of making experimentation accessible to everyone in the organization, not just a select few.
Reflecting on the loss of Google Optimize, Shiva acknowledged its role as a gateway into the world of experimentation. “I got into experimentation through Google Optimize,” Shiva recalled, recognizing the tool’s importance in lowering the barrier to entry for newcomers. He urged companies to lower barriers to entry and enable more people to engage with experimentation, thereby fostering a more mature and widespread culture of testing.
The role of curiosity and data in experimentation
Another critical point Shiva raised was the importance of curiosity in experimentation. He believes that genuine curiosity drives the desire to ask “why” and dig deeper into user behavior, which is essential for effective experimentation.
“If you’re not genuinely curious about the why behind many things, I don’t know if experimentation is the field for you,” Shiva stated, underscoring curiosity as a crucial soft skill in the field.
Shiva also highlighted the foundational role of being data-driven in any experimentation strategy. However, he cautioned that having data isn’t enough—it must be effectively used to drive decisions.
“If you’re in a business setting and the business looks at your program and this is zero test wins, right? And then after two years, they would rightfully say, ‘Is this the way it’s supposed to go?’” Shiva remarked, pointing out that data-driven decisions are key to sustaining a culture of experimentation.
What else can you learn from our conversation with Shiva Manjunath?
- Why it’s crucial to critically evaluate industry buzzwords and ensure they align with real practices.
- How true personalization in experimentation goes beyond just adding a user’s name.
- The need for thorough analysis to genuinely support data-driven decisions.
- Shiva’s take on the future of experimentation after Google Optimize and how companies can adapt.
