The seventh framework of the Data Leadership Toolkit is: Building a Culture of Experimentation.

The Goal

Experimentation is one of the purest forms of being truly data driven. It enables practitioners to assess the likelihood of success between 2 or more options and to make statistically sound decisions from it.
‍
A culture of experimentation is one in which businesses acknowledge that gut instincts and experience may not be the best things for making crucial business and product decisions. It’s a culture in which teams actively put their assumptions and ideas to the test in order to assess and validate them. 

Of course, testing in and of itself is not the end goal. The end goal is to ensure that we base the decisions that we make from the results we observe through testing. 

The goal also is not to test every conceivable thing, lest we end up unable to make timely decisions when fast, business critical decisions need to be made. 

The goal behind developing an experimentation culture is to ensure that when and where possible we allow our assumptions and points of view to be challenged and validated through sound statistical analysis. 

The Challenge

Building a culture of experimentation is not easy, but it is none-the-less a worth and necessary pursuit for any data leader. Whilst the specific challenges you may face will likely change for company to company, here are common challenges you may encounter:

‍Cultural resistance to change:
Advocating for change within an organisation is always difficult. You will encounter the classic mindset of “we’ve always done it this way”.  Comfort with established methods is a hard mindset to shift. 

You’ll also likely encounter resistance when implementing a culture of experimentation as it has a tendency to expose bias and ego. People may be reluctant to put their perspectives to the test in case those are proven to be incorrect via the process. 

‍Lack of data literacy and analytical skills:
Implementing experimentation led thinking requires a team that understands data and can use it to make informed decisions.

It can be very tough to speak in terms of statistical significance, sample sizes and variants etc. when these concepts are not broadly understood by your colleagues and partners. 

‍Insufficient resource and infrastructure:
A culture of experimentation can be established with very little technical capability and resources. It would be a mistake to think that it can only be achieved by experimenting at scale with expensive testing tools. Experimentation can start at a very small scale, with small business tests handled manually. The purpose of this framework is not to turn your organisation into an experimentation superhouse, but rather to encourage the conversation around the purpose and validity of testing your assumptions and writing hypotheses.

But having said that, to succeed at scale, experimentation programs require an increase in capability and often, expensive investments in tools. This can be a barrier to scaling for a lot of organisations.

But building a culture of experimentation, where we look at how we might test our assumptions before diving headlong into them, is free. 

The Framework

As with any good drive for change, we must start with ‘why’.

The why’s of experimentation are likely infinite, but here are a few.

• Facilitates and improves data driven decision making

• Fosters a culture of innovation and adaptability: AKA evolution

• Is the catalyst for continuous improvement and optimisation

• Helps reduce risk  - stops us investing in/building things customers don’t want

• Helps us to acquire a competitive advantage

• Increases the feeling of contribution and collaboration amongst teams

As a data leader trying to kickstart an experimentation culture, communicating these goals and how achieving them can contribute to efficiency and the bottom line will be your goal. 

But how can you get started doing this? Try some of the follow approaches:

Start small and scale gradually:
Begin with small, low-risk experiments to demonstrate the value of the approach. Once initial successes are evident, you can scale the experimentation framework across more departments or processes.

‍Emphasise a learning mindset:
Shift the focus from success or failure to learning. Encourage the gathering of valuable insights, regardless of the outcome. Highlight that every experiment provides useful data that can guide future decisions.

‍Create safe spaces for failure:
‍Teams need to feel comfortable taking risks. Encourage a psychological safety net where failure is seen as a natural part of innovation. Celebrate efforts and learning from experiments.

‍Encourage cross-functional collaboration:
Promote collaboration between teams and departments to foster diverse ideas and perspectives. Ensure that these teams share insights and results across the organisation.

‍Provide the right tools and resources:
Invest in the necessary tools, data infrastructure, and training to support experimentation. Employees need access to data analytics platforms, testing frameworks, and education on how to design experiments properly. Provide guidance on how to measure success and interpret results.

‍Lead by example:
Leaders who openly discuss their own experiments and what they’ve learned set the tone for the rest of the organisation. If teams see leaders embracing experimentation, they’re more likely to adopt the same mindset.

The Outcomes

Successfully building a culture of experimentation will likely result in a whole host of positive outcomes for businesses that do so. These are listed above, so I won’t repeat them here again, but suffice to say experimentation is a driving force for so many of the world’s top performing companies, with regard to their culture, efficiency, profits and even share prices!
Don’t believe me? Google: The Experimenters Index.   

The Experimenters Index is a basket of S&P 500 companies whose collective share prices are pitched against the remaining companies in the S&P 500 index. Those included are the companies that have adopted experimentation as a core component of their culture. 

Suffice to say, their stocks have massively outperformed their S&P 500 peer’s.