Experimentation Program Governance

Running an experimentation program at scale presents both great opportunities and significant challenges. In mid-to-large organizations, these challenges can often become overwhelming, leading to poorly run programs and wasted potential. This is where governance plays a critical role. Without proper governance, experimentation can quickly become inefficient, chaotic, and fail to deliver the desired business impact.

Congratulations! You have just been hired (or promoted) as an Experimentation Leader / Manager / Lead and are raring to go! The real work starts now and getting to value is all about speed and doing things correctly. But where do you start? What should you tackle first?

Towards the end of 2023, Kameleoon, an A/B testing software vendor published an article on their blog with interviews of CROs and experimentation leaders in the industry asking them what trends to expect in 2024. Everyone interviewed made bold predictions but here's why we believe they could be wrong and what experimentation teams must focus on.

Your experiment learnings and insights are the most powerful part of the experimentation process. If not handled correctly or overlooked, it can nullify all the work you have put in to running the experiment. The default state for a lot of experimentation teams is to lead with the actual experiment…

If your organization is running an experimentation program, you must track your ideas and experiments. It should be a no brainer. This is company IP. They’ve invested in experimentation and they own the output. It shocks me to hear that even now (at the time of writing this in 2023),…

An experimentation program is more than just running tests and there are lots of different moving parts including the people involved that you need to watch out for. When running your experimentation program properly, you need to keep a close eye on the metrics that can help you determine if your program is healthy or not.

In this final part of a three part series on Vanity Metrics in Experimentation programs, we will look at how the activities centred around Culture of Experimentation is a vanity metric.

There are clear indicators of successful experimentation programs. The biggest of these is how an organization manages the documentation aspect of their experimentation program. Experimentation documentation cannot be an afterthought. There must be a clear strategy and plan behind it to ensure it is valid and useful to the organization. A spreadsheet is not a strategy. Neither is hooking up random project management tools together with APIs in your spare time. What will separate a lot of organizations running mediocre and poor experimentation programs from those that do it well is how they view and structure documentation as part of their experimentation process.

The challenge with organizations trying to grow their experimentation program across the wider business is it faces an uphill challenge to get a disparate group of people to run experiments at the same level of quality. Often, there is no visibility of the quality of the experiments run. This is why created the Experiment Health Score.

Experimentation program documentation – the process from idea to result – are (or rather should be) a significant part of an organization’s experimentation program. It is something we at Effective Experiments have been talking about since our inception in 2014 when the idea of tracking experiments seemed like an alien…

Experimentation teams could be changing their hypothesis or data after an experiment is run. Here's how to stop this malpractice in your team.

A hamster on a hamster wheel can run for long periods of time but at the end of the day, it doesn’t go anywhere beyond the confines of that place. Experimentation programs are stuck on their own kind of hamster wheel. They do good work and run lots of tests…