The Role of Leadership & Governance in Experimentation Programs

A hamster on a hamster wheel can run for long periods of time but at the end of the day, it doesn’t go anywhere beyond the confines of that place.

Experimentation programs are stuck on their own kind of hamster wheel. They do good work and run lots of tests but ultimately, the real litmus test is whether business strategy is determined because of experimentation or independent of it.

The initiative to take the experimentation practice beyond the silos of the testing teams becomes a challenging one but it’s nothing to do with the lack of enthusiasm of the team.

The real reason why these initiatives fail to take off is because the teams tasked with it have no authority or ability to set the remit or rules that they want other teams to follow. Trying to drive change without authority is like trying to hammer a nail with a screwdriver. There might be some progress but it won’t be what you want it to be.

In one of the companies we talked to, the head of experimentation said this

“We wanted to collaborate with the UX team to use their research to drive new test ideas. We showed them the tests we run and how they could also contribute. After some initial interest, the teams stopped communicating or sharing their information blaming lack of time and other deadlines”

You might be forgiven in thinking they would listen to a “head of experimentation”. The “head of..” title doesn’t matter much as they still sit on what we term as a practitioner level.

The two critical pieces sorely missing in Experimentation Programs are

• Leadership
• Governance

Strong Leadership helps pave the way for strong, scalable experimentation programs

Let’s clear something up. Leadership is not lip service from senior management where they give you a bit of time in meetings or budget to buy tools and hire people.

Experimentation programs need someone in a senior position – VP or above – who understands the value of experimentation and is able to set the guidance for the teams that answer to them. The person in the leadership position doesn’t have to know every aspect of testing but must know how to tie it to business strategy. They must be involved in learning the insights gained from experimentation. They must also actively engaged by communicating the business initiatives and goals that the experimentation teams should work towards.

Autonomy is a good thing to have but when you want to orchestrate change at scale, there has to be leadership shown where the rules are clearly set and there are consequences for not following it.

If we revisit the story from earlier where the head of experimentation had a challenge convincing the UX team to work closely with them. Let’s see how it could have gone differently if there was strong leadership.

Before approaching the UX team or any team, the person in the leadership position (let’s say it’s the VP of digital) would have the plan of action ready. This initiative would have started from the top down rather than from bottom up.

The VP of experimentation would then bring the managers of both the experimentation and UX teams along with anyone else they may report to. Change requires a shared understanding and agreement from all parties who are part of it. They would then explain the initiative and why it needs to happen – the need for greater collaboration between UX and Experimentation to use customer insights in experimentation and experimentation insights in UX.

They would then lay down the expectations from each of the parties. UX must catalogue and make their research shareable in a certain way. Experimentation must clearly keep a log of post experimentation analysis and pass that back to research. There must also be a monthly lunch and learn where attendance and participation is needed.

This is also the time where you can expect objections. The UX team lead might complain about lack of time or too many projects. This is where the leader can help alleviate these issues by addressing those challenges and changing their workload to facilitate their initiative.

The biggest reason teams do not like change is because change is hard but more importantly, its additional work where they don’t have any incentive to do it.

Once the roadblocks are removed and the guidance set, the teams will work towards that initiative as it has been set by someone in authority. It’s no longer a nice to have but rather a must do because there are consequences if not done.

Governance helps teams operate predictably

Governance is about setting guidelines and guardrails within which the team operates. When an experimentation program lacks governance, they decide how they will work and eventually corners get cut. This is not because they set out to do bad work but time constraints or other factors start creeping and the quality of the work takes a backseat.

When the quality is unpredictable it’s harder to ascertain whether the experiments being run are reliable or not.

Talking to numerous experimentation teams, we have seen that governance is never a priority at the practitioner level. It’s not something they consider as necessary. They need to get the job done i.e. running experiments. It’s only when the program starts to scale, the cracks start becoming visible.

Governance shouldn’t be a scary topic. It’s not about micromanaging people or making their lives hard.

It’s simply creating predictable systems that everyone follows to achieve a trustworthy output.

Examples of Governance are –

• Using a hypothesis toolkit framework to guide practitioners to form strong hypotheses
• Methodology for prioritising and selecting test ideas
• Experiment scorecards to evaluate the quality of experiments run

This allows for objective evaluation of the inputs and outputs of an experimentation team.

Experimentation programs need to be structured with clear governance and strong leadership with oversight and involvement.

CLICK HERE TO REGISTER FOR OUR UPCOMING WEBINAR “GOVERNANCE IN EXPERIMENTATION PROGRAMS” on 13th January 2022.