Five stages every CRO team goes through when choosing a platform to manage their experimentation program

An experimentation program management tool is a vital platform for mature optimization teams. It helps glue together the disparate activities involved in a testing program into one place. Whilst Effective Experiments is one of the most advanced centralised testing management platform on the market going from strength to strength over the last five years.

However, we have noticed patterns in the prospective customers we talk to and the ones that eventually become our customers and in this blog post I want to highlight some of our learnings and insights.

Every organisation we talk to has gone through these five steps and in rare cases, they do bypass a couple of steps. I have listed the five steps below based on the five stages of grief.

Denial

Most optimization teams don’t start with any structured processes or intentions to store the learnings. Over 83% of teams we speak to have gone through years of testing before the need sets in.

This is initially brought on because of requirements from stakeholders who want more transparency and visibility of the work done.

At this stage, the business is unaware of the fact that they have lost vital insights and customer behaviour data that the team should have otherwise been catalogueing.

The Optimization team looks to tools they are already familiar with – the spreadsheets, the kanban tools and generic project management as they are unaware of the bottlenecks and challenges that these tools pose. Psychologically speaking, humans are unable to picture future challenges so it’s no doubt that these tools are chosen.

The tools don’t do everything so the team settles on a patchwork of tools. One for ideation and prioritisation. Another for crafting a test plan. Another for managing the workflow. To add to this, multiple email threads and communications via different channels.

Even with all of that, they are satisfied because the goal is to do a data dump of the work they do. Nothing more.

Frustration

After using this patchwork of tools for a few months, the cracks start to show and the bottlenecks of these solutions start to rear its head.

The amount of effort to keep multiple systems in sync increases. Optimizers start noticing that they spend more time managing these systems.

They now have to create reports by combing through information from multiple systems. Stakeholder reports and team updates start taking longer and eating through the time they could be spending on research and tests.

We were stuck with a spreadsheet that earned a nickname “the spreadsheet of doom”

An unnamed optimizer (and a current customer of Effective Experiments)

A quick note to mention that the tools chosen by the optimization team are not bad. They work well but for different use cases. In an optimization program, there are a lot of nuances involved in its management and generic tools don’t rise up to that challenge. Whilst most of these tools, spreadsheets and the likes might cost very little, the stakeholders would be shocked by the actual cost of all the inefficiencies and time wasted.

Acceptance

At this stage, the optimization team has realised they need a centralised way of managing their work. The tabs in the spreadsheet and numerous documents and emails simply won’t cut it anymore.

Now starts the search to find a centralized CRO program management tool.

From our experience, most teams are still in the mindset of finding one single tool to dump all their data. The processes still lack strong foundations and the criteria for any tool considered is basically a replica of what they currently do.

I call this process “Retrofitting”. It’s trying to do exactly what you were doing before in a new setting. This discounts the chance for improvements or exploring the potential benefits of doing it differently.

When we talk to customers at this stage, we share the potential benefits and our approach to helping businesses run better experimentation programs. This includes the mindset of Transformational optimization programs over Transactional ones.

We educate them on the real purpose of any centralized experimentation program management platform. Not as a data entry tool but one that goes beyond it.

It needs to help them think about their processes, to identify gaps and establish clear foundations and standards which will help them scale up in a predictable manner. The most important benefit however is the seamless visibility and transparency that will help them engage with their stakeholders and other team members without manual effort.

Back to the Status Quo

This stage is when, even after seeing the benefits of a platform they opt to stay where they are. They decide that the Status Quo is the easier option.

Change is hard. Switching to a new system could be seen as upheaval by some.

We looked into why this might be the case and we also found that Optimizers are unable to build business cases or are not able to authoritatively ask for the right tool set. Project management, as they pitch it to their stakeholders is bound to get an apathetic reply.

There is a need for an executive sponsor or a good relationship with C-level to help them make a case.

The lack of buy in, lack of authority to make decisions and the pull of the status quo sets them firmly back to square one.

Realization

Once the team realises the mounting challenges from using inefficient tools, they are ready to make the leap to a centralized experimentation management platform with renewed enthusiasm.

Closing Thoughts

It’s surprising how many CRO teams predictably go through all the stages even with all facts presented.

The small fraction of teams that have bypassed all the stages have gone on to work with Effective Experiments and make big changes to their testing programs and internal test and learn culture. They have saved their company a lot of time and money in making the decision earlier rather than later.

It reminds of the time I tried to put up a picture frame but couldn’t find a hammer for the nail. I used a screwdriver to hammer in the nail. The outcome : It worked. I managed to hammer the nail but not before hammering my own fingers a couple of times, cracking the plastic end of the screwdriver and bending the nail quite a bit. A few days later, the picture frame fell down after the bent screw gave way.

Will you continue to use a screwdriver (tools that are not built for purpose) or the right tool (one that is specifically built for the purpose of managing experimentation programs)