Five mistakes experimentation teams make when it comes to test documentation

Experimentation program documentation – the process from idea to result – are (or rather should be) a significant part of an organization’s experimentation program.

It is something we at Effective Experiments have been talking about since our inception in 2014 when the idea of tracking experiments seemed like an alien concept.

Stakeholders are now starting to ask questions about the value their experimentation program brings them and teams who have not been doing it correctly are getting caught out.

In this blog post, we will share the findings of our research into the industry. We looked at testing teams at varying levels of program maturity and found these common mistakes coming up time and again.

There is no clear plan or strategy when it comes to documentation

Documentation of experiments and insights are unfortunately not given the importance or time when an experimentation program is set up. This is evident with so many experimentation teams that we speak to at Effective Experiments that are suffering from chaotic and haphazard documentation practices.

It typically starts off with teams either not documenting or doing the bare minimum. The tools of choice are whatever is available – Excel, online whiteboards.

Experimentation teams are still very much practitioner centric. Their time skews heavily towards the technical side of experimentation – research, ideation, experimentation etc. In the absence of any organizational or senior management remit, documentation becomes a box ticking exercise where practitioners don’t spend time thinking about the bigger picture.

Practitioners don’t have the time or bandwidth to care about it. They only look at the fact that they have stored the data. There are no KPIs or OKRs in place to ensure data is captured, synthesised and shared in a certain way. They don’t consider the “WHY” about proper documentation and anything that seems like too much work is a “tomorrow problem”.

In putting off fixing the documentation issues and challenges, practitioners are creating organizational debt when it comes to their experimentation program which becomes harder to change as time goes on. This manifests with companies having patchy data and insight or worse still, losing years of insights when the team members leave.

Now, as many organizations are looking to expand their experimentation practices to the wider business, a strategy for documentation is sorely needed.

Onboarding teams amplifies the documentation chaos

Experimentation is now moving out from the initial circle of CRO and testing specialists to the wider business. There is a requirement for product managers (amongst other departments) to start running their own tests.

When the documentation strategy is lacking as highlighted in the first point, rolling out experimentation to the wider organization adds fuel to the fire.

When new teams are onboarded, they are not shown the importance of documentation as this wasn’t seen as important to the testing specialists themselves. Moreover, the testing specialists tend not to have any authority to set the requirements or remits for these new teams.

What ends up happening is that every new team onboarded document their ideas, learnings and insights in whatever way they see fit. There is no centralised and unified plan for these new teams build on. The lack of authority from the specialists also means that these new teams can conveniently ignore any suggestions or guidelines.

In many organizations, we have seen different teams having different mediums for tracking their experiments. Some track ideas, others only experiments. Some prioritize ideas. Others don’t.

Having such as scattered approach to documentation drastically reduces the organization’s ability to scale their experimentation program. It hampers the organization’s true innovation potential as they will constantly have to revisit the basics and attempt to start again.

Practitioners knowingly or unknowingly create Data Siloes

We have seen experimentation teams and the wider team document experiments in the medium and format that they feel most comfortable with. This is often done without any strategy or centralised co-ordination as we discussed before.

What is most comfortable to those at the practitioner level is not always the most useful to the wider business because of one factor.

Data Siloes.

A Data Silo is a repository of data that’s controlled by one department or business unit and isolated from the rest of an organization.

When different teams have their own method of tracking information it’s like talking different languages. The others won’t always understand you.

Practitioners also create data silos knowingly as a flex to hold power over the accessibility of the data. To feed a sense of self importance, any individual seeking answers must ask it from them first. This poses a big challenge for the business as the practitioner is the single point of failure and a bottleneck to the free flow of insights and data. Should they leave, all that knowledge is lost.

Data siloes become an obstacle as the people not involved in experimentation cannot self serve or discover and interpret the data themselves. They have to go to the practitioner.

What the practitioners don’t realise is in doing so, their time becomes stretched as they try to cater to the demands of all the interested parties.

Their documentation is riddled with data integrity issues

We once talked to an experimentation manager in a well known brand that described their experimentation tracking spreadsheet as

I’d like to say they were being overly dramatic but after seeing it with my own eyes, I had to agree.

There were experiments with lots of information missing in random cells – hypothesis, test planning information, some experiments did not even have any information about the results of the experiments save for the description of the outcome.

Is this data even trustworthy? Can senior leadership make business decisions based on faulty data?

Spreadsheets and general project management tools are notorious for data integrity issues because there is never an audit trail.

I could easily delete the data from a couple of cells and the entire spreadsheet would be rendered useless. No one would know. There is no audit trail or oversight in these system.

How can you fix something when you don’t even have visibility on what is going on?

Usefulness of the documentation beyond the practitioner circles is limited

As mentioned before, when there is no governance or guardrails put in place for documenting experiments, you will find that the quality within the company is all over the place.

Even more so, practitioners forget the reason they are documenting their experiments and insights.

They mistakenly assume that this is for their purposes only.

However, the purpose of documenting MUST BE about providing this information to the wider business.

To the stakeholders. To marketing. To sales. To the strategy makers.

Data siloes already prevent them from accessing it. But even if these data siloes were not present, how will someone who is completely disconnected from the experimentation process know what the data is saying and what actions are needed?

Practitioners take a lot of pride in their Powerpoints and dashboards which contains (selectively added) experiments and results with their p-values and statistical significance. But what can the reader take away from it?

A test you ran won. Great. I guess. What good does that do for me?

I use the example of a stock market dashboard.

There are people out there who will know exactly what these numbers mean and what they should do about it.

What good does this do to me if I’m time stretched and only looking for insights on what I should do or where I should invest. I could take the time to learn everything from the basics to the nuanced details but I don’t have the time.

You documentation data and insights are not there for you as a practitioner.

It’s company IP and customer insights that needs to be made visible and available to the decision makers.

How can organizations escape these mistakes?

The best way to fix these issues is for a senior leader in the business to take the reigns of this part. This could be someone Senior VP or Director level who will oversee and set the remits for the documentation.

This way, practitioners are freed from the burden of having to manage the documentation practices of other teams. When it comes from the top down, the teams will quickly fall in line.

If you’re a practitioner reading this, get the backing of a senior leader to help you with this. If you truly want your company to build a culture of experimentation, you have let go or change your role completely to that of an ambassador or orchestrator. Even then, it’s hard if you don’t have the authority over the other teams.

You can start by getting a baseline of your organization’s current Documentation Maturity through our free Experimentation Documentation Maturity Assessment