Five Important Experimentation Documentation Practices Of Successful Experimentation teams

When done right, experimentation can be a vital tool in a company’s decision making process. It can act like a compass that gives the decision maker a direction to move towards. Experiments can help the business improve their outputs (or conversion rates) and mitigate any risks.

Experimentation documentation cannot be an afterthought. There must be a clear strategy and plan behind it to ensure it is valid and useful to the organization. A spreadsheet is not a strategy. Neither is hooking up random project management tools together with APIs in your spare time.

What will separate a lot of organizations running mediocre and poor experimentation programs from those that do it well is how they view and structure documentation as part of their experimentation process.

For it to be useful, the practice of documenting, maintaining and using the insights is as important.

These five key practices are indicative of the long term success of experimentation in that organization

• Documenting the Experimentation Program not just the experiment – The most successful experimentation teams know that every aspect of the experimentation program is a data point that can be observed and improved upon. They track the source of the idea, connect it to relevant research and have an audit trail all the way through the completion of the experiment and the its analysis.
  
  Experimentation documentation is more than documenting individual experiments and their outcomes. It’s not about dumping data into a spreadsheet whenever someone has free time. It needs to be structured and coordinated to capture data of the entire experimentation process as it shows up in the real world. Every customer insight that is generated by the research team, the idea is generated, it is captured. Your processes need to dictate what happens when an idea is created, when can it be prioritised and by whom, what information needs to be in an experiment plan and who can call the outcome. Every aspect of documentation must follow these processes rigorously.
  
• Structured onboarding – As organizations attempt to grow the experimentation capabilities across the business, they face challenges onboarding people who are new to experimentation. As such, it is vital to create a robust onboarding program that not only educates them on what’s possible with experimentation but also equips them with the “Why are we testing”.Spending more time focussing on this aspect instead of just the technical know how will reap long term benefits. Do the teams being onboarded know how their jobs will be impacted by running experiments?
  
  The onboarding must include a monitoring phase which helps those individuals new to the practice of experimentation get support and guidance as they “grow the muscle” until they are ready to do it independently. Leaving them to their own devices after passive training sessions will result in.
  
• Everyone is responsible for it – If experimentation is being done by multiple individuals and teams across the business, everyone should be responsible for the quality of that documentation. This means that all individuals have a remit for inputting the information at the correct moment in the process and are accountable to it. In contrast, organizations that struggle are those poor established processes where the burden falls on the one or two people to do the the heavy lifting for everyone else. In some scenarios, information isn’t input or forgotten about because of conflicting priorities.
  
• Governance & Guardrails monitored by senior managers – It is important to have clear guardrails defined to ensure data integrity. However, it is more important to have that governance in the hands of someone who sits above the teams responsible for documentation. If that isn’t in place, the program is susceptible to Hark-ing (basically, practitioners marking their homework)
  
• Dissemination of insights – Documentation is only good if the information is seen and utilised. The insights and meta analyses of the data can provide teams, stakeholders and decision makers with useful information. That said, the way information is disseminated in the organization is important. For many teams, this is limited to PowerPoints sent via email and that’s their job done. A lot more can done in this aspect to get information shared in a way that it gets engagement. The ones that are more successful keep a close eye on these engagement metrics rather than vanity metrics.

These five practices will set an organizations experimentation program far ahead of their competitors which may still be stuck on a hamster wheel of running test after test and seeing only micro improvements but not really making the organizational impact that experimentation can make.