The Effective Experiments Sandwich Model – The Importance of your Roadmap
Building an experiment roadmap can be tricky but can also be highly beneficial for your organisation and by the same token; having a badly maintained roadmap can be incredibly costly.
A poorly designed roadmap is, for all intent and purpose, a poorly designed experimentation program.
The roadmap is the core of why, what, who and when; so designing a strong roadmap should be at the forefront of your business.
The Benefits
- Greater visibility
By building an effective, well-designed roadmap your organisation will gain greater visibility on the experimentation pipeline, the goals and the eventual outcomes.
All teams that contribute to, or are involved in testing, will be aligned on company vision / team goals and how the experimentation team are trying to solve them.
- Cross-Team Collaboration
From greater visibility, grows stronger organisational collaboration. If all teams share the same information, understand the goals and what’s being achieved then naturally cross-team collaboration will improve.
By providing visibility on the expected work with timelines, the separate teams can all start pulling in the same direction, enabling growth.
- Resource Management
When the organisation benefits from visibility, understanding & collaboration increase, the management and allocation of resources becomes much more efficient.
When there is clarity over what everyone is meant to be doing and more importantly, when they’re supposed to be doing it…there will be less mistakes made.
No more thinly spread resources, no more changing to priorities and no more misunderstandings made across teams of workflow.
- A Well-oiled Insights Machine
When your team is all pulling in the same direction and firing on all cylinders, the experimentation program becomes an insights machine.
By removing doubt, misunderstanding, timescale change, movement of resource and confusion, the program can do what it’s been designed to do…help to gain insight(s) based on overall business objectives.
The Risks
- Test clashes
Test clashes are a nightmare for unorganised experimentation programs globally. This is the discovery that two (or more) tests due to run are now clashing with one another. They might be due to run on the same page at the same time for example.
Upon discovery three things then need to happen. Re-prioritisation, movement of timelines and communication to the team
It wastes resources, time and otherwise prevents the business from testing.
Imagine you have 10 experiments and 2 now can’t run. How many insights have you lost?
How costly is 20% of your outcome to the business?
- Overworked Resources
Confusion breeds the mismanagement of resources.
When mismanaged; they can often become overworked through the changing of design, development and additional QA.
Resources are arguably what defines a programs’ success, why would overwork your most valuable asset?
Overwork leads to declined productivity and eventually burnout.
- Process bottlenecks
A bottleneck is a setback or blocker that causes delay or changes to your roadmap. This could be something as simple as a director not signing off on time or the QA process taking longer than expected.
Without visibility of your roadmap (what is expected and when), the organisation loses the foresight of required action, and the process is delayed.
When the process is delayed, timelines are forced to change, experiments timelines and potential insights are missed. - Missed Opportunity
Why do organisations test in the first place? For valuable insight gain leading to consistent improvement.
If your program is experiencing test clashes, overworked resource, lack of visibility and bottlenecks then the organisation is missing the opportunity to gain value insights from the tests.
Without insights, what’s the reason for having an experiment program?
The Effective Experiments Sandwich Model
Planning a roadmap is easy…if you know how best to approach it. It’s a very common approach to dive into a roadmap headfirst, place the dates for experiments and cross your fingers that there aren’t too many obstacles in the way. This is not best practice.
Effective Experiments have designed the Sandwich Model for building your roadmap. Think of your roadmap as the filling to a delicious sandwich, maybe it’s your favourite sandwich (bacon and mango chutney if you’re wondering).
You wouldn’t get the filling out and throw it onto your plate without first establishing the what the two slices of bread holding it together were, would you?
Putting together your sandwich
The slices of bread have significant structural integrity to your sandwich and also to your experimentation roadmap.
The first slice of bread signifies the overall business objectives. It’s best practice to tie all your experiments to overall business objectives. This allows the team to understand what problem the business wants to solve?
The second slice signifies the desired outcomes. What do we want to achieve from our experiments and how do they contribute to the overall business objectives?
Understanding these two critical pieces of information will then help fill the roadmap with the right ingredients to be able to deliver the right outcomes against the business objectives.
Two slices of bread and some filling does not a sandwich make, there are a couple of additional ingredients to complete a rather delicious sandwich (fully functioning experimentation program).
Buttering up
As any good sandwich maker will tell you, you need lubrication for your sandwich filling to ensure they are placed into the sandwich correctly. In our analogy our lubrication is butter…also known as the prioritisation of experiments.
Prioritisation is very important and will differ hugely from company to company depending on your business goals, KPIs and resources available.
Prioritising your experiments can be done using many models, PIE, ICE, PXL, MoSCoW method and RICE are all well-known and proven effective ways of prioritising.
PIE and ICE may be a bit more subjective models to use, owing to their scoring framework.
An experimentation program that designs its tests based on business objectives will be able to prioritise impactful tests making the roadmap much more effective to the wider business.
Check out our article about prioritisation frameworks
A Splash of Flavour
The art of sandwich-making is ensuring the flavours complement each other and are complex.
Therefore, we need to add a condiment to shake things up…mango chutney is added…welcome the analysis of results.
This is often an overlooked part of the experiment process and vanity metrics are used to validate the experiment:
Was the test a winner or loser?
Did we get more clicks?
Did we get more conversions?
Did we reach significance?
All valid questions.
But the missing piece is “what impact will this have on our business and what are our next steps?” Looking at your results in this way will enable your results to become insight-based and to become actionable.
The action takes the form of the plate to place your sandwich on.
What is the point in doing all the planning, prioritising, testing and analysis if we’re not going to use the information?
Without action then all your hard work is worthless.
Is more research required?
Are there any iterations available?
Can we move the needle?
How can it be improved?
What needs to be implemented?
What can we do with this information?
Now that you have all the ingredients together and plated up; the sandwich should now be complete, and your experimentation program should look like this:
Pretty tasty looking sandwich don’t you think?
How do you serve-up experimentation to your stakeholders?
Now that you’ve designed your roadmap, tested, gained insight and have your next actions the work is done right?
Wrong, now comes arguably the hardest part of running an effective experimentation program: showcasing the incredible work you do to stakeholders.
Effective Experiments have seen a wide variety of efforts in this department; emails, screenshots, spreadsheets and experiment show-pages.
The sandwich model puts the objectives, outcomes, and actions in clear sight, improving overall understanding and uniting the entire organisation.
Why change that when presenting the insights and impact?
The best way, in our humble opinion, to showcase your work is by using the sandwich model in essence.
- Define overall business objectives = problem to solve
- What tests were run = what we did to try and solve
- Showcase insights, not results = from this we learned that…
- Action = using this knowledge we will now do…
Using this model, stakeholders from all levels in the business will find it easier to:
- Understand the test context
- Why they’re being run
- What the outcomes were
- What impact they have on the business
- What the next steps are
Experimentation is often misunderstood by stakeholders and needs to be made relatable.
Linking testing to business goals and the clarity on learning and action; the program becomes a credible, relatable and valuable part of the business
Remember; building an effective roadmap can be tricky and it can take time to get it working smoothly, but just like making a good sandwich…if you pour in your heart and soul then you’ll end up making a pretty tasty dish.
Now go forth and eat!
