Product Review : Wasabi – An Open Source A/B Testing Framework

PLEASE NOTE: This is an outdated article and Wasabi is no longer active in development since 2019.

This is a guest blog post contribution from Online Dialogue ( Individual contributors listed at the end of the article)

Despite the fact that A/B testing has become a standard method in market research, there are only a few open source platforms that offer these functionalities. However, in 2016, software company Intuit announced they were open sourcing their real-time, enterprise-grade A/B Testing platform named Wasabi. Intuit Inc. develops and sells finance, accounting and tax software for small business accountants and individuals. Wasabi is since the introduction of Facebook’s PlanOut in 2015 the first open source platform for A/B testing. That we had to test!

What is Wasabi?

Wasabi is a real-time, 100% API driven, A/B Testing platform. The platform offers companies and individuals the opportunity to perform experiments on the web, mobile and desktop for back-end and front-end, products and marketing. According to Intuit the platform is quick, easy to use (“You can spin up your Wasabi docker in 5 minutes and deploy it in distributed production mode within a day.”), cheap and scalable. The platform also protects your user data and at the same time maintains your competitive advantage since you do not have to hand over your sensitive data to 3rd party services like Google Analytics or Adobe Analytics. Another advantage can be the flexible UI for managing your experiments.

Let’s test

After installing a local server and entering the test data, we build a few A/B tests to see if the promising description of Intuit appeared to be true. At first sight, Wasabi looks like a solid framework that offers an excellent foundation, especially for server-side testing. A big advantage is the anonymity the platform maintains towards competitors and consumers since it is not visible who is using the platform (which is often the case for other A/B testing tools i.e. Optimizely and Visual Website Optimizer). However, you do have to programme all tests by yourself since a WYSIWYG editor is not available. Another feature we would like to see in a platform like Wasabi is a preview mode during the development of the tests.

Because the data of your visitors are saved in-house, the experiment-data for each visitor has to be collected from the Wasabi server. As a result, an on-premise installation seems the most logical solution. If you would run the Wasabi server in the cloud, you run the risk of network-latency effects that influence the speed of your website in a negative way. However, intuit describes the platform as fast (30ms response) and able to handle ‘enterprise-grade’ applications.

In spite of the promise for ease of use, the platform requires a rather high level of IT knowledge: before you are able to start testing you need the correct settings for the installation and updates and write your own code to get everything working (the entire platform is API driven). For our test we got the platform running locally on macOS, but running a live production instance is in our opinion very complicated. And because the tool’s infancy, the documentation is fairly brief and online information scarce. On top of that, you are responsible for implementing and testing any updates.

Our conclusion

There is a lot to look at if you want to use Wasabi the right way. At first sight Wasabi seems mostly suited for companies with a large IT department who are able to construct serverside A/B tests. This way it becomes a framework for companies who are looking to develop an in-house A/B test solution that includes standard features like a management interface and different A/B test options. If we compare the platform with GTMTesting.com (testing through Google Tag Manager) Wasabi seems to be a lot more complicated. Also, GTM runs independently of the code and server of the customer. In our opinion it is easier to use existing client-side solutions (like GTM, Optimizely and Visual Website Optimizer) if you want to start testing quickly.

The tests for this review are executed by Data Driven Designer Roy Schieving and Analytics and Optimization Expert Hans Nauta. Content by Kyra Delsing.