PROJECT
Content A/B testing
Date
November 2023 - April 2024
When I first started at Wix, we had a tool that allowed us to do content A/B tests through our CMS, without involving developers. Then, for nearly 2 years, it stopped working and no one knew why. The ability to test content on our own is very important for showing the value of content and to learn more about what resonates with users. So, I set out to figure out what went wrong and who could fix it.
After months of back and forth, we found someone to fix the system. Once it was fixed, I put together resources and did outreach to make sure other stakeholders knew about this ability.
Since it was fixed, we have launched many content A/B tests and have further improved our ability to show the value of content.
Resources created
A guide to content A/B testing
I put together a document that could be used by all stakeholders. It covered when a content A/B test is appropriate, the limitations of it, an overview of how it works and the overall process, then a step-by-step guide for all stakeholders.
To make it as easy as possible for writers, I also put together a deck that had a screenshot of every step needed to set up and close a test in our CMS.
Test plan template
We want writers to own and lead the testing process, so we created a test plan template document that they can use when preparing the test with a Business Analyst.
Examples of successful past tests
To make sure that stakeholders understood the value of these tests, I put together a deck of examples of successful past content-only A/B tests.
Recent tests
Multilingual
We changed the CTA on a main dashboard from
"Translate" to "Edit Translations".
We saw a 10.1% increase in our main KPI of users being ready to go live with their Multilingual site.
There was a 6.8% increase in the number of Multilingual users upgrading their sites to Premium.
Events
We changed the name of a tab under our Events product from "Tickets" to "Tickets and Seating".
We saw a 19.2% increase in people who clicked to create a seat map and a 4.8% increase in tickets assigned to seats.