One of the often overlooked places where A/B testing can contribute to an organization is new feature or functionality development. The main reason this is the case usually boils down to a CRO program failing to establish a testing culture within a company, where testing becomes one of the first stops for making business decisions.
I can recount a time at a large e-commerce site where I managed testing, there was a new mobile platform design revealed that was going to make shopping so much easier for mobile users, and purchase rates were going to skyrocket. Luckily, we had the infrastructure in place to actually A/B test this new platform with live users, only to see it underperform the old version, tanking purchases to the tune of what would have worked out to be millions of dollars a year. We spent months backward testing to find the issues with the new platform, and ultimately netting out to positive gains. The morale of the story here though is without testing, we could have very easily rolled out this new mobile platform, then been in the difficult spot of trying to understand why company performance is slipping. Was it the new platform, marketing, site issues, etc.
Here are three areas we find A/B testing being the most beneficial when it comes to feature testing:
1) Guide Decisions on Feature Design (or solve decision paralysis)
Anytime the conversation of adding a new feature, or tweaking a current one comes up in most companies, there will be no shortage of “expert opinions”. In many cases, this can lead to strife on the team and create a toxic environment, which can stifle collaboration. What we have seen great success in is using A/B testing to help arbitrate these arguments, and allow the decision-agnostic test data steer the team.
Also, if your testing department is keeping results warehoused in an organized fashion (as they should), you can quickly consult past test results to help guide conversations in these meetings with what is likely to work well or a good starting place. So instead of just offering another opinion in the room, yours is backed up by testing results from your company’s actual users.
2) Usability or Customer Panels Can Give Misleading Information
The ease with which many companies can conduct usability panels has led many to rely on these to be driving forces when looking at new features and functionality, in some cases instead of AB testing. Unfortunately, we have seen many times where experiences which performed well in panel conversations, not live up to expectations when split tested and exposed to the entire site population.
This can happen for a few reasons:
- Non-representative panel sample to general population
- The experience in a usability panel is often very controlled, whereas your website is not
- Small user sample
- Poor usability panel question or experience construction
Proper usability paneling is not the point of this article; just know that spot checking some of the findings/recommendations with A/B tests are recommended. Be aware, that split testing can become politically charged if it challenges previous findings derived from usability panels. If you wait too long after usability resources has touted to be a winner internally, there may be some resistance from those who’ve become attached to those beliefs.
3) Save Development Cycles with Prototyping with Testing
This is one area CRO teams are starting to leverage to better the working relationship with the IT team. With testing tools, you can often arrive at 70-90% of a feature experience with less development work to run a simple prototype test, and to observe how site performance is affected. This can be a great way to test a concept out before committing many development cycles to fully fleshing it out. Companies can spend many hours developing a new feature, only to find out it is a performance loser. The problem this creates is given the amount of sunk work, it can become an internal battle where the Conversion Group is pitted against IT. This is where internal relationships can become strained. Save the political battles and work with IT to get ahead of this situation, collaborating to flesh out the initial concepts.
Another way of getting testing buy-in is to sell A/B Testing as a way for IT to test some of their “cooking in the labs” ideas. One of the struggles for IT teams is retaining good talent, and developers feeling like they never have input into site development. Carving out some testing resources to allow for this experimentation can help IT feel connected, provide a creative outlet, keep talented developers around, and ultimately provide a better site experience