A Comprehensive Guide to Preference Testing

What is Preference Testing

Preference Testing is a process where researchers present users with 2-3 variations of a design or copy and analyze which one they choose and why.

It is a way to get proof of concept before you actually spend your money creating something that users may or may not like. Writing copy for a marketing campaign always has us wondering if we are picking the right words, phrases, tone – you name it – for our audience. No reason to leave it to luck – that’s what preference testing is for!

How it works

First things first, identify your goals. Are you aiming only to see which variation most users will pick? Or, are you planning on getting to the bottom of why they chose a specific variation? Make sure to make it clear to the people you test from the start, so they can put more thought into their decision.

Then, you are gonna need your designs (no more than 3!). If it is designs you are testing, they do not need to be live to test it. That’s the beauty of preference testing! Unlike A/B testing, where:

• your product needs to be live
• you can test only very small changes

in Preference Testing:

• you can have just the wireframe
• your versions can be drastically different

Setting up your test, you’ll want to follow it up with questions. That’s how you’ll gather all that sweet, qualitative data that will help you get into the users’ minds.

Your questions can be:

• open-ended, asking the participant to write a short text on why they chose what they chose or even describe what they like about it.
• multiple-choice, for example, providing different adjectives for the participant to choose which associates to the design/copy.
• opinion or Likert scale, asking the participants to rate aspects of what they chose.

You can also mix and match. Two open-ended and one multiple-choice, one open-ended, one multiple-choice and a Likert, or any combination really that works for you.

Now you are ready to send out your test. But how do you choose your participants? That can be a long discussion. It is important that the people you choose are as close to your target audience as possible.

Try to avoid people that would be biased, like your bosses or your colleagues. For statistical significance, aim to have at least 20, or even better 40. The phrase “the more the merrier” does apply here for better quantitative insights.

When is the best time for Preference Testing?

As early as possible. When you are still in that light bulb over the head moment, and you want to see what your (potential) users think of it. It is a way to get proof of concept before you throw your hard-earned money at designers and developers and say Go!

Alternatively, you could be thinking of a redesign. Your website/app/etc. works well so far, but it may be time to modernize things around here. Or even, compare it to your competitors and see what you are doing great, and where you could work a bit more.

Preference Testing in Marketing

Preference testing allows marketers to create more effective marketing campaigns that will be more successful for their company and avoid spending money creating something that won’t be well received by users.

Every marketer has done A/B testing. If you are like me, you love seeing the difference small changes make in your numbers. But I always, always, always want more information. I want to test bigger changes. I want to test before things are live. And most importantly, I want to understand why people choose what they do and if my changes actually played a part or if it was dumb luck. That is why I started using Preference Testing.

Maintaining brand consistency across your marketing channels is more than using the same palette and font. It is the tone of voice, the visual style, and sometimes, it can be hard to define for growing companies.

Preference Testing can help you find that sweet spot that makes you, the stockholders, and the users happy, and keep it consistent. You don’t have to keep using the same templates over and over when you can create fresh content and test it for consistency.

Not sure about your copy for that new big campaign you are planning? Test it out with your team. You can not only find out which their favorite version is (numbers don’t lie!) but why as well.

Using Preference Testing to test designs, can also massively improve collaboration between the marketing and design teams and reduce the frustrating back and forth.

Tips & Tricks

• Use conditional logic in your test to show participants different questions based on the design they choose to get the information you need.
• You can use Preference tests for almost anything! Pictures, text, videos, music files, color palettes.
  •  To improve the accessibility of your videos, you might find tools like VEED’s ‘add subtitle to video‘, ‘auto subtitle generator‘, and ‘video caption generator‘ useful. These tools are designed to help your videos reach a broader audience by ensuring they are more inclusive and easier to understand for everyone.
• To minimize recency bias, have your designs appear in a different order to half the participants.

Limitations of Preference Testing

In closing, it is only fair to discuss the limitations of the Preference test as well. It surely is a great plus that your design does not need to be a live product to test it, but from this stems the issue of the participants not being able to interact with it in any meaningful way.

Thus, you will need to test for usability once you have a prototype. User research is an important part of development (and one that can save you a lot of time and money down the road) and cannot be completed with only one type of testing. 

It is also prudent to keep in mind that we as people are not always good at stating why we like something. We may have made our choice based on a feeling of familiarity or even by chance, and it’s not easy to put into words. Make sure to add specific questions after your testing and maybe even have some multiple-choice ones, to make it easier for your participants to express.