If you have been paying attention to the recent updates from some of the most popular UX testing platforms out there, you might have noticed a common trend about the new “randomization” feature that is being released. So now that we have a few good options out there, we decided it would be a good time to compare the capabilities of the new feature on each tool. Before we start comparing the randomization features that UX platforms offer, it would be good to start by explaining why we would like randomization in our UX studies?
Why Randomize?
Randomization is essential in UX research because it helps control for biases and confounding variables that might influence study results. By randomly assigning participants to different conditions or varying the presentation of tasks and questions, we ensure that each participant’s experience is unbiased and fair. This approach prevents patterns and predictability, reduces order effects, and mitigates participant fatigue. Ultimately, randomization enhances the validity and reliability of our findings, providing a clearer, more accurate picture of user behavior and preferences.
If you would like to have a more in depth look at the impact of randomization on UX studies and how it enhances your findings and insights, you can check out our other blog article on the subject, “Introducing Randomization: How to Reduce Bias in UX Research”
Who Will We Compare?
In this article, we will take a look at the randomization options offered by Useberry, Maze, Lyssna, and Userzoom. Let’s first take a brief look at each feature individually, starting with Useberry.
- Useberry launched its randomization feature in January 2024. It handles randomization through what is called the “Group & Randomize” block. Useberry refers to the individual research methods and steps that they use to setup their UX studies, “blocks”. Through the Group & Randomize block, you gain access to an advanced randomization with many capabilities. For a detailed explanation of how the group & randomize block works, you can view our tutorial here:
- Maze has just released a limited randomization feature. With Maze, you have to add the “Variant” block and create the test structure for the variants you would like to test. Then select if the study is within subjects or between subjects.
- Userzoom has had its Randomization feature for a few years now. It is a detailed randomization feature similar to Useberry although not all the functionalities are done in one place like the “Group & Randomize” block of Useberry.
- Lyssna has also recently released a randomization feature with limited options. With their feature, you need to add follow-up questions after a task and enable the randomization for the follow-up questions or add a questions section and randomize the questions within this section.
Time to Compare Features
So now that we have taken a brief look at all the platforms we will compare, it is time to go into the capabilities of each feature.
Questions
| Questions | Multiple choice options | Ranking questions options | Between different questions | Between different groups of questions |
|---|---|---|---|---|
| Lyssna | ✔️ | ✔️ | ✔️ | ❌ |
| Maze | ✔️ | ✔️- Via Card Sorting | ❌ | ❌ |
| UserZoom | ✔️ | ✔️ | ✔️ | ✔️ |
| Useberry | ✔️ | ✔️- Via Card Sorting | ✔️ | ✔️ |
Multiple choice options – Starting with an easy capability, all platforms allow you to randomize the options of multiple choice questions, both single select and multiple answers.
Ranking question options – All platforms allow you to randomize the options of ranking questions as well. Userzoom and Lyssna use a specific question type called “ranking questions”, while Useberry and Maze have these capabilities through their card sorting options.
Between different questions – Here we start to see a split. Although we have the option to randomize questions at a basic level for all platforms, there is no option to randomize a subset of questions such as post-task questions or survey questions with Maze.
Between different groups of questions – Sometimes questions will be grouped into “themes” within a single study. You have the option to randomize each of these themes or subset of questions with Useberry and Userzoom but with Lyssna and Maze, you wouldn’t have the option to randomize these groups of questions.
In addition, with Lyssna and Maze, you wouldn’t be able to show only a certain number of questions from each theme to participants. This could limit your options for reducing bias as well as participant fatigue.
Card Sorting
| Card Sorting | Cards | Categories |
|---|---|---|
| Lyssna | ✔️ | ✔️ |
| Maze | ✔️ | ✔️ |
| UserZoom | ✔️ | ✔️ |
| Useberry | ✔️ | ✔️ |
So this step of our comparison will be short. All competitors allow randomization in card sorting studies. Things might look different for a UX study with a lot of questions or a usability study in general but for a simple card sorting study, either one of these features will be able to reduce biases and effects through randomization.
Cards – All tools allow randomizing the order in which cards appear to participants.
Categories – All tools allow randomizing the order in which categories appear to participants.
Tree Testing
| Tree Test | Tasks | Randomize tree order (nodes) |
|---|---|---|
| Lyssna | ❌ | ❌ |
| Maze | ❌ | ❌ |
| UserZoom | ✔️ | ✔️ |
| Useberry | ✔️ | ✔️ |
Although we had a pretty balanced comparison when it came to card sorting, things look a bit different when we are comparing UX randomization for Tree Tests:
Tasks – Lyssna and Maze do not allow you to randomize tree testing tasks. Thus, the participants will always see the tree test tasks in the same order, which means they won’t be able to address the biases caused by randomization in this research method.
Randomize tree order (nodes) – Lyssna and Maze do not allow you to randomize the order in which the nodes of the tree test appear to participants either which again causes problems when you would like to reduce biases in your UX study.
Preference Tests
| Preference Test | Tasks |
|---|---|
| Lyssna | ✔️ |
| Maze | ❌ |
| UserZoom | ✔️ |
| Useberry | ✔️ |
Maze does not offer preference testing. However, the rest of the platforms allow randomization in their preference tests. On these platforms, you can randomize the order in which the stimuli (or assets) of the preference test appear to participants. Offering randomization on a preference test is crucial for reducing order effects in your UX studies to have better findings.
Comparison Studies (A/B Testing)
| Comparison Studies | Within Subjects | Between Subjects |
|---|---|---|
| Lyssna | ❌ | ✔️ |
| Maze | ✔️ | ✔️ |
| UserZoom | ✔️ | ✔️ |
| Useberry | ✔️ | ✔️ |
There are two ways to design a comparison study which we call “within subject” and “between subject” designs. This distinction is based on whether your participants see a single variant or both. To know more about within and between subject design, you can visit the section of our article “Guide to Effective User Testing in the UX Industry” where we explain the differences between these study designs.
Within Subjects – All platforms except for Lyssna allow you to randomize within-subjects comparison studies. Which means, if you want your participants to see both variants on a UX study with Lyssna, you wouldn’t be able to randomize the order in which those variants appear.
*Between Subjects – All platforms have the capability to randomize a between-subjects study. Meaning that you would be able to randomly assign one variant to a participant while running a comparison UX study. *With Lyssna, you have to create two separate studies and share them to achieve between-subjects randomization.
Usability Testing & Additional Capabilities
| Usability Testing | Tasks | Assigning a subset option |
|---|---|---|
| Lyssna | ❌ | ❌ |
| Maze | ❌ | ❌ |
| UserZoom | ✔️ | ✔️ |
| Useberry | ✔️ | ✔️ |
Usability testing is the backbone of UX Research and probably a must-run UX method for any business. When we compare the randomization capabilities we see that 2 of the 4 tools that we are reviewing don’t have the option to randomize their usability testing tasks.
Tasks – Unfortunately, Lyssna and Maze don’t offer the option to randomize tasks which limits the randomization options when it comes to usability testing. For example, you can’t choose to add randomization to your overall usability testing study, or randomize the order of tasks for specific methods like tree testing, or first click.
Assigning a subset of tasks, questions, or groups – This is probably one of my favorite capabilities when it comes to randomization features because of its significant impact on reducing fatigue and order effect. It is only available on Useberry and Userzoom. What does it actually do? Let’s go over it with an example:
With this capability, you can choose to show only a certain number of questions or sections out of your questionnaire or even a certain number of questions from each subsection of your questionnaire to each participant. Of course, the same applies for each task of a study.
This means that you can build a single overarching study with dozens or hundreds of questions, group those questions under “themes” or categories, and let your participants only answer a custom number of questions or tasks.. You will be safe knowing that your participants are not affected by the length of a long study and get tired towards the end or be affected by the order in which they saw the questions or tasks. On top of this, you don’t have to spend time building multiple smaller questionnaires just to manually avoid this issue and then have to export and compile the results of separate studies, saving valuable time.
Which UX Testing Platform Comes Up on Top?
#1 – Useberry
Based on our comparison, Useberry is the clear winner here. Not only it has one of the most advanced and capable randomization features out of the four platforms that we compared, it also has a very clear structure on how to run randomization with the Group & Randomize block. Although some of the advanced randomization features require you to be on one of the paid plans, with a free plan you can still randomize multiple choice options, card and categories in card sorting, tree testing nodes, and preference testing options.
#2 – Userzoom
Userzoom is a close second to Useberry with its randomization feature. It has very similar capabilities but its pricing options and execution of the randomization feature are slightly below Useberry.
While running a research for the purpose of this article, I had to run through a convoluted help section on their website to understand the feature and pricing options meanwhile Useberry has a clear blog article with detailed explanations of the feature, a how-to guide video which I shared earlier, as well as a randomization webinar recording on their YouTube Page.
#3 – Lyssna
Lyssna was the clear third based on the comparison results. Their randomization feature is not as capable as Useberry or Userzoom but the feature is available with their basic monthly plan as well as free options.
#4 – Maze
Although Maze had similar capabilities to Lyssna, having their randomization feature locked behind the “organization” plan which is one of their most expensive pricing models, is a major barrier to entry for such an important feature that is a must-have for all sizes of customers for reducing biases and other negative effects.
Try Randomization Today with Useberry!
Reduce biases and get better insights
