In order to gain actionable user insights for your product, the first step is ensuring that the participants you’re testing are the ones that can actually provide those insights. Your choice of respondents will have a huge impact on the quality of the data you receive – testing the wrong users can lead to low data quality, and low data quality inevitably leads to bad business decisions.
For example, if you’ve designed a website catering exclusively to 20-30 year old females, insights gained by testing older or male participants are unlikely to provide you with data that can help drive your product forward. Instead, you’ll have wasted both time and financial resources collecting data that you simply cannot use.
Those wasted financial resources can be significant. Research in 2016 found that the annual cost of data quality issues in the U.S. for commercial interests amounts to $3.1 trillion (IBM, 2016), with an average financial impact per organization of $9.7M per year (Gartner Research, 2018). Remember, sometimes no data is better than bad data.
So how do you avoid this trap and ensure that you get access to the perfect participants? Luckily, Useberry and Prolific have teamed up to make it easy to source and recruit the exact respondents you need. Read on for three key questions that you should be asking yourself to ensure that the participants you’re finding are the right participants for your research.
What are the technical requirements for my study?
Don’t forget this often overlooked aspect of research design: the devices a participant will use to complete your study. When it comes to online sampling, participants may be logging in using a desktop/laptop, a tablet, or even using their mobile phone.
Have you tested your study experience on a mobile phone, a tablet, and a desktop computer? This is critical – and be sure that your prototype either works on all device types, or that you have screened your sample to make sure only valid device types are used. If you do not do this then you risk not getting insightful data on which to base your conclusions.
You can use the ‘supported devices’ selection when you create your target audience in Useberry to specify the exact device you want your participants to take your study on. You can either select a single device, multiple devices or, if your prototype supports all device types, you can leave them all selected. It is also important to state your research’s device compatibility clearly in your task description.
How do I want participants to engage?
Depending on the aims of your research, you can request different levels of participant feedback and engagement. If you are looking at basic usability such as prototype navigation or troubleshooting, then simply allowing participants to click through your study may be enough data for your requirements. However, if you are looking for richer data to drive insights, you may want participants to provide more detailed feedback in the form of audio, webcam, and screen recordings. To get this, you’ll need participants who can provide this engagement and are happy to do so. Prolific’s internal screening tool enables you to specifically select participants who are willing to use their audio, webcam, and record video. You can do this by using the ‘Webcam’, ‘Record Audio’, and ‘Record Video’ attributes in Useberry. With Useberry’s Video Shoots feature, you can get back rich video insights by capturing your user’s face, voice, and screen.
As with device specifications, you will also need to add some information to your study description to describe exactly what participants will be required to do and, more importantly, how. For example, if a study requires participants to verbally dictate their journey through your prototype, then make sure to say that in the task description and provide examples of how they would do this. It is also important to inform participants how you’ll use, store, and discard video or audio recordings. You can find more information about what should be included in your description here.
Who is my key demographic?
Once you have selected the right devices and the right methods of participant engagement, you need to select participants that fit the key demographic you are interested in. Choosing the wrong demographics will produce insights that are effectively useless, so this step is maybe the most critical of all. Remember, not all participants can answer all research questions.
For an example of why participant selection is so important, look no further than the replication crisis in psychology. Although this crisis has been driven by a number of factors (small sample sizes, incorrect analyses, etc.), one of those factors was that most published psychological research only investigated so-called ‘WEIRD’ (Western, Educated, Industrialized, Rich, and Democratic) participants (for more detail of this see here). This has meant that findings from the psychological literature may only apply to a specific subset of the population, rather than being generalizable.
So, to avoid this trap, you need to ensure that the participants you are recruiting are the ones best placed to provide you with actionable insights. Using the attributes selection gives you access to 250+ participant demographic criteria that you can use free of charge, so you can continue to add attributes until you have your ideal sample.
Even if the attribute you are looking for doesn’t exist you may be able to find your target audience by recruiting your own custom sample. To do this, you will first need to create a screening survey with Useberry. For more detail on how to create a custom sample please see our guidance in this article.
Once you have received the answers from your screening survey you can filter the participants that meet your criteria and then add their Prolific IDs to the ‘Custom Allowlist’ attribute on Useberry. This will ensure that your study is only advertised to those participants.
Conclusion
By answering these three questions and making the appropriate sampling decisions, you can be confident that the participants taking part in your research will give you the best insights. Now you simply need to see how many participants you want to test and set your study live!
Useberry offers a vetted and verified participant pool.
- Recruit users from 34 countries around the world.
- Profiled on hundreds of attributes, to match even the most niche and demanding testing needs.
- Recruit as many representative users as you need to answer your research questions.
- Get results with no delays.
