Adult Studies Online
We run studies online through Prolific and Testable. We also have an mTurk account for adult participants, but have not recruited through mTurk thus far (as of 7/15/20). Currently, we primarily use Prolific for adult study recruitment. Testable is used when reaction time is required in a measure (for e.g. cognitive control task), as Qualtrics' reaction time function isn't the best.
Both Prolific and Testable will be covered on this page in some detail but both have comprehensive guides, manuals, and/or FAQs, as well as a very responsive and helpful team for additional support.
Qualtrics
The lab primarily uses Qualtrics in order to administer survey-style studies to adult participants online. All researchers in the lab running adult studies online should become familiar with Qualtrics. This section will only outline some basics all studies have in common, under the assumption that researchers already know how to use Qualtrics. Please sign into Qualtrics through UCI using the following link: https://uci.co1.qualtrics.com/
Blocks
The lab uses several common blocks across all studies. These blocks can be found in the Qualtrics Block Library when signing in using the dosclab SSO login. Blocks for each study are typically structured as follows:
-
Study Consent (including captcha and general instructions for the study)
-
Demographics Information
-
[any additional demographics information collected, e.g. Covid questionnaire]
-
Writing Prompt (used to weed out bot responses)
-
[any study-related blocks; i.e. all study measures]
-
Payment (with instructions to redirect to Prolific/Testable)
Quick things to keep in mind:
-
Putting a study online: Make sure to automatically redirect participants back to the recruitment website when setting up the study.
-
Downloading data after collection is complete: Make sure to export the question order information for randomized surveys (just in case).
Go to Survey Options on the menu at the top of the survey
Add the website redirect link here
Enter Prolific IDs for participants who completed pilot studies here
Prolific
The lab uses Prolific for studies consisting of only survey-style measures. Setting up studies on Prolific is fairly simple:
Adding a custom prescreening to exclude pilot participants:
In order to automatically input Prolific IDs (highly recommended) follow the instructions on this webpage:
https://researcher-help.prolific.co/hc/en-gb/articles/360009220993-Recording-participant-IDs-in-your-study-survey
For adult studies, we pay participants $3/15 mins estimated, which is $12/hr. Set up the payment as below in order to get the cost of publishing:
Once you have the cost of publishing, save the study as a draft and email Nadia the required amount.
Adding funds to Prolific:
You can add funds using a credit card. The funds reflected on the menu bar include all amounts currently on the account. Bear in mind that not all of these funds may be accessible to use for new studies. For a complete breakdown of funds, click on the "DL" account logo and scroll to view "Balances."
Accepting /Rejecting Participants
Once you click on the study name, all participants with IDs and completion codes should be displayed. Use the downloaded datafile to filter participants who should be accepted and rejected according to preset rejection criteria.
Participants can also be bulk approved using the following method:
Describe the study here
1) Add the study URL
2) Make sure to redirect the Qualtrics link to include parameters in order to auto-input Prolific ID (click configure)
Custom prescreenings are required if:
1) You want to restrict by any variables (e.g. restrict to US participants).
2) Exclude pilot participants
Testable
The lab uses Testable for studies where measures require reaction time. The survey-style measures are still presented through Qualtrics, while the RT measures are presented through Testable. Therefore, studies presented on Testable have two parts: 1) Survey measures on Qualtrics 2) RT measures on Testable.
Publishing Studies on Testable
Quick notes: When putting studies on Testable, make sure the first part of the study (on the Testable trial file) links to the second part of the study (on Qualtrics). Additionally, make sure to redirect participants automatically from the Qualtrics survey back to the Testable link.
Setting up a study:
When the study is ready, set it up on Testable before setting up recruitment.
Fill out all of the relevant fields.
Once the project is set up, you can add the trial file. If you do not have the trial file for a study, ask Nadia. If you are trying to amend the trial file to reflect changes for a particular study, the Testable Manual, found here is extremely helpful. For examples of trial files, or study setup, all previous studies run on Testable are available through the lab Testable account.
You MUST Activate the study in order for the link to be visible, and to recruit through Testable Minds. Once you activate the study, make sure to edit the redirect link in Qualtrics.
Recruiting through Testable Minds:
Testable allows the recruitment of participants through their Testable Minds feature, which is functions separately from the Testable projects.This is a step-by-step process of how to recruit through Testable (including payment).
If you are running a more recent iteration of a previously run study, you will need to exclude participants from previous iterations. This can be done by creating a panel of previous participants:
Write in the name of the new panel as convenient. Then save the panel and go to list of participants run on the previous study.
Check all participants and then click "Add to panel" and add all participants to the newly created panel. Make sure your panel includes participants from all previous iterations by repeating the process for all previous studies.
Once the panel is created, you can go back to Testable Minds > Manage Studies and create a new study for recruitment:
Fill in all of the required fields (title, description, number of participants, participation fee etc.), and upload the IRB ethics approval. Double check all of your links to make sure both parts of the study are working correctly.
Adding funds to Testable:
In order to recruit participants, Nadia will need to add funds to post the study.
These are recruitment funds and can be added via paypal or credit card. Once Nadia has added funds, you should submit the study for review. Review usually takes less than 24 hours, so the study will be up online and data collection complete very soon after submission. This is important because Testable requires all participants be accepted/rejected by 72 hours. At the 72 hour mark, all participants will be automatically accepted. Only submit a study for review if you have time within the next 72 hours to filter through participants.
Accepting/Rejecting Participants:
In order to access data, you must also add in Testable credits (a total of $0.50/data point). These must be added not to Testable Minds (outlined in yellow), but Testable Credits (outlined in red) for access.
Funds for Testable Credits can also be added via paypal or credit card.
Participants on Testable must complete both parts of the study in order to be paid. Both the study data from Qualtrics and the data from Testable must be cross-checked with participant Testable IDs in order to confirm participants have completed, and successfully passed 2/3 attention checks in, both parts of the study. Check the Qualtrics data as required and specified in the accepting/rejecting participants section. The Testable data can be checked by downloading the correct answers in the wide format from testable.
(To be amended by Sifana soon)
-
Delete by date
-
Go through attn checks with conditional formatting
-
Highlight rows (participants with 2/3 attn checks incorrect)
-
Go through prompt responses for low effort and bot responses
-
Reject incorrect attn check participants
-
Accept all participants who filled it out correctly
