Scientific Integrity Policies

Introduction

Our lab's aim is to uncover something true about the world. But, doing so is very hard! There are many ways to incidentally introduce bias during data collection, coding, and data analysis. For example, a research assistant who knows that a child is supposed to get the "right" answer at age 4 but not age 3 might accidentally spend more time giving 4-year-olds the chance to give the right answer, be more likely to code 4-year-olds as giving the right answer in the case of ambiguous or close-call situations, and be more likely to look for the data to be consistent with this expectation. All of these are problems because they blur the distinction between what's true and what we think is true. Our goal is to produce high-quality data that is bias-free. The field of psychology is still evolving and figuring out the best ways to do this and part of our goal as a lab is to continually find new and improved best practices to adopt. As a start, here are the principles we will abide by:

Study Design

  • All studies should be piloted (i.e., "tested") prior to finalizing study design. The lead researcher decides when the piloting phase is done and actual data collection begins. Piloted participants may not be included in the final sample size. As a rule of thumb, you should plan to pilot at least 5 child participants.

  • All sample sizes should be set in advance of any data collection (excluding piloting - you can pilot, re-pilot and re-re-pilot as needed).

  • As a general rule of thumb, we aim for N=40 per cell, though when it comes to sample size, more is always better. A cell is defined as an area of interest (e.g., conditions; genders). If you are comparing 3 conditions, and also 2 cultures, you will need 3 (conditions) x 2 (cultures) x 40 = 240 children minimum. Of course, sometimes practical constraints stop us from collecting as much data as we would like. For example, we may have particularly cumbersome studies that require multiple visits, or the lead researcher may be unavailable to collect data after a certain date (and no one else will be able to take over). In these cases, we may need to go below our minimum, but always follow the rule of setting sample sizes in advance.

  • When possible, data collectors should try to stay blind to conditions or study hypotheses. This is frequently not possible, in which case you should consider how you can make sure to collect good data anyway.

  • Lead researchers should plan to collect all or part of the data on their own projects, unless it is in the project's best interest that this not happen (e.g., the project calls for a hypothesis-blind researcher). Lead researchers may also train research assistants to assist with data collection.

  • All study decisions should be logged in a Project Notebook.

  • We aim to pre-register at least one study within a project.

  • Prior to collecting data, all lead researchers should take the following steps:

    • Discuss and finalize project idea with Dr. Chernyak

    • Create a Project Binder with the following sheets (copies of these should also be kept on the server at all times)

      • Fill out the Study Opening Sheet for each new study

      • Create all of the Protocol Sheets (i.e., scripts for running the study)

      • Create all of the Live Coding Sheets (sheets to record live responses)

      • Create a Counterbalancing Sheet

      • Create a Participants Tracking Sheet for tracking all participants

      • Create a Study Checklist for all researchers to cross-reference prior to data collection

    • Create a digital Project Notebook logging all study decisions

    • Create a Data Sheet defining all your variables

    • Create a Project Box with the Project Binder and all stimuli and materials

    • Train research assistants (see below)

  • Any undergraduate research assistants must complete the following steps prior to being eligible to collect data on any project. All lead researchers are responsible for ensuring research assistants complete steps on projects they are leading.

  • Read through all study protocols and the Study Opening Sheet

  • Read the relevant articles listed on the Study Opening Sheet

  • Watch/shadow the lead researcher run at least 3 child participants on the project they are placed on

  • Practice running the project protocol with a senior lab member (usually the lead researcher) and get feedback

  • Pilot 1 child participant and get feedback

  • Complete the Research Assistant Training Form

Data Collection

Data collection refers to interacting with child participants.​​

  • We video and audiotape our data for record-keeping as much as possible. Videos should be set up such that they show the child, researcher, and stimuli. Videos should be audible enough to hear child's responses.

  • At least 2 researchers must be present during data collection. One researcher conducts the experiment. The second researcher works the camera, hands the stimuli, consents parents, and fills out the Data Coding Sheet

  • We need to take steps to ensure we keep accurate records and don't lose any data. Within 24 hours after each data collection session, researchers must:

    • upload all videos​ and delete them from the cameras once you ensure they are safely on the server

    • update the Project Notebook

    • file Consent Forms and Live Coding Sheets

    • update the participant database

Data Coding

Once we collect data, we extract the relevant information by coding that data into data sheets. Here, we need to take steps to minimize coding the data in a way that conforms to our expectations.

  • All videos must be coded by at least 2 research assistants and then checked for discrepancies. Discrepancies should be resolved by a third coder.

  • At least one coder and the third coder must be condition-blind (not know the conditions the child participated in). One simple way to do this is to create a set of "Condition Blind" videos by cutting out all information about condition for coders.

  • Data should be open by default (placed on the Open Science Framework or another publically-acccessible platform so that other researchers may access it).

  • Coders must always check their own work to ensure they have not made typos or mistakes.

Data Analysis

Once data has been collected and analyzed, we take the following steps to ensure that our statistics are accurate and that we are not tempted to look for patterns that aren't there:

  • All final analyses should be easily reproducible using data code.

  • All data and code should be made publicly available by default.

  • All analyses should be double-checked by another senior lab member (ideally, an author on the paper).

  • Clearly distinguish which analyses you planned to do (confirmatory) and which you did not (exploratory).

Manuscript Write-Up

Once data is analyzed and all authors have met to discuss the next steps, the first author should plan to proceed with writing up the first draft of the manuscript.

  • Report all experimental conditions and exclusions

  • Report where data, protocols, and code may be found online

  • Justify deviations from any pre-registered plan

  • Make use of your co-authors! Ask questions, send drafts, and request meetings if all authors are not on the same page. Co-authors from this lab (PI included) should expect to give feedback within 1-2 weeks of receiving a manuscript.

  • Make sure all authors have signed off on the final copy of the manuscript before it is submitted