Sampling biases in developmental studies

Are parents who agree to let their children participate in scientific experiments really a representative sample of all human parents?

At some point, perhaps developmental researchers all have wondered, are parents who agree to let their children participate in scientific experiments really a representative sample of all human parents? Anecdotally, it seems to me that those who choose to sign up for studies have exceptional passion for or at least trust in science, curiosity about how their children think, devotion to early learning and education, etc.. Do their children learn differently compared to those whose parents do not consent? The answer is of the utmost importance to the generalizability of developmental studies : on the one hand, we can only test children whose parents agree for them to participate (How else are we going to pass IRB reviews?); on the other, we wish our results apply to all children.

Yue Yu and his colleagues at Rutgers University (Yu, Bonawitz, & Shafto, 2017) solved this apparent conundrum with a clever design and a popular statistical technique.

The experiment

In the beginning, two experimenters secretly observe pairs of parents and children (“parent-child dyads”) in a racially diverse local zoo or a playground. They keep track of both the quantity1 and the quality2 of parent-children interactions. A total of 109 dyads were observed.

Five minutes later, a third experimenter approaches the observed dyads, asking if the parents want their children to take part in a study. 78 pairs were invited (the other 31 were excluded for various reasons), among which 59 agreed and 19 refused.

For those who agreed, two experimenters introduced their children to a novel toy with five functions: “A tower that lights up when a button was pushed, a knob that produces a squeaking sound when squeezed, a lady bug pin light that flashes in three different patterns when pushed, a flower magnet that moves between three different places on the toy, and a turtle hidden in a pipe that is visible through a magnifying window” (p. 3 [emphasis added; changed to present tense]).

One experimenter who claims to be knowledgeable about the toy points to the tower (the target function) and says, “I’m asking you to think about: What does this button do?” Then children are given the toy to play with until bored. Children’s test performance is coded by several well-established measures3.

The relevant finding

The question is, do children with or without parental consent differ in their test performance? The problem is, the researchers don’t have data of those without consent. The solution is “model-based multiple imputation” (Rubin, 2004), a statistical technique that deals with missing data.

The “magical” imputation process works in roughly five steps:

1. Yu et al. (2017) found that children’s pre-test interactions with their parents are correlated with their test performance later. As a result, we can predict test performance reasonably well using pre-test parent-children interactions.

2. Let’s assume that for both consented and non-consented children, the relationship between test performance and parent-children interactions is the same.

3. So we can model the above relationship based on consented children

4. and use the model to predict non-consented children’s test performance from their interactions with parents (which are available!).

5. Repeat Step 1–4 100 times, each time adding a random noise.

Results of these 100 simulations suggest that non-consented children consistently differ from consented children in test performance, the mean differences of which range from .09 to .20.

In a nutshell, children with parental consent are a biased sample and their test performance may actually differ from that of the whole population!!


Reference

Yu, Y., Bonawitz, E., & Shafto, P. (2017). Inconvenient samples: Modeling the effects of non-consent by coupling observational and experimental results. Proceedings of the 39th Annual Conference of the Cognitive Science Society.


Footnotes

  1. Including: 1) the length of dyadic activities, 2) the length of supervised activities, and 3) the length of unsupervised activities.
  2. Including: (continue numbering) 4) the number of parents’ pedagogical questions, 5)the number of parents’ information-seeking questions, 6) the number of parents’ statements, and 7) the number of parents’ commands.
  3. Including: 1) the total time spent on the toy, 2) whether the target function is activated during the whole process, 3) the number of unique actions performed on the toy during the whole process, 4) the number of non-target functions activated during the whole process, 5) whether the target function is activated in the first minute, 6) the number of unique actions performed on the toy in the first minute, and 7) the number of non-target functions activated in the first minute.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s