As a non-fMRI researcher who consumes fMRI studies from time to time, I’ve wondered how one can spot loose or even audacious claims relying on basic logic and scientific thinking. In one of her “brain talks”, Nancy Kanwisher warned readers against three common issues in fMRI studies that they should watch out for.
Here I summarize the gist of Professor Kanwisher’s talk—While doing so, I realized that many of the same principles can be used to understand scientific findings in general, not just fMRI studies.
So here we go!
Tip 1: Thoroughly compare differences between conditions
- Claim: Brain region X activates when one does Y. Therefore, X is responsible for Y.
- Example: A certain area in the brain activates when people see faces. Therefore, this area is responsible for face processing.
- Issue: “Compared to what”? — If activation in the experimental condition is compared to “rest” (where people basically don’t do or see anything), then it could literally result from anything, including seeing objects, just seeing, exerting mental efforts, and a lot more. Even when the control group performs another task rather than resting, their task may still differ from the experimental task in more than one way—one should be cautious about which factor actually drives the activation!
- Advice: Imagine you’re a participant! — You can mentally run through all the tasks to see how they differ. Then, you can go on to examine which difference might be the real cause behind the observed activation.
As Kanwisher put it,
“Introspection isn’t perfect but it’s a damn good tool.”
Tip 2: Be aware of “reverse inference”
- Claim: Task X activates brain region Y. Y is found to be responsible for task Z. Therefore, X is equivalent to Z.
- Example: When people see their loved ones in pain, a certain area in their brains activates. According to past research, this area is responsible for pain perception. Therefore, “empathy for pain” shares the same neural substrate with experiencing pain oneself.
- Issue #1: “How specific is Y?” — Y may be responsible for 20 other things. Without further evidence, we can’t conclude that X is equivalent to Z rather than A, B, C, D, etc..
- Advice: One way to be sure that a brain region is exclusively responsible for one purpose is to painstakingly rule out all the other possibilities through years of experimentation. Kanwisher has done that for the fusiform face area so we can be fairly sure that when it activates, the person is processing faces.
- Issue #2: “What counts as the same region across individuals?” — People have different brains so it’s hard to say that one region in one person’s brain is the same as this other region in another person’s. Kanwisher compared the difficulty of mapping a region in different brains to mapping a speckle on different faces — “What do you mean that we have a speckle on the same spot on our faces?”
- Advice: If possible, run your experiments on the same individuals—if the same voxels in the same person’s brain activate when doing X and Z, then you can be much more confident that X is probably equivalent to Z.
Tip 3: Don’t use transitive inference!
- Claim: Brain region X activates when one but not the other group of people do Y. Therefore, the other group may use different brain regions for Y.
- Example: When people without autism spectrum disorders (ASD) see actual faces, their brain region for face recognition activates significantly more than when they see face-like stimuli. However, this difference isn’t significant in people with ASD. Therefore, people with ASD may use different brain regions to recognize faces.
- Issue: “A difference in significances is not a significant difference!” — The fact that one group differs from chance and the other group doesn’t is NOT the same as saying that there’s a significant difference between these two groups! (There may not be!)
- Advice: To see whether groups indeed differ, you should compare them directly, NOT each to their baseline condition!