Uncorrected statistics in brain imaging

I admit I’m prejudiced, and that I’ve enjoyed watching the recent blast of public exposure for problems in statistical inference in FMRI, undermining many of the central papers of the field, spurred most centrally by Dorothy Bishop’s (“Time for neuroimaging (and PNAS) to clean up its act”) spanking of some often-cited works. click through for xkcd on statistical significance

Daniel Bor (“The dilemma of weak neuroimaging papers”) does a very nice job of explaining the corrected vs. uncorrected stats error along with offering some chastising of the field. And others have picked on these problems recently: Bennett et. al (“Neural correlates of interspecies perspective taking in the post-mortem Atlantic Salmon: An argument for multiple comparisons correction”) made a similar point in their terrific and very funny salmon study, and Vul et al captured some worries about way-too-high correlations in social psych w/FMRI studies (“Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition“) in a widely-circulated paper.

I think it’s pretty clear that this is a pretty big smackdown of something going on regularly in the actual practice of brain imaging studies. But I want to offer a couple of quick and off-the-cuff comments on the underlying biases that in a way supported the errors. Bor does a good job of saying “boy, we should be way more careful with our stats” (and who could disagree with that?), but if we don’t understand why we were so careless and mistaken, we won’t really be any better off, even if we fix this particular statistical error.

I believe that such errors were so widespread among so many practitioner who should (and do) know better is because of the reductive biases at work here. If you assume that there are or must be regular neural correlates for whatever human activity or behavior you’re interested in (as opposed to thinking such links are often fragile, temporary, holistic, and contextual), then the FMRI becomes the telescope or microscope with which we see those things. And failures to see just mean that you haven’t adjusted the focus correctly yet. After all, it takes me a while to get a nice clear focus on the bacteria with my microscope; but the blurry looks before that don’t count against what I see once the focus is set. We assume the fact that there is something to be seen there, and that correlates that we find should be regular and even causal, and so we tweak and twiddle until we see the thing that we are looking for — in these cases, a correlation between the target behavior and the brain activity that’s less than 5% likely by mere chance. Which, as xkcd so nicely reminds us, should turn up about 5% of the time just by chance.

The point isn’t that we don’t have ways to avoid this particular statistical error (clearly, we do), but that the assumption about neural correlates of behaviors and their causal significance masked us from seeing and attending to an error that we easily should have detected. In our rush and excitement to discover the (assumed to be there) neural correlates (and causes) of human activity (with, I’ll add, no actual model whatsoever of how those brain states are actually implicated in the causal story of human behavior), we were distracted from giving careful and critical attention to whether we had actually discovered what we thought we had.

The basic moral of the story is not that we carelessly screwed up our statistics; it’s that those screw-ups were completely in keeping with and likely in part motivated by our reductionistic localization fallacy biases about cognitive/behavioral neuroscience and the supposed neural correlates and causes of complex human behavior. It’s only when we keep those assumptions on the surface and explicitly call them into question that we will be doing our best to avoid these kinds of mistakes in the future.

15. March 2012 by Ron
Categories: philosophy, Uncategorized | Comments Off

The end of Facebook

For me, anyway. Facebook never really worked for me anyway; but on top of that, there are lots of serious privacy concerns — see, for example: Facebook gives your info to advertisers, gets caught in the act, promises to stop; EFF’s Bill of Privacy Rights for Social Network Users; MoveOn’s “Did you see what Facebook is trying to do?“; Openbook; Facebook and “radical transparency” (a rant); and the many links at Quit Facebook Day.

Which brings us to today. I decided to quit, and figured I might as well do it on Quit Facebook Day

I support the move to create an open-source, de-centralized, not-for-whoring-to-corporations alternative to Facebook; maybe Diaspora will be that; we’ll see.

31. May 2010 by Ron
Categories: whatever | Comments Off

Anti-materialism in 4th-grade science

My son Oz (in 4th grade) brought home some homework covering the study of matter in his 4th grade science lesson. Here’s a couple of lines from the narrative page “explaining” atoms and matter:

And on the next page, they quizzed him, to make sure he’d accepted the anti-materialist dogma.

When I asked Oz what was wrong with that, he thought for a minute, and then said “They’re in your brain!”

I’ve raised him well.

12. April 2010 by Ron
Categories: whatever | Comments Off

New machinery

I’ve put up new springs and weights to hold up the virtual facade of profron.net.  Things might be funky here and there while I get everything tweaked and balanced.

14. January 2010 by Ron
Categories: whatever | Comments Off

Varieties of PowerPoint minimalism

I often run across advice on PowerPoint presentations of the following sort: Keep your slides very simple, with as few words as possible. Lawrence Lessig might be the king radical of this sort of strategy — he’s notorious for the talks with over 200 slides of only a few words each. But the advice is everywhere in the “state of the art” talk about presentations. But I think it’s often the wrong way to go – especially in the context of the sort of teaching I do. Let me say why.

A good example of this advice that spurred me to jot this post down is given at Presentation Advisors in the post Reducing the Amount of Text on your PowerPoint Slides. There’s much good advice in this post; e.g., “Filling slides with useless text can be detrimental to your presentation’s health… since the audience can’t read and listen at the same time, they’ll sometimes do neither”.

The important and valuable advice here (that’s also well-grounded in lots of research) is that you don’t want the listener to be trying to read and listen to text at the same time. So keep the amount of printed text low; most of the content and detail should be in what the speaker says. Excellent advice, I think.

But agreeing to that doesn’t entail adopting slides with the minimal amout of text on them. The valuable point is that the amount of reading should be minimal compared to the amount of listening. So my 55 minutes of 100-words-per-minute talking (5500 words if you’re keeping score at home) should be accompanied by a pretty small number of written words. Suppose I follow this plan and get my number of printed words down to, say, 200.

Great. But that doesn’t resolve whether those 200 words should be (more or less) 50 four-word slides, or 4 fifty-word slides. We’ve agreed to keep text down to about 4 printed words per minute. The question now is about how those words should be shown. And to answer that, the question has to be asked: What’s the point of having words up there at all?

Images and drawings have their place, and for people who are selling rather than teaching, or teaching certain kinds of material, pictures may matter a lot. But for those of us who are teaching, and especially teaching abstractly described rather than concretely pictured points (like me, a philosophy teacher), you’re going to want words — key concepts and terminology, important questions, conclusions, and steps along the way.

In short, you often want the words up there to summarize where you are in the medium-to-big picture; to remind them of their current place in the macro-level structure of the discussion; to keep telling them that the ephemera of what you happen to be saying right that second is significant because of the place it holds in the larger-scale discussion of the topic being considered. When it comes to words on the overhead, it’s all about summarizing and locating where you are.

So why not just a handout? Handouts have their place as well, and sometimes a handout is better. But often a lecture tells a story and takes you from one place to another; with a handout, the later chapters of the story are in front of them already, which may well undermine your ability to tell the story in the best way possible. I want the students to feel the force of an argument for a view, or an objection to that argument; a handout allows them to see exactly where we’re going, and relax about whether the current twist of logic might be the last word or not.

None of this means the more Lessig-ish style can’t work for some things, or that we should backslide into long discursive slides that have students tuning out just to read the text in front of them. But it does mean we should be careful about a potentially illegitimate inference from the reasonable advice of “minimize the ratio of projected words to spoken words” to the conclusion “minimize the number of words per overhead”. There are a variety of styles that embrace the good “minimize audience reading” advice; and not all of them minimize the amount of text per slide.

Update: There’s some thoughtful comments and discussion on this issue and this post over at Presentation Advisorsoriginal post.

30. December 2009 by Ron
Categories: teaching, tech | Comments Off

This year's "Mindset" list

As you may know, Beloit College makes up a “Mindset List” each year to reflect the world of the new incoming college students (born in about 1991) — what their life has included, and what it hasn’t. Understandably, reading it makes those of us over, um, thirty feel old and alienated. And who doesn’t like that? So check it out; here’s a few of my favorites from this year:

  • 5. Salsa has always outsold ketchup.
  • 6. Earvin “Magic” Johnson has always been HIV-positive.
  • 9. Rap music has always been mainstream.
  • 15. Babies have always had a Social Security Number.
  • 22. The European Union has always existed.
  • 30. There has always been a Cartoon Network.
  • 32. Their folks could always reach for a Zoloft.
  • 34. Women have always outnumbered men in college.
  • 38. Belarus, Moldova, Ukraine, Uzbekistan, Armenia, Latvia, Georgia, Lithuania, and Estonia have always been independent nations.
  • 68. Official racial classifications in South Africa have always been outlawed.

18. August 2009 by Ron
Categories: fun, teaching | Comments Off

Not me, but I’ve heard this

What Your Prof. Read” (in your email), from Ph.D. Comics:

What Your Prof Read

12. August 2009 by Ron
Categories: fun, teaching | Leave a comment

Leiter's 10 Notable Philosophy Blogs

Brian Leiter has done a nice little list of 10 Notable Philosophy Blogs, which gives a nice place to start if you like such things. And Dave Chalmers maintains a huge list of philosophy-related blogs, if you really want to go nuts.

31. July 2009 by Ron
Categories: philosophy | Comments Off

Funny philosophy videos?

There’s the beginning of a collection of funny philosophy videos on YouTube. I’ve pointed to a couple before (like the excellent Kant Attack Ad and the Noam Chomsky Show), but hopefully it will grow into a nice collection.

31. July 2009 by Ron
Categories: fun, philosophy | Leave a comment

They’re Made Out Of Meat

Terry Bisson’s classic short about minds and meat.

24. July 2009 by Ron
Categories: movies-n-tv, Sci-Fi | Leave a comment

← Older posts