The Mythbusters and Statistics

I love the TV show "Mythbusters". If you've never seen the show, it's great. (A new season starts this week on the Discovery Channel in the US.) Every week, the team describes one or more myths or urban legends (example: a penny dropped from the top of the Empire State Building could kill a pedestrian), and then they attempt to confirm or "bust" the myth with experimentation. (Wind tunnel experiments showed that the terminal velocity of a penny isn't fast enough to seriously injure someone.)

One thing that has mildly bugged me about the show is the lack of replication in the experiments: typically, they only ever attempt to replicate the myth once or maybe twice, and rarely discuss the variability in their measurements. So I was pleased to see the issue come up in this New Scientist interview:

You often have sample sizes of one or two, but science is all about replication. How do you respond to that criticism?

JH: People simply wouldn't watch it if we were just repeating things over and over again. We do them as compactly as we can to keep up the energy level and flow. We intend these shows to be thought-provoking, not definitive.

AS: I think the part of the scientific enterprise that we do illuminate is that it's a messy, creative process that changes your whole understanding. We'll spend half an episode finding that we're asking the wrong question.

A fair response. I think in general they're aware of the variability and significance issues, but I agree that more important contribution of the show is the experimental process: encouraging kids in particular to actually measure and compare things to answer questions. Issues like controls, replication, and significance are moot without data, after all. (Co-host Adam Savage talks more about the effect of the show on kids in this video interview from reason.com. Heartwarming stuff.)  

I was also pleased to see that they do seek the input of statisticians from time to time, too:

When you are testing your own reactions, might you bias your results because you have expectations about the outcome?

AS: That's a good point and makes me think that we should demonstrate experimental bias on the show. It was an issue when we investigated "beer goggles": whether drinking alcohol can make people seem more attractive. I spent a long time with a friend of mine who's a statistician to try and remove as much of the bias as possible.

New Scientist: MythBusters: 'Using your head is a lot of fun' 

Comments

[Jose C Silva]

Good post, as usual. But I'd like to point out that in the early episodes (and some later ones as well), the Mythbusters did have some replication and statistics. (Very minor, though.) Examples: the Helium football one and the Toast always falls with butter down one.

The MBs have made quite a few whoppers, including "negative proofs," like when they proved that you couldn't scale the inside of a heat duct because Adam couldn't. One might point out that cat burglars are in better shape than the MBs...

Overall the MBs are a positive influence (more than "TV scientists" who create the impression that science is just knowledge): they point out that curiosity, skepticism, and experimentation are important tools in looking at the world and that you shouldn't just think about the big important things but rather make thinking and reasoning a part of everyday life.

(Of course, if they could address the many many many fallacies that come from basic innumeracy in the population, that would be swell. But that might make for boring TV, unlike -- say -- a rocket propelled car collision.)

My 0.02c (minus agent commission).

[David Smith]

Yeah, the negative "proof" get me, too. I rationalize it that "busted" means something different than "disproven", i.e. that *they* couldn't recreate the myth *this* way. (I'm sure the producers like having a definitive conclusion, too.) TO be fair, they've revisited a good number of their "busted" myths with new information or methods, which is actually a good illustration of the scientific method. There are no proofs, just hypotheses to be verified - or not - by others.

[Jeffrey Hulten]

xkcd.com said it best... "'Ideas are tested by experiment.' Everything else is bookkeeping"

http://xkcd.com/397/