If you’re geeky enough to listen to this podcast, you’re probably also a fan of the “XKCD” webcomic by Randall Munroe, which bills itself as “A webcomic of romance, sarcasm, math, and language”. (If not, be sure to check it out at xkcd.com.) Recently I was especially amused as I browsed comic 1132, titled “Frequentists vs Bayesians”, which contains a hilarious example of what is known as the “Base Rate Fallacy”.
Here’s how the comic goes. In Frame 1, a character states that he has a detector to tell him if the sun just went nova. Remember that it takes light from the sun around 8 minutes to reach the Earth, so theoretically if this happened, you might not know yet. However, the detector always rolls two dice, and if both come up 6s, it lies- giving a 1 in 36 chance of a wrong answer. The detector has just displayed the word “Yes”, claiming that the sun did indeed go nova. In the 2nd frame, a character points out that this means there is a 35 in 36 chance that the sun has indeed exploded— and since this is greater than 95%, the “p value” usually accepted as the standard in scientific papers, we must accept this answer as accurate. In the 3rd frame, another character says “Bet you $50 it hasn’t”.
As is often the case in XKCD comics, this humor works on several levels. In particular, if ever offered the chance to bet on whether or not the sun has just exploded, I would bet on the “no” side regardless of the odds. Money just won’t be that useful in a universe where you have less than 8 minutes to live. I’m also not so sure about the feasibility of the nova-detection machine, though the xkcd discussion page does claim that it might be possible using neutrinos, which are expelled slightly before the actual nova and travel at nearly the speed of light. Anyway, for the moment let’s assume we’re some kind of faster-than-light capable and nova-immune alien spacefaring society, and think about this bet.
Something probably bothers you about believing the sun has exploded based on the word of a machine that occasionally lies. But how do you get around the fact that the machine is right 35/36 of the time? Doesn’t the math tell you directly which side to bet on? This is the core of the base rate fallacy: when trying to detect a specific incidence of an extremely rare event, you must consider both the independent probability of the event itself occurring AND the accuracy of your detection method. In this case, any time we use the hypothetical machine, we are facing essentially four possibilities: A. The sun exploded, and our detector tells the truth. B. The sun exploded, and our detector lies. C. The sun is fine, and our detector tells the truth. D. The sun is fine, and our detector lies. Since the machine said yes, we know we’re in situation A or D.
Now let’s look at the probabilities. For the moment, let’s assume the sun had a 1 in 10000 chance of going nova. (It’s actually a lot less than that, since our scientists are very sure our sun has a few billion more years in it, but this should suffice for our illustration.) Situation A, where the sun exploded and the detector tells the truth, has a probability of 1/10000 times 35/36, or 35/360000. Situation D, where the sun is fine and the detector lies, has a probability of 9999/10000 times 1/36, or 9999/360000. So we can see that in this situation, we are 9999/35, or 287 times more likely to be fine than to be facing a nova. Thus, even if we are all-powerful aliens, we should still be betting on the side that the machine is wrong and the sun is fine.
This comic makes us laugh, but actually makes a very important point. There are many more concrete applications of this principle of the base rate fallacy in real life, as pointed out by the Base Rate Fallacy;s Wikipedia page. The classic one is AIDS testing— if, say, a test quoted as “95%-accurate” claims you are HIV-positive, but you are in a very low-risk population, you are probably fine, and should arrange another independent test. A scarier one is random “95% accurate” breathalyzer tests for drunk drivers— if there are very few drunk drivers on the road, but police set up a roadblock and test everyone, chances are that the innocent non-drunks falsely flagged by the machine will far outnumber the actual drunks. This actually could apply to any police technique, such as finding terrorists based on profile data, that attempts to identify rare criminals in the general population.
Another common case of this fallacy that has reached epidemic proportions lately is the use of supposedly “scientific” studies to justify exotic alternative medicine techniques. For example, suppose you run a study of sick people given homeopathy, a method that violates hundreds of well-understood properties of chemistry and physics, such as Avogadro’s Number and core biochemical reactions. Let’s say you get results indicating that it works with a “p value” showing a 95% probability that your test was accurate. You can’t just quote that 95% without taking into account the independent probability that a treatment that violates so many known scientific laws would work— and when you take this into account, the probability that such a study has really given useful information is vanishingly small. Thus the occasional studies that show good results for these scientifically-infeasible techniques are almost certainly false positives.
So, any time someone is discussing the probability of some extremely unlikely event or result with you in real life, regardless of the context, think about whether you might be ignoring some key factors and taking part in a Base Rate Fallacy. If that might be the case, take a few homeopathic brain-enrichment pills and listen again to this podcast.
And this has been your math mutation for today.
References:
- https://www.xkcd.com/1132/
- https://www.explainxkcd.com/wiki/index.php/1132:_Frequentists_vs._Bayesians
- https://en.wikipedia.org/wiki/Base_rate_fallacy
