Counting Carefully - The Base Rate Fallacy - YouTube

Hello Internet, I want to talk to you about counting.

No, not kind.

Not that kind either.

I want to talk to you about the kind of counting that can easily fool you into believing something

is obviously true when it's neither obvious nor true.

Here's Lambda.

Lambda just learned his country has a thousand cases of some disease, let's say vampirism,

each year.

Being a little paranoid, he badgers his doctor to test him for it.

The test has a 99.99% accuracy.

A few days later, Lambda gets a positive test result.

Should he be worried?

What are the chances that he is sick?

If you guessed that Lambda actually has a less than a 1 in 10 chance of being sick,

congratulations!

Wait... what?

How can a such a highly accurate test produce such vague results?

Let's find out.

What do we know?

We know how many people usually get vampirism in Lambda's country, how many people live

there otherwise and how good the test is.

But that's not what Lambda cares about.

The question he really cares about is "Am I infected?"

or rather "What are the odds/likelihood that I am infected?".

The answer to this comes down to counting very carefully.

The procedure is this - we first gather everyone relevant to our question.

Then, we split them up repeatedly into groups based on all the information we have.

Finally, identify the group that we care about and check how many members it is expected

to have - and perhaps, compare it with other groups that we care about.

Remember, the odds of something happening is simply the ratio between the various ways

in which it can happen and the ways it cannot.

So, here are the 100 million residents of Lambda's country.

Just for comparison, the United States has a little over 3 times as many people.

Of these, we know that usually about 1000 of them are infected at any given time.

Now, remember that our test has a 99.99% accuracy.

If we applied our test on everyone, we expect the test to mess up once in every 10000 trials

more or less.

It is important to remember that these failures occur randomly - the test doesn't care if

you are sick or healthy when messing up.

The result will look like this: Amongst the 1000 who are infected, the test

would almost certainly identify every single one of them as sick.

It's a really good test after all.

Amongst everyone remaining, the test would correctly identify the vast majority of them

as healthy - well, 99.99% of them.

Unfortunately, the test will mistakenly declare 0.01% of the healthy people as infected - that's

10,000 of them.

That's what you get when dealing with such large numbers.

The question now is, if the test declared you as infected, how likely is it that you

are infected.

Let's group everyone that the test declared as infected - healthy people and vampires.

Lambda is somewhere in here.

Now what do you now think his chances of being infected are?

A lot less than 99.99% for sure.

It's less than one in ten.

Weird right?

Now, you might be wondering why we had to start with the entire population and whittle

our way down.

This is because when doing these counting arguments, we limit our grouping based only

on the information we have.

Lambda was not showing any signs of infection.

For all we knew, he was just another resident of his country.

What if Lambda was showing signs of infection such as, oh I don't know, glittering or burning

under sunlight or looking like this?

We can add that to our diagram.

For the sake of argument, let's say that such symptoms are universal amongst vampires and

uncommon amongst non-vampires affecting only 1 in 1000 people.

If we didn't know this number, we could have set up a survey and found out.

Then, let's test people, but only if they are symptomatic.

The test will still catch a few glittery but otherwise healthy people amongst all the vampires

because it still has its 0.01% error rate but now there are far fewer of them.

Now, take a look at the set of people who were symptomatic and were declared infected

by the test.

If Lambda had been symptomatic and had gotten a positive test result, we could be almost

certain that he was infected.

This is why doctors don't order tests indiscriminately.

This curious result wasn't because the test for vampirism was bad.

It was because the test was applied inappropriately; the error rate of the test was comparable

to the fraction of the test subjects that the test was trying to categorize.

Only a tiny fraction of the population was infected and the test had a tendency to mess

up more often than that.

It was akin to knitting lace while donning oven-mitts.

If we instead manage to filter the test subjects even a little bit, then the combined effect

becomes powerful enough to give us the certainty that we want.

"Why would I care?", you cry.

Because this variety of "weird math" shows up in everything from spam filtration and

quality control to medical testing and cyber security.

It's important to recognize situations where your own intuition is about to fool you, particularly

on issues that are important to you.

Here are real world hot button issues where this knowledge of math is critical.

And remember, once you eliminate the impossible, whatever remains, no matter how improbable,

must be the truth.