Beginner’s guide to Bayesian Statistics: Bayes’ Theorem

statistics

bayesian statistics

bayes' theorem

probability

inference

introduction

Published

July 7, 2026

Imagine a man named Steve. He’s shy, helpful, tidy, loves order, and has a passion for details. If I asked you: “Is Steve more likely to be a librarian or a farmer?” — your gut probably screams librarian. He fits the stereotype perfectly.

But here’s the Bayesian twist: the known fact is that there are more farmers than librarians in this world — let’s say 20 farmers for every 1 librarian in the population. So, even though Steve matches the librarian stereotype, the base rate of farmers is much higher.

Let’s do the rough math. Suppose 80% of librarians fit that shy/tidy description, but only 10% of farmers do. Out of 200 farmers, 20 fit the description. Out of 10 librarians, 8 fit. So, out of the 28 people who fit the description, only 8 are actually librarians.

In the language of conditional probability, that means:

\[ \begin{aligned} P(\text{Librarian} | \text{Shy and tidy}) = \frac{8}{28} \approx 28.6\% \\ P(\text{Farmer} | \text{Shy and tidy}) = \frac{20}{28} \approx 71.4\% \end{aligned} \]

Despite matching the stereotype, Steve is actually more likely to be a farmer (20 out of 28)! This is the famous mental trap developped by psychologist Daniel Kahneman. He cleverly illustrates that our brains often fixate on the description (the evidence) and completely ignore the common knowledge (the prior). We tend to prove our hypothesis with the observation. Bayesian statistics, on the other hand, is about how to combine what we already know with what we just observed, or, even better, how to update our beliefs in light of new observations.

The Core Idea: Bayes’ Theorem

At the heart of Bayesian statistics lies Bayes’ theorem. It tells us how to reverse conditional probabilities — how to go from “what’s the chance of seeing this data if my hypothesis is true?” to “what’s the chance my hypothesis is true, given this data?”

Here’s the formula:

\[ P(H|D) = \frac{P(H) \cdot P(D|H)}{P(D)} \]

Each piece has a name:

\(P(H)\) — Prior probability: your belief about the hypothesis before seeing the data.
\(P(D|H)\) — Likelihood: the probability of observing the data if your hypothesis is true.
\(P(H|D)\) — Posterior probability: your updated belief after seeing the data.
\(P(D)\) — Evidence (or marginal likelihood): the total probability of the data, which normalizes the posterior so it sums to 1.

Deriving Bayes’ Rule: The Counting Way

Let’s generalize what we just did with Steve.

We want to know: Given the observed data D (e.g., they are shy), what is the probability of our hypothesis H (e.g., a librarian)?

By definition, that’s just the number of people who are both H and D (librarians who are shy), divided by the total number of people who are D (shy people):

\[ P(H|D) = \frac{N(H \cap D)}{N(D)} \]

Now, divide the top and bottom of that fraction by the total population N:

\[ P(H|D) = \frac{N(H \cap D) / N}{N(D) / N} \]

But counts divided by the total population are just probabilities! So:

\[ P(H|D) = \frac{P(H \cap D)}{P(D)} \]

And we know from the chain rule of probability (also called the “AND” rule) that the chance of both H and D happening is the chance of H, multiplied by the chance of D given H:

\[ P(H \cap D) = P(H) \cdot P(D|H) \]

Substitute that into our fraction, and boom — Bayes’ rule appears:

\[ P(H|D) = \frac{P(H) \cdot P(D|H)}{P(D)} \]

If you prefer to apply bayes’ theorem to Steve puzzle, let’s try: our hypothesis \(P(\text{Librarian}) = 10/210\); given the hypothesis, the probability of observing shy person is \(P(\text{Shy}|\text{Librarian}) = 0.8\), and \(P(\text{Shy}) = (200 \cdot 0.1+10 \cdot 0.8)/210\). Plug them in:

\[ \begin{aligned} P(\text{Librarian}|\text{Shy}) &= \frac{P(\text{Librarian}) \cdot P(\text{Shy}|\text{Librarian})}{P(\text{Shy})} \\ &= \frac{10/210 \cdot 0.8}{28/210} = \frac{8/210}{28/210} = \frac{8}{28} \approx 28.6\%. \end{aligned} \]

Same result. The counting method is just Bayes’ rule without the fractions getting in the way.

The Algebraic Shortcut (For Completeness)

If you prefer the formal conditional probability derivation, here it is in brief:

If the above does not make your head spin, here’s another way to see it through the lens of joint probabilities. The joint probability of H and D can be expressed in two equal ways:

\(P(H \cap D) = P(D \cap H)\). Therefore, \(P(H) \cdot P(D|H) = P(D) \cdot P(H|D)\). Rearranging gives us the Bayes’ theorem. All roads lead to Rome!

Another example: The Medical Test Horror Story

You have joined a screening test for a rare disease that affects 1 in 10,000 people. The test is incredibly accurate—99.9% sensitive (it catches 99.9% of sick people) and 99.9% specific (it correctly clears 99.9% of healthy people with 0.1% false positive rate).

Now picture 10 million people getting tested, what is the probability that someone who tests positive is actually sick (\(P(\text{Sick} | \text{Positive})\))? The followings are the bayes’ theorem inputs:

\(P(\text{Sick})=1/10000\)
\(P(\text{Positive}|\text{Sick})=0.999\)
\(P(\text{Positive})= (9999000 * 0.001 + 1000 * 0.999) / 10000000\)

\[ \begin{aligned} P(\text{Sick} | \text{Positive}) &= \frac{P(\text{Sick}) \cdot P(\text{Positive}|\text{Sick})}{P(\text{Positive})} \\ &= \frac{1/10000 \cdot 0.999}{(9999000 * 0.001 + 1000 * 0.999) / 10000000} \approx 9.08\% \end{aligned} \]

A 99.9% accurate test, and you have less than a 10% chance of being sick given a positive test result. That’s the base rate fallacy in action—ignoring the prior (the disease is incredibly rare) leads us to massively overestimate the meaning of a positive test result.

Bayesian Statistics is not just about Bayes’ Theorem

Before we go any further, I need to come clean about something.

The two examples we just walked through—Steve the librarian and the medical test—are striking. They shock our intuition. They teach us to respect prior knowledge. And yes, they use Bayes’ theorem.

But they are not fully Bayesian statistics.

They are point estimates. We plugged in single numbers and got a single number out. We treated every probability as if it were known with absolute certainty:

80% of librarians are shy. Exactly 80%. No uncertainty.
The test is 99.9% sensitive. Exactly 99.9%. No margin of error.
The disease prevalence is 1 in 10,000. Exactly 1 in 10,000. No variation across populations.

This is useful for learning the formula, but it’s not how Bayesian statisticians actually work.

What Makes Bayesian Inference Truly Bayesian?

In a fully Bayesian approach, we don’t treat parameters as fixed numbers. We treat them as random variables drawn from probability distributions. This allows us to express our uncertainty about the parameters themselves.

Here’s what that looks like in practice (yet simplified):

Prior as a Distribution

Instead of saying “the disease prevalence is exactly 0.01%,” we say:

“I believe the prevalence is around 0.01%, but I’m not certain. My uncertainty can be represented by a Beta distribution, say Beta(1, 9999), which puts most of its mass near 0.01% but allows for other values.”

\[ \text{Sick} \sim \text{Beta}(1, 9999) \]

This expresses our probabilistic belief about the prior, not a single number.

Likelihood as a Distribution

Instead of saying “the test sensitivity is exactly 99.9%,” we acknowledge that this estimate comes from studies with limited sample sizes. Maybe it’s 99.9% with a standard error of 0.05%. We might model it as:

\[ P(\text{Positive}|\text{Sick}) \sim \text{Beta}(999, 1) \]

or even more flexibly.

The Denominator: Marginal Likelihood

In the point estimate version, we calculated \(P(\text{Positive})\) as a single number—the total probability of the data across all hypotheses (e.g. probability of test positive given sick or not sick). In a full Bayesian model, the denominator is the marginal likelihood (or “evidence”):

\[ P(\text{Positive}) = \int P(\text{Positive}|\text{Sick}) \cdot P(\text{Sick}) \, dP(\text{Sick}) \]

This integrates over all possible values of being \(\text{Sick}\), weighted by their prior probability. It’s not a single arithmetic operation—it’s an integral over a distribution.

Posterior as a Distribution

The output isn’t a single number like 9.08%. It’s a full posterior distribution over the parameter:

\[ P(\text{Sick} | \text{Positive}) = \frac{P(\text{Positive}|\text{Sick}) \cdot P(\text{Sick})}{\int P(\text{Positive}|\text{Sick}) \cdot P(\text{Sick}) \, dP(\text{Sick})} \]

From this distribution, we can compute:
- The mean (our best point estimate).
- The median (another point estimate).
- Credible intervals (e.g., “there’s a 95% chance the true prevalence is between 8.5% and 9.8%”).
- The full shape of uncertainty.

Strictly speaking, the denominator \(P(\text{Positive}) = \int P(\text{Positive}|\text{Sick}) \cdot P(\text{Sick}) \, dP(\text{Sick})\) is a shorthand. The full expansion includes both the sick and no-sick possibilities:

\[ P(\text{Positive}) = \int_0^1 \left[ P(\text{Positive}|\text{Sick}) \cdot P(\text{Sick}) + P(\text{Positive}|\text{No Sick}) \cdot P(\text{No Sick}) \right] \, dP(\text{Sick}) \]

Since \(P(\text{No Sick}) = 1 - P(\text{Sick})\), this becomes:

\[ P(\text{Positive}) = \int_0^1 \left[ P(\text{Positive}|\text{Sick}) \cdot P(\text{Sick}) + P(\text{Positive}|\text{No Sick}) \cdot (1 - P(\text{Sick})) \right] \, dP(\text{Sick}) \]

In this simplified version, I’m treating \(P(\text{Positive}|\text{No Sick})\) (the false positive rate) as a known constant — for example, from the test manufacturer’s specifications. So it doesn’t get its own prior distribution or integral. The integral above averages over all possible values of \(P(\text{Sick})\), and for each value, it accounts for both the sick and no-sick cases. This is why the shorthand \(\int P(\text{Positive}|\text{Sick}) \cdot P(\text{Sick}) \, dP(\text{Sick})\) implicitly includes the no-sick term — because \(P(\text{No Sick}) = 1 - P(\text{Sick})\) is encoded in the integration over \(P(\text{Sick})\).

Define a full Bayesian model

In the post that follows, we revisit this rare-disease example and build the Bayesian model up step by step:

Understanding Priors — We’ll explore different types of priors (informative, weak, uninformative) and why choosing them is both controversial and essential. We’ll also discuss the danger of unfalsifiable priors—beliefs that can never be overturned by data.
Likelihood and Posterior — We’ll build the likelihood and infer the posterior, fully incorporating uncertainty from all parameters: prevalence, sensitivity, and false positive rate. I’ll demonstrate the Bayesian workflow based on observed data and show how observations shift prior beliefs into updated posterior distributions.
The Posterior Predictive Distribution — We’ll move beyond point estimates and learn how to generate predictions that fully account for parameter uncertainty, giving us not just a single answer but a full distribution of what we should expect.

By the end of the next post, you won’t just know the formula for Bayes’ theorem. You’ll understand why Bayesian statistics is a completely different way of thinking about uncertainty, learning, and decision-making—not just a formula you memorize.