Probability vs. Likelihood: A Deeper Dive

Probability quantifies uncertainty about an outcome or parameter, often as a distribution, while likelihood measures how well observed data supports a specific parameter value. The key difference lies in what is fixed and what varies: probability applies to outcomes given a fixed model, whereas likelihood is a function of the model's parameters given fixed data. This distinction is crucial in statistics, where related concepts like density, proportion, and possibility also play distinct roles.

What is Probability? A Formal Look

At its core, probability is a measure of uncertainty. It provides a formal framework for reasoning about the plausibility of different outcomes or parameters. The mathematical meaning of probability is built on a few key ideas:

• Sample Space: The set of all possible outcomes of an experiment (e.g., for a coin toss, the sample space is {Heads, Tails}).
• Event: A specific outcome or a set of outcomes we are interested in (e.g., getting Heads).
• Axioms: Probabilities are governed by fundamental rules. They are always non-negative numbers, and the total probability of all outcomes in the sample space must sum to 1.

In statistical inference, we often express probability as a distribution. For instance, if we are uncertain about the true proportion of water on a globe, which we can call parameter P, a probability distribution for P would indicate the plausibility of every possible value from 0 to 1. This distribution represents our state of knowledge or belief about the parameter.

Types of Probability

While the mathematical rules are consistent, the interpretation of what probability represents can differ:

• Classical Probability: Assumes all outcomes in a sample space are equally likely. The probability of an event is the ratio of favorable outcomes to the total number of outcomes (e.g., the probability of rolling a 4 on a fair six-sided die is 1/6).
• Empirical Probability: Based on observed frequencies from data. The probability of an event is the limit of its relative frequency in a large number of trials.
• Subjective Probability: Represents a degree of belief about a proposition, based on all available evidence. This is the perspective used in Bayesian inference, where probabilities for parameters are updated as more data becomes available. This article primarily uses this interpretation, where probability distributions represent our beliefs about parameters both before (prior) and after (posterior) observing data.

What is Likelihood? The Role of Data

While probability quantifies our uncertainty about a parameter, likelihood addresses a different question: "Given a particular parameter value, how plausible is the data we actually observed?" Likelihood measures how strongly the data supports a specific parameter value under a chosen generative model.

A generative model is a story for how the data is produced from the parameters. Without this story, the concept of likelihood is meaningless. For example, let's continue with the globe-tossing experiment to determine the proportion of water, P. Our generative model assumes each toss is an independent event that results in 'Water' (W) with probability P or 'Land' (L) with probability 1-P.

The likelihood function, written as L(P | data), is the probability of the observed data viewed as a function of the parameter P. If we observe W water tosses and L land tosses, the likelihood is proportional to:

L(P | W, L) ∝ P^W * (1-P)^L

Here, the data (W and L) are fixed, and the parameter P is the variable. We can plug in different values of P to see which one makes our observed data most probable. For instance, if our data includes at least one land observation, a model where P=1 (the globe is 100% water) has a likelihood of zero. As we collect more data, the likelihood function becomes sharper, concentrating around the parameter values that best explain the data.

Probability vs. Likelihood: The Core Distinction

It is a common and critical mistake to confuse probability with likelihood. They are mathematically related but conceptually distinct tools used for statistical inference. Probability describes the plausibility of different outcomes or parameter values, while likelihood quantifies how well the data supports a specific parameter value.

The relationship is central to Bayesian updating, where we combine our initial beliefs with evidence from data:

Posterior Probability ∝ Prior Probability × Likelihood

The prior is our probability distribution for the parameter before seeing data. We multiply it by the likelihood (the plausibility of the data given the parameter) and then normalize the result to get the posterior distribution. The posterior is our new, updated probability distribution for the parameter. Likelihood alone does not account for our prior beliefs.

The following table summarizes the key differences:

Maximum Likelihood Estimation (MLE)

A primary application of the likelihood function is Maximum Likelihood Estimation (MLE). In MLE, the goal is to find the specific parameter value that maximizes the likelihood function for the observed data. This value is considered the most plausible estimate for the parameter because it's the one that makes the observed data "most likely" to have occurred. This is a common method in frequentist statistics for estimating parameters.

Expanding the Comparison: Related Concepts

To fully grasp the meaning of probability, it's helpful to contrast it with other related statistical terms.

Probability vs. Density

For continuous variables, the probability of the variable taking on any single, exact value is zero. Instead of assigning probabilities to specific points, we use a Probability Density Function (PDF).

• Probability is the area under the PDF curve across a specific range. It is a value between 0 and 1.
• Density is the value of the PDF at a specific point. It is not a probability and can be greater than 1. It represents the relative plausibility of values in that region.

Probability vs. Statistics

Probability and statistics are two sides of the same coin. They both deal with analyzing random events, but from opposite directions.

• Probability is a branch of mathematics that starts with a known model or parameters and predicts the probability of various outcomes. It is a forward-looking process (Model → Data). For example: "If a coin is fair, what is the chance of getting 8 heads in 10 tosses?"
• Statistics (specifically, inferential statistics) starts with observed data and works backward to make inferences about the model or parameters that generated it. It is a backward-looking process (Data → Model). For example: "If I tossed a coin 10 times and got 8 heads, how likely is it that the coin is fair?"

Probability vs. Possibility

The terms 'probability' and 'possibility' are often used loosely, but they have distinct technical meanings.

• Possibility is a binary concept: an event is either possible or impossible. It doesn't quantify uncertainty.
• Probability is a continuous measure, ranging from 0 to 1, that quantifies the uncertainty of possible events.

An event must be possible to have a non-zero probability. An impossible event always has a probability of 0. However, an event with a probability of 0 is not necessarily impossible, particularly in the context of continuous variables where any single point has zero probability, but is still part of the possible range of outcomes.

Chance vs. Probability vs. Proportion

These terms are closely related but serve different functions.

• Chance vs. Probability: In everyday language, 'chance' and 'probability' are used interchangeably. In a technical context, probability is the formal mathematical measure of uncertainty. 'Chance' is best thought of as the informal, layman's term for the same idea.
• Probability vs. Proportion: A proportion is a descriptive fact about a population (e.g., the actual proportion of water on Earth is ~71%). Probability is used to express our uncertainty about that proportion when it is unknown. In our globe-tossing example, P is the true, unknown proportion. The probability distribution over P represents our belief about what that proportion might be, which we update as we gather data.

Frequently Asked Questions

Conclusion

While probability and likelihood are deeply connected, they are not interchangeable. Probability quantifies uncertainty about outcomes or parameters, serving as the language of belief and prediction. Likelihood, in contrast, is a tool for inference, measuring how well our observed data supports different hypotheses about the parameters that generated it. Understanding this distinction—along with related concepts like density, proportion, and possibility—is fundamental to correctly interpreting statistical results and reasoning coherently in the face of uncertainty.

Sources & References

• Can somebody please say what Gibbsian statistical ...
• J. Stat. Mech. (2016) 064007 Generalized Gibbs ensemble in
• [1507.02993] Complete Generalized Gibbs Ensemble in an interacting Theory
• [1604.03990] Generalized Gibbs ensemble in integrable lattice models
• [1809.08268] Equilibration towards generalized Gibbs ensembles in non-interacting theories
• [1905.09501] Bayesian Item Response Modeling in R with brms and Stan
• Fast Kernel Summation in High Dimensions via Slicing and Fourier Transforms
• Thermodynamic Perspectives on Computational Complexity: Exploring the P vs. NP Problem
• Automated Kernel Discovery Towards Understanding High-dimensional Bayesian Optimization
• Condensed Matter