Probability Meaning: Uncertainty, Likelihood & Updating

Probability is a measure of uncertainty. In the Bayesian framework, it quantifies the plausibility of different parameter values, which is updated from a prior belief to a posterior belief using evidence from data, encapsulated by the likelihood. This process allows for a formal, mathematical way to learn from experience and revise our understanding of the world.

Subjective vs. Objective Interpretations of Probability

The meaning of probability itself is a subject of philosophical debate, leading to different statistical frameworks. The two most prominent are the Bayesian (subjective) and frequentist (objective) interpretations, which assign probability to different things.

In the Bayesian view, probability is a subjective measure of belief or plausibility about a parameter. It quantifies your uncertainty about quantities like the true proportion of water on a globe. This uncertainty is represented by a probability distribution, which can be updated as you collect data. The prior probability represents your initial belief, and the posterior probability represents your updated belief after seeing evidence.

The frequentist interpretation, in contrast, typically treats parameters as fixed, unknown constants. Probability is seen as the long-run frequency of an event over many repeated trials. In this view, you don't assign a probability distribution to a parameter. Instead, you use likelihood—the probability of the observed data given a specific parameter value—to make inferences. While Bayesians use likelihood as one component to update their beliefs about a parameter, frequentists often use it more directly to find the parameter value that makes the data most probable.

Probability vs. Likelihood: Distinct Roles in Bayesian Inference

While interpretations differ, the Bayesian framework gives probability and likelihood distinct, practical roles. Understanding this distinction is fundamental to avoiding common misconceptions.

Probability: Quantifying Uncertainty

In Bayesian inference, probability directly addresses the question, "How uncertain am I about the parameter?". For a parameter like the proportion of water (P), a probability distribution indicates the plausibility of different P values.

Key aspects of probability:

• Prior Probability: Encodes initial beliefs about parameters before any data is observed.
• Posterior Probability: Represents the updated distribution of beliefs about parameters after combining prior knowledge with observed data. It is the new probability distribution over the parameter.
• Normalization: A posterior distribution must sum or integrate to 1, ensuring it represents a valid probability distribution.

Likelihood: Measuring Data Plausibility

Likelihood, in contrast, answers, "Given a parameter value, how plausible is the observed data?". It quantifies the probability of observing the data under a fixed parameter value within a generative model.

Key aspects of likelihood:

• Data and Model Dependent: Likelihood depends on the specific generative model and the data that has been observed.
• Ranking Parameter Values: Likelihood ranks parameter values for fixed data, but it should not be interpreted as the "probability of the parameter".
• Generative Model: A generative model is essential for likelihood to have meaning, as it provides the story of how data is generated from parameters. Without this story, likelihood is undefined.

Comparison of Probability and Likelihood

Bayesian Updating: Combining Prior and Likelihood

Bayesian updating is the mechanism by which prior beliefs are reshaped by evidence from data. This process follows the formula: posterior ∝ prior × likelihood.

The Mechanism of Updating

1. Generative Model: A generative model is established, which describes how data is produced from parameters. For example, in the globe-tossing example, the model specifies how likely observing water or land is for each possible proportion of water (P).
2. Prior Distribution: An initial probability distribution over the parameter (P) is defined, representing prior beliefs.
3. Likelihood Calculation: For each possible parameter value, the likelihood of observing the actual data is calculated based on the generative model. For instance, if observing 'W' water and 'L' land, the likelihood for a particular P is proportional to P^W * (1-P)^L.
4. Pointwise Multiplication: The prior probability of P is multiplied by the likelihood of the observed data under P. This step reshapes uncertainty, making parameter values that poorly explain the data less plausible.
5. Normalization: The resulting product (unnormalized posterior scores) is then normalized so that the posterior distribution sums or integrates to 1. This normalized posterior distribution represents the updated plausibility of different parameter values given the model and the observed evidence.

Impact of Data on Posterior

• Small Dataset: With limited data, the likelihood remains broad, leading to a posterior that still reflects substantial uncertainty.
• More Data: As more data is collected, the likelihood sharpens, causing the posterior to concentrate around parameter values that best explain the data.
• Inconsistent Data: If the data contradicts a parameter value (e.g., observing land when P=1), that parameter value will have zero likelihood, resulting in a near-zero probability in the posterior.

Common Pitfalls in Applying Bayesian Updating

While powerful, the Bayesian updating process is susceptible to several common mistakes that can lead to flawed conclusions.

Confusing Probability and Likelihood

A primary pitfall is confusing the roles of probability and likelihood. Likelihood measures how well a specific parameter value explains the data; it is not a probability distribution over the parameter. Treating the likelihood function as if it were the posterior distribution is a significant error because it completely ignores the influence of prior beliefs. The posterior is the product of the prior and the likelihood, properly normalized.

Using an Incorrect Generative Model

The generative model provides the "story" for how data is created from parameters, which is what makes the likelihood meaningful. If this model is wrong, the inference will be misleading. For example, if your model assumes the proportion of water on a globe (P) can be 1 (all water), but you observe a "land" outcome, the likelihood for P=1 becomes zero. This correctly forces the posterior probability for P=1 to also be zero. However, if the generative model is fundamentally flawed and cannot produce the patterns seen in the data, the resulting posteriors can have misleading zero regions or be confidently concentrated on incorrect parameter values.

Errors in Posterior Predictive Checks

Posterior predictive checks are used to validate a model by seeing if it can generate data that looks like the real data. A frequent error is to perform these checks using only the posterior means of the parameters (a "point plug-in" approach). The correct method involves a two-step simulation:

1. Draw a full set of parameter values (θ) from the entire posterior distribution p(θ|y).
2. Simulate a new, replicated dataset (y_rep) from the likelihood, using the drawn parameter values: y_rep ~ p(y|θ).

Repeating this process thousands of times builds up the posterior predictive distribution p(y_rep|y), which properly accounts for all parameter uncertainty. Using only point estimates ignores this uncertainty and can make a model appear to fit better than it actually does.

Generative Modeling: The Foundation of Likelihood

Generative modeling is crucial because it provides the "story" of how data is generated from parameters, making the concept of likelihood meaningful. Without a generative model, likelihood cannot be defined.

Components of a Generative Model

A generative model specifies P(data | θ), where θ represents parameters like P, regression coefficients, or latent class rates. It defines:

1. Likelihood Structure: This describes the noise or randomness that corrupts observations.
2. Mapping from Parameters to Observations: This details how outcomes depend on the parameters (θ).

This choice directly influences the shape of the likelihood and, consequently, which regions of the posterior distribution receive high weight. A well-constructed generative model encodes the believed data-generating process. If the generative model poorly matches reality, the resulting posterior can be confidently wrong.

Example: Item Response Theory (IRT)

Item Response Theory (IRT) is a Bayesian generative model used for test data. It explains response data by assuming each person has a latent ability and each item has parameters describing its behavior across that ability.

In a Bayesian IRT setup (e.g., 2PL), the model:

• Draws person ability values and item parameters.
• Generates each observed response using a probabilistic rule, often a logistic function of ability and item parameters.

This generative structure allows for:

• Predicting new responses.
• Quantifying uncertainty in ability estimates.
• Adding hierarchical priors to share statistical strength across items or persons.

Frequently Asked Questions

Conclusion

Probability, in the context of statistical inference, is fundamentally about quantifying uncertainty. The Bayesian interpretation treats probability as a measure of plausibility about parameters, which can be formally updated in light of new evidence. This stands in contrast to the frequentist view, which sees probability as a long-run frequency. In the Bayesian framework, probability is distinct from likelihood, which assesses the plausibility of observed data given specific parameter values and a generative model. The process of Bayesian updating systematically combines prior probabilities with likelihoods to produce a posterior probability distribution, thereby reshaping initial beliefs. Understanding this mechanism, building sound generative models, and avoiding common pitfalls are essential for robust statistical reasoning and the honest quantification of uncertainty.

Sources & References

• [1905.09501] Bayesian Item Response Modeling in R with brms and Stan
• [2106.01963] A Survey on Optimal Transport for Machine Learning: Theory and Applications
• [2202.00821] Optimizing Sequential Experimental Design with Deep Reinforcement Learning
• [2505.06589] Optimal Transport for Machine Learners
• A Survey on Algorithmic Developments in Optimal Transport Problem with Applications
• Optimal Transport for Machine Learners Course notes
• Improving Protein Sequence Design through Designability Preference Optimization
• Optimal and Diffusion Transports in Machine Learning
• An econometrician’s guide to optimal transport
• Bayesian Item Response Modeling in R with brms and Stan