Review: Multivariate Probability
This module covered how to handle multiple random variables simultaneously, from their joint behavior to their individual marginals and the structures that connect them (conditional independence).
Key Takeaways
- Curse of Dimensionality: A full joint probability table grows exponentially (KN). We must use factorization (Conditional Independence) to make storage feasible.
- Marginalization: Recovering the distribution of a single variable P(X) by summing/integrating out others from the Joint P(X, Y).
- Naive Bayes: An algorithm that assumes conditional independence to reduce model complexity from exponential to linear.
- Covariance Matrix: A symmetric, positive semi-definite matrix that summarizes the spread and direction of linear relationships between variables.
- Hardware Reality: Matrix memory layout (Row-major vs Column-major) critically impacts the performance of covariance calculations due to CPU cache locality.
1. Interactive Flashcards
Test your understanding of the core concepts.
2. Cheat Sheet
| Concept | Formula / Definition |
|---|---|
| Joint Probability | P(X, Y) |
| Marginal Probability | P(X) = Σy P(X, Y) |
| Conditional Probability | P(X | Y) = P(X, Y) / P(Y) |
| Independence | P(X, Y) = P(X)P(Y) |
| Cond. Independence | P(X, Y | Z) = P(X | Z)P(Y | Z) |
| Expectation (Linearity) | E[aX + bY] = aE[X] + bE[Y] |
| Covariance | Cov(X, Y) = E[(X - μX)(Y - μY)] |
| Correlation | ρXY = Cov(X, Y) / (σX σY) |
| Variance Sum Law | Var(X + Y) = Var(X) + Var(Y) + 2Cov(X, Y) |
Matrix Operations
| Operation | Result |
|---|---|
| Covariance Matrix | Σ = E[(X - μ)(X - μ)T] |
| Transformation Y = AX | ΣY = A ΣX AT |
| Mahalanobis Distance | d2 = (x - μ)T Σ-1 (x - μ) |
3. Quick Revision
- Joint Probability P(X, Y): The entire probability space describing two variables together. Curse of dimensionality makes large joint tables unfeasible.
- Marginal Probability P(X): The distribution of one variable, found by summing out the other variable from the joint distribution.
- Conditional Independence (X ⊥ Y | Z): Knowing Z fully explains any apparent correlation between X and Y. Critical for Naive Bayes.
- Covariance Matrix: A symmetric, positive semi-definite matrix storing variances on the diagonal and covariances off-diagonal. It describes the shape of multivariate distributions.
Next Steps
Explore Bayesian Probability in the next module: Bayesian.