PCA (Covariance)
Finds directions of maximum variance via eigenvectors of the covariance matrix.
Formula
Variables
Example
Top eigenvectors are the principal components
How it works
Principal Component Analysis can be done by forming the covariance matrix of mean-centered data, C = (1/n)X^T X, then finding its eigenvectors and eigenvalues. The eigenvector equation C v = λ v identifies directions in feature space where the data spread is maximized; the associated eigenvalue tells you how much variance lies along that direction. This works because covariance measures how features vary together, and the eigenvectors of a symmetric covariance matrix give the orthogonal axes that diagonalize that variation.
When to use it
Use this when you want to reduce dimensionality while preserving as much variance as possible, such as compressing high-dimensional sensor data or visualizing clustered data in 2D/3D. It is also useful before training models on correlated features, where rotating into principal-component space can simplify structure and remove redundancy.
Worked example, step by step
Suppose we have 3 centered observations in 2D: X = [[2, 0], [0, 2], [-2, -2]] Step 1: Compute C = (1/3)X^T X. X^T X = [[2,0,-2],[0,2,-2]] · [[2,0], [0,2], [-2,-2]] = [[8, 4], [4, 8]] So C = (1/3)[[8, 4], [4, 8]] = [[8/3, 4/3], [4/3, 8/3]]. Step 2: Solve C v = λ v by finding eigenvalues. For [[8/3, 4/3], [4/3, 8/3]], the eigenvalues are λ1 = 4 and λ2 = 4/3. Step 3: Find eigenvectors. For λ1 = 4, an eigenvector is v1 = [1, 1]/√2. For λ2 = 4/3, an eigenvector is v2 = [1, -1]/√2. Step 4: Interpret the result. The first principal component is along the line x = y, and it explains variance 4. The second principal component is orthogonal to it, along x = -y, and explains variance 4/3. Since 4 > 4/3, the first direction captures the most spread in the data.
Common mistakes to avoid
- Forgetting to center the data before computing X^T X, which changes the covariance directions
- Using X X^T instead of X^T X and then interpreting the wrong space as feature space
- Assuming the largest eigenvalue gives the most important original feature rather than the most important rotated direction
Did You Know?
PCA compresses data by keeping only the directions that carry the most variance.
Share this formula
More in Machine Learning & AI
View allLinear Regression Model
BasicPredicts a continuous value as a weighted sum of input features plus a bias.
Gradient Descent Update
BasicIteratively moves parameters in the direction that most reduces the loss.
ReLU Activation
BasicRectified Linear Unit: outputs the input if positive, else zero.
Leaky ReLU
BasicA ReLU variant that lets a small gradient flow for negative inputs.
Tanh Activation
BasicSquashes input to the range (-1, 1); zero-centred activation.
Softmax
BasicTurns a vector of scores into a probability distribution that sums to 1.