PCA, visually.
Three questions, answered with pictures rather than algebra: what does “maximum variance along PC1” actually mean, why are the principal components guaranteed to be orthogonal, and what does each dot in a PC1 vs PC2 scatter plot tell you about a data sample?
What “maximum variance along PC1” means
Take a 2-feature cloud (the same idea holds in 100 dimensions). A “direction” is a line through the middle of the data. Every sample can be projected onto that line — it collapses to a single number, its score. PCA asks: which direction makes those projected scores spread out the most? Drag the slider and watch.
One direction is better than another when the projected samples spread out more on it — because then the scores genuinely distinguish the samples from each other. The same direction is also the line that passes closest to all the points (least perpendicular error). Maximizing variance kept and minimizing information thrown away are two views of the same optimum, which is why keeping the top PCs is the least-distorting compression of the data.
Orthogonality is built in, not luck
Once PC1 is found, PC2 is defined as the variance-maximizing direction subject to being uncorrelated with PC1 — and “uncorrelated scores” means the weight vectors are perpendicular. The right angle you see below is guaranteed by the eigendecomposition of the covariance matrix.
The covariance matrix of the data, and what PCA computes from it:
1 · Symmetry. The covariance matrix C is symmetric (cov of feature A with B = cov of B with A). The spectral theorem says every real symmetric matrix has a full set of mutually orthogonal eigenvectors. PCA just takes those eigenvectors, ordered by eigenvalue.
2 · Constraint. PC2 maximizes variance subject to w₂·w₁ = 0. Zero dot product is the definition of perpendicular. It’s not that orthogonality emerges — it’s imposed at every step, then the Lagrange-multiplier math hands you back an eigenvector.
3 · SVD. Libraries actually compute X = UΣVᵞ; the columns of V are the PCs, and SVD produces orthonormal columns by construction.
If two eigenvalues are exactly equal (e.g. isotropic, noise-like data), the PCs inside that subspace are not unique — any rotation of them explains the same variance. You can still always choose them orthogonal, but which particular pair a library returns is arbitrary. Well-separated eigenvalues → unique, stable PCs.
Same dots, new coordinates
PCA is a rotation, not a selection: no samples are dropped. Each dot on the left is one sample with coordinates (x₁, x₂); the same sample on the right has coordinates (PC1, PC2) — its projections onto the two new axes. Hover or click any dot.
It is one sample (row) of your data — the same patient / image / customer as before — summarized by two numbers instead of 100: where it sits along the dataset’s biggest pattern (PC1) and its second-biggest, independent pattern (PC2). Dots that cluster together are samples whose 100 features behaved similarly; a far-out dot is an unusual sample.
Your actual case: 100 features → 2 PCs
Below: a synthetic dataset of 150 samples × 100 features, standardized, then reduced to 2 PCs — exactly the pipeline you described. The samples come from three hidden groups (shown by color). Note that PCA never saw those labels.
Reading the plot — and what PC1 is made of
PC1 is a linear combination of the original features with learned weights (the loadings). For the dataset above, the six features with the largest PC1 weights:
PC1 ≈
The 2D plot faithfully preserves only what PC1+PC2 capture (32% here). Two samples far apart on screen are genuinely different; but two samples that look close might differ on PC3+.
Variance is unit-dependent. Without standardization, one feature measured on a large scale can dominate PC1 all by itself — and you’ve merely rediscovered that feature, not a pattern.
Nothing guarantees PC1/PC2 align with your class labels or outcome. Sometimes the structure you care about lives on PC5 — or in none of them. Check the scree plot before trusting the picture.
- Each dot = one sample (row) of the original data — nothing is averaged or dropped.
- Dot position = that sample’s scores on the two highest-variance patterns of all 100 features.
- Clusters = groups of samples that behave similarly across the features; outliers = atypical samples.
- PCs = orthogonal linear combinations of the features; PC1 maximizes projected variance, each next PC maximizes what’s left under the orthogonality constraint.
- Loadings = the per-feature weights that tell you which features drive each axis.