PCA: Reducing Dimensions Without Guessing

Principal component analysis (PCA) replaces many correlated coordinates with a smaller set of directions that preserve as much variance as possible. It is a projection, not a discovery that the data was “really” two-dimensional.

Center first

Let the data points be $x_1,\ldots,x_n\in\mathbb{R}^d$. Compute their mean

and center each point: $z_i=x_i-\mu$. Centering makes PCA describe variation around the data cloud rather than distance from the origin.

For a unit direction $v$, the scalar projection of $z_i$ is $z_i\cdot v$. PCA chooses the direction with the largest projected variance:

The next direction maximizes the remaining variance while being perpendicular to $v_1$. The process continues until there are $d$ principal directions.

Covariance and eigenvectors

Collect the centered rows into a matrix $Z$. The covariance matrix is proportional to

The principal directions are eigenvectors of $C$. Their eigenvalues $\lambda_1\geq\lambda_2\geq\cdots\geq\lambda_d$ measure variance along those directions.

If the first two eigenvalues are $7$ and $2$, while the remaining eigenvalues sum to $1$, then a two-dimensional projection explains

or 90% of the total variance.

A geometric example

Imagine points close to the line $y=2x$. The original $x$ and $y$ coordinates both change, but much of that change describes the same movement along the diagonal. The first principal component points roughly along $(1,2)$. The second points across the thin width of the cloud.

Projecting onto the first component compresses each two-coordinate point to one number. This preserves its position along the main trend but discards its small deviation from the line.

What PCA does not promise

• High variance is not always the information relevant to a label.
• Principal components are linear combinations, so curved structure may need many of them.
• A low-dimensional plot can change distances and neighborhoods.
• Feature scaling changes the covariance matrix and therefore the result.
• A component is a direction, not automatically a meaningful human concept.

Suppose one feature is measured in dollars and another in proportions. Without standardization, the dollar feature may dominate simply because of its units. Standardize when comparable relative variation is intended; keep original scale when absolute magnitude is meaningful.

Reconstruction error

After keeping $r$ principal components, a point can be projected and reconstructed. The difference from the original point measures lost information. PCA chooses the rank-$r$ linear projection with the smallest total squared reconstruction error.

That gives a concrete evaluation: choose the number of components based on explained variance, reconstruction error, and—most importantly—performance on the downstream task. A compact representation is useful only if it preserves what the next step needs.