Dimensionality Reduction with PCA

Principal component analysis, usually shortened to PCA, is a dimensionality reduction method for replacing many correlated numeric features with a smaller number of new numeric features. In this lesson the outcome is concrete: given a feature matrix, you will center it, find the directions where the data varies most, project rows onto those directions, and decide how much information is lost when fewer directions are kept.

Within the unsupervised learning part of this machine learning course, PCA is useful because it discovers structure without labels. It is not a classifier and it does not know which variation is useful for a later prediction task. It only sees geometry: distances, variances, and correlations among columns. That makes PCA a powerful preprocessing and inspection tool, but also a method that must be matched carefully to the data and objective.

Purpose and Result

A dataset with dozens or thousands of columns can be hard to visualize, slow to model, and noisy when many columns describe almost the same underlying movement. PCA constructs a new coordinate system. The first principal component is the axis through the centered data with the largest possible variance. The second principal component is the highest-variance axis that is perpendicular to the first. Later components continue the same pattern.

After fitting PCA, each original row can be represented by component scores. If you keep all components, the representation is just a rotation of the centered data, with no information loss apart from numerical rounding. If you keep only the first few components, you compress the row. The discarded components define the reconstruction error: the part of each row that cannot be recovered from the compressed scores.

Mechanism Inside PCA

PCA begins with a matrix X whose rows are observations and columns are numeric features. The first operation is centering: subtract each column mean so every feature has mean zero. Centering matters because PCA looks for directions through the origin. Without centering, the first component can point toward the feature means instead of describing variation among observations.

The common covariance view then builds C = X_centered.T @ X_centered / (n - 1). The covariance matrix records how feature pairs move together. A large positive covariance means two features tend to rise together; a negative covariance means one tends to rise as the other falls. PCA performs an eigenvalue decomposition of this covariance matrix. Eigenvectors become component directions, and eigenvalues give the variance captured by each direction. Sorting eigenvalues from largest to smallest orders the components from most to least variance.

The singular value decomposition view reaches the same result more directly for many implementations. It factorizes the centered matrix into orthogonal row-space directions and singular values. The right singular vectors are principal axes, and squared singular values divided by n - 1 are the component variances. Libraries often use SVD because it is numerically stable and avoids explicitly materializing a huge covariance matrix when the feature count is large.

The fitted PCA object therefore contains column means, component vectors, explained variances, and explained variance ratios. Transforming new data repeats the same centering using the training means, then multiplies by the learned component matrix. It is a mistake to recompute means or components separately on validation, test, or production data, because that changes the coordinate system and makes scores incomparable.

API Anatomy

In Python libraries, the usual PCA API has a small set of ideas. fit(X) learns means and components. transform(X) returns component scores. fit_transform(X) does both for the training matrix. inverse_transform(scores) maps scores back into the original feature space, which is how reconstruction loss can be inspected.

The most important configuration is n_components. As an integer, it means keep exactly that many components. As a variance target in some APIs, it means choose the smallest number of components whose cumulative explained variance ratio reaches the target. Whitening is another option: it scales transformed components to unit variance. Whitening can help algorithms that assume isotropic inputs, but it also removes the natural variance scale that made the first components dominant.

Example 1: Centering and Covariance

This first example computes the quantities PCA starts from. The two features might be measurements that tend to rise together. The output shows nonzero covariance between the columns, which is the signal PCA will rotate into a more compact coordinate system.

import numpy as np

X = np.array([
    [2.5, 2.4], [0.5, 0.7], [2.2, 2.9], [1.9, 2.2], [3.1, 3.0],
    [2.3, 2.7], [2.0, 1.6], [1.0, 1.1], [1.5, 1.6], [1.1, 0.9],
])

means = X.mean(axis=0)
centered = X - means
covariance = np.cov(centered, rowvar=False, ddof=1)

print(np.round(means, 2))
print(np.round(covariance, 3))

The deterministic output is [1.81 1.91] for the feature means and a covariance matrix close to [[0.617 0.615] [0.615 0.717]]. Because the off-diagonal covariance is large and positive, one diagonal direction should explain most of the movement.

Example 2: Fitting Components

The second example performs the eigenvalue step explicitly. It sorts component directions by decreasing variance and fixes vector signs so the display is stable. Sign deserves attention: multiplying a component by -1 describes the same axis, so different software may print opposite signs while producing an equivalent PCA model.

import numpy as np

X = np.array([
    [2.5, 2.4], [0.5, 0.7], [2.2, 2.9], [1.9, 2.2], [3.1, 3.0],
    [2.3, 2.7], [2.0, 1.6], [1.0, 1.1], [1.5, 1.6], [1.1, 0.9],
])
centered = X - X.mean(axis=0)
covariance = np.cov(centered, rowvar=False, ddof=1)

values, vectors = np.linalg.eigh(covariance)
order = values.argsort()[::-1]
values = values[order]
vectors = vectors[:, order]
for column in range(vectors.shape[1]):
    if vectors[0, column] < 0:
        vectors[:, column] *= -1

ratios = values / values.sum()
scores = centered @ vectors

print(np.round(values, 3))
print(np.round(vectors, 3))
print(np.round(ratios, 3))
print(np.round(scores[:3], 3))

The eigenvalues are approximately [1.284 0.049], so the explained variance ratios are approximately [0.963 0.037]. The first component captures about 96.3 percent of total variance. The first three rows receive scores around [[0.828 0.175], [-1.778 -0.143], [0.992 -0.384]], which are coordinates in component space rather than original feature values.

Example 3: Compression and Reconstruction

The third example shows the trade-off directly. Keeping one component compresses two original features into one score per row. Reconstructing from that one score produces an approximation. Keeping both components reconstructs the original matrix exactly for this two-feature dataset, aside from floating-point rounding.

import numpy as np

X = np.array([
    [2.5, 2.4], [0.5, 0.7], [2.2, 2.9], [1.9, 2.2], [3.1, 3.0],
    [2.3, 2.7], [2.0, 1.6], [1.0, 1.1], [1.5, 1.6], [1.1, 0.9],
])
means = X.mean(axis=0)
centered = X - means
values, vectors = np.linalg.eigh(np.cov(centered, rowvar=False, ddof=1))
order = values.argsort()[::-1]
vectors = vectors[:, order]
for column in range(vectors.shape[1]):
    if vectors[0, column] < 0:
        vectors[:, column] *= -1
scores = centered @ vectors

for kept in (1, 2):
    reconstructed = scores[:, :kept] @ vectors[:, :kept].T + means
    mse = np.mean((X - reconstructed) ** 2)
    print(kept, round(float(mse), 6), np.round(reconstructed[0], 3))

The expected output is 1 0.022088 [2.371 2.519] and 2 0.0 [2.5 2.4]. The one-component reconstruction moves the first row onto the best one-dimensional line through the data. The two-component reconstruction restores the original point because no dimension was discarded.

Design Choices and Trade-offs

The first choice is scaling. PCA is variance based, so features measured in large units dominate unless you standardize. If one column is annual income and another is a zero-to-one ratio, raw PCA mostly explains income variance. Standardizing to zero mean and unit variance makes components describe correlation structure instead of raw measurement scale. That is often appropriate, but not always. If units are meaningful and comparable, scaling can overemphasize noisy small-range features.

The second choice is how many components to keep. A cumulative explained variance threshold such as 90 or 95 percent is a useful starting point, but it is not a guarantee of downstream model quality. Low-variance directions can contain rare but important information. Choose component count by combining explained variance, reconstruction error, visualization needs, and validation performance of the next model in the pipeline.

The third choice is interpretability. Principal components are linear combinations of original features. Loadings reveal which features contribute to each component, but a component can mix many signals with positive and negative weights. PCA may make a model faster while making explanations harder. When interpretability of original features is the goal, feature selection or domain-driven aggregation may be better than PCA.

Failure Modes and Troubleshooting

Symptom: one component explains nearly all variance, but the loading vector mostly points at a single high-magnitude column. Cause: the features were not scaled and one unit dominated the covariance matrix. Diagnose: inspect column standard deviations and component loadings before fitting the downstream model. Correction: standardize numeric columns inside a pipeline fitted only on training data.

Symptom: validation scores look excellent, but production scores shift sharply. Cause: PCA was fit separately on train and validation data, or refit on each batch in production. Diagnose: compare saved training means and components with those used during transform. Correction: persist the fitted PCA state and use exactly the same state for validation, test, and production transforms.

Symptom: transformed values contain nan or fitting fails with a linear algebra error. Cause: missing values, infinite values, constant columns, or an ill-conditioned matrix entered the decomposition. Diagnose: count missing and infinite values, inspect zero-variance columns, and reproduce the failure on a small slice. Correction: impute or remove missing values, drop constant columns, and use a numerically stable solver appropriate for the matrix shape.

Reliability and Performance

PCA can improve performance by reducing feature count before clustering, nearest-neighbor search, or linear modeling. It can also reduce storage for embeddings or sensor streams. The cost is fit-time linear algebra, memory for the matrix or covariance, and possible loss of rare signals. For very wide or very tall matrices, randomized or incremental PCA variants trade a small amount of exactness for lower memory use.

Security and privacy issues come from the data, not from PCA being unsupervised. Components can still encode sensitive information, and reconstructed values may reveal approximate original features. Treat fitted means, components, and scores as derived data with the same retention and access rules as the source dataset when the source contains personal or regulated fields.

Hands-on Lab

Prerequisites: Python with NumPy installed, a terminal, and a small working directory where you can run a script. Create a script named pca_lab.py and paste the code from Example 3. Run it with python3 pca_lab.py. Verify that the one-component mean squared reconstruction error is about 0.022088 and that the two-component error rounds to 0.0.

1. Change the loop to test only kept = 1 and print all reconstructed rows. Confirm that every reconstructed point lies on a single line in the original two-feature space.
2. Add a third column equal to the first column multiplied by 100. Run the script again and inspect the first component. The large-scale column should dominate unless you standardize.
3. Standardize each column by subtracting its mean and dividing by its standard deviation before computing covariance. Rerun the script and compare the explained variance pattern.
4. Rollback by removing the artificial third column and standardization changes, returning to the original two-column dataset.

The lab verifies the core behavior: PCA learns axes from centered training data, projection creates component scores, and inverse projection exposes the error introduced by discarded components.

Assessment Exercises

• A dataset has two features: height in centimeters and annual spending in dollars. Explain whether raw PCA or standardized PCA is more appropriate for customer segmentation, and defend the trade-off.
• You keep enough components to explain 95 percent of variance, but a downstream fraud detector gets worse. Give two plausible reasons and one experiment to distinguish them.
• Given component loadings with large positive weights on engine_size and vehicle_weight, and a large negative weight on fuel_efficiency, describe what a high score on that component means.
• Design a check that would catch accidental refitting of PCA on validation data before metrics are reported.

Summary

PCA reduces dimensionality by centering numeric data, finding orthogonal directions of maximum variance, and projecting rows onto the leading directions. Eigenvalues or singular values quantify how much variance each component captures, while reconstruction error shows what the compressed representation loses. Use PCA when correlated numeric features can be summarized linearly, but make deliberate choices about scaling, component count, interpretability, and persistence of the fitted transform.