Clustering with K-Means

K-Means is an unsupervised learning algorithm for splitting numeric observations into k compact groups. It does not know the true labels. Instead, it searches for cluster centers, called centroids, so that each point is close to the centroid of its assigned cluster.

The practical outcome is a repeatable way to summarize structure in unlabeled data: customer segments, image colors, document embeddings, sensor states, or groups of similar products. In the Unsupervised Learning section, this matters because the algorithm can reveal candidate patterns, but it cannot prove those patterns are meaningful. You must still inspect features, scale, stability, and domain consequences.

How K-Means Works Internally

K-Means minimizes within-cluster squared distance. For each observation, the algorithm computes the distance to every centroid, assigns the observation to the nearest one, then recomputes each centroid as the mean of the observations assigned to it. These two phases repeat until assignments stop changing, centroid movement becomes tiny, or an iteration limit is reached.

The most common distance is Euclidean distance. In two dimensions, it measures straight-line separation. In many dimensions, it still sums squared feature differences, so a feature measured in large units can dominate a feature measured in small units. A salary column can overwhelm an age column unless the data is scaled.

The objective value is called inertia in many libraries: the sum of squared distances from each point to its assigned centroid. Lower inertia means points are closer to their centroids, but it always improves as k increases. With one cluster per point, inertia can approach zero, but the model is useless as a summary. That is why choosing k is a modeling decision, not a mechanical search for the smallest number.

API Anatomy

A K-Means call has four important inputs. First, n_clusters is the desired number of clusters. Second, initialization controls the starting centroids; poor starts can lead to weaker local optima. Third, the stopping rule controls how long the assignment-update loop runs. Fourth, a random seed makes experiments reproducible.

In scikit-learn, fit learns centroids from a matrix of numeric features. predict assigns new points to the nearest learned centroid. labels_ contains the training assignments, cluster_centers_ contains centroid coordinates, and inertia_ reports the final objective. A typical workflow combines scaling and clustering in a pipeline so the same transformation is applied during training and prediction.

Example 1: Manual Assignment

This small example shows the assignment step without any library. Three points are compared against two fixed centroids. The deterministic output shows which centroid is nearest.

from math import dist

points = [(1.0, 1.0), (1.5, 2.0), (8.0, 8.0)]
centroids = [(1.0, 1.0), (9.0, 9.0)]

assignments = []
for point in points:
    distances = [dist(point, center) for center in centroids]
    assignments.append(distances.index(min(distances)))

print(assignments)

The first two points are nearer to centroid 0. The point near (8, 8) is nearer to centroid 1. K-Means repeats this assignment for every point on every iteration.

Example 2: One Update Step

After assignment, K-Means moves each centroid to the mean of its assigned points. This is why centroids can have coordinates that do not match any real observation.

points = [(1.0, 1.0), (1.5, 2.0), (8.0, 8.0)]
assignments = [0, 0, 1]

new_centroids = []
for cluster_id in sorted(set(assignments)):
    members = [p for p, a in zip(points, assignments) if a == cluster_id]
    x_mean = sum(p[0] for p in members) / len(members)
    y_mean = sum(p[1] for p in members) / len(members)
    new_centroids.append((x_mean, y_mean))

print(new_centroids)

The updated first centroid becomes the average of (1.0, 1.0) and (1.5, 2.0), which is (1.25, 1.5). The second centroid stays at (8.0, 8.0) because it has one member.

Example 3: Scaling Changes Clusters

K-Means is scale-sensitive. In this example, annual spend is much larger than visit count, so unscaled distances mostly reflect spending. Scaling puts both features on comparable footing.

from sklearn.cluster import KMeans
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X = [
    [2, 100],
    [3, 120],
    [20, 130],
    [22, 125],
    [4, 900],
    [5, 920],
]

plain = KMeans(n_clusters=2, random_state=7, n_init=10).fit(X)
scaled = make_pipeline(
    StandardScaler(),
    KMeans(n_clusters=2, random_state=7, n_init=10),
).fit(X)

print(list(plain.labels_))
print(list(scaled.named_steps["kmeans"].labels_))

The exact labels may be swapped because cluster numbers are arbitrary. The important behavior is that the unscaled run tends to isolate high-spend rows, while the scaled run can also account for visit-count separation. When interpreting results, compare cluster profiles rather than trusting numeric label names.

Choosing K

The cluster count controls compression. Small k gives broad summaries; large k gives detailed groups that may be unstable. The elbow method plots inertia against k and looks for a point where improvement slows. Silhouette score compares how close points are to their own cluster versus the nearest other cluster. Neither method understands business meaning. A customer segmentation with a beautiful score is still weak if the segments cannot support different decisions.

Good practice is to test several values of k, inspect centroid profiles, measure stability across random seeds or samples, and ask whether each cluster leads to a distinct action. In unsupervised learning, validation often combines statistics with human interpretation.

Design Choices and Trade-Offs

K-Means is fast, simple, and easy to explain. Its centroids are compact summaries, and assignment for new data is cheap. The trade-off is that it prefers round, similarly sized clusters under Euclidean distance. It struggles with curved shapes, uneven densities, heavy outliers, and categorical variables that have been encoded carelessly.

Initialization is another trade-off. Random starts are simple but can converge to a poor local solution. K-Means++ spreads initial centroids more deliberately and usually improves results. Running multiple initializations reduces risk but costs more compute. Feature selection is equally important: adding irrelevant dimensions can make distances less meaningful, especially when many weak features dilute the signal.

Failure Modes and Troubleshooting

Symptom: one feature appears to define every cluster. Cause: the feature has a much larger numeric scale. Diagnosis: print feature ranges and centroid coordinates. Correction: standardize or otherwise transform features before fitting.

Symptom: results change noticeably between runs. Cause: random initialization finds different local optima. Diagnosis: compare inertia and cluster profiles across seeds. Correction: set a random seed, use multiple initializations, and report stability.

Symptom: a cluster contains only a few extreme points. Cause: outliers pull centroids or deserve separate handling. Diagnosis: inspect distances to centroids and high-leverage rows. Correction: cap, transform, remove, or separately model outliers when justified.

Symptom: clusters look statistically clean but are useless. Cause: the features do not represent the decision you need to make. Diagnosis: profile clusters against external outcomes or expert labels not used for fitting. Correction: revise features or choose a different unsupervised method.

Performance and Reliability

K-Means scales well because each iteration mainly computes distances between n points and k centroids. Cost grows with rows, features, clusters, and iterations. For large datasets, mini-batch K-Means estimates updates from smaller batches and can be much faster, with some loss of precision.

Reliability depends on reproducible preprocessing. The scaler, selected features, centroid array, random seed, and training data snapshot should be tracked together. If new production data drifts far from training centroids, assignment may still produce a label, but the label can be misleading. Monitor distance-to-centroid distributions and retrain when they shift materially.

Hands-On Lab

Prerequisites: Python with scikit-learn installed, plus a terminal where you can run a short script. The lab fits K-Means on the Iris feature matrix, compares several values of k, and verifies that the pipeline can assign a new observation.

1. Create a file named kmeans_lab.py.
2. Add the script below.
3. Run python kmeans_lab.py.
4. Verify that inertia decreases as k increases and that the prediction is a single cluster id.
5. Cleanup by deleting the file when finished.

from sklearn.cluster import KMeans
from sklearn.datasets import load_iris
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, _ = load_iris(return_X_y=True)

for k in [2, 3, 4]:
    model = make_pipeline(
        StandardScaler(),
        KMeans(n_clusters=k, random_state=11, n_init=10),
    )
    model.fit(X)
    kmeans = model.named_steps["kmeans"]
    print(f"k={k} inertia={kmeans.inertia_:.2f}")

final_model = make_pipeline(
    StandardScaler(),
    KMeans(n_clusters=3, random_state=11, n_init=10),
)
final_model.fit(X)
print(final_model.predict([[5.1, 3.5, 1.4, 0.2]]).tolist())

Verification is not that k=3 must be perfect. The point is to confirm the mechanics: preprocessing occurs before clustering, inertia is available after fitting, and new observations are assigned by the trained centroids. If the script fails with an import error, install scikit-learn in the active environment. If prediction fails due to feature count, check that the new observation has exactly four values.

Assessment Exercises

• You have age, income, and purchase count. Which feature is most likely to dominate unscaled K-Means, and how would you confirm it?
• Why can inertia alone make a larger k look better even when the clustering is less useful?
• A run with k=5 produces two nearly identical centroid profiles. What might you change before presenting the result?
• Design a stability check that compares cluster profiles across three random seeds.
• Explain why cluster labels 0 and 1 should not be interpreted as ranked categories.

Summary

K-Means alternates between nearest-centroid assignment and centroid recomputation until it reaches a stable local solution. Its strengths are speed, simplicity, and interpretable center points. Its main risks are scale sensitivity, arbitrary cluster labels, local optima, outliers, and overinterpreting patterns without domain validation. Use it when compact distance-based groups match the problem, and treat evaluation as both a numerical and domain-specific exercise.