Unsupervised Learning: Finding Structure Without Labels

Chapters 16 and 17 had an answer key. Someone had described the core, so every training row came with the porosity or the facies already known, and the model only had to learn the mapping. Most data does not arrive that way. You have logs from a hundred wells and a core description from three of them; you have a year of sensor readings and no one has marked which are real. Unsupervised learning is what you reach for when there is no answer key. It finds the structure already present in the data, with nobody telling it what to look for.

That structure has two faces, and both pay off in petroleum. The first is grouping: which logged feet belong together? Cluster them and you get electrofacies, data-driven rock types, for a well no geologist has yet sat with. The second is the mirror image: which feet belong to nothing? A reading that fits no group is usually not a rare rock; it is a washed-out borehole, a stuck tool, an electrical spike, bad data that will silently poison every calculation downstream. This chapter takes the exact logs from Chapter 17, throws the facies labels away, and asks both questions: can clustering recover the rock types on its own, and can it flag the readings that should never have been trusted?

The most direct unsupervised question is: do the logged samples fall into natural groups? K-means answers it by placing k cluster centres in log space and shuffling them until each sample sits with its nearest centre and the centres rest at the mean of their members. The catch, the one that makes this harder than supervised learning, is that you must supply k, the number of rock types, without being told it.

Two pieces of preparation matter first. The logs span very different ranges, so we standardise them (K-means measures straight-line distances, and an unscaled gamma ray in the hundreds would drown out a porosity near zero). And resistivity ranges over decades, so we cluster on its logarithm, the scale on which a petrophysicist actually reads it.

In a real project the swap is the same one as the last two chapters (lasio.read("well.las") in place of the generator) except that now you would not need the core description at all, which is the entire point: clustering runs on wells you could never afford to core.

Pick k too low and you blur real rock types together; too high and you split one rock into meaningless slivers. Two diagnostics guide the choice. Inertia, the total squared distance from each sample to its cluster centre, always falls as k rises, so you look for the elbow where it stops falling steeply. The silhouette score (−1 to 1) measures how much better each sample fits its own cluster than the next nearest; higher is tighter, better-separated grouping.

Neither diagnostic points to four. The silhouette is highest at k=2 (the data's strongest signal is simply shale versus reservoir) and the elbow is a soft bend, not a clean corner, which honestly tells you these rock types are not crisply separated. We will cluster at k=3 anyway: it is the most rock types the data will support as interpretable groups, and it lets us ask the sharper question of whether the gas pay survives.

Now reveal the labels we hid and score the clustering against ground truth. Plain accuracy is meaningless here: cluster "1" and facies "1" share a digit by accident, so any honest score must ignore how the integers were assigned. The adjusted Rand index (ARI) does exactly that: a label-agnostic agreement score from 0 to 1, adjusted so a random relabelling lands near 0 rather than at a flatteringly high number. We pair it with a contingency table of cluster against true facies.

An ARI around 0.62 says the clustering recovered the broad rock structure without ever seeing a label. The contingency table shows where it succeeded and where it could not: read the shale row and nearly every count falls in one column, while the gas-sand counts scatter across the two sandy clusters. One cluster is almost pure shale (over 90% of its members) cleanly separated because shale's high gamma ray and density set it apart. The other two clusters are both sand, and there lies the limit: the gas sand has no cluster of its own. Its samples scatter across the sandy clusters, mixed in with brine sand, because in log space gas sand simply is a sand with a slightly softer density. Clustering groups what looks alike, and gas sand looks like sand.

That is the defining lesson of unsupervised learning, and it is the exact inverse of Chapter 17: there, labels let you teach the model the faint gas signature; here, with no labels, that signature is not strong enough to separate on its own. A clustering surfaces the variation that dominates the data, not the rare distinction you most want; it shows you the structure that is there, not the structure you wish were there. Knowing that difference is what keeps you from over-trusting a tidy-looking cluster map.

Having measured the overlap in a table, it helps to see it, and the same tool pays off twice. Correlated logs are a liability for any downstream model: curves that mostly repeat each other add noise and computation without adding information. Principal component analysis (PCA) finds the few directions along which the data actually varies and projects onto them, compressing the logs with little lost. Dropping four dimensions to two is also exactly what lets us plot the whole dataset on a page.

Two components hold about 82% of the variation: a four-log space flattened to a page with little lost. The picture makes the earlier numbers visible: shale sits apart, the clusters follow the genuine grouping, and the gas sand (red, right panel) stays smeared through the brine sand rather than standing on its own. Those two components are also the compression promised above: feeding them into a downstream model in place of the raw, correlated curves cuts noise and computation with almost nothing given up.

Clustering asks what belongs together. Flip the question (what belongs to nothing?) and you have anomaly detection, the unsupervised tool that may matter most day to day. A reading that sits far from every cluster is rarely an exciting new rock; it is almost always bad data, and bad data is corrosive. A washed-out borehole drops the density log, a stuck tool flatlines a curve, an electrical spike throws resistivity from a few ohm-m to thousands, and every one of those feeds, unflagged, straight into your porosity, your saturation, your reserves. Catching them by hand across thousands of feet is hopeless; an isolation forest does it automatically, learning what normal looks like and flagging the points that are easiest to isolate from the crowd.

The detector catches about 95% of the injected bad readings with no false alarms, and it never needed a rule for "washout" or "spike"; it learned the shape of normal and flagged the rest. The contamination setting is the one judgement call: it is your estimate of how dirty the data is, and it sets how aggressively the forest flags. One honest limit closes the loop with the clustering result: an isolation forest catches readings of the wrong magnitude, but a stuck tool that flatlines a curve at a perfectly normal value is a contextual anomaly (every point looks fine on its own) and slips through. Those need a different check, a rolling variance that notices a log has stopped moving. No single detector catches every failure, which is exactly why you look at the flags, not blindly trust them.

You have now seen both faces of learning without labels: clustering, which groups what belongs together and recovers rock structure but not the rare pay, and anomaly detection, which isolates what belongs to nothing and turns thousands of feet of logs into a short list a petrophysicist can actually review. Together they are how you start on data nobody has had time to interpret, which, in this industry, is most of it.

• Unsupervised learning finds structure with no answer key, the right tool when you have logs from many wells but core from few, which is the normal state of affairs.
• K-means groups logs into electrofacies, but you must choose k without being told it; the elbow and silhouette guide the choice, and a soft elbow honestly means the rock types are not crisply separated.
• Clustering recovers the structure that dominates the variance, not the rare distinction you care about. Here shale separated cleanly while the gas pay dissolved into the sand: the headline limit of learning without labels, and the reason labels mattered in Chapter 17.
• PCA compresses correlated logs into a readable picture (and a cleaner model input), holding most of the variation in two dimensions.
• Anomaly detection flags what fits nothing: automated log QC that catches washouts, spikes, and out-of-range readings without a rule for each, while contextual failures like a flatline still need a complementary variance check.

This closes Part IV. You have run the full fundamental ML loop (regression, classification, and clustering) always on the same petroleum data and always with the engineering judgement that decides what a model's output is worth. Part V puts these tools to work end to end, from a raw dataset to a deployed application.