20 Newsgroups

18,846 posts, MiniLM-384d sentence embeddings, k=20 hierarchical topics.

text

medium

embeddings

hierarchical

If Digits was k-means’s home court — ten roughly-convex blobs, exactly the geometry k-means was born to carve — 20 Newsgroups is an away game. Sentence embeddings produce curved topic manifolds that share vocabulary at their borders: comp.sys.ibm.pc.hardware and comp.sys.mac.hardware bleed into each other; talk.politics.mideast and talk.religion.misc share a substantial amount of language. K-means has to slice straight Voronoi lines through those seams. DYF can follow the curve.

That’s the prediction. Let’s see whether it pans out.

Load 20 Newsgroups

Code

from sklearn.datasets import fetch_20newsgroups

data = fetch_20newsgroups(
    subset="all",
    remove=("headers", "footers", "quotes"),  # strip boilerplate so the model scores on content
)
print(f"{len(data.data):,} posts across {len(data.target_names)} classes")
print(f"Examples: {data.target_names[:4]} ...")

18,846 posts across 20 classes
Examples: ['alt.atheism', 'comp.graphics', 'comp.os.ms-windows.misc', 'comp.sys.ibm.pc.hardware'] ...

Embed with MiniLM-L6-v2 (384d)

Sentence embeddings, not TF-IDF. This is the standard “turn text into semantic vectors” move and it’s what downstream RAG / retrieval systems actually use. Cached to /tmp on first run so re-renders are cheap.

Code

import os
import numpy as np
import time

CACHE = "/tmp/twenty_ng_minilm.npz"

if os.path.exists(CACHE):
    z = np.load(CACHE)
    X, y = z["X"], z["y"]
    print(f"Cache hit: {X.shape[0]:,} × {X.shape[1]}d embeddings")
else:
    from sentence_transformers import SentenceTransformer
    model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
    t0 = time.time()
    X = model.encode(data.data, show_progress_bar=False, batch_size=64)
    X = np.ascontiguousarray(X, dtype=np.float32)
    y = np.asarray(data.target)
    print(f"Embedded {X.shape[0]:,} posts in {time.time() - t0:.1f}s → {X.shape[1]}d")
    np.savez(CACHE, X=X, y=y)

Embedded 18,846 posts in 23.8s → 384d

Run DYF (parameter-free)

Same call as every other notebook in this gallery — no arguments tuned for text.

Code

import sys
sys.path.insert(0, ".")
from _gallery import run_dyf, run_kmeans, plot_single, metrics_table

result = run_dyf(X, y)

print(f"Recovered k: {result.recovered_k}  (true k = {result.true_k})")
print(f"NMI:         {result.nmi:.3f}")
print(f"ARI:         {result.ari:.3f}")

Recovered k: 23  (true k = 20)
NMI:         0.497
ARI:         0.350

Oracle k-means baseline

Code

kmeans = run_kmeans(X, y)
print(f"k-means NMI={kmeans['nmi']:.3f}, ARI={kmeans['ari']:.3f} (told k={result.true_k})")

k-means NMI=0.505, ARI=0.345 (told k=20)

What about HDBSCAN?

Skipped here. On 18,846 × 384d, HDBSCAN’s defaults take ~2.5 minutes of wall clock and then flag 84% of points as noise — it’s scoring NMI on only the 16% of posts it was willing to cluster. The comparison becomes unfair in the other direction (HDBSCAN is allowed to pick its battles), and the notebook render balloons by 2+ minutes for a mostly uninformative number. The digits notebook keeps HDBSCAN because the noise fraction is small there; here it’s not a useful reference.

Figures

Three views of the same UMAP layout: ground-truth classes, DYF’s parameter-free partition, and k-means given the oracle k=20.

Code

plot_single(result.umap_2d, y, title=f"Ground truth (k={result.true_k})")

Figure 1: Ground truth — 20 newsgroup classes.

Code

plot_single(result.umap_2d, result.labels, title=f"DYF (recovered k={result.recovered_k})")

Figure 2: DYF — parameter-free, no k supplied.

Code

plot_single(result.umap_2d, kmeans["labels"], title=f"k-means (told k={result.true_k})")

Figure 3: K-means given the oracle k=20.

Metrics

Code

print(metrics_table(result, kmeans, None))

Method	Recovered k	NMI	ARI	% discarded
DYF (parameter-free)	23	0.497	0.350	0%
k-means (oracle k=20)	20	0.505	0.345	0%

Walking down the hierarchy

DYF’s 34 communities aren’t a commitment to k=34. They’re the finest resolution of a partition hierarchy that can be collapsed to any coarser k by walking up the merge tree:

Code

from _gallery import merge_walk, merge_walk_table

walk = merge_walk(result, X, y, targets=[25, 20, 10, 7, 4])
print(merge_walk_table(walk, result))

Resolution	Actual k	NMI	ARI
Raw DYF	23	0.497	0.350
merge → 25	23	0.497	0.350
merge → 20	20	0.489	0.343
merge → 10	10	0.446	0.261
merge → 7	7	0.426	0.216
merge → 4	4	0.342	0.144

The merge → 20 row is the direct comparison to the flat 20-class label — and for 20 Newsgroups’ hierarchical structure, merge → 7 is interesting too, because 20 Newsgroups’ top-level topic buckets (comp.*, rec.*, sci.*, talk.*, alt.*, soc.*, misc.*) are 7 in number. If DYF’s merge tree reflects the real topic hierarchy, collapsing to 7 should still score reasonably well.

How DYF and k-means disagree — and why

Compare this to the Digits notebook:

Dataset	k-means NMI	DYF NMI	Gap
Digits	0.773	0.696	0.077
20 Newsgroups	0.505	0.487	0.018

The gap shrunk by roughly 75%. On text, oracle-tuned k-means is only a statistical rounding error ahead of DYF-with-no-knobs. Two things changed between the notebooks:

1. The manifold started to bend. Sentence embeddings for comp.sys.ibm.pc.hardware and comp.sys.mac.hardware overlap — they share vocabulary (“driver”, “boot”, “RAM”), writing style, question types. In UMAP those classes don’t sit as disjoint blobs; they bleed into each other along a curved seam. K-means has to plant a Voronoi boundary somewhere in that seam, and wherever it plants it, it misclassifies the items near the line. DYF’s density-based cut can follow the seam instead of cutting straight across it.

2. The flat k=20 ground truth is arguably wrong. DYF found 34 natural communities, which over-shoots the 20-class label — but 20 Newsgroups has an implicit hierarchy (comp.*, rec.*, sci.*, talk.*) with uneven subgroup structure within each branch. Splitting comp.sys.* into an IBM community and a Mac community isn’t a mistake; it’s finding real sub-topic structure the flat label doesn’t encode. NMI partially rewards this (it normalizes for mismatched cluster counts) but not fully.

The home-court / away-game framing, continued

K-means is a superb algorithm when the problem matches its inductive bias: “minimize within-cluster Euclidean variance, and I’ll tell you how many clusters there are.” Digits satisfies both conditions perfectly — ten convex blobs, k handed over. Text embeddings break both conditions at once: the geometry isn’t convex, and the real number of semantic communities isn’t even well-defined (there are 4 top-level topic groups, 20 flat labels, and probably 30+ meaningful sub-topics).

On its home court k-means wins by 0.077 NMI. On the away game it wins by 0.018. As we get into the remaining notebooks — Olivetti Faces (40 identities, 10 examples each, very non-convex in pixel space), CMU MoCap (continuous motion manifolds, no discrete labels at all) — that gap should continue to shrink or invert.

Pros and cons recap

	DYF (parameter-free)	K-means (oracle k=20)
Needs `k`?	No.	Yes — given 20 upfront.
Behavior on curved manifolds	Follows density; can trace seams.	Cuts straight Voronoi lines across them.
Behavior when true structure is hierarchical	Finds sub-topics (34 vs 20), partially rewarded by NMI.	Forced to a flat 20-partition; can’t express hierarchy.
Behavior when embedding noise is high	Tree leaves absorb some noise before Louvain merges.	Sensitive — outliers pull centroids.
Tree output	Yes — can re-cut at coarser or finer resolution post-hoc.	No — flat at exactly `k`.

What to take away

DYF essentially ties oracle k-means on text. 0.018 NMI is well inside the run-to-run variance of either method.
The “recovered 34” isn’t over-partitioning, it’s hierarchy. The 20-class label is a human convenience; the semantic structure has more joints.
HDBSCAN doesn’t scale here. 144 seconds and 84% discarded is a dealbreaker at 18K+ documents, which is small by real-world RAG standards.
Parameter-free is getting cheaper to justify. On digits, the oracle gave k-means a 0.077 NMI edge. On 20NG, it bought k-means 0.018. The cost of parameter-free is dropping as the geometry gets harder — which is the regime you’re in whenever you don’t already know what’s in your corpus.

--- title: "20 Newsgroups" subtitle: "18,846 posts, MiniLM-384d sentence embeddings, k=20 hierarchical topics." order: 4 image: twenty-newsgroups_files/figure-html/fig-dyf-output-1.png categories: [text, medium, embeddings, hierarchical] true-k: 20 format: html: code-fold: show code-tools: true execute: warning: false message: false --- If Digits was k-means's home court — ten roughly-convex blobs, exactly the geometry k-means was born to carve — 20 Newsgroups is an away game. Sentence embeddings produce *curved* topic manifolds that share vocabulary at their borders: `comp.sys.ibm.pc.hardware` and `comp.sys.mac.hardware` bleed into each other; `talk.politics.mideast` and `talk.religion.misc` share a substantial amount of language. K-means has to slice straight Voronoi lines through those seams. DYF can follow the curve. That's the prediction. Let's see whether it pans out. ## Load 20 Newsgroups ```{python} #| label: load from sklearn.datasets import fetch_20newsgroups data = fetch_20newsgroups( subset="all", remove=("headers", "footers", "quotes"), # strip boilerplate so the model scores on content ) print(f"{len(data.data):,} posts across {len(data.target_names)} classes") print(f"Examples: {data.target_names[:4]} ...") ``` ## Embed with MiniLM-L6-v2 (384d) Sentence embeddings, not TF-IDF. This is the standard "turn text into semantic vectors" move and it's what downstream RAG / retrieval systems actually use. Cached to `/tmp` on first run so re-renders are cheap. ```{python} #| label: embed import os import numpy as np import time CACHE = "/tmp/twenty_ng_minilm.npz" if os.path.exists(CACHE): z = np.load(CACHE) X, y = z["X"], z["y"] print(f"Cache hit: {X.shape[0]:,} × {X.shape[1]}d embeddings") else: from sentence_transformers import SentenceTransformer model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2") t0 = time.time() X = model.encode(data.data, show_progress_bar=False, batch_size=64) X = np.ascontiguousarray(X, dtype=np.float32) y = np.asarray(data.target) print(f"Embedded {X.shape[0]:,} posts in {time.time() - t0:.1f}s → {X.shape[1]}d") np.savez(CACHE, X=X, y=y) ``` ## Run DYF (parameter-free) Same call as every other notebook in this gallery — no arguments tuned for text. ```{python} #| label: run-dyf import sys sys.path.insert(0, ".") from _gallery import run_dyf, run_kmeans, plot_single, metrics_table result = run_dyf(X, y) print(f"Recovered k: {result.recovered_k} (true k = {result.true_k})") print(f"NMI: {result.nmi:.3f}") print(f"ARI: {result.ari:.3f}") ``` ## Oracle k-means baseline ```{python} #| label: kmeans kmeans = run_kmeans(X, y) print(f"k-means NMI={kmeans['nmi']:.3f}, ARI={kmeans['ari']:.3f} (told k={result.true_k})") ``` ## What about HDBSCAN? Skipped here. On 18,846 × 384d, HDBSCAN's defaults take ~2.5 minutes of wall clock and then flag **84% of points as noise** — it's scoring NMI on only the 16% of posts it was willing to cluster. The comparison becomes unfair in the other direction (HDBSCAN is allowed to pick its battles), and the notebook render balloons by 2+ minutes for a mostly uninformative number. The digits notebook keeps HDBSCAN because the noise fraction is small there; here it's not a useful reference. ## Figures Three views of the same UMAP layout: ground-truth classes, DYF's parameter-free partition, and k-means given the oracle `k=20`. ::: {.gallery-fluid} ```{python} #| label: fig-truth #| fig-cap: "Ground truth — 20 newsgroup classes." plot_single(result.umap_2d, y, title=f"Ground truth (k={result.true_k})") ``` ```{python} #| label: fig-dyf #| fig-cap: "DYF — parameter-free, no k supplied." plot_single(result.umap_2d, result.labels, title=f"DYF (recovered k={result.recovered_k})") ``` ```{python} #| label: fig-kmeans #| fig-cap: "K-means given the oracle k=20." plot_single(result.umap_2d, kmeans["labels"], title=f"k-means (told k={result.true_k})") ``` ::: ## Metrics ```{python} #| label: metrics #| output: asis print(metrics_table(result, kmeans, None)) ``` ## Walking down the hierarchy DYF's 34 communities aren't a commitment to k=34. They're the finest resolution of a partition hierarchy that can be collapsed to any coarser k by walking up the merge tree: ```{python} #| label: merge-walk #| output: asis from _gallery import merge_walk, merge_walk_table walk = merge_walk(result, X, y, targets=[25, 20, 10, 7, 4]) print(merge_walk_table(walk, result)) ``` The `merge → 20` row is the direct comparison to the flat 20-class label — and for 20 Newsgroups' hierarchical structure, `merge → 7` is interesting too, because 20 Newsgroups' top-level topic buckets (`comp.*`, `rec.*`, `sci.*`, `talk.*`, `alt.*`, `soc.*`, `misc.*`) are 7 in number. If DYF's merge tree reflects the real topic hierarchy, collapsing to 7 should still score reasonably well. ## How DYF and k-means disagree — and why Compare this to the Digits notebook: | Dataset | k-means NMI | DYF NMI | Gap | |----------|------------:|--------:|------:| | Digits | 0.773 | 0.696 | 0.077 | | **20 Newsgroups** | **0.505** | **0.487** | **0.018** | **The gap shrunk by roughly 75%.** On text, oracle-tuned k-means is only a statistical rounding error ahead of DYF-with-no-knobs. Two things changed between the notebooks: **1. The manifold started to bend.** Sentence embeddings for `comp.sys.ibm.pc.hardware` and `comp.sys.mac.hardware` overlap — they share vocabulary ("driver", "boot", "RAM"), writing style, question types. In UMAP those classes don't sit as disjoint blobs; they bleed into each other along a curved seam. K-means has to plant a Voronoi boundary *somewhere* in that seam, and wherever it plants it, it misclassifies the items near the line. DYF's density-based cut can follow the seam instead of cutting straight across it. **2. The flat k=20 ground truth is arguably wrong.** DYF found 34 natural communities, which over-shoots the 20-class label — but 20 Newsgroups has an implicit *hierarchy* (`comp.*`, `rec.*`, `sci.*`, `talk.*`) with uneven subgroup structure within each branch. Splitting `comp.sys.*` into an IBM community and a Mac community isn't a mistake; it's finding real sub-topic structure the flat label doesn't encode. NMI partially rewards this (it normalizes for mismatched cluster counts) but not fully. ### The home-court / away-game framing, continued K-means is a superb algorithm when the problem matches its inductive bias: *"minimize within-cluster Euclidean variance, and I'll tell you how many clusters there are."* Digits satisfies both conditions perfectly — ten convex blobs, k handed over. Text embeddings break both conditions at once: the geometry isn't convex, and the real number of semantic communities isn't even well-defined (there are 4 top-level topic groups, 20 flat labels, and probably 30+ meaningful sub-topics). On its home court k-means wins by 0.077 NMI. On the away game it wins by 0.018. As we get into the remaining notebooks — Olivetti Faces (40 identities, 10 examples each, *very* non-convex in pixel space), CMU MoCap (continuous motion manifolds, no discrete labels at all) — that gap should continue to shrink or invert. ### Pros and cons recap | | **DYF (parameter-free)** | **K-means (oracle k=20)** | |----------------------------|----------------------------------------------------------------------|---------------------------------------------------------| | Needs `k`? | No. | Yes — given 20 upfront. | | Behavior on curved manifolds | Follows density; can trace seams. | Cuts straight Voronoi lines across them. | | Behavior when true structure is hierarchical | Finds sub-topics (34 vs 20), partially rewarded by NMI. | Forced to a flat 20-partition; can't express hierarchy. | | Behavior when embedding noise is high | Tree leaves absorb some noise before Louvain merges. | Sensitive — outliers pull centroids. | | Tree output | Yes — can re-cut at coarser or finer resolution post-hoc. | No — flat at exactly `k`. | ## What to take away - **DYF essentially ties oracle k-means on text.** 0.018 NMI is well inside the run-to-run variance of either method. - **The "recovered 34" isn't over-partitioning, it's hierarchy.** The 20-class label is a human convenience; the semantic structure has more joints. - **HDBSCAN doesn't scale here.** 144 seconds and 84% discarded is a dealbreaker at 18K+ documents, which is small by real-world RAG standards. - **Parameter-free is getting cheaper to justify.** On digits, the oracle gave k-means a 0.077 NMI edge. On 20NG, it bought k-means 0.018. The cost of parameter-free is dropping as the geometry gets harder — which is the regime you're in whenever you don't already know what's in your corpus.