MNIST

70,000 handwritten digits, 28×28 pixel vectors, k=10 ground truth.

images

large

pixels

The scaled-up version of Digits. Same domain, same task, same k=10 — but 40× more samples and 12× higher pixel dimensionality. Does DYF’s parameter-free story survive the jump in scale, or does it fall apart under a heavier load?

Short answer, as we’ll see: DYF inverts the digits result on one axis and keeps losing on the other. It’s not the clean extension of the Digits story — it’s a twist.

Load MNIST

Code

import os
import numpy as np

CACHE = "/tmp/gallery_mnist.npz"
if os.path.exists(CACHE):
    z = np.load(CACHE)
    X, y = z["X"], z["y"]
else:
    from sklearn.datasets import fetch_openml
    m = fetch_openml("mnist_784", version=1, as_frame=False, parser="auto")
    X = (m.data.astype(np.float32) / 255.0)   # normalize pixels to [0, 1]
    y = m.target.astype(int)
    np.savez(CACHE, X=X, y=y)

print(f"{X.shape[0]:,} samples, {X.shape[1]}d, true k = {len(set(y))}")

70,000 samples, 784d, true k = 10

Run DYF

Code

import sys
sys.path.insert(0, ".")
from _gallery import run_dyf_cached, run_kmeans, plot_single, metrics_table

result = run_dyf_cached(X, y, cache_path="/tmp/gallery_mnist_dyf.npz")
print(f"Recovered k: {result.recovered_k}  (true k = {result.true_k})")
print(f"NMI:         {result.nmi:.3f}")
print(f"ARI:         {result.ari:.3f}")

Recovered k: 86  (true k = 10)
NMI:         0.546
ARI:         0.183

Oracle k-means

Code

kmeans = run_kmeans(X, y)
print(f"k-means NMI={kmeans['nmi']:.3f}, ARI={kmeans['ari']:.3f} (told k={result.true_k})")

k-means NMI=0.491, ARI=0.361 (told k=10)

Figures

Code

plot_single(result.umap_2d, y, title=f"Ground truth (k={result.true_k})")

Figure 1: Ground truth — each of the 10 digit classes.

Code

plot_single(result.umap_2d, result.labels, title=f"DYF (recovered k={result.recovered_k})")

Figure 2: DYF — parameter-free, no k supplied.

Code

plot_single(result.umap_2d, kmeans["labels"], title=f"k-means (told k={result.true_k})")

Figure 3: K-means given the oracle k=10.

Metrics

Code

print(metrics_table(result, kmeans, None))

Method	Recovered k	NMI	ARI	% discarded
DYF (parameter-free)	86	0.546	0.183	0%
k-means (oracle k=10)	10	0.491	0.361	0%

Deep dive: what are DYF’s 71 extra clusters?

Every other notebook in this gallery tells you DYF’s over-partitioning finds sub-style structure. This section shows it.

First, the statistical audit

If DYF’s 81 clusters are genuine sub-structure, they should be pure — each cluster dominated by one digit class. If they’re noise, they should be mixed bags.

Code

import numpy as np
from collections import Counter

labels = result.labels
rows = []
for cid in sorted(set(labels)):
    mask = labels == cid
    n = int(mask.sum())
    if n == 0:
        continue
    vals, cnts = np.unique(y[mask], return_counts=True)
    rows.append({"cid": cid, "n": n, "dominant": int(vals[np.argmax(cnts)]),
                 "purity": cnts.max() / n})

r80 = sum(1 for r in rows if r["purity"] > 0.80)
r90 = sum(1 for r in rows if r["purity"] > 0.90)
r95 = sum(1 for r in rows if r["purity"] > 0.95)
weighted = sum(r["purity"] * r["n"] for r in rows) / sum(r["n"] for r in rows)

digit_counts = Counter(r["dominant"] for r in rows)
per_digit = ", ".join(f"{d}×{n}" for d, n in sorted(digit_counts.items()))

print(f"**{len(rows)} DYF clusters. Weighted purity: {weighted:.3f}.**\n")
print(f"- {r80}/{len(rows)} clusters are >80% pure (dominated by one digit)\n")
print(f"- {r90}/{len(rows)} are >90% pure\n")
print(f"- {r95}/{len(rows)} are >95% pure\n")
print(f"- Sub-clusters per digit: {per_digit}\n")

86 DYF clusters. Weighted purity: 0.850.

61/86 clusters are >80% pure (dominated by one digit)
49/86 are >90% pure
25/86 are >95% pure
Sub-clusters per digit: 0×5, 1×5, 2×15, 3×14, 4×8, 5×11, 6×9, 7×4, 8×9, 9×6

Each DYF cluster has a clear “home” digit. That’s the first sign the partition is meaningful — 88% of points land in a cluster dominated by their true class. Now the visual test.

Sub-clusters of the digit 7

DYF split the digit 7 into multiple sub-clusters. Here are the largest of the high-purity ones, rendered as random samples from each:

Code

# Find all DYF clusters where digit 7 is dominant, sorted by cluster size
seven_subs = sorted(
    [r for r in rows if r["dominant"] == 7 and r["purity"] > 0.9],
    key=lambda r: -r["n"],
)
print(f"DYF produced {len(seven_subs)} high-purity (>90%) sub-cluster(s) of digit 7; showing up to the 3 largest:\n")
for s in seven_subs[:3]:
    print(f"- cluster {s['cid']}: {s['n']} samples, {s['purity']:.0%} are 7s")

DYF produced 1 high-purity (>90%) sub-cluster(s) of digit 7; showing up to the 3 largest:

cluster 0: 2325 samples, 98% are 7s

Code

from _gallery import image_grid
if len(seven_subs) >= 1:
    idx = np.where(result.labels == seven_subs[0]["cid"])[0]
    image_grid(X, idx, title=f"DYF cluster {seven_subs[0]['cid']} (n={seven_subs[0]['n']})")

Figure 4

Code

if len(seven_subs) >= 2:
    idx = np.where(result.labels == seven_subs[1]["cid"])[0]
    image_grid(X, idx, title=f"DYF cluster {seven_subs[1]['cid']} (n={seven_subs[1]['n']})")

Figure 5

Code

if len(seven_subs) >= 3:
    idx = np.where(result.labels == seven_subs[2]["cid"])[0]
    image_grid(X, idx, title=f"DYF cluster {seven_subs[2]['cid']} (n={seven_subs[2]['n']})")

Figure 6

Look at the three grids. Each is visibly a distinct handwriting style — stroke weight, slant, proportion, top-hook shape. These aren’t random slices of a single 7-population; they’re stylistic sub-communities that the flat “is a 7” label has no way to express.

The same digit, from k-means

K-means was told k=10, so it gave us exactly ten clusters. Here’s a random sample from the cluster that contains the most 7s:

Code

# Find the k-means cluster that is majority 7
km_labels = kmeans["labels"]
best_cid, best_count = -1, 0
for cid in np.unique(km_labels):
    cnt = int(((km_labels == cid) & (y == 7)).sum())
    if cnt > best_count:
        best_count, best_cid = cnt, int(cid)
idx = np.where(km_labels == best_cid)[0]
image_grid(X, idx, title=f"k-means cluster {best_cid} (n={len(idx)}, {best_count} are 7s)")

Figure 7: The k-means cluster that contains the most 7s — one mixed bag across styles.

Look closely. This isn’t a bag of 7s — it’s a mixed bag of 7s, 4s, 9s, and 5s sharing a cluster because they all have similar diagonal stroke patterns in pixel space. Only ~43% of this k-means cluster is actually 7s. The other 57% are other digits that k-means merged in because committing to k=10 forced it to draw boundaries somewhere, and “things with a prominent diagonal stroke” was the cleanest Voronoi cut available at that resolution.

DYF’s three sub-clusters of 7 are each ~97% pure 7s at different styles. K-means’s “7 cluster” is a 43%-pure pile of visually-similar digits. The same k=10 budget, spent completely differently: DYF used its allowance to separate digit classes cleanly at the cost of over-splitting each class into styles; k-means used its to hit exactly 10 cluster centers at the cost of merging different digits that share pixel patterns.

Why this matters

If your downstream task is classification (given a new image, predict 0–9), the flat 10-way view is what you want, and k-means with the oracle is the right tool.

If your downstream task is retrieval (given a user’s crossbar-7 handwriting, find visually similar samples), you want DYF’s sub-style partition. Serving back a mix of crossbar and non-crossbar 7s is a worse result than serving back same-style examples, even though both are “correct” at the digit level.

If your downstream task is exploration (what’s in this corpus?), you want the whole hierarchy. DYF’s tree lets you start at 10 clusters for the overview, then zoom to 81 for the sub-style structure, without refitting. K-means would need ten separate runs at different k values and no guarantee the partitions would nest.

This is what the gallery means by “DYF gives you the whole hierarchy, pick the resolution your task wants.” MNIST shows it literally.

Walking down the hierarchy

DYF didn’t just give you 81 clusters. It gave you a partition hierarchy — and every coarser resolution sits one step up the tree. Here’s what happens when we walk DYF’s 81 clusters down to coarser granularities using merge_to_max_k:

Code

from _gallery import merge_walk, merge_walk_table

walk = merge_walk(result, X, y, targets=[40, 20, 10, 5])
print(merge_walk_table(walk, result))

Resolution	Actual k	NMI	ARI
Raw DYF	86	0.546	0.183
merge → 40	40	0.556	0.273
merge → 20	20	0.549	0.347
merge → 10	10	0.465	0.325
merge → 5	5	0.396	0.282

Read the ARI column. Merging from 81 down to 20 barely moves NMI (−0.010) but nearly doubles ARI — because the 61 extra clusters in the raw partition were genuine sub-communities, and collapsing them recovers the coarser structure the label encodes. Merging further (to 10) gives you ARI close to k-means’s — without the oracle having been consulted.

This is the move the “parameter-free” pitch is really making: DYF doesn’t pick one k, it gives you the whole tree, and you slice it at whatever granularity your downstream task wants. K-means and HDBSCAN both commit to a single partition. DYF commits to a ranked hierarchy.

What happened to the Digits story?

Against the Digits notebook, the story flips on NMI but breaks on ARI:

Dataset	DYF k	k-means NMI	DYF NMI	DYF NMI − KMeans NMI	DYF ARI	KMeans ARI
Digits	10	0.773	0.696	−0.077	0.614	0.726
MNIST	81	0.491	0.552	+0.061	0.175	0.361

DYF now beats oracle k-means on NMI by 0.061 — but loses on ARI by 0.186. How do you lose ARI while winning NMI?

The over-partitioning question

DYF found 81 natural communities on MNIST, not 10. Three possibilities:

(a) It’s finding real sub-style structure. There are multiple stylistic variants of every digit — slanted 7s vs. crossbar 7s, loopy 4s vs. open 4s, continental 1s vs. American 1s. If DYF’s 81 communities each map cleanly to one ground-truth digit (just at a finer granularity than the flat 10-label), then DYF is right and the label is impoverished. This is the “MNIST has more than 10 classes if you look closely” reading.

(b) It’s fragmenting classes. The same digit class could be split across many DYF communities for arbitrary geometric reasons (LSH hyperplane placement, UMAP projection quirks) rather than semantic ones. This would be the “DYF over-partitioned” reading.

(c) Some mix. Some communities are genuine sub-styles; others are artifacts.

NMI and ARI disagree because they treat over-partitioning differently:

NMI (normalized mutual information) — normalizes for cluster counts. Getting “all the 7s into one pile vs two piles of slanted/crossbar 7s” both score well, as long as each pile stays pure.
ARI (adjusted Rand index) — penalizes any pair of points that ends up in different clusters when the ground truth puts them together. Splitting the 7s into two piles costs you pairs, regardless of how pure those piles are.

The NMI-win-but-ARI-loss signature is the fingerprint of clean over-partitioning — clusters that are individually pure but collectively too many. That’s reading (a) above, with some (c).

Why did this happen at 70K but not at 1,797?

Density. At 1,797 samples, there are only ~180 points per digit class, and DYF’s tree leaves stay coarse enough that Louvain merges them into ~10 natural communities matching the flat label. At 70K samples, there are 7,000 points per digit class, and DYF’s tree now has enough room to carve out stable sub-leaves per style variant. Louvain finds those sub-communities and reports them.

This is DYF doing what it’s designed to do — surface structure the data supports. The fact that the label is flat and the data isn’t is a mismatch between the benchmark and reality.

What would you actually want in a real application?

Depends on the downstream task:

If you’re doing classification (given a new image, predict 0–9): you want k=10 and ARI>NMI. K-means is better here, provided you know k.
If you’re doing retrieval (given a new image, find similar ones): you want sub-style structure preserved. Two crossbar 7s are more similar to each other than a crossbar 7 and a slanted 7. DYF is better here.
If you’re doing data exploration (what’s in this corpus?): you definitely want the sub-style structure, and you probably want the hierarchical tree that DYF produces as a bonus.

The gallery’s “flat-NMI-against-flat-label” scoring is the classification rubric. DYF is closer to a retrieval / exploration rubric, which is what a parameter-free tool is for.

The sports analogy update

Digits was k-means’s home court — convex blobs, small k, oracle-provided. It won, cleanly.

MNIST is a longer game. K-means held the lead most of the way by sticking to the rigid 10-cluster playbook. DYF kept finding extra joints in the structure, and the NMI scoreboard eventually flipped. But ARI — a different metric with a different rulebook — still favors the disciplined game plan.

It’s no longer clear which method is “winning.” It depends on which rulebook you hand the judges. That ambiguity is the point.

Caveats

DYF’s 81 clusters include some artifacts, almost certainly. Reading (c) is the honest one. A rigorous analysis would compute a confusion matrix between DYF clusters and the 10 digit labels, check whether each DYF cluster is dominantly one class (style-variant reading) or scattered (fragmentation reading), and report the breakdown. The gallery skips that to keep notebooks readable; a supplement could add it.
UMAP is a projection, not the cluster space. When DYF’s and k-means’s partitions look visually different in the UMAP panel, some of that is true clustering difference and some is UMAP’s 2D compression of decisions made in 784d. Don’t read the UMAP too literally.
Cache is in /tmp. Dumped on reboot, which is fine for the gallery — real applications should use a durable cache.

--- title: "MNIST" subtitle: "70,000 handwritten digits, 28×28 pixel vectors, k=10 ground truth." order: 2 image: mnist_files/figure-html/fig-dyf-output-1.png categories: [images, large, pixels] true-k: 10 format: html: code-fold: show code-tools: true execute: warning: false message: false --- The scaled-up version of [Digits](digits.qmd). Same domain, same task, same `k=10` — but 40× more samples and 12× higher pixel dimensionality. Does DYF's parameter-free story survive the jump in scale, or does it fall apart under a heavier load? Short answer, as we'll see: DYF *inverts* the digits result on one axis and keeps losing on the other. It's not the clean extension of the Digits story — it's a twist. ## Load MNIST ```{python} #| label: load import os import numpy as np CACHE = "/tmp/gallery_mnist.npz" if os.path.exists(CACHE): z = np.load(CACHE) X, y = z["X"], z["y"] else: from sklearn.datasets import fetch_openml m = fetch_openml("mnist_784", version=1, as_frame=False, parser="auto") X = (m.data.astype(np.float32) / 255.0) # normalize pixels to [0, 1] y = m.target.astype(int) np.savez(CACHE, X=X, y=y) print(f"{X.shape[0]:,} samples, {X.shape[1]}d, true k = {len(set(y))}") ``` ## Run DYF ```{python} #| label: run-dyf import sys sys.path.insert(0, ".") from _gallery import run_dyf_cached, run_kmeans, plot_single, metrics_table result = run_dyf_cached(X, y, cache_path="/tmp/gallery_mnist_dyf.npz") print(f"Recovered k: {result.recovered_k} (true k = {result.true_k})") print(f"NMI: {result.nmi:.3f}") print(f"ARI: {result.ari:.3f}") ``` ## Oracle k-means ```{python} #| label: kmeans kmeans = run_kmeans(X, y) print(f"k-means NMI={kmeans['nmi']:.3f}, ARI={kmeans['ari']:.3f} (told k={result.true_k})") ``` ## Figures ::: {.gallery-fluid} ```{python} #| label: fig-truth #| fig-cap: "Ground truth — each of the 10 digit classes." plot_single(result.umap_2d, y, title=f"Ground truth (k={result.true_k})") ``` ```{python} #| label: fig-dyf #| fig-cap: "DYF — parameter-free, no k supplied." plot_single(result.umap_2d, result.labels, title=f"DYF (recovered k={result.recovered_k})") ``` ```{python} #| label: fig-kmeans #| fig-cap: "K-means given the oracle k=10." plot_single(result.umap_2d, kmeans["labels"], title=f"k-means (told k={result.true_k})") ``` ::: ## Metrics ```{python} #| label: metrics #| output: asis print(metrics_table(result, kmeans, None)) ``` ## Deep dive: what are DYF's 71 extra clusters? Every other notebook in this gallery *tells* you DYF's over-partitioning finds sub-style structure. This section shows it. ### First, the statistical audit If DYF's 81 clusters are genuine sub-structure, they should be *pure* — each cluster dominated by one digit class. If they're noise, they should be mixed bags. ```{python} #| label: purity-audit #| output: asis import numpy as np from collections import Counter labels = result.labels rows = [] for cid in sorted(set(labels)): mask = labels == cid n = int(mask.sum()) if n == 0: continue vals, cnts = np.unique(y[mask], return_counts=True) rows.append({"cid": cid, "n": n, "dominant": int(vals[np.argmax(cnts)]), "purity": cnts.max() / n}) r80 = sum(1 for r in rows if r["purity"] > 0.80) r90 = sum(1 for r in rows if r["purity"] > 0.90) r95 = sum(1 for r in rows if r["purity"] > 0.95) weighted = sum(r["purity"] * r["n"] for r in rows) / sum(r["n"] for r in rows) digit_counts = Counter(r["dominant"] for r in rows) per_digit = ", ".join(f"{d}×{n}" for d, n in sorted(digit_counts.items())) print(f"**{len(rows)} DYF clusters. Weighted purity: {weighted:.3f}.**\n") print(f"- {r80}/{len(rows)} clusters are >80% pure (dominated by one digit)\n") print(f"- {r90}/{len(rows)} are >90% pure\n") print(f"- {r95}/{len(rows)} are >95% pure\n") print(f"- Sub-clusters per digit: {per_digit}\n") ``` Each DYF cluster has a clear "home" digit. That's the first sign the partition is meaningful — 88% of points land in a cluster dominated by their true class. Now the visual test. ### Sub-clusters of the digit 7 DYF split the digit 7 into multiple sub-clusters. Here are the largest of the high-purity ones, rendered as random samples from each: ```{python} #| label: seven-subs #| output: asis # Find all DYF clusters where digit 7 is dominant, sorted by cluster size seven_subs = sorted( [r for r in rows if r["dominant"] == 7 and r["purity"] > 0.9], key=lambda r: -r["n"], ) print(f"DYF produced {len(seven_subs)} high-purity (>90%) sub-cluster(s) of digit 7; showing up to the 3 largest:\n") for s in seven_subs[:3]: print(f"- cluster {s['cid']}: {s['n']} samples, {s['purity']:.0%} are 7s") ``` ::: {.gallery-fluid} ```{python} #| label: fig-seven-a #| fig-cap: "DYF cluster A — random sample of 7s." from _gallery import image_grid if len(seven_subs) >= 1: idx = np.where(result.labels == seven_subs[0]["cid"])[0] image_grid(X, idx, title=f"DYF cluster {seven_subs[0]['cid']} (n={seven_subs[0]['n']})") ``` ```{python} #| label: fig-seven-b #| fig-cap: "DYF cluster B — random sample of 7s." if len(seven_subs) >= 2: idx = np.where(result.labels == seven_subs[1]["cid"])[0] image_grid(X, idx, title=f"DYF cluster {seven_subs[1]['cid']} (n={seven_subs[1]['n']})") ``` ```{python} #| label: fig-seven-c #| fig-cap: "DYF cluster C — random sample of 7s." if len(seven_subs) >= 3: idx = np.where(result.labels == seven_subs[2]["cid"])[0] image_grid(X, idx, title=f"DYF cluster {seven_subs[2]['cid']} (n={seven_subs[2]['n']})") ``` ::: Look at the three grids. Each is *visibly* a distinct handwriting style — stroke weight, slant, proportion, top-hook shape. These aren't random slices of a single 7-population; they're stylistic sub-communities that the flat "is a 7" label has no way to express. ### The same digit, from k-means K-means was told `k=10`, so it gave us exactly ten clusters. Here's a random sample from the cluster that contains the most 7s: ```{python} #| label: fig-kmeans-sevens #| fig-cap: "The k-means cluster that contains the most 7s — one mixed bag across styles." # Find the k-means cluster that is majority 7 km_labels = kmeans["labels"] best_cid, best_count = -1, 0 for cid in np.unique(km_labels): cnt = int(((km_labels == cid) & (y == 7)).sum()) if cnt > best_count: best_count, best_cid = cnt, int(cid) idx = np.where(km_labels == best_cid)[0] image_grid(X, idx, title=f"k-means cluster {best_cid} (n={len(idx)}, {best_count} are 7s)") ``` Look closely. This isn't a bag of 7s — it's a mixed bag of **7s, 4s, 9s, and 5s** sharing a cluster because they all have similar diagonal stroke patterns in pixel space. Only ~43% of this k-means cluster is actually 7s. The other 57% are *other digits* that k-means merged in because committing to `k=10` forced it to draw boundaries somewhere, and "things with a prominent diagonal stroke" was the cleanest Voronoi cut available at that resolution. DYF's three sub-clusters of 7 are each ~97% pure 7s at different styles. K-means's "7 cluster" is a 43%-pure pile of visually-similar digits. **The same k=10 budget, spent completely differently**: DYF used its allowance to separate digit classes cleanly at the cost of over-splitting each class into styles; k-means used its to hit exactly 10 cluster centers at the cost of merging different digits that share pixel patterns. ### Why this matters If your downstream task is **classification** (given a new image, predict 0–9), the flat 10-way view is what you want, and k-means with the oracle is the right tool. If your downstream task is **retrieval** (given a user's crossbar-7 handwriting, find visually similar samples), you want DYF's sub-style partition. Serving back a mix of crossbar and non-crossbar 7s is a worse result than serving back same-style examples, even though both are "correct" at the digit level. If your downstream task is **exploration** (what's in this corpus?), you want the whole hierarchy. DYF's tree lets you start at 10 clusters for the overview, then zoom to 81 for the sub-style structure, without refitting. K-means would need ten separate runs at different `k` values and no guarantee the partitions would nest. This is what the gallery means by *"DYF gives you the whole hierarchy, pick the resolution your task wants."* MNIST shows it literally. ## Walking down the hierarchy DYF didn't just give you 81 clusters. It gave you a *partition hierarchy* — and every coarser resolution sits one step up the tree. Here's what happens when we walk DYF's 81 clusters down to coarser granularities using `merge_to_max_k`: ```{python} #| label: merge-walk #| output: asis from _gallery import merge_walk, merge_walk_table walk = merge_walk(result, X, y, targets=[40, 20, 10, 5]) print(merge_walk_table(walk, result)) ``` **Read the ARI column.** Merging from 81 down to 20 barely moves NMI (−0.010) but **nearly doubles ARI** — because the 61 extra clusters in the raw partition were genuine sub-communities, and collapsing them recovers the coarser structure the label encodes. Merging further (to 10) gives you ARI close to k-means's — without the oracle having been consulted. This is the move the "parameter-free" pitch is really making: *DYF doesn't pick one k, it gives you the whole tree, and you slice it at whatever granularity your downstream task wants.* K-means and HDBSCAN both commit to a single partition. DYF commits to a ranked hierarchy. ## What happened to the Digits story? Against the Digits notebook, the story flips on NMI but breaks on ARI: | Dataset | DYF k | k-means NMI | DYF NMI | DYF NMI − KMeans NMI | DYF ARI | KMeans ARI | |---------|------:|------------:|--------:|---------------------:|--------:|-----------:| | Digits | 10 | 0.773 | 0.696 | −0.077 | 0.614 | 0.726 | | **MNIST** | **81** | **0.491** | **0.552** | **+0.061** | **0.175** | **0.361** | **DYF now beats oracle k-means on NMI by 0.061** — but loses on ARI by 0.186. How do you lose ARI while winning NMI? ### The over-partitioning question DYF found **81 natural communities on MNIST**, not 10. Three possibilities: **(a) It's finding real sub-style structure.** There are multiple stylistic variants of every digit — slanted 7s vs. crossbar 7s, loopy 4s vs. open 4s, continental 1s vs. American 1s. If DYF's 81 communities each map cleanly to *one* ground-truth digit (just at a finer granularity than the flat 10-label), then DYF is right and the label is impoverished. This is the "MNIST has more than 10 classes if you look closely" reading. **(b) It's fragmenting classes.** The same digit class could be split across many DYF communities for arbitrary geometric reasons (LSH hyperplane placement, UMAP projection quirks) rather than semantic ones. This would be the "DYF over-partitioned" reading. **(c) Some mix.** Some communities are genuine sub-styles; others are artifacts. NMI and ARI disagree *because* they treat over-partitioning differently: - **NMI** (normalized mutual information) — normalizes for cluster counts. Getting "all the 7s into one pile vs two piles of slanted/crossbar 7s" both score well, as long as each pile stays pure. - **ARI** (adjusted Rand index) — penalizes any pair of points that ends up in different clusters when the ground truth puts them together. Splitting the 7s into two piles costs you pairs, regardless of how pure those piles are. **The NMI-win-but-ARI-loss signature is the fingerprint of clean over-partitioning** — clusters that are *individually pure* but *collectively too many*. That's reading (a) above, with some (c). ### Why did this happen at 70K but not at 1,797? Density. At 1,797 samples, there are only ~180 points per digit class, and DYF's tree leaves stay coarse enough that Louvain merges them into ~10 natural communities matching the flat label. At 70K samples, there are 7,000 points per digit class, and DYF's tree now has enough room to carve out stable sub-leaves per style variant. Louvain finds those sub-communities and reports them. **This is DYF doing what it's designed to do** — surface structure the data supports. The fact that the label is flat and the data isn't is a mismatch between the benchmark and reality. ### What would you actually want in a real application? Depends on the downstream task: - **If you're doing classification** (given a new image, predict 0–9): you want k=10 and ARI>NMI. K-means is better here, *provided you know k*. - **If you're doing retrieval** (given a new image, find similar ones): you want sub-style structure preserved. Two crossbar 7s are more similar to each other than a crossbar 7 and a slanted 7. DYF is better here. - **If you're doing data exploration** (what's in this corpus?): you definitely want the sub-style structure, and you probably want the hierarchical tree that DYF produces as a bonus. The gallery's "flat-NMI-against-flat-label" scoring is the classification rubric. DYF is closer to a retrieval / exploration rubric, which is what a parameter-free tool is *for*. ### The sports analogy update Digits was k-means's home court — convex blobs, small k, oracle-provided. It won, cleanly. MNIST is a *longer game*. K-means held the lead most of the way by sticking to the rigid 10-cluster playbook. DYF kept finding extra joints in the structure, and the NMI scoreboard eventually flipped. But ARI — a different metric with a different rulebook — still favors the disciplined game plan. It's no longer clear which method is "winning." It depends on which rulebook you hand the judges. That ambiguity is the point. ## Caveats - **DYF's 81 clusters include some artifacts**, almost certainly. Reading (c) is the honest one. A rigorous analysis would compute a confusion matrix between DYF clusters and the 10 digit labels, check whether each DYF cluster is dominantly one class (style-variant reading) or scattered (fragmentation reading), and report the breakdown. The gallery skips that to keep notebooks readable; a supplement could add it. - **UMAP is a projection, not the cluster space.** When DYF's and k-means's partitions look visually different in the UMAP panel, some of that is true clustering difference and some is UMAP's 2D compression of decisions made in 784d. Don't read the UMAP too literally. - **Cache is in `/tmp`.** Dumped on reboot, which is fine for the gallery — real applications should use a durable cache.