Parameter-Free Clustering Gallery

DYF produces a partition hierarchy, not a single k. Here’s what that buys you across eight classic datasets.

Every other clustering algorithm asks you to pick a number.

K-means wants k. HDBSCAN wants min_cluster_size. DBSCAN wants eps. Louvain wants resolution. Each is a commitment to a single granularity — make the wrong pick and you either split a natural cluster in half or blur two together.

DYF’s pitch is different. Build the tree once, and you get the whole hierarchy. At its finest leaves, DYF’s Louvain step finds many fine-grained communities. Walk up the tree and those merge into progressively coarser partitions. You pick the resolution your downstream task needs — at query time, not at training time.

What the gallery shows

Each notebook below runs the same three lines of DYF against a dataset with known class structure. The metric columns that matter are:

• Recovered k — the finest-resolution partition DYF finds from the data.
• Merge walk — how NMI and ARI evolve as you collapse that partition to coarser k values.
• NMI / ARI at matched granularity — how closely DYF’s partition aligns with the ground truth at the resolution the label is expressed at.

Each notebook also scores k-means (told the oracle k) as a ceiling, and HDBSCAN (parameter-light, but not parameter-free) as a second baseline. Where DYF loses, the notebook says so.

The notebooks

How to read the results

The gallery’s cross-notebook findings cluster into four patterns:

DYF behavior depends on two axes, not one.

Axis	Where DYF struggles	Where DYF wins
Cluster shape	Low-dim curved / topologically nested (HDBSCAN wins — see Moons)	Convex or hierarchical clusters in medium-to-high dim
Data scale	Very small n, very high d (Olivetti 400 × 4096 breaks it)	Medium-to-large n, moderate d — where HDBSCAN’s O(n²) breaks

Over-partitioning is often a feature. On MNIST DYF finds 81 communities for 10 classes, and on 20 Newsgroups it finds 34 for 20. The merge walk shows those extra splits are genuine hierarchy — collapsing 81 → 20 on MNIST roughly doubles ARI while barely moving NMI. The finer grain isn’t noise; it’s sub-style / sub-topic / sub-motion structure that the flat ground-truth label ignores.

Merging walks hierarchy; it can’t fix bad geometry. On Moons merging 3 → 2 decreases NMI. That’s diagnostic: the raw over-partition must contain the coarser structure as a sub-partition for the merge to help.

“True k” is frequently a fiction. CMU MoCap has three plausible ground-truth labels at different granularities (7 / 25 / 80 classes). DYF’s partition beats oracle k-means on NMI against every one of them, because it doesn’t commit to a single k at fit time.

Who should be in your toolbox?

Based on the nine notebooks:

• HDBSCAN for low-dimensional, clean, small-to-medium datasets where cluster shape matters and you can tolerate a “noise” bucket.
• K-means when you genuinely know k, your clusters are convex-ish, and you need the fastest possible fit.
• DYF for medium-to-large datasets where you don’t know k, the structure is hierarchical, or you want multi-resolution output for downstream routing / retrieval / exploration.

These are compositional, not competitive. The recipe “pretrained embedder → DYF tree → merge-walk to whatever resolution the downstream task wants” is the pattern CIFAR-10 + CLIP demonstrates end-to-end.

What “parameter-free” actually means here

The gallery’s title uses “parameter-free” loosely. DYF has parameters — it just doesn’t ask the user for the data-dependent ones that make other clustering methods fiddly.

Parameters you don’t supply

• k (number of clusters). DYF discovers it. This is the big one — k-means and spectral clustering both need it.
• eps (DBSCAN-style distance threshold). No such knob; you don’t need to know your data’s scale to run DYF.
• min_cluster_size (HDBSCAN-style). DYF has a structural min_leaf_size analog, but it isn’t tied to what you think your cluster count should be.

Not needing k or eps is the real claim DYF makes. The ones below are structural choices that stay fixed across every gallery dataset.

Parameters DYF fixes at defaults

Every notebook uses these same values — not tuned per-dataset, not adaptive. Where they break, the relevant notebook says so honestly.

Parameter	Default	What it does	Known failure mode
max_depth	4	Tree depth cap (→ 4096 leaves max)	Needs deeper tree at n > ~250K
num_bits	3	Branching factor (8-way splits)	Alternatives untested
min_leaf_size	20	Leaves must contain ≥20 points	Breaks below ~10 points per latent class (Olivetti)
Louvain resolution	1.0	Modularity granularity	Fails on extreme imbalance (10:1–20:1 minority-size range)
Louvain leaf_k	10	kNN on leaf centroids	No observed failure mode
Metric	cosine	Implicit in LSH tree construction	Wrong for low-dim Euclidean (Moons partial failure)
seed	42	Reproducibility	Run-to-run variance uncharacterized

Parameters the reader does supply

• target_k in the merge walk or slider — the resolution at which you want to view the hierarchy. This is the only knob a gallery user touches, and it’s downstream of clustering, not upstream of it.
• The embedding model for text and image data — not a DYF parameter, but the pre-embedding step is an unavoidable user choice. CIFAR-CLIP documents the raw pixels → pretrained embedder → DYF recipe.

So what’s the actual claim?

Not “DYF has no parameters.” The honest claim is:

The same three lines of DYF, with the same defaults, work across every dataset in this gallery — from 400 faces to 120,000 news articles, from 2D synthetic shapes to 62-dim motion capture. You don’t tune per dataset. You don’t have to know k or eps. The fit runs everywhere.

That’s weaker than “parameter-free” but it’s a true claim, and it’s the one the gallery actually demonstrates. The cases where defaults fail (Olivetti’s density floor, Moons’ metric mismatch, imbalanced-class Louvain) are documented in the notebooks, not hidden.