DYF produces a partition hierarchy, not a single k. Here’s what that buys you across eight classic datasets.
Every other clustering algorithm asks you to pick a number.
K-means wants k. HDBSCAN wants min_cluster_size. DBSCAN wants eps. Louvain wants resolution. Each is a commitment to a single granularity — make the wrong pick and you either split a natural cluster in half or blur two together.
DYF’s pitch is different. Build the tree once, and you get the whole hierarchy. At its finest leaves, DYF’s Louvain step finds many fine-grained communities. Walk up the tree and those merge into progressively coarser partitions. You pick the resolution your downstream task needs — at query time, not at training time.
Each notebook below runs the same three lines of DYF against a dataset with known class structure. The metric columns that matter are:
k values.Each notebook also scores k-means (told the oracle k) as a ceiling, and HDBSCAN (parameter-light, but not parameter-free) as a second baseline. Where DYF loses, the notebook says so.
The gallery’s cross-notebook findings cluster into four patterns:
DYF behavior depends on two axes, not one.
| Axis | Where DYF struggles | Where DYF wins |
|---|---|---|
| Cluster shape | Low-dim curved / topologically nested (HDBSCAN wins — see Moons) | Convex or hierarchical clusters in medium-to-high dim |
| Data scale | Very small n, very high d (Olivetti 400 × 4096 breaks it) | Medium-to-large n, moderate d — where HDBSCAN’s O(n²) breaks |
Recovered k tracks the label once the tree has enough leaves. On MNIST DYF finds 12 communities for 10 classes and on CIFAR-10 + CLIP 11 for 10 — in both cases beating oracle k-means on ARI without being told k. The failure mode runs the other way on small data: Digits (1,797 points, ~190 leaves) recovers 8 for 10 and 20 Newsgroups 12 for 20, because a modularity optimiser on a graph of a few hundred leaf centroids hits its resolution limit and merges real classes. Raising Louvain resolution fixes that (Digits at resolution=2: 10 communities, better on both metrics); the default is left at 1.0 because no single value won across the gallery.
(Earlier versions of this page said the opposite — “over-partitioning is often a feature”, citing 81 MNIST communities. That count came from a single-level Louvain in dyf-rs before 0.12.0 that could not merge communities; see dyf’s KNOWN_ISSUES.md, issue 10.)
Merging walks hierarchy; it can’t fix bad geometry. On Moons merging 3 → 2 decreases AMI. That’s diagnostic: the raw over-partition must contain the coarser structure as a sub-partition for the merge to help.
“True k” is frequently a fiction. CMU MoCap has three plausible ground-truth labels at different granularities (7 / 25 / 80 classes). DYF recovers 15 communities and edges oracle k-means on ARI against the 25-class label (0.116 vs 0.103) while trailing it on AMI (0.292 vs 0.348) — and both are far from the leaf-level ceiling, because trial labels describe trajectories through pose space, not blobs in it. Frame-level clustering is the wrong tool for that label at any k; the notebook says so.
Every page’s metrics table has a wall (s) column: DYF’s tree build + index write + Louvain against one k-means++ init at the oracle k (the shared UMAP layout is excluded from both). The honest reading at gallery scale — 400 to 70,000 points — is that DYF is the slower tool on every page, by a factor that ranges from noise on Digits to roughly an order of magnitude on Olivetti and CIFAR-10. That is not a surprise: k-means with k handed to it and a single init is close to the cheapest partition there is, and DYF is paying for an on-disk index and a hierarchy that k-means does not produce at any k.
The comparison where DYF is the faster tool is a different one. At 1.3M cells the shipped pipeline matches a kNN-graph + Leiden partition (ARI 0.603 vs 0.608 against scanpy’s labels) in roughly a seventh of the wall time — dyf KNOWN_ISSUES.md #10 has the measurement — and at that scale a graph pipeline, not k-means, is what people actually run. The gallery does not include a dataset that size, so it does not make that claim on its own evidence.
Based on the nine notebooks:
k, your clusters are convex-ish, and you need the fastest possible fit.k, the structure is hierarchical, or you want multi-resolution output for downstream routing / retrieval / exploration.These are compositional, not competitive. The recipe “pretrained embedder → DYF tree → merge-walk to whatever resolution the downstream task wants” is the pattern CIFAR-10 + CLIP demonstrates end-to-end.
The gallery’s title uses “parameter-free” loosely. DYF has parameters — it just doesn’t ask the user for the data-dependent ones that make other clustering methods fiddly.
k (number of clusters). DYF discovers it. This is the big one — k-means and spectral clustering both need it.eps (DBSCAN-style distance threshold). No such knob; you don’t need to know your data’s scale to run DYF.min_cluster_size (HDBSCAN-style). DYF has a structural min_leaf_size analog, but it isn’t tied to what you think your cluster count should be.Not needing k or eps is the real claim DYF makes. The ones below are structural choices that stay fixed across every gallery dataset.
Every notebook uses the library’s own defaults — build_dyf_tree(num_bits=3, max_depth=4, min_leaf_size=20), which _gallery.run_dyf passes unchanged. Nothing is tuned per dataset against the labels. (_gallery.auto_tune_tree_params(n), a rule that derives the depth from the sample count, used to be the default here; measured on the seven labelled datasets with the multilevel Louvain it loses to the fixed defaults — mean ARI 0.436 vs 0.440, winning 3 of 7 — so it is kept only for explicit experiments. dyf KNOWN_ISSUES.md #9.) Where the defaults break, the relevant notebook says so honestly.
| Parameter | Default | What it does | Known failure mode |
|---|---|---|---|
num_bits |
3 | Branching factor (8-way splits per level) | Alternatives (2 bits, deeper trees) measured and not better |
max_depth |
4 | Tree depth (→ ≤ 8⁴ = 4,096 leaves) | Fixed regardless of n; at 1.3M points leaves average ~350 points and the partition still matches the incumbent, because the centroid graph, not leaf purity, is the bottleneck |
min_leaf_size |
20 | Leaves must contain ≥20 points | Breaks below ~10 points per latent class (Olivetti) |
Louvain resolution |
1.0 | Modularity granularity | Under-splits small leaf graphs (≲ a few hundred leaves — Digits, 20 Newsgroups); resolution=2 fixes Digits |
Louvain leaf_k |
10 | kNN on leaf centroids | No observed failure mode |
| Row normalisation | always | Every tree fit L2-normalises rows first, so the tree partitions directions (dyf KNOWN_ISSUES.md #11) |
Erases the norm — fatal when the norm is the structure, as in low-d raw coordinates (Moons / Circles); harmless for embeddings, which live on the sphere anyway |
seed |
42 | Reproducibility | Run-to-run variance uncharacterized |
target_k in the merge walk or slider — the resolution at which you want to view the hierarchy. This is the only knob a gallery user touches, and it’s downstream of clustering, not upstream of it.raw pixels → pretrained embedder → DYF recipe.Not “DYF has no parameters.” The honest claim is:
The same three lines of DYF, with the same defaults, work across every dataset in this gallery — from 400 faces to 120,000 news articles, from 2D synthetic shapes to 62-dim motion capture. You don’t tune per dataset. You don’t have to know
koreps. The fit runs everywhere.
That’s weaker than “parameter-free” but it’s a true claim, and it’s the one the gallery actually demonstrates. The cases where defaults fail (Olivetti’s density floor, Moons’ metric mismatch, imbalanced-class Louvain) are documented in the notebooks, not hidden.