Home
Getting Started
How It Works
API Reference
Split Keywords
tokenize
API Reference
Core
DensityClassifierFull
Tree Methods
build_pca_tree
cut_tree_to_labels
extract_boundary_persistence
boundary_persistence_scores
build_dyf_tree
refine_dyf_tree
cut_dyf_tree_to_labels
refine_clusters
Retrieval (RAG)
BridgeIndex
find_super_connectors
select_orthogonal_anchors
diversify_by_facet
get_kmeans_init
compute_neighbor_diversity
compute_hub_score
mine_dag_chains
build_dag_taxonomy
build_unified_ontology
build_rog_ontology
Result Types
SuperConnectorResult
OrthogonalAnchorResult
FacetDiverseResult
DAGChain
DAGMiningResult
DAGTaxonomy
UnifiedOntologyResult
HubScoreResult
ROGLayer
ROGResult
Re-ranking
rerank_standard
rerank_mmr
rerank_bridge_boost
rerank_bridge_mmr
Chunk Analysis
DocSpread
chunk_redundancy
deduplicate_chunks
doc_spread
cluster_quality
neighbor_coherence
Lazy Index
LazyIndex
write_lazy_index
rewrite_lazy_index
split_dyf3
from_faiss
SearchResult
AdaptiveProbeConfig
Fisher Weighting
compute_fisher_weights
apply_fisher_weights
Split Keywords
compute_split_keywords
compute_embedding_keywords
label_clusters_frequency
build_tree_maps
collect_descendant_indices
compute_domain_stopwords
format_split_path
tokenize
TextDiversityReport
assess_text_diversity
Cluster Tree
build_cluster_tree_dag
compute_sibling_keywords
derive_path_labels
format_cluster_context
Agglomeration
agglomerate_tree_leaves
louvain_cluster_leaves
merge_to_max_k
Categorical
CategoryGraph
coarsen
multi_level_fisher_weights
store_category_graph
load_category_graphs
AxisDiagnostic
diagnose_axes
diagnostics_to_metadata
discover_categorical_columns
embed_with_diagnostics
Colors
spatial_rgb_map
spatial_color_map
tree_rgb_map
Provenance
Provenance
create_provenance
file_hash
params_hash
check_compatible
provenance_to_dict
provenance_from_dict
Configuration
EmbedderConfig
LabelerConfig
list_configs
On this page
tokenize
Split Keywords
tokenize
tokenize
tokenize(text)
Lowercase, extract words of 3+ chars, filter stop words.