How Ward clustering becomes a topic hierarchy AI

The difference between the fitted binary dendrogram and the topic branches exposed in the viewer.

The atlas starts with individual papers and repeatedly merges groups until all papers belong to one tree. The viewer exposes selected branches of that tree as topics. Its levels are a navigation choice layered over the fitted hierarchy.

What Ward merges

Ward’s method selects the merge with the smallest increase in the sum of squared distances from members to their group mean. For two groups, that increase can be written as

$$\Delta(A,B)=\frac{n_A n_B}{n_A+n_B}\lVert\mu_A-\mu_B\rVert^2.$$

Here $n$ is group size and $\mu$ is its mean vector. In this atlas, the vectors are the unwhitened PCA-128 scores. Euclidean geometry is part of the method; an arbitrary matrix of cosine dissimilarities is not an interchangeable input. The SciPy linkage documentation describes Ward’s variance objective and Euclidean requirement.

This objective favors groups that are compact in the chosen representation. It does not require equally sized topics, and it supplies no scientific topic names.

Why an expansion can show more than two topics

The underlying agglomerative tree is binary. Showing every merge would produce many small, unhelpful navigation steps. The atlas therefore collapses intermediate merges and exposes a frontier of descendant branches. A visible parent can have two, five or more children even though every fitted merge combined two groups.

The visible branches are selected using branch size, minimum child support, the relative gap in merge heights, and the variance reduction associated with a split. A size override can expose divisions in very large branches even when the other signals are weak. These are explicit exploration heuristics, not significance tests or evidence that each displayed division is a natural scientific boundary.

What a stopping point means

A topic with no ordinary subtopics may have reached a size target or failed a split criterion. It need not be homogeneous. Additional inspection levels expose finer branches of the same tree.

The child groups partition their parent: each sampled paper has one path through this hierarchy. Interdisciplinary papers may therefore fit more than one useful description while still receiving one location in the tree.

Topic names summarize these memberships; changing a name does not change the fitted branch.