Hypothesis test based document clustering

It is well-known in the literature that the main limitations of document clustering techniques are that they operate in a high-dimensional space, and it is difficult to interpret the different clusters once a partition has been obtained. The proposed methods for computing document clustering employ a two-stage process. Initially, it can be observed that the information contained within the document-term matrix exhibits significant sparsity, so a direct application of a clustering technique would be highly inefficient. Consequently, dimensionality reduction is applied. The proposed strategy involves employing latent Dirichlet allocation (LDA) to identify the main topics in the corpus under analysis. To determine the similarity between two documents, the p-value of a hypothesis test of the homogeneity of topic distributions between two documents is computed. This p-value is used as a similarity measure, upon which three different clustering procedures are built. The first two directly employ the new dissimilarity using a hierarchical approach and a fuzzy relational clustering approach, while the other is a test-based approach to clustering. The performance of the clustering methods is then assessed using some benchmark datasets in order to understand the advantages and disadvantages of the proposals.