A Model Selection Approach to Hierarchical Shape Clustering with an Application to Cell Shapes

Shape is an important phenotype of living species that contain different environmental and genetic information. Clustering living cells using their shape information can provide a preliminary guide to their functionality and evolution. Hierarchical clustering and dendrograms, as a visualization tool for hierarchical clustering, are commonly used by practitioners for classification and clustering. The existing hierarchical shape clustering methods are distance based. Such methods often lack a proper statistical foundation to allow for making inference on important parameters such as the number of clusters, often of prime interest to practitioners. We take a model selection perspective to clustering and propose a shape clustering method through linear models defined on Spherical Harmonics expansions of shapes. We introduce a BIC-type criterion, called CLUSBIC, and study consistency of the criterion. Special attention is paid to the notions of over- and under-specified models, important in studying model selection criteria and naturally defined in model selection literature. These notions do not automatically extend to shape clustering when a model selection perspective is adopted for clustering. To this end we take a novel approach using hypothesis testing. We apply our proposed criterion to cluster a set of real 3D images from HeLa cell line.