Estimating the predominant number of clusters in a dataset

Al Shaqsi, Jamil and Wang, Wenjia (2013) Estimating the predominant number of clusters in a dataset. Intelligent Data Analysis, 17 (4). pp. 603-626. ISSN 1088-467X

Full text not available from this repository. (Request a copy)

Abstract

In cluster analysis, finding the number of clusters, K, for a given dataset is an important yet very tricky task, simply for the facts that there is no universally accepted correct or wrong answer for most real world problems and it all depends on the context and purpose of a cluster study. Numerous methods have been developed for estimating K, but most are not widely used in practice due to their poor performance. Thus, it is still quite common that human user is required to select a specific value or a range for K for many clustering methods before they are used. Inappropriate predetermination for K can result in poor clustering results. This paper presents a new method for estimating the most probable number of clusters automatically. It firstly calculates the length of constant similarity intervals, L, and then considers the longest ones as the representations of the most probable numbers of the clusters under the set context and the chosen similarity measure. An error function is defined to measure and evaluate the goodness of estimations. The proposed method has been tested on 3 synthetic datasets and 8 real-world benchmark datasets, and compared with some other popular methods including particularly the TwoStep implemented in IBM/SPSS Modeler software package. The experimental results showed that the proposed method is able to find the "desired" predominant number of clusters for all the simulated datasets and most of the benchmark datasets, and the statistical tests indicate that our method is significantly better.

Item Type:	Article
Uncontrolled Keywords:	clustering, k-means, cluster number,similarity measure
Faculty \ School:	Faculty of Science > School of Computing Sciences
UEA Research Groups:	Faculty of Science > Research Groups > Data Science and Statistics
Depositing User:	Pure Connector
Date Deposited:	19 Aug 2014 08:28
Last Modified:	31 Oct 2022 12:31
URI:	https://ueaeprints.uea.ac.uk/id/eprint/49923
DOI:	10.3233/IDA-130596

Actions (login required)

View Item