Stability for selecting the number of clusters: literature review, questions, and ideas

Posted in Science on August 29, 2008

Lecture slides:

Clustering Stability — a literature review, <br>many questions, and a few ideas for answers
Overview
The principle of stability
Stability as a tool for model selection in clustering
Stabilty – the general principle
The toy figure in favor of stability
Generating artificial data sets
How to use the clustering algorithm
Distances between the clusterings
Which clusterings to compare?
Stability scores
Normalization
Selecting K, finally
Stability in theory
Negative results on stability
First catch: large vs. small sample size
Possible solution: “stability window”
Second catch: attaining the global minimum
Possible solution: exploring objective function
Catch 3: What is “the right K”,<br> actually?
The “correct K”, first approach
The “correct” K, second approach
Idea: hierarchy of cluster core sets
Summary

Author: Ulrike Von Luxburg, Max Planck Institute For Biological Cybernetics