In brief - A reliable guide?

Evaluating the performance of approaches for extraction of image features independent of scale, orientation and photometrical changes is an important issue in vision research. The most widely employed metric is repeatability rate, however it has been seen that this does not necessarily match actual performance. Researchers in the UK present an improved repeatability formulation and find that this alters the choice of ‘best’ detector.

Improved repeatability measures for evaluating performance of feature detectors
Electron. Lett. -- 8 July 2010 -- Volume 46, Issue 14, p.998–1000

S. Ehsan (1), N. Kanwal (1), A.F. Clark (1) and K.D. McDonald-Maier (1)
(1) University of Essex, School of Computer Science and Electronic Engineering, Colchester, United Kingdom

The most frequently employed measure for performance characterisation of local feature detectors is repeatability, but it has been observed that this does not necessarily mirror actual performance. Presented are improved repeatability formulations which correlate much better with the true performance of feature detectors. Comparative results for several state-of-the-art feature detectors are presented using these measures; it is found that Hessian-based detectors are generally superior at identifying features when images are subject to various geometric and photometric transformations.

Full text available on the IET Digital Library.