Mode testing in difficult cases

Usually, when testing the null hypothesis that a distribution has one mode against the alternative that it has two, the null hypothesis is interpreted as entailing that the density of the sampling distribution has a unique point of zero slope, which is a local maximum. In this paper we argue that a more appropriate null hypothesis is that the density has two points of zero slope, of which one is a local maximum and the other is a shoulder. We show that when a test for a mode-with-shoulder is properly calibrated, so that it has asymptotically correct level, it is generally conservative when applied to the case of a mode without a shoulder. We suggest methods for calibrating both the bandwidth and dip-excess mass tests in the setting of a mode with a shoulder. We also provide evidence in support of the converse: a test calibrated for a single mode without a shoulder tends to be anticonservative when applied to a mode with a shoulder. The calibration method involves resampling from a "template" density with exactly one mode and one shoulder. It exploits the following asymptotic factorization property for both the sample and resample forms of the test statistic: all dependence of these quantities on the sampling distribution cancels asymptotically from their ratio. In contrast to other approaches, the method has very good adaptivity properties.