Representing uncertainty in ranking by single or multiple indicators

This paper introduces a method for representing uncertainty in ranking by single or multiple indicators. The method can potentially integrate parametric and structural uncertainty of model outputs. It requires estimating the range of conditions over which a ranking of items should be robust. The ranking is then subjected to perturbation tests, and the results displayed graphically. Ranking a set of measurements, or ranking a set of model outputs, is a generic task for decision support. In the case of multiple indicators, a composite index is often defined. However, as Patil and Taillie (2004) point out, "every such composite involves judgements (often arbitrary or controversial) about tradeoffs or substitutability among indicators." These concerns are addressed by the concept of partial order. Partially-ordered sets can be used to identify items that are objectively comparable, in the sense that all indicators favour one item over the other. If there is a tradeoff between two items (i.e. their indicators are inconsistent) then they are not inherently comparable. The concept of partial order been used recently to rank multiple indicators. For example, Hollert et al. (2002) used it to rank ecotoxicological contamination of small streams according to different chemical and biological tests. This paper extends the use of partial order, from representating ambiguity, to also representing uncertainty. Outputs from perturbed models can be treated as additional indicators, alongside outputs from alternative model structures. Another possibility is the use of data resampling (jackknife or bootstrap tests) to generate perturbed indicators. An example of a robust partial order is shown on Figure 1, where sites in a river system are ranked by their median flow magnitude. For this analysis, river flow time-series were used from 9 sites in a common 18-year period. For the ranking to be robust, it should not change when a single year is included or excluded (Figure Presented) from the common period. Additionally, it should be equivalent for any percentiles between 40% and 60% (not just the exact median, 50%). The partial order on Figure 1 shows the comparisons that are robust under these conditions - in this case, it is almost a complete order. There are only three sites with ambiguous ranks. This paper also gives a more complex case study, combining multiple indicators. Representing uncertainty in ranking should provide an improved basis for decision-making. The lack of agreement between indicators, or their lack of robustness, lead naturally to reconsidering and revising the modelling process.