Agreement Analysis: A Decision Chart and Its Evaluation

Abstract

<p>Existing statistical measures of agreement are reviewed and organized into a series of flow charts to facilitate their selection for analysis. A subset of "useful" measures is selected on the basis of their measuring meaningful agreement, hating interpretable referent values, being subject to hypothesis testing, being hand calculable and continuous (categorical) measures having categorical (continuous) analogues. "Useful" measures for continuous data sets are all intraclass correlation coefficients, applicable to different ANOVA models depending on the assumptions involved. "Useful" measures for categories data include various chance corrected (kappa) types, sensitivity-specificity and predictive value measures. The "useful" measures are displayed in a single condensed flow chart. A strategy to evaluate the use of this flow chart is also developed.</p>