THE POTENTIAL FOR MACHINE LEARNING IN MENTAL HEALTH POLICING: PREDICTING OUTCOMES OF MENTAL HEALTH RELATED CALLS FOR SERVICE

Abstract

My objective was to predict outcomes following police interactions with PMIs, and compare the predictive accuracy of logistic regression models and Random Forests learning algorithms. Additionally I evaluated if predictive accuracy of Random Forests changed when applied to merged versus region-specific data. I conducted a retrospective cohort study of reports completed by police in 13 communities between 2015 and 2018. 13,058 reports were analyzed. Random Forests learning algorithms were compared against logistic regression models for predictive accuracy in a merged dataset (13 communities) and 3 regional datasets. Outcomes for prediction were high risk of harm to self, risk of harm to others, and risk of failure to care for self within 24 and 72 hours following police contact. Random Forests learning algorithms were trained on merged and regional datasets, and compared against merged and regional holdout datasets. Performance was compared by area under the curve. For Random Forests learning algorithms, confusion matrix statistics were calculated for each outcome and predictive utility was examined by calculating conditional probabilities. Prediction accuracy was modest across all methods. Random Forests achieved better predictive accuracy than logistic regression. Random Forests accuracy varied between merged and regional holdout data. Sensitivity of Random Forests learning algorithms were moderate (74% average, 6 outcomes, merged holdout set). Specificity was low (53% average, 6 outcomes, merged holdout set). Conditional probabilities were modestly improved by the use of the Random Forests learning algorithm. The rareness of the target outcomes created a situation where even predictions with moderate likelihood ratios had only modest predictive value. Though the Random Forests learning algorithms did outperform the logistic regression learning algorithms, the clinical significance of those benefits were limited when conditional probabilities were calculated. These findings are limited to the outcomes considered, and may not apply to more common outcomes.