A Cross-cultural Exploration of Physician Assessment

Abstract

We conduct an evaluation of the cross-cultural ‘export’ of the Objective Structured Clinical Examination (OSCE), a well-established Western medical education assessment tool that is in keeping with Competency-Based Medical Education (CBME) principles, into the new socio-economic setting of Rwanda. The evaluation framework of ‘assessment utility’ is applied, where the utility of an assessment is described conceptually as the multiplicative function of its validity (V), reliability (R), educational impact (E), cost/feasibility (C) and acceptability (A). A mixed-methods approach of both quantitative and qualitative data analysis is used. The quantitative findings support high content and face validity, high reliability, high acceptability and achievable cost and feasibility of the OSCE, all of which would suggest high utility. The analysis of qualitative data identifies some important threats to validity, namely perceived significant gaps in training in the internship program that were thought to likely be the underlying reason for the low mean assessment scores. This threat to the validity of the results appears to influence and limit the acceptability of the assessment in this context. While it is believed that it would be suitable as a formative assessment, primarily for the purpose of ‘assessment for learning’, it was not felt that it was currently acceptable as a summative or high-stakes ‘assessment of learning’, until and unless training deficits are addressed. Currently, the OSCE is seen to have greatest value in its potential for educational impact by acting as both a driver and a marker for change both at the individual and programmatic levels. Many principles of CBME and the concept of ‘entrustability’ as a criterion-referenced assessment standard were well-received cross-culturally, when training and assessment were viewed in tandem. Our study highlights the importance of using a comprehensive evaluation framework that includes both quantitative and qualitative methods to accurately characterize the utility of an assessment.