An Evaluation of the Utility of a Hybrid Objective Structured Clinical Examination for the use of Assessing Residents Enrolled in McMaster University's Orthopaedic Surgery Residency Program

Abstract

<p><strong>Introduction: </strong>McMaster University’s orthopaedic surgery residency program implemented the OSCE as an assessment tool in 2010; this study evaluates the first four OSCEs administered to residents. The OSCEs were composed of knowledge-testing stations, which are normally not included in this testing format, and performance-testing stations. Recruiting enough faculty evaluators challenged the ability to feasibly implement this examination format. Knowledge-testing stations were incorporated since they do not require evaluators to be present. Reliability was assessed, and the correlation between knowledge-testing station scores and performance-testing station scores was determined. The ability of the OSCE to discriminate between residents in different post-graduate years (PGYs) was assessed. Residents’ acceptability of the OSCE was also assessed. <strong>Methods: </strong>Reliability was assessed using generalizability theory. The correlation of knowledge-testing and performance-testing station scores was measured with Pearson’s r. A two-way ANOVA was used to analyze whether the OSCE can discriminate between residents in different PGYs. An exit survey was administrated after each OSCE to assess acceptability. <strong>Results: </strong>The generalizability estimates of each OSCE ranged from 0.71 to 0.87. The disattenuated correlation between knowledge- and performance-testing stations for senior residents was 1.00, and 0.89 for junior residents. A significant effect of year of residency was found for the October 2010 OSCE in the ANOVA (F(1,30) = 11.027, p = 0.005), but the remaining OSCEs did not replicate this finding. In general, residents felt that they were able to present an accurate portrayal of themselves in the OSCEs and that the examination covered appropriate topics. <strong>Discussion: </strong>The OSCEs were reliable and acceptable to residents. The high correlations between knowledge- and performance-testing station scores suggest that the examination can be made more feasible by including knowledge-testing stations. The small sample sizes made significant differences difficult to detect between levels of training, resulting in inconclusive evidence for this construct validation measure.</p>