Effects of differential time and difficulty weighting on the improvement of OSCE quality metrics

Vpliv časovnih in težavnostnih diferencialnih uteži na izboljšanje kazalcev kakovosti OSKI

Abstract

Purpose: Objective structured clinical examination (OSCE) has become a leading method for assessing clinical skills. A common problem observed in OSCE is the discrepancy between checklist scores and global scale scores. To reduce this discrepancy, checklists have been unsuccessfully weighted. However, differential weighting based on a combination of time and difficulty weighting had not been investigated. The goals of our study were to: (a) develop new formulas for time and difficulty weights, (b) design an experimental checklist considering the calculated weights, and (c) evaluate the effectiveness of the method in improving OSCE quality metrics.
Methods: First, we created formulas for time and difficulty weights for each checklist item. Second, we formulated an experimental checklist considering
the calculated weights of each item. Simultaneously, a control checklist was independently weighted by OSCE board members. Third, we compared the experimental and control checklists during the OSCE, wherein students (n=55) were graded by ten assessors using both checklists and the holistic global rating form. Finally, we performed statistical analysis with paired samples t-test, simple linear regression, and scale reliability statistics.
Results: A higher correlation coefficient between the checklist and the holistic scoring form (r=0.622, p<0.001 vs. r=0.496, p<0.001), and significantly lower checklist scores (p<0.001), were obtained with the experimental checklist than the control checklist.
Conclusion: Assigning time and difficulty differential weights to checklist items improved OSCE quality metrics and reduced the discrepancy between
global and checklist scores.