Oral Examinations in Undergraduate Medical Education – What is the ‘Value Added’ to Assessment? Luise I.M. Pernar, MD1,2; Amy M. Sullivan, EdD3; Katherine Corso, MPH2; Elizabeth Breen, MD1,2 1. Department of Surgery; Brigham and Women’s Hospital; Boston, MA
2. Center for Surgery and Public Health; Brigham and Women’s Hospital; Boston, MA
Background and Purpose: Oral examinations have a long standing tradition as an assessment tool used in the evaluation of learners at all levels of surgical training, from student to resident to Board Eligible Surgeons. The value of these oral examinations, however, has been questioned, particularly in regard to whether and how oral examinations assess competence in ways not evaluated in other assessment tools. The aim of this study was to identify and describe the implicit criteria faculty examiners use in their grading of medical students’ oral exams during their Core Surgery Clerkship, and to provide a preliminary determination of whether the oral exam may assess qualities and competencies beyond less resource-intensive assessments such as shelf exams.
Methods: This study is a retrospective, qualitative analysis of comments provided by faculty examiners (n=43) on the oral examination score sheets evaluating performance of all medical students completing their core surgery clerkship at the Brigham and Women’s Hospital during the years 2005-2011. After immersion in the data to develop a set of themes and concepts, we developed a final set of codes, assessed reliability of coding, and coded all exam data using Atlas.ti® qualitative data analysis software. All comments were reviewed and analyzed to determine what qualities examiners detect or naturally comment on when administering the oral examinations. Codes were also labeled as either “positive,” “negative,” or “neutral” to describe the valence given to the quality observed by the examiner. Codes were described qualitatively and counted to assess frequency of use overall and by examiner.
Results: A total of 629 oral exams for 315 students were examined. Oral exam comments addressed three major areas: (1) clinical knowledge (expressed as fund of knowledge; demonstration of appropriate steps in a patient work up [history and physical exam]; differential diagnosis; and patient management [pre-operative, operative, and post-operative]); (2) cognitive processes (described as critical thinking; focus; organization; thoroughness); and (3) professional/interpersonal skills (including demeanor, communication, decisiveness, pace, and level of prompting needed). Three coders coded a random sample of 10% of comments and achieved a moderately high reliability of .83 (Fleiss’ Kappa). The most frequent references were to how students described conducting the patient work up, observed in 21% (n=437) of all comments. The most universally applied comments (that is, used by all examiners [n=43] at least once) related to students’ fund of knowledge. Both positive (n=1146, 55.2%) and negative (n=879, 42.3%) comments were identified in the data.
Conclusions: Oral examinations provide rich opportunities for testing many competencies that are important in clinical practice. A number of the qualities evaluated by examiners--such as communication, decisiveness, and critical thinking--are not well addressed or not tested at all in other assessment tools. Our identification of these testable competencies can aid in the development of a standardized scoring rubric, and provide faculty guidelines that may aid in improving efficiency and minimizing subjectivity and bias in what otherwise is a rich assessment tool.