Objectives: To develop and pilot test an objective assessment of technical skills instrument for evaluation of residents' surgical performance of pediatric direct laryngoscopy and rigid bronchoscopy, with emphasis on feasibility, validity, and interrater agreement.
Study design: Prospective, unblinded educational quality-improvement project combining paired and unpaired observations.
Methods: Objective Structured Assessment of Technical Skills (OSATS) evaluation instruments were developed from detailed faculty input. Two instruments were created: a surgical checklist and a global assessment of surgical performance. The instruments were tested in a simulation bronchoscopy course and in the operative suite. Paired observations were used to determine interrater agreement, whereas multiple evaluations were used to calculate construct validity and internal consistency.
Results: Forty-four assessments were completed for 19 residents. Seven faculty members evaluated residents as they performed laryngoscopy and bronchoscopy in an animal simulation laboratory and in the operating room. The evaluation tool was found to be feasible. The interrater agreement for both instruments was significant at 80.4% to 84.6% (P ≤.0001). Construct validity was confirmed with increasing mean global and task specific scores by postgraduate year (P <.0001). Internal consistency, measured with Cronbach alpha, was high at 0.968.
Conclusions: This pilot study suggests that a reliable and valid instrument for objective evaluation of surgical competency can be developed for pediatric direct laryngoscopy and rigid bronchoscopy. This instrument can be used for formative and summative feedback of operative performance. In addition, it was easy to use and valid in a limited evaluation; however, larger studies are required to validate its utility.