Anesthesia assessment in the era of competences: state of the art
Abstract
Introduction: Anesthesiology requires procedure fulfillment, problem, and real-time crisis resolution, problem, and complications forecast, among others; therefore, the evaluation of its learning should center around how students achieve competence rather than solely focusing on knowledge acquisition. Literature shows that despite the existence of numerous evaluation strategies, these are still underrated in most cases due to unawareness.
Objective: The present article aims to explain the process of competency-based anesthesiology assessment, in addition to suggesting a brief description of the learning domains evaluated, theories of knowledge, instruments, and assessment systems in the area; and finally, to show some of the most relevant results regarding assessment systems in Colombia.
Methodology: The results obtained in “Characteristics of the evaluation systems used by anesthesiology residency programs stakeholders in the educational process, a fact that motivated the publishing of this discussion around the topic of competency-based assessment in anesthesiology. Following a bibliography search with the keywords through PubMed, OVID, ERIC, DIALNET, and REDALYC, 110 articles were reviewed and 75 were established as relevant for the research’s theoretical framework.
Results and conclusion: Anesthesiology assessment should be conceived from the competency’s multidimensionality; it must be longitudinal and focused on the learning objectives.
References
Van Der Vleuten CP. Revisiting assessing professional competence: from methods to programmes. Med Educ 2016;50:885–888.
Celman S. ¿Es posible mejorar la evaluación y transformarla en herramienta de conocimiento? En Camilloni AW: La evaluación de los aprendizajes en el debate didáctico contemporáneo. Paidos, Buenos Aires; 1998.
Colbert CY, Dannefer EF, French JC. Clinical competency committees and assessment: changing the conversation in graduate medical education. J Grad Med Educ 2015;7:162–165.
Tetzlaff JE. Assessment of competency in anesthesiology. Anesthesiology 2007;106:812–825.
Tetzlaff JE. Assessment of competence in anesthesiology. Curr Opin Anaesthesiol 2009;22:809–813.
Tezlaff JE. Evaluation of Anesthesia Residents. En Frost EAM. Comprehensive Guide to Education in Anesthesia. 2014;Springer, New York:129-145.
Fraser AB, Stodel EJ, Jee R, et al. Preparing anesthesiology faculty for competency-based medical education. Surv Anesthesiol 2017;61:32–33.
Bould MD, Naik VN, Hamstra SJ. Review article: new directions in medical education related to anesthesiology and perioperative medicine. Can J Anesth 2012;59:136–150.
Boet S, Pigford AAE, Naik VN. Program director and resident perspectives of a competency-based medical education anesthesia residency program in Canada: a needs assessment. Korean J Med Educ 2016;28:157–168.
Jonker G, Manders L, Marty A, et al. Variations in assessment and certification in postgraduate anaesthesia training: a European survey. Br J Anaesth 2017;119:1009–1014.
Ross FJ, Metro DG, Beaman ST, et al. A first look at the Accreditation Council for Graduate Medical Education anesthesiology milestones: implementation of self-evaluation in a large
residency program. J Clin Anesth 2016;32:17–24.
Yamamoto S, Tanaka P, Madsen MV, et al. Comparing anesthesiology residency training structure and requirements in seven different countries on three continents. Cureus 2017;9:e1060.
Durante E. Algunos métodos de evaluación de las competencias: Escalando la pirámide de Miller. Revista del Hospital Italiano 2006;55–61.
Ebert TJ, Fox CA. Competency-based education in anesthesiology: history and challenges. Anesthesiology 2014;120:24–31.
Frost E. Comprehensive Guide to Education in Anesthesia. New York: Springer science + business Media; 2014.
Baker K. Determining resident clinical performance: getting beyond the noise. Anesthesiology 2011;115:862–878.
Boulet JR, Murray D. Review article: assessment in anesthesiology education. J Can Anesth 2012;59:182–192.
The Accreditation Council for Graduate Medical Education and The American Board of Anesthesiology. The Anesthesiology Milestone Project. [Internet]. Accreditation Council for Graduate Medical Education. [Cited 2020 May 10]. Available at: https://www.acgme.org/
The Royal College of Physicians and Surgeons of Canada [Internet]. Royal College of Physicians and Surgeons of Canada. [Cited 2020 May 12]. Available at: http://www.royalcollege.ca/rcsite/canmeds-e
The Standing Committee on Education and Professional Development of the Section and Board of Anaesthesiology. European Training Requirement ETR in Anesthesiology [Internet]. Available at: https://www.uems.eu/about-us/medical-specialties
Larsson J, Holmström I. Understanding anesthesia training and trainees. Curr Opin Anaesthesiol 2012;25:681–685.
Gessel EV, Mellin-Olsen J, Østergaard HT, et al. Postgraduate training in anaesthesiology, pain and intensive care. Eur J Anaesthesiol 2012;29:165–168.
Chiu M, Crooks S, Tarshis J, et al. Simulation-based assessment of anesthesiology residents’ competence: development and implementation of the Canadian National Anesthesiology Simulation Curriculum (CanNASC). Can J Anesth 2016;63:1357–1363.
Carlsson C, Keld D, Van Gessel E, et al. Education and training in anaesthesia – revised guidelines by the European Board of anaesthesiology, reanimation and intensive care: SECTION and BOARD OF ANAESTHESIOLOGY1, European Union of Medical specialists. Eur J Anaesthesiol 2008;25:528–530.
Rebel A, Dilorenzo A, Nguyen D, et al. Should objective structured clinical examinations assist the clinical competency committee in assigning anesthesiology milestones competency? Anesth Analg 2019;129:226–234.
Cate OT. Entrustability of professional activities and competency based training. Med Educ 2005;39:1176–1177.
Cate OT. Nuts and bolts of entrustable professional activities. J Grad Med Educ 2013;5:157–158.
Wisman-Zwarter N, Schaaf MVD, Cate OT, et al. Transforming the learning outcomes of anaesthesiology training into entrustable professional activities. Eur J Anaesthesiol 2016;33:559–567.
Jonker G, Hoff RG, Cate OTJT. A case for competency-based anaesthesiology training with entrustable professional activities. Eur J Anaesthesiol 2015;32:71–76.
The Anesthesiology Milestone Project. Assessment of procedural skills in anesthesiology trainees: changing trends. J Grad Med Educ 2014;6 (1 suppl 1):15–28.
Sivaprakasam J, Purva M. CUSUM analysis to assess competence: what failure rate is acceptable? Clin Teach 2010;7:257–261.
Neira VM, Bould MD, Nakajima A, et al. GIOSAT: a tool to assess CanMEDS competencies during simulated crises. Can J Anesth 2013;60:280–289.
Jaramillo S, Vargas R. Cañadas R, Vargas R, Rinc on R, et al. Cómo aprenden los adultos: una Aproximación desde la enseñanza médica. Currículo nuclear en endoscopia digestiva: fundamentos teóricos y propuesta curricular Bogotá: Panamericana; 2018; 15–25.
Sociedad Colombiana de Anestesiología y Reanimación. Documento marco del Plan de Estudios y Competencias para un Programa de Anestesiología en Colombia. Bogotá: SCARE; 2017.
Van der Vleuten C, Schuwirth L, Driessen E, et al. A model for programmatic assessment fit for purpose. Med Teach 2012;34:205–214.
Van Der Vleuten CPM, Schuwirth LWT, Scheele F, et al. The assessment of professional competence: building blocks for theory development. Best Pract Res Clin Obstet Gynaecol 2010; 24:703–719.
Bilotta F, Titi L, Lanni F, et al. Training anesthesiology residents in providing anesthesia for awake craniotomy: learning curves and estimate of needed case load. J Clin Anesth 2013;25:359–366.
Ramírez LJ, Moreno MA, Gartdner L, et al. Modelo de enseñanza de las habilidades psicomotoras básicas en anestesia para estudiantes de ciencias de la salud: sistematización de una experiencia. Colombian Journal of Anesthesiology 2008;36:85–92.
Aguirre Ospina OD, Ríos Medina ÁM, Calder on Marulanda M, et al. Cumulative Sum learning curves (CUSUM) in basic anaesthesia procedures. Colombian Journal of Anesthesiology 2014;42:142–153.
Stiegler MP, Tung A. Cognitive processes in anesthesiology decision making. Anesthesiology 2014;120:204–217.
Enser M, Moriceau J, Abily J, et al. Background noise lowers the performance of anaesthesiology residents’ clinical reasoning when measured by script concordance. Eur J Anaesthesiol 2017;34:464–470.
Echevarría Moreno M, Prieto Vera C, Martin Telleria A, et al. The objective structured clinical evaluation of teaching in anaesthesiology and resuscitation. Rev Esp Anestesiol Reanim 2012;59:134–141.
Ben-Menachem E, Ezri T, Ziv A, et al. Objective structured clinical examination-based assessment of regional anesthesia skills: the israeli national board examination in anesthesiology experience. Anesth Analg 2011;112:242–245.
Ahmed O, O’Donnell B, Gallagher A, et al. Development of performance and error metrics for ultrasound-guided axillary brachial plexus block. Adv Med Educ Pract 2017;5:257–263.
Cheung JJH, Chen EW, Darani R, et al. The creation of an objective assessment tool for ultrasound-guided regional anesthesia using the delphi method. Reg Anesth Pain Med 2012;37:329–333.
Chin KJ, Tse C, Chan V, et al. Hand motion analysis using the Imperial College surgical assessment device: validation of a novel and objective performance measure in ultrasound-guided peripheral nerve blockade. Reg Anesth Pain Med 2011;36:213–219.
Chuan A, Thillainathan S, Graham PL, et al. Reliability of the direct observation of procedural skills assessment tool for ultrasoundguided regional anaesthesia. Anaesth Intensive Care 2016;44:201–209.
Watson MJ, Wong DM, Kluger R, et al. Psychometric evaluation of a direct observation of procedural skills assessment tool for ultrasound-guided regional anaesthesia. Anaesthesia 2014;69:604–612.
Laurent DA, Niazi A, Cunningham M, et al. A valid and reliable assessment tool for remote simulation-based ultrasound-guided regional anesthesia. Reg Anesth Pain Med 2014;39:496–501.
Corvetto MA, Fuentes C, Araneda A, et al. Validation of the imperial college surgical assessment device for spinal anesthesia. BMC Anesthesiol 2017;17:131.
Chuan A, Wan AS, Royse C, et al. Competency-based assessment tools for regional anaesthesia: a narrative review. Br J Anaesth 2017;120:264–273.
Chuan A, Graham PL, Wong DM, et al. Design and validation of the Regional Anaesthesia Procedural Skills Assessment Tool. Anaesthesia 2015;70:1401–1411.
Hastie MJ, Spellman JL, Pagnano PP, et al. Designing and implementing the objective structured clinical examination in anesthesiology. Anesthesiology 2014;120:196–203.
Farnan JM, Petty LA, Georgitis E, et al. A systematic review: the effect of clinical supervision on patient and residency education outcomes. Acad Med 2012;87:428–442.
Moore DL, Ding L, Sadhasivam S. Novel real-time feedback and integrated simulation model for teaching and evaluating ultrasound-guided regional anesthesia skills in pediatric anesthesia trainees. Paediatr Anaesth 2012;22:847–853.
Riveros R, Kimatian S, Castro P, et al. Multisource feedback in professionalism for anesthesia residents. J Clin Anesth 2016; 34:32–40.
Smith SE, Tallentire VR. The right tool for the right job: the importance of CUSUM in self-assessment. Anaesthesia 2011; 66:747.
Gaba DM, Howard SK, Flanagan B, et al. Assessment of clinical performance during simulated crises using both technical and behavioral ratings. Anesthesiology 1998;89:8–18.
Gaba DM, Howard SK, Gan BF, et al. Assessment of clinical performance during simulated crises using both technical and behavioral ratings. Surv Anesthesiol 1999;43:111–112.
Murray DJ, Boulet JR, Avidan M, et al. Performance of residents and anesthesiologists in a simulation-based skill assessment. Anesthesiology 2007;107:705–713.
Murray DJ, Boulet JR, Kras JF, et al. A simulation-based acute skills performance assessment for anesthesia training. Anesth Analg 2005;101:1127–1134.
Rábago JL, L opez-Doueil M, Sancho R, et al. Learning outcomes evaluation of a simulation-based introductory course to anaesthesia. Rev Esp Anestesiol Reanim 2017;64:431–440.
Fehr JJ, Boulet JR, Waldrop WB, et al. Simulation-based assessment of pediatric anesthesia skills. Anesthesiology 2011;115:1308–1315.
Lammers RL, Davenport M, Korley F, et al. Teaching and assessing procedural skills using simulation: metrics and methodology. Acad Emerg Med 2008;15:1079–1087.
Rebel A, DiLorenzo A, Fragneto RY, et al. Objective assessment of anesthesiology resident skills using an innovative competition based simulation approach. A A Case Rep 2015;5:79–87.
Schwid HA, Rooke GA, Carline J, et al. Evaluation of anesthesia residents using mannequin-based simulation: a multiinstitutional study. Anesthesiology 2002;97:1434–1444.
Byrne AJ, Greaves JD. Assessment instruments used during anaesthetic simulation: review of published studies. Br J Anaesth 2001;86:445–450.
Corvetto MA, Bravo MP, Montaña RA, et al. Bringing clinical simulation into an anesthesia residency training program in a university hospital. Participants’ acceptability assessment. Rev Esp Anestesiol Reanim 2013;60:320–326.
Hindman BJ, Dexter F, Smith TC. Anesthesia residents’ global (departmental) evaluation of faculty anesthesiologists’ supervision can be less than their average evaluations of individual anesthesiologists. Anesth Analg 2015;120:204–208.
Mitchell JD, Holak EJ, Tran HN, et al. Are we closing the gap in faculty development needs for feedback training? J Clin Anesth 2013;25:560–564.
O’Sullivan O, Shorten GD. Formative assessment of ultrasoundguided regional anesthesia. Reg Anest Pain Med 2011;36: 522–523.
Hindman BJ, Dexter F, Kreiter CD, et al. Determinants, associations, and psychometric properties of resident assessments of anesthesiologist operating room supervision. Anesth Analg 2013;116:1342–1351.
Bindal N, Goodyear H, Bindal T, et al. DOPS assessment: a study to evaluate the experience and opinions of trainees and assessors. Med Teach 2013;35:e1230–e1234.
De Oliveira Filho GR, Dal Mago AJ, Garcia JHS, et al. An instrument designed for faculty supervision evaluation by anesthesia residents and its psychometric properties. Anesth Analg 2008;107:1316–1622.
Ahmed K, Miskovic D, Darzi A, et al. Observational tools for assessment of procedural skills: a systematic review.AmJ Surgery 2011;202:469–80e6.
Rebel A, DiLorenzo AN, Fragneto RY, et al. A competitive objective structured clinical examination event to generate an objective assessment of anesthesiology resident skills development. A Case Rep 2016;6:313–319.
Norris A, McCahon R. Cumulative sum (CUSUM) assessment and medical education: a square peg in a round hole. Anaesthesia 2011;66:250–254.
Khaliq T. Reliability of results produced through objectively structured assessment of technical skills (OSATS) for endotracheal intubation (ETI). J Coll Physicians Surg Pak 2013;23:51–55.
Flin R, Patey R, Glavin R, et al. Anaesthetists’ non-technical skills. Br J Anaesth 2010;105:38–44.
Flin R, Patey R. Non-technical skills for anaesthetists: developing and applying ANTS. Best Pract Res Clin Anaesthesiol 2011;25:215–227.
Graham J, Hocking G, Giles E. Anaesthesia non-technical skills: can anaesthetists be trained to reliably use this behavioural marker system in 1 day? Br J Anaesth 2010;104:440–445.
Ahmed A. Assessment of procedural skills in anesthesiology trainees: changing trends. Anaesth, Pain & Intensive Care 2014;18:135–36.
Bould MD, Crabtree NA, Naik VN. Assessment of procedural skills in anesthesia. Br J Anaesth 2009;103:472–483.
Witt A, Iglesias S, Ashbury T. Evaluation of Canadian family practice anesthesia training programs: can the resident logbook help? Can J Anesth 2012;59:968–973.
Weller JM, Castanelli DJ, Chen Y, et al. Making robust assessments of specialist trainees’ workplace performance. Br J Anaesth 2017;118:207–214.
Weller JM, Jones A, Merry AF, et al. Investigation of trainee and specialist reactions to the mini-clinical evaluation exercise in anaesthesia: implications for implementation. Br J Anaesth 2009;103:524–530.
Kathirgamanathan A, Woods L. Educational tools in the assessment of trainees in anaesthesia. Contin Educ Anaesth Crit Care Pain 2011;11:138–142.
Castanelli DJ, Castanelli DJ, Jowsey T, et al. Perceptions of purpose, value, and process of the mini-clinical evaluation exercise in anesthesia training. Can J Anesth 2016;63:1345–1356.
Colbert-Getz J, Ryan M, Hennessey E, et al. Measuring assessment quality with an assessment utility rubric for medical education. MedEdPORTAL 2017;13:10588.
Van Meeuwen LW, Brand-Gruwel S, Kirschner PA, et al. Fostering self-regulation in training complex cognitive tasks. Educ Tech Res Dev 2018;66:53.
Downloads
| Article metrics | |
|---|---|
| Abstract views | |
| Galley vies | |
| PDF Views | |
| HTML views | |
| Other views | |
