Development and evaluation of an assessment tool for identification of elements of situation awareness in OSCEs in undergraduate medical curricula

Fischer, Markus Andreas
Situation awareness (SA) is recognised as an important factor in patient safety and clinical decision making in conjunction with diagnostic and therapeutic reasoning. Missed diagnosis and treatment errors have been identified as intensely influenced by cognitive abilities of clinical physicians. Medical education aims to qualify physicians for diagnostic accuracy and the ability to select the most appropriate treatment. Research accentuated that medical students have little insight into cognitive processing and SA in clinical scenarios. Furthermore, literature does not suggest a methodology to quantify students´ cognitive processing in a clinical encounter. SA is described as having the proficiency to obtain awareness of the surrounding and to integrate this consciousness into the situational context and potential future development. The reduction of fatal outcomes and critical incidents in high-risk environments such as aviation, was attributed to an acknowledgement of the impact of SA on task performance. Pilots undergo practical simulation exercises mirroring critical scenarios during their training and subsequent professional careers. Endsley`s model explains SA as an interdependent three-levelled concept, which enables one to distinguish the individual processes and the product of these processes when evaluating the cognitive abilities of examinees. The model has been used amongst multiple medical disciplines for training clinicians utilising simulation scenarios. However, the educational benefit has not achieved an overall integration of the concept of SA into medical curricula. Objective Structured Clinical Examinations (OSCEs) have been suggested to facilitate the assessment of certain elements of SA similar to high-risk simulation. Though, the underlying cognitive processing to obtain and maintain SA is difficult to measure and to quantify. Medical educators and clinical tutors might not be aware of essential steps in collecting, interpreting and incorporating necessary information during patient encounters. The initial purpose of this thesis was to review the literature with a view to identifying whether levels of SA based on Endsley’s model can be assessed utilising OSCEs during undergraduate medical training. Therefore, a systematic search was performed pertaining to SA and OSCEs, to identify studies published between January 1975 (first paper describing an OSCE) and February 2017, in peer reviewed international journals published in English. Selected databases were searched for papers that described the assessment of SA using OSCEs among undergraduate medical students. Findings suggested that whole-task OSCEs enable the evaluation of SA associated with clinical reasoning skills. Furthermore, if these assessments address the levels of SA, these OSCEs can provide supportive feedback and strengthen educational measures associated with higher diagnostic accuracy and reasoning abilities. This suggests an early exposure of medical students to OSCEs mirroring clinical practice to evaluate and facilitate SA in clinical encounters. At the same time, it highlights the need for examiners and developers of these whole-task assessments to be able to understand the model of SA and to identify assessment criteria which can be assigned to the sequence and the product of the cognitive information processing. Upon the identification of elements of OSCE assessments which can be assigned to the levels of SA, the next study explored the ability of clinical professionals to identify and characterise SA in OSCE guides and OSCE score sheets. Due to the unavailability of an instrument, a self-developed tool was used, thereby yielding an inferential measure of SA. The outcome of the study revealed a strong internal validity of the tool, however, only a moderate interrater reliability has been identified. In order to improve the level of interrater agreement, and thus the objectivity of the measurement tool, a consecutive guided training was developed. An expected outcome was the beneficial impact on assessors’ ability to select parameters of OSCE assessments which can be assigned as process markers for SA or the level achieved in this type of assessment. In order to investigate underlying factors for any disagreement between raters rather than their subjectivity in evaluating OSCE forms, the Generalisability Theory (G- Theory) was suggested. The evaluation study carried out upon completing of the training programme indicated the beneficial influence on the ability of raters to identify and categorise elements of SA within OSCE forms. The G-Theory revealed key facets for variance: OSCE score sheets, Levels of SA, Items embedded in the Levels of SA, Interaction between Forms and Levels and Forms and Items embedded within Levels. Thus, it was demonstrated that the consecutive guided training improved the identification of elements within OSCE assessments, which can be attributed to individual levels of SA. Outcomes of the G-Study highlighted the need to improve the readability of OSCE forms, suggested to be achieved by a clear description of the assessment markers for students` performance and behaviour. Further research can contribute to fostering the development of educational and assessment strategies in undergraduate medical curricula for SA, and thus, improving patient safety.
NUI Galway
Publisher DOI
Attribution-NonCommercial-NoDerivs 3.0 Ireland