Year : 2013 | Volume
| Issue : 2 | Page : 122-127
Assessment of current undergraduate anesthesia course in a Saudi University
Tarek Shams1, Ragaa El-Masry2, Hamed al Wadani3, Mostafa Amr4
1 Department of Anesthesia and ICU, Faculty of Medicine, Mansoura University, Egypt
2 Department of Public Health and Community Medicine, Faculty of Medicine, Mansoura University, Egypt
3 Department of Surgery, Faculty of Medicine, King Faisal University, Saudi Arabia
4 Department of Psychiatry, Faculty of Medicine, Mansoura University, Egypt
Department Anesthesia and ICU, Faculty of Medicine, Mansoura University
Source of Support: None, Conflict of Interest: None
|Date of Web Publication||26-Jun-2013|
Background: The assessment of the anesthesia course in our university comprises Objective Structured Clinical Examinations (OSCEs), in conjunction with portfolio and multiple-choice questions (MCQ). The objective of this study was to evaluate the outcome of different forms of anesthesia course assessment among 5 th year medical students in our university, as well as study the influence of gender on student performance in anesthesia. Methods: We examined the performance of 154, 5 th year medical students through OSCE, portfolios, and MCQ. Results: The score ranges in the portfolio, OSCE, and MCQs were 16-24, 4.2-28.9, and 15.5-44.5, respectively. There was highly significant difference in scores in relation to gender in all assessments other than the written one (P=0.000 for Portfolio, OSCE, and Total exam, whereas P=0.164 for written exam). In the generated linear regression model, OSCE alone could predict 86.4% of the total mark if used alone. In addition, if the score of the written examination is added, OSCE will drop to 57.2% and the written exam will be 56.8% of the total mark. Conclusions: This study demonstrates that different clinical methods used to assess medical students during their anesthesia course were consistent and integrated. The performance of female was superior to male in OSCE and portfolio. This information is the basis for improving educational and assessment standards in anesthesiology and for introducing a platform for developing modern learning media in countries with dearth of anesthesia personnel.
Keywords: Anesthesiology, assessment, gender, objective structured clinical examination, portfolios and multiple-choice questions
|How to cite this article:|
Shams T, El-Masry R, al Wadani H, Amr M. Assessment of current undergraduate anesthesia course in a Saudi University. Saudi J Anaesth 2013;7:122-7
|How to cite this URL:|
Shams T, El-Masry R, al Wadani H, Amr M. Assessment of current undergraduate anesthesia course in a Saudi University. Saudi J Anaesth [serial online] 2013 [cited 2020 Oct 20];7:122-7. Available from: https://www.saudija.org/text.asp?2013/7/2/122/114049
| Introduction|| |
In Saudi Arabia, the College of Medicine at King Faisal University offers a 6 year medical curriculum to selected Saudi students who have successfully completed 1 year of requisite general university studies following secondary school education. The first 4 years of the curriculum are devoted to preclinical (medical sciences and family medicine).  The anesthesia course involves fifteen topics in anesthesia and intensive care for 5th year students. It covers seven basic concepts in anesthesia and intensive care including general anesthesia, anesthetic agents, local anesthesia, postoperative pain management, fluid therapy, blood transfusion, and basic life support.
The assessment of the course comprises Objective Structured Clinical Examinations (OSCE), in conjunction with portfolio andmultiple-choice questions (MCQ). The maximum marks for the written paper MCQ, portfolio, and OSCE are 45, 25, and 30, respectively, making a total of 100 points.
The OSCE has been well received by the students and is perceived by them to be a fair reflection of their level of knowledge and skill attainment during the course.  OSCEs have enjoyed much popularity since their introduction in 1975 by Harden, which replaced the clinical.  In our university, OSCE was introduced in the academic year 2011-2012, which is why we conducted this research to evaluate these multi-assessment tools. Because of its validity and reliability, OSCE has been increasingly used for the assessment of medical students 'clinical skills. , A preliminary study in our institution in psychiatry applied the OSCE to 5 th year medical students and reported that OSCE is a reliable measure in the formative assessment of students. 
Leung et al.  have used an adaptation of the OSCE to assess learning. Question stations included the use of manikins to assess practical skills, such as cardiopulmonary resuscitation and airway management, identification and description of the use of equipment, and interpretation of clinical scenarios and investigations. They believe that adaptation of the OSCE is better than traditional methods of examination and it has allowed identifying shortcomings in teaching methods.
Earlier studies suggest that female undergraduates perform better than male undergraduates in clinical clerkship. Amr and Amin  showed non-significant differences in male and female attainment in psychiatry in our institute. Our subjective impressions and a previous study suggested that women had more positive attitude and performed better than males in King Faisal University.  Moreover, there is a rising trend in German universities to address the structure of the respective curriculum, learning goals, teaching, assessment and evaluation methods, and facultative courses. 
The objective of this study was to evaluate the outcome of different forms of anesthesia course assessment among 5 th year medical students and at the same time study the influence of gender on student performance at King Faisal University, Saudi Arabia.
| Methods|| |
Male students attended the first semester ( n=90) and females ( n=64) attended the second one. Lectures in each semester have to cover general anesthesia, anesthetic agents, local anesthesia, postoperative pain management, fluid therapy, blood transfusion, and basic life support. The students in each semester were classified into three batches, with each batch divided into three sub-batches. Each sub-batch had 5 sessions, including one in the simulation lab where students are allowed to practice basic airway management and spinal analgesia on a mannequin and at the same time familiarize themselves with the anesthetic instruments prior to being sent to operating room OR. Three hours in addition to half an hour break each day for 4 days are spent in OR. The students will be encouraged in OR to participate in preoperative assessment, anesthesia machine, monitoring, intraoperative anesthetic management including intubation and regional analgesia, fulfilled anesthesia sheet, and patient management in postanesthesia care unit (PACU) were also recorded. There were presentation exams at the end of 5 th day, which were assessed by 2 faculty anesthesia staff. Portfolio also included checking the logbook, individual group discussions, and attendance. OSCEs were conducted at the end of clinical rotation for each batch (5 weeks); moreover, a quiz and MCQs in addition to two essays (5 marks for all batches) were conducted at the same time at the end of each semester. Short essays are corrected by 2 faculty staff. There is a similar assessment methods for male and female students.
The MCQ paper at each examination contained 20 questions, with each item carrying one and half mark. The initial item bank of 500 questions was designed to cover all lectures. Two to three items were included to represent each lecture. One item was answered through simple recall, and the other was designed to be answered interpretatively and commonly involved a brief, one-to four-sentence case vignette. Each MCQ item consisted of a stem of short sentence, along with four response options. Haladyna et al. developed test items following standard, well-described MCQ writing procedures, and they were designed to avoid ambiguity, vagueness, and value-laden language.  Reliability (Cronbach's alpha) and concurrent validity (Pearson r) coefficients were obtained by correlating the scores of MCQ papers with the overall outcome of the examination. The MCQ answer sheet was corrected by the correction machine.
OSCE was developed to capture the clinical competencies of students and involved seven main items: Anesthesia machine, monitoring, instruments, spinal anesthesia, general anesthesia, drugs, and basic life support. The OSCE comprisessix questions providing 6 min each, in addition to 6 min revision and any photographs used were to be displayed in both the OSCE paper and two large screens. Portfolio consisted of multiple presentations of the students combined with a logbook, feedback. The assessment of the student included 7 items that were scored from a total of 15 marks. Logbook is assessed and scored from a total of 5 marks. Feedback and attendance include 5 marks, and if attendance is less than 75%, the student will not be allowed to attend any anesthesia exam and has to repeat the course in the nearest semester according to university regulations.
Medical students' perceptions of anesthesia-teaching evaluations were evaluated by a questionnaire that was based on the survey conducted by Ursula and Galway, 2010,  where the students expressed their opinion of their clerkship on a five-point Likert scale, 1 being strongly disagree and 5 being strongly agree. The questionnaire included statements targeting students' perceptions around the following themes:
Data analysis was carried out using the Statistical Software for the Social Sciences package (Version 11, Chicago, Illinois, USA). Median, minimum, maximum, mean, and standard deviations were calculated for examination marks. Statistical comparison was carried out using the Mann-Whitney test. Zero order and partial correlations were performed between test marks, and regression models were fitted to evaluate the predictive value of the total clinical score or total final marks as the dependent variables. To assess reliability and credibility, Cronbach's alpha, Kappa, and Pearson's correlation coefficient were used.
- My expectations for this rotation were met
- My expectations for this rotation were exceeded
- This rotation has increased my desire to pursue a career in anesthesiology
- This rotation has helped improve my skills that relate to all areas of medicine (airway skills, intravenous line placement, etc.)
- This rotation has helped improve my knowledge of anesthesia
- This rotation has helped improve my knowledge of general medicine.
| Results|| |
[Table 1] displays the students' scores along with the different assessment methods used to evaluate the outcome. The score ranges in the portfolio, OSCE, and MCQs were 16-24, 4.2-28.9, and15.5-44.5, respectively. There was a highly significant difference in scores regarding genders, with the exception of insignificant difference for the written exam. Strong positive correlations were found between all forms of anesthesia examinations ( r=0.000). [Table 2], [Figure 1], [Figure 2], [Figure 3] present the correlations of different types of assessment forms in the anesthesia course.
|Figure 1: Correlation between objective structured clinical examination and portfolio exam in anesthesia course, College of Medicine, King Faisal University|
Click here to view
|Figure 2: Correlation between written and portfolio exam in anesthesia course, College of Medicine, King Faisal University|
Click here to view
|Figure 3: Correlation between written exam and objective structured clinical examination in anesthesia course, College of Medicine, King Faisal University|
Click here to view
|Table 1: Students' scores along with the different types of assessment in anesthesia course|
Click here to view
[Table 3] displays student anesthesia satisfaction. There were highly significant differences for gender in the following areas: "Expectations were met", "Increased the desire to have anesthesiology career", and "Improved knowledge of general medicine"(0.005, 0.000, and 0.004, respectively). There were significant differences in "Expectations was exceeded" and "Improved invasive skills" (0.024 and 0.017, respectively) but insignificant difference with regard to "Improved knowledge of anesthesia Females showed higher mean satisfaction scores than males with the exception of "Decreased the desire to have anesthesiology a career".
The Kappa concordance coefficient and the correlation between the scores of examinees were computed; they ranged from 0.79 to 0.68. The overall Cronbach's alpha coefficient was 0.872, Cronbach's alpha for portfolio exam, OSCE, and MCQs in case of corrected item total correction were0.8471, 0.8136, and 0.7781, respectively and if the item was deleted it was 0.8910, 0.8505, and 0.8298, respectively. In the generated linear regression model, OSCE alone could predict 86.4% of the total marks. In addition, if the written exam is included, OSCE will drop to 57.2% and the written exam will be 56.8% of the total marks. Moreover, if we add portfolio, written exam, OSCE, and portfolio, respectively, it will account for 54%, 49.8%, and 15.7%, respectively, of the variance in total marks ( P=0.001).
| Discussion|| |
Currently, medical student curricula are focusing less on mastery of knowledge and clinical skills and more on achieving multiple competencies that will provide students with a solid foundation to practice in complex healthcare environments. Anesthesiologists are uniquely positioned to teach towards a number of competencies. To do so, innovations in the traditional assessments need to be considered. Assessment of competency in traditional graduate medical education has been based on observation of clinical care and tests that measure the effectiveness of didactic teaching. Assessment of anesthesia in our university is in the form of OSCE, MCQs, and portfolio. In anesthesiology, direct observation of student performance by staff is the norm, and assessment of competence is often based on global impression. 
The acquisition of clinical skills is paramount to the development of a safe and competent practitioner.  OSCE as a performance-based assessment is a well-established assessment tool for several reasons: Competency-based valid, practical, and effective means of assessing clinical skills that are fundamental to the practice of medicine and other healthcare-related professions.  OSCE is in use in many medical disciplines in Saudi Arabia, particularly in general surgery, , orthopedics,  internal medicine, and psychiatry. 
Findings from this study showed that the results of the MCQs are the most important predictors of final scores, as they accounted for 54% of student variability, followed by OSCE. It has been observed that general ability is the foundation of most performance measures and a well-constructed MCQ is the best estimator of this general ability (covers the area of 'knows' and 'knows how' of Miller's pyramid of assessment) and this could span the levels of Bloom's taxonomy of educational objectives from the level of comprehension to the level of evaluation. However, OSCE covers the area of 'shows how' of the Miller's pyramid of assessment, which is a prerequisite for physician performance in real life, such as history taking and physical examinations. ,
The most important finding of our work is that there were highly significant differences in portfolio and OSCE scores regarding gender, with the exception of the written exam (MCQs).
Previous studies demonstrated that female students showed greater adaptability to clinical situations and concepts.  Other possible factors could be social factors, because female students were less influenced by negative and external peer pressure, and had to prove themselves in a male-dominated society, which resulted in them being more hardworking and motivated. 
Female students also possess better vigilance, perceptual speed, and associative memory and are also better at scanning the physical environment.  These might have provided them with an edge over males; consequently, they show a better ability to recall more details from learning exposure in anesthesia. The brains of men and women do differ physically; mens' brains are larger womens' by about 8-10% and possess deeper fissures and sulci.  At a cellular level, women have 9 times more white matter in areas of the brain associated with intelligence than men. ,,, Moreover Morley and Toga attributed gender difference to endocrinal factors. ,
On the other hand, this finding contradicted a previous study that no statistically significant difference exists between male and female students regarding OSCE, portfolio, and MCQs in the psychiatry department in our university. ,, In addition, females had higher significant mean satisfaction scores than males, with the exception of decreased desire to pursue anesthesiology as a career, where males had the upper hand; there was also insignificant gender difference regarding improved knowledge of anesthesia.
| Conclusion|| |
Our study provides detailed information about the current status of undergraduate evaluation of anesthesiology in Saudi University. It shows a remarkable consistency of structure, contents, and methods of assessment in addition to the positive influence of anesthesia clerkship on changing the attitude of medical students towards anesthesia specialty in a developing country This information is the basis for triggering synergistic effects, for improving educational and assessment standards in anesthesiology, and for introducing a platform for developing modern learning media in countries with a dearth of anesthesia personnel.
| References|| |
|1.||Amr M, Amin T. Assessment methods of an undergraduate psychiatry course at a saudi university. Sultan Qaboos Univ Med J 2012;12:214-20. |
|2.||El-Gilany AH, Amr M, Iqbal R. Students' attitudes toward psychiatry at Al-hassa medical college, Saudi Arabia. Acad Psychiatry 2010;34:71-4. |
|3.||Vaidya NA. Psychiatry clerkship Objective Structured Clinical Examination is here to stay. Acad Psychiatry 2008;32:177-9. |
|4.||Hodges B, Hanson M, McNaughton N, Regehr G. University of Toronto Psychiatric Skills Assessment Project. Creating, monitoring, and improving a psychiatry OSCE: A guide for faculty. Acad Psychiatry 2002;26:134-61. |
|5.||Leung JY, Critchley LA, Yung AL, Kumta SM. Introduction of virtual patients onto a final year anesthesia course: Hong Kong experience. Adv Med Educ Pract 2011;2:71-83. |
|6.||Scott J, Allardyce J, Ferrier N. Gender differences in undergraduate performance in psychiatry. Psychiatr Bulletin 1993;17:599-600. |
|7.||Haladyna TM, Downing SM. A taxonomy of multiple choice item-writing rules. Appl Meas Educ 1989;2:37-50. |
|8.||Galway UA. designing an optimally educational anesthesia clerkship for medical students - Survey results of a new curriculum. JEPM 2012;12:1-812. |
|9.||Sullivan KR, Rollins MD. Innovations in anaesthesia medical student clerkships. Best Pract Res Clin Anaesthesiol 2012;26:23-32. |
|10.||Zyromski NJ, Staren ED, Merrick HW. Surgery residents' perception of the Objective Structured Clinical Examination (OSCE). Curr Surg 2003;60:533-7. |
|11.||Bakhsh TM, Sibiany AM, Al-Mashat FM, Meccawy AA, Al-Thubaity FK. Comparison of students' performance in the traditional oral clinical examination and the objective structured clinical examination. Saudi Med J 2009;30:555-7. |
|12.||Guraya S, Alzobydi A, Salman S. Objective structured clinical examination: Examiners' bias and recommendations to improve its reliability. J Med Med Sci 2010;1:269-72. |
|13.||Almoallim H. (OSCE) in Internal medicine for undergraduate students newly encountered with clinical training. Med Ed Portal; 2006. Available from: http://www.mededportal.org/publication/429. [Last accessed on 2006 Dec 7]. |
|14.||Hijazi M, Downing SM. Objective structured clinical examinations as an assessment method in residency training: Practical considerations. Ann Saudi Med 2008;28:192-9. |
|15.||Fabrega H Jr, Ulrich R, Keshavan M. Gender differences in how medical students learn to rate psychopathology. J Nerv Ment Dis 1994;182:471-5. |
|16.||Maatsch JL, Huang RR, Downing S, Barker D, Munger B. The predictive validity of test formats and a psychometric theory of clinical competence. Res Med Educ 1984;23:76-82. |
|17.||Amr M, Raddad D, Afifi Z. Objective structured clinical examination (OSCE) during psychiatry clerkship in a Saudi university. Arab J Psychiatry 2012;23:69-73. |
|18.||Al-Tamimi DM. Saudi women in academic medicine. Are they succeeding? Saudi Med J 2004;25:1564-7. |
|19.||Boulet JR, McKinley DW, Norcini JJ, Whelan GP. Assessing the comparability of standardized patient and physician evaluations of clinical skills. Adv Health Sci Educ Theory Pract 2002;7:85-97. |
|20.||Deepak KK, Al-Umran KU, Al-Sheikh MH, Al-Rubaish A. The influence of gender on undergraduate performance in multiple choice testing in clinical disciplines at University of Dammam, Saudi Arabia. Al Ame en J Med Sci 2011;4:123-30. |
|21.||Haier RJ, Jung RE, Yeo RA, Head K, Alkire MT. The neuroanatomy of general intelligence: Sex matters. Neuroimage 2005;25:320-7. |
|22.||Luders E, Narr KL, Thompson PM, Rex DE, Jancke L, Steinmetz H, et al. Gender differences in cortical complexity. Nat Neurosci 2004;7:799-800. |
|23.||Benbow CP, Lubinski D, Shea DL, Eftekhari-Sanjani H. Sex differences in mathematical reasoning ability at age 13: Their status 20 years later. Psychol Sci 2000;11:474-80. |
|24.||Leagey E. Gender differences in mathematical trajectories. Soc Forces 2001;80:713-32. |
|25.||Morley KI, Montgomery GW. The genetics of cognitive processes: Candidate genes in humans and animals. Behav Genet 2001;31:511-31. |
|26.||Toga AW, Thompson PM. Genetics of brain structure and intelligence. Annu Rev Neurosci 2005;28:1-23. |
[Figure 1], [Figure 2], [Figure 3]
[Table 1], [Table 2], [Table 3]