van Der P.M

van Der P.M
Are you van Der P.M?

Claim your profile, edit publications, add additional information:

Contact Details

van Der P.M

Publications Authored By van Der P.M

Narrative feedback documented in performance evaluations by the teacher, i.e. the clinical supervisor, is generally accepted to be essential for workplace learning.Read More

Many studies have examined factors of influence on the usage of mini-clinical evaluation exercise (mini-CEX) instruments and provision of feedback, but little is known about how these factors influence teachers' feedback-giving behaviour. In this study, we investigated teachers' use of mini-CEX in performance evaluations to provide narrative feedback in undergraduate clinical training.
We designed an exploratory qualitative study using an interpretive approach. Focusing on the usage of mini-CEX instruments in clinical training, we conducted semi-structured interviews to explore teachers' perceptions. Between February and June 2013, we conducted interviews with 14 clinicians participated as teachers during undergraduate clinical clerkships. Informed by concepts from the literature, we coded interview transcripts and iteratively reduced and displayed data using template analysis.
We identified three main themes of interrelated factors that influenced teachers' practice with regard to mini-CEX instruments: teacher-related factors; teacher-student interaction-related factors, and teacher-context interaction-related factors. Four issues (direct observation, relationship between teacher and student, verbal versus written feedback, formative versus summative purposes) that are pertinent to workplace-based performance evaluations were presented to clarify how different factors interact with each other and influence teachers' feedback-giving behaviour. Embedding performance observation in clinical practice and establishing trustworthy teacher-student relationships in more longitudinal clinical clerkships were considered important in creating a learning environment that supports and facilitates the feedback exchange.
Teachers' feedback-giving behaviour within the clinical context results from the interaction between personal, interpersonal and contextual factors. Increasing insight into how teachers use mini-CEX instruments in daily practice may offer strategies for creating a professional learning culture in which feedback giving and seeking would be enhanced.

Perspect Med Educ
Perspect Med Educ 2016 Oct;5(5):303-7
Department of Pathology, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, The Netherlands.

Programmatic assessment requires labour and cost intensive activities such as feedback in a quantitative and qualitative form, a system of learner support in guiding feedback uptake and self-directed learning, and a decision-making arrangement that includes committees of experts making a holistic professional judgment while using due process measures to achieve trustworthy decisions. This can only be afforded if we redistribute the resources of assessment in a curriculum. Several strategies are suggested.Read More

One is to introduce progress testing as a replacement for costly cognitive assessment formats in modules. In addition, all assessments should be replaced by assessment formats that are maximally aligned with the learning tasks. For performance-based assessment, OSCEs should be sparsely used, while education and work-embedded assessment should be maximized as part of the routine of ongoing instruction and assessment. Information technology may support affordable feedback strategies, as well as the creation of a paper trail on performance. By making more dramatic choices in the way we allocate resources to assessment, the cost-intensive activities of programmatic assessment may be realized.

Perspect Med Educ
Perspect Med Educ 2016 Oct;5(5):276-84
Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, The Netherlands.

Feedback after assessment is essential to support the development of optimal performance, but often fails to reach its potential. Although different assessment cultures have been proposed, the impact of these cultures on students' receptivity to feedback is unclear. This study aimed to explore factors which aid or hinder receptivity to feedback.Read More

Using a constructivist grounded theory approach, the authors conducted six focus groups in three medical schools, in three separate countries, with different institutional approaches to assessment, ranging from a traditional summative assessment structure to a fully implemented programmatic assessment system. The authors analyzed data iteratively, then identified and clarified key themes.
Helpful and counterproductive elements were identified within each school's assessment system. Four principal themes emerged. Receptivity to feedback was enhanced by assessment cultures which promoted students' agency, by the provision of authentic and relevant assessment, and by appropriate scaffolding to aid the interpretation of feedback. Provision of grades and comparative ranking provided a helpful external reference but appeared to hinder the promotion of excellence.
This study has identified important factors emerging from different assessment cultures which, if addressed by programme designers, could enhance the learning potential of feedback following assessments. Students should be enabled to have greater control over assessment and feedback processes, which should be as authentic as possible. Effective long-term mentoring facilitates this process. The trend of curriculum change towards constructivism should now be mirrored in the assessment processes in order to enhance receptivity to feedback.

Med Educ
Med Educ 2016 Sep;50(9):885-8
Department of Educational Development and Research, Maastricht University, Maastricht, the Netherlands.
Adv Health Sci Educ Theory Pract
Adv Health Sci Educ Theory Pract 2016 Sep 20. Epub 2016 Sep 20.
Centre for Health Education Scholarship, University of British Columbia, Vancouver, BC, Canada.

Whenever multiple observers provide ratings, even of the same performance, inter-rater variation is prevalent. The resulting 'idiosyncratic rater variance' is considered to be unusable error of measurement in psychometric models and is a threat to the defensibility of our assessments. Prior studies of inter-rater variation in clinical assessments have used open response formats to gather raters' comments and justifications.Read More

This design choice allows participants to use idiosyncratic response styles that could result in a distorted representation of the underlying rater cognition and skew subsequent analyses. In this study we explored rater variability using the structured response format of Q methodology. Physician raters viewed video-recorded clinical performances and provided Mini Clinical Evaluation Exercise (Mini-CEX) assessment ratings through a web-based system. They then shared their assessment impressions by sorting statements that described the most salient aspects of the clinical performance onto a forced quasi-normal distribution ranging from "most consistent with my impression" to "most contrary to my impression". Analysis of the resulting Q-sorts revealed distinct points of view for each performance shared by multiple physicians. The points of view corresponded with the ratings physicians assigned to the performance. Each point of view emphasized different aspects of the performance with either rapport-building and/or medical expertise skills being most salient. It was rare for the points of view to diverge based on disagreements regarding the interpretation of a specific aspect of the performance. As a result, physicians' divergent points of view on a given clinical performance cannot be easily reconciled into a single coherent assessment judgment that is impacted by measurement error. If inter-rater variability does not wholly reflect error of measurement, it is problematic for our current measurement models and poses challenges for how we are to adequately analyze performance assessment ratings.

Gibbs and Coffey (2004) have reported that teaching practices are influenced by teachers' conceptions of learning and teaching. In our previous research we found significant differences between teachers' conceptions in two medical schools with student-centred education. Medical school was the most important predictor, next to discipline, gender and teaching experience.Read More

Our research questions for the current study are (1) which specific elements of medical school explain the effect of medical school on teachers' conceptions of learning and teaching? How? and (2) which contextual and personal characteristics are related to conceptions of learning and teaching? How?
Individual interviews were conducted with 13 teachers of the undergraduate curricula in two medical schools. Previously their conceptions of learning and teaching were assessed with the COLT questionnaire. We investigated the meanings they attached to context and personal characteristics, in relation to their conceptions of learning and teaching. We used a template analysis.
Large individual differences existed between teachers. Characteristics mentioned at the medical school and curriculum level were 'curriculum tradition', 'support by educational department' and 'management and finances'. Other contextual characteristics were 'leadership style' at all levels but especially of department chairs, 'affordances and support', 'support and relatedness', and 'students' characteristics'. Personal characteristics were 'agency', 'experience with PBL (as a student or a teacher)','personal development', 'motivation and work engagement'and 'high content expertise'.
Several context and personal characteristics associated with teachers' conceptions were identified, enabling a broader view on faculty development with attention for these characteristics, next to teaching skills.

Credible evaluation of the learning climate requires valid and reliable instruments in order to inform quality improvement activities. Since its initial validation the Dutch Residency Educational Climate Test (D-RECT) has been increasingly used to evaluate the learning climate, yet it has not been tested in its final form and on the actual level of use - the department.
Our aim was to re-investigate the internal validity and reliability of the D-RECT at the resident and department levels.Read More

D-RECT evaluations collected during 2012-2013 were included. Internal validity was assessed using exploratory and confirmatory factor analyses. Reliability was assessed using generalizability theory.
In total, 2306 evaluations and 291 departments were included. Exploratory factor analysis showed a 9-factor structure containing 35 items: teamwork, role of specialty tutor, coaching and assessment, formal education, resident peer collaboration, work is adapted to residents' competence, patient sign-out, educational atmosphere, and accessibility of supervisors. Confirmatory factor analysis indicated acceptable to good fit. Three resident evaluations were needed to assess the overall learning climate reliably and eight residents to assess the subscales.
This study reaffirms the reliability and internal validity of the D-RECT in measuring residency training learning climate. Ongoing evaluation of the instrument remains important.

Adv Health Sci Educ Theory Pract
Adv Health Sci Educ Theory Pract 2016 May 27. Epub 2016 May 27.
Center for Research and Innovation in Medical Education, University Medical Center Groningen, University of Groningen, Groningen, The Netherlands.

Students feel insufficiently supported in clinical environments to engage in active learning and achieve a high level of self-regulation. As a result clinical learning is highly demanding for students. Because of large differences between students, supervisors may not know how to support them in their learning process.Read More

We explored patterns in undergraduate students' self-regulated learning behavior in the clinical environment, to improve tailored supervision, using Q-methodology. Q-methodology uses features of both qualitative and quantitative methods for the systematic investigation of subjective issues by having participants sort statements along a continuum to represent their opinion. We enrolled 74 students between December 2014 and April 2015 and had them characterize their learning behavior by sorting 52 statements about self-regulated learning behavior and explaining their response. The statements used for the sorting were extracted from a previous study. The data was analyzed using by-person factor analysis to identify clusters of individuals with similar sorts of the statements. The resulting factors and qualitative data were used to interpret and describe the patterns that emerged. Five resulting patterns were identified in students' self-regulated learning behavior in the clinical environment, which we labelled: Engaged, Critically opportunistic, Uncertain, Restrained and Effortful. The five patterns varied mostly regarding goals, metacognition, communication, effort, and dependence on external regulation for learning. These discrete patterns in students' self-regulated learning behavior in the clinical environment are part of a complex interaction between student and learning context. The results suggest that developing self-regulated learning behavior might best be supported regarding individual students' needs.

BMC Med Educ
BMC Med Educ 2016 20;16:117. Epub 2016 Apr 20.
Medical Education Unit, School of Medicine, Western Sydney University, Sydney, Australia.

Entrustable Professional Activities (EPAs) are increasingly used as a focus for assessment in graduate medical education (GME). However, a consistent approach to guide EPA design is currently lacking, in particular concerning the actual content (knowledge, skills and attitude required for specific tasks) for EPAs. This paper describes a comprehensive five stage approach, which was used to develop two specialty-specific EPAs in emergency medicine focused on the first year of GME.Read More

The five stage approach was used to gain consensus on the task, content and entrustment scale for two specialty-specific EPAs in emergency medicine. The participants consisted of twelve clinical supervisors working in the emergency department. The five stages were: 1) Selecting the EPA topic; 2) Developing the EPA content by collecting data from participants using focus group and individual interviews; 3) Drafting the EPAs based on analysis of collected data; 4) Seeking feedback on the draft EPAs from the participants and other stakeholders; 5) Refining and finalising the EPAs based on feedback.
Two specialty-specific EPAs were developed using the five stage approach. The participants reached consensus on the specific tasks and criteria for performance for the two EPAs. They also agreed that both day-to-day (ad hoc) and formal (summative) entrustment decisions were put into practice through the intensity of supervision provided to PGY1 doctors. As a result, a three level entrustment and supervision scale consisting of direct active, indirect active, passive was developed reflecting the shift in the intensity of supervision from close supervision to minimal supervision.
The five stage approach described in this paper was used successfully to develop two specialty-specific EPAs in emergency medicine along with a three level entrustment scale.We propose that the five stage approach is transferable to a range of medical training contexts to design specialty-specific EPAs.

Acad Med
Acad Med 2016 Mar;91(3):365-70
K.D. Könings is assistant professor, Department of Educational Development and Research and Graduate School of Health Professions Education, Maastricht University, Maastricht, the Netherlands. J. van Berlo is senior ICT developer, Department of Educational Development and Research and Graduate School of Health Professions Education, Maastricht University, Maastricht, the Netherlands. R. Koopmans is professor of general internal medicine, Department of Educational Development and Research and Graduate School of Health Professions Education, Maastricht University, Maastricht, the Netherlands. H. Hoogland was associate professor, Department of Educational Development and Research, Maastricht University, Maastricht, the Netherlands, at the time this work was completed. He is now retired. I.A.E. Spanjers is researcher, ResearchNed, Nijmegen, the Netherlands. J.A. ten Haaf is project manager, E-learning, University Library, Maastricht University, Maastricht, the Netherlands. C.P.M. van der Vleuten is professor of education, Department of Educational Development and Research and Graduate School of Health Professions Education, Maastricht University, Maastricht, the Netherlands. J.J.G. van Merriënboer is professor of learning and instruction, Department of Educational Development and Research and Graduate School of Health Professions Education, Maastricht University, Maastricht, the Netherlands.

Reflecting on workplace-based experiences is necessary for professional development. However, residents need support to raise their awareness of valuable moments for learning and to thoughtfully analyze those learning moments afterwards.
From October to December 2012, the authors held a multidisciplinary six-week postgraduate training module focused on general competencies.Read More

Residents were randomly assigned to one of four conditions with varying degrees of reflection support; they were offered (1) a smartphone app, (2) coaching group sessions, (3) a combination of both, or (4) neither type of support. The app allowed participants to capture in real time learning moments as a text note, audio recording, picture, or video. Coaching sessions held every two weeks aimed to deepen participants' reflection on captured learning moments. Questionnaire responses and reflection data were compared between conditions to assess the effects of the app and coaching sessions on intensity and frequency of reflection.
Sixty-four residents participated. App users reflected more often, captured more learning moments, and reported greater learning progress than nonapp users. Participants who attended coaching sessions were more alert to learning moments and pursued more follow-up learning activities to improve on the general competencies. Those who received both types of support were most alert to these learning moments.
A simple mobile app for capturing learning moments shows promise as a tool to support workplace-based learning, especially when combined with coaching sessions. Future research should evaluate these tools on a broader scale and in conjunction with residents' and students' personal digital portfolios.

Perspect Med Educ
Perspect Med Educ 2015 Dec;4(6):323-5
Department of Educational Development and Research, Faculty of Health, Medicine and Life Sciences, Maastricht University, PO Box 616, 6200 MD, Maastricht, The Netherlands.
BMC Med Educ
BMC Med Educ 2015 12;15:203. Epub 2015 Nov 12.
Radboud University Medical Center, Institute for Health Sciences, IQ healthcare, Geert Grooteplein 21, 6525 EZ, Nijmegen, The Netherlands.

Clinical practice guidelines are intended to improve the process and outcomes of patient care. However, their implementation remains a challenge. We designed an implementation strategy, based on peer assessment (PA) focusing on barriers to change in physical therapy care.Read More

A previously published randomized controlled trial showed that PA was more effective than the usual strategy "case discussion" in improving adherence to a low back pain guideline. Peer assessment aims to enhance knowledge, communication, and hands-on clinical skills consistent with guideline recommendations. Participants observed and evaluated clinical performance on the spot in a role-play simulating clinical practice. Participants performed three roles: physical therapist, assessor, and patient. This study explored the critical features of the PA program that contributed to improved guideline adherence in the perception of participants.
Dutch physical therapists working in primary care (n = 49) organized in communities of practice (n = 6) participated in the PA program. By unpacking the program we identified three main tasks and eleven subtasks. After the program was finished, a questionnaire was administered in which participants were asked to rank the program tasks from high to low learning value and to describe their impact on performance improvement. Overall ranking results were calculated. Additional semi-structured interviews were conducted to elaborate on the questionnaires results and were transcribed verbatim. Questionnaires comments and interview transcripts were analyzed using template analysis.
Program tasks related to performance in the therapist role were perceived to have the highest impact on learning, although task perceptions varied from challenging to threatening. Perceptions were affected by the role-play format and the time schedule. Learning outcomes were awareness of performance, improved attitudes towards the guideline, and increased self-efficacy beliefs in managing patients with low back pain. Learning was facilitated by psychological safety and the quality of feedback.
The effectiveness of PA can be attributed to the structured and performance-based design of the program. Participants showed a strong cognitive and emotional commitment to performing the physical therapist role. That might have contributed to an increased awareness of strength and weakness in clinical performance and a motivation to change routine practice.

Adv Health Sci Educ Theory Pract
Adv Health Sci Educ Theory Pract 2015 Aug 15;20(3):727-44. Epub 2014 Oct 15.
Department of Primary and Community Care, Radboud University Nijmegen Medical Centre, ELG 117, Postbus 9101, 6500 HB, Nijmegen, The Netherlands,

Self-regulated learning is essential for professional development and lifelong learning. As self-regulated learning has many inaccuracies, the need to support self-regulated learning has been recommended. Supervisors can provide such support.Read More

In a prior study trainees reported on the variation in received supervisor support. This study aims at exploring supervisors' perspectives. The aim is to explore how supervisors experience self-regulated learning of postgraduate general practitioners (GP) trainees and their role in this, and what helps and hinders them in supervising. In a qualitative study using a phenomenological approach, we interviewed 20 supervisors of first- and third-year postgraduate GP trainees. Supervisors recognised trainee activity in self-regulated learning and adapted their coaching style to trainee needs, occasionally causing conflicting emotions. Supervisors' beliefs regarding their role, trainees' role and the usefulness of educational interventions influenced their support. Supervisors experienced a relation between patient safety, self-regulated learning and trainee capability to learn. Supervisor training was helpful to exchange experience and obtain advice. Supervisors found colleagues helpful in sharing supervision tasks or in calibrating judgments of trainees. Busy practice occasionally hindered the supervisory process. In conclusion, supervisors adapt their coaching to trainees' self-regulated learning, sometimes causing conflicting emotions. Patient safety and entrustment are key aspects of the supervisory process. Supervisors' beliefs about their role and trainees' role influence their support. Supervisor training is important to increase awareness of these beliefs and the influence on their behaviour, and to improve the use of educational instruments. The results align with findings from other (medical) education, thereby illustrating its relevance.

Acad Med
Acad Med 2015 Aug;90(8):1093-9
J.M.W. Moonen-van Loon is postdoctoral researcher, Department of Educational Development and Research, Maastricht University, Maastricht, The Netherlands. K. Overeem is postdoctoral researcher, Department of Educational Development and Research, Maastricht University, Maastricht, The Netherlands. M.J.B. Govaerts is assistant professor, Department of Educational Development and Research, Maastricht University, Maastricht, The Netherlands. B.H. Verhoeven is pediatric surgeon, Department of Surgery, Radboud University Medical Center, Nijmegen, and assistant professor, Department of Educational Development and Research, Maastricht University, Maastricht, The Netherlands. C.P.M. van der Vleuten is professor of education, Department of Educational Development and Research, Maastricht University, Maastricht, The Netherlands. E.W. Driessen is associate professor of education, Department of Educational Development and Research, Maastricht University, Maastricht, The Netherlands.

Residency programs around the world use multisource feedback (MSF) to evaluate learners' performance. Studies of the reliability of MSF show mixed results. This study aimed to identify the reliability of MSF as practiced across occasions with varying numbers of assessors from different professional groups (physicians and nonphysicians) and the effect on the reliability of the assessment for different competencies when completed by both groups.Read More

The authors collected data from 2008 to 2012 from electronically completed MSF questionnaires. In total, 428 residents completed 586 MSF occasions, and 5,020 assessors provided feedback. The authors used generalizability theory to analyze the reliability of MSF for multiple occasions, different competencies, and varying numbers of assessors and assessor groups across multiple occasions.
A reliability coefficient of 0.800 can be achieved with two MSF occasions completed by at least 10 assessors per group or with three MSF occasions completed by 5 assessors per group. Nonphysicians' scores for the "Scholar" and "Health advocate" competencies and physicians' scores for the "Health advocate" competency had a negative effect on the composite reliability.
A feasible number of assessors per MSF occasion can reliably assess residents' performance. Scores from a single occasion should be interpreted cautiously. However, every occasion can provide valuable feedback for learning. This research confirms that the (unique) characteristics of different assessor groups should be considered when interpreting MSF results. Reliability seems to be influenced by the included assessor groups and competencies. These findings will enhance the utility of MSF during residency training.

BMC Med Educ
BMC Med Educ 2015 15;15:133. Epub 2015 Aug 15.
Department of Educational Development and Research, Faculty of Health, Medicine and Life Sciences, Maastricht University, P.O. Box 616, 6200, MD, Maastricht, The Netherlands.

It is generally assumed that learning in context increases performance. This study investigates the relationship between the characteristics of a paper-patient context (relevance and familiarity), the mechanisms through which the cognitive dimension of context could improve learning (activation of prior knowledge, elaboration and increasing retrieval cues), and test performance.
A total of 145 medical students completed a pretest of 40 questions, of which half were with a patient vignette.Read More

One week later, they studied musculoskeletal anatomy in the dissection room without a paper-patient context (control group) or with (ir)relevant-(un)familiar context (experimental groups), and completed a cognitive load scale. Following a short delay, the students completed a posttest.
Surprisingly, our results show that students who studied in context did not perform better than students who studied without context. This finding may be explained by an interaction of the participants' expertise level, the nature of anatomical knowledge and students' approaches to learning. A relevant-familiar context only reduced the negative effect of learning the content in context. Our results suggest discouraging the introduction of an uncommon disease to illustrate a basic science concept. Higher self-perceived learning scores predict higher performance. Interestingly, students performed significantly better on the questions with context in both tests, possibly due to a 'framing effect'.
Since studies focusing on the physical and affective dimensions of context have also failed to find a positive influence of learning in a clinically relevant context, further research seems necessary to refine our theories around the role of context in learning.

Acad Med
Acad Med 2015 Jul;90(7):921-9
M. Jippes was, at the time of this research, a PhD candidate, Department of Educational Development and Research, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, the Netherlands. Currently, she is a plastic surgery resident, Diakonessenhuis, Utrecht, the Netherlands. E.W. Driessen is associate professor, Department of Educational Development and Research, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, the Netherlands. N.J. Broers is assistant professor, Department of Methodology and Statistics, Faculty of Psychology and Neuroscience, Maastricht University, Maastricht, the Netherlands. G.D. Majoor is associate professor (retired), Institute for Education, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, the Netherlands. W.H. Gijselaers is full professor, Department of Educational Research and Development, School of Business and Economics, Maastricht University, Maastricht, the Netherlands. C.P.M. van der Vleuten is full professor, Department of Educational Development and Research, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, the Netherlands.

National culture has been shown to play a role in curriculum change in medical schools, and business literature has described a similar influence of organizational culture on change processes in organizations. This study investigated the impact of both national and organizational culture on successful curriculum change in medical schools internationally.
The authors tested a literature-based conceptual model using multilevel structural equation modeling.Read More

For the operationalization of national and organizational culture, the authors used Hofstede's dimensions of culture and Quinn and Spreitzer's competing values framework, respectively. To operationalize successful curriculum change, the authors used two derivates: medical schools' organizational readiness for curriculum change developed by Jippes and colleagues, and change-related behavior developed by Herscovitch and Meyer. The authors administered a questionnaire in 2012 measuring the described operationalizations to medical schools in the process of changing their curriculum.
Nine hundred ninety-one of 1,073 invited staff members from 131 of 345 medical schools in 56 of 80 countries completed the questionnaire. An initial poor fit of the model improved to a reasonable fit by two suggested modifications which seemed theoretically plausible. In sum, characteristics of national culture and organizational culture, such as a certain level of risk taking, flexible policies and procedures, and strong leadership, affected successful curriculum change.
National and organizational culture influence readiness for change in medical schools. Therefore, medical schools considering curriculum reform should anticipate the potential impact of national and organizational culture.

Med Educ
Med Educ 2015 Jun;49(6):589-600
Centre for Evidence-Based Education, Academic Medical Centre, University of Amsterdam, Amsterdam, the Netherlands.

The importance of self-regulated learning (SRL) has been broadly recognised by medical education institutions and regulatory bodies. Supporting the development of SRL skills has proven difficult because self-regulation is a complex interactive process and we know relatively little about the factors influencing this process in real practice settings. The aim of our study was therefore to identify factors that support or hamper medical students' SRL in a clinical context.Read More

We conducted a constructivist grounded theory study using semi-structured interviews with 17 medical students from two universities enrolled in clerkships. Participants were purposively sampled to ensure variety in age, gender, experience and current clerkship. The Day Reconstruction Method was used to help participants remember their activities of the previous day. The interviews were transcribed verbatim and analysed iteratively using constant comparison and open, axial and interpretive coding.
Self-regulated learning by students in the clinical environment was influenced by the specific goals perceived by students, the autonomy they experienced, the learning opportunities they were given or created themselves, and the anticipated outcomes of an activity. All of these factors were affected by personal, contextual and social attributes.
Self-regulated learning of medical students in the clinical environment is different for every individual. The factors influencing this process are affected by personal, social and contextual attributes. Some of these are similar to those known from previous research in classroom settings, but others are unique to the clinical environment and include the facilities available, the role of patients, and social relationships pertaining to peers and other hospital staff. To better support students' SRL, we believe it is important to increase students' metacognitive awareness and to offer students more tailored learning opportunities.

Adv Health Sci Educ Theory Pract
Adv Health Sci Educ Theory Pract 2015 May 14;20(2):559-74. Epub 2014 Jun 14.
Department of Intensive Care Medicine, Maastricht University Medical Centre+, P. Debyelaan 25, 6202 AZ, Maastricht, The Netherlands,

Although several examples of frameworks dealing with students' unprofessional behaviour are available, guidance on how to deal locally or regionally with dysfunctional residents is limited (Hickson et al. in Acad Med 82(11):1040-1048, 2007b; Leape and Fromson in Ann Intern Med 144(2):107-115, 2006). Any 'rules' are mostly unwritten, and often emerge by trial and error within the specialty training programme (Stern and Papadakis in N Engl J Med 355(17):1794-1799, 2006).Read More

It is nevertheless of utmost importance that objectives, rules and guidelines comparable to those existing in undergraduate training (Project Team Consilium Abeundi van Luijk in Professional behaviour: teaching, assessing and coaching students. Final report and appendices. Mosae Libris, 2005; van Mook et al. in Neth J Crit Care 16(4):162-173, 2010a) are developed for postgraduate training. And that implicit rules are made explicit. This article outlines a framework based on the lessons learned from contemporary postgraduate medical training programmes.

Med Educ
Med Educ 2015 May;49(5):487-98
Department of Pathology, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, The Netherlands; Faculty of Health, Medicine and Life Sciences, School of Health Professions Education, Maastricht University, Maastricht, The Netherlands.

It is widely acknowledged that assessment can affect student learning. In recent years, attention has been called to 'programmatic assessment', which is intended to optimise both learning functions and decision functions at the programme level of assessment, rather than according to individual methods of assessment. Although the concept is attractive, little research into its intended effects on students and their learning has been conducted.Read More

This study investigated the elements of programmatic assessment that students perceived as supporting or inhibiting learning, and the factors that influenced the active construction of their learning.
The study was conducted in a graduate-entry medical school that implemented programmatic assessment. Thus, all assessment information, feedback and reflective activities were combined into a comprehensive, holistic programme of assessment. We used a qualitative approach and interviewed students (n = 17) in the pre-clinical phase of the programme about their perceptions of programmatic assessment and learning approaches. Data were scrutinised using theory-based thematic analysis.
Elements from the comprehensive programme of assessment, such as feedback, portfolios, assessments and assignments, were found to have both supporting and inhibiting effects on learning. These supporting and inhibiting elements influenced students' construction of learning. Findings showed that: (i) students perceived formative assessment as summative; (ii) programmatic assessment was an important trigger for learning, and (iii) the portfolio's reflective activities were appreciated for their generation of knowledge, the lessons drawn from feedback, and the opportunities for follow-up. Some students, however, were less appreciative of reflective activities. For these students, the elements perceived as inhibiting seemed to dominate the learning response.
The active participation of learners in their own learning is possible when learning is supported by programmatic assessment. Certain features of the comprehensive programme of assessment were found to influence student learning, and this influence can either support or inhibit students' learning responses.

Phys Ther
Phys Ther 2015 Apr 18;95(4):600-12. Epub 2014 Sep 18.
M.W.G. Nijhuis-van der Sanden, PhD, Radboud University Medical Center, Scientific Institute for Quality of Healthcare (IQ Healthcare).

Clinical practice guidelines (CPGs) are not readily implemented in clinical practice. One of the impeding factors is that physical therapists do not hold realistic perceptions of their adherence to CPGs. Peer assessment (PA) is an implementation strategy that aims at improving guideline adherence by enhancing reflective practice, awareness of professional performance, and attainment of personal goals.Read More

The purpose of this study was to compare the effectiveness of PA with the usual case discussion (CD) strategy on adherence to CPGs for physical therapist management of upper extremity complaints.
A single-masked, cluster-randomized controlled trial with pretest-posttest design was conducted.
Twenty communities of practice (n=149 physical therapists) were randomly assigned to groups receiving PA or CD, with both interventions consisting of 4 sessions over 6 months. Both PA and CD groups worked on identical clinical cases relevant to the guidelines. Peer assessment focused on individual performance observed and evaluated by peers; CD focused on discussion.
Guideline adherence was measured with clinical vignettes, reflective practice was measured with the Self-Reflection and Insight Scale (SRIS), awareness of performance was measured via the correlation between perceived and assessed improvement, and attainment of personal goals was measured with written commitments to change.
The PA groups improved more on guideline adherence compared with the CD groups (effect=22.52; 95% confidence interval [95% CI]=2.38, 42.66; P=.03). The SRIS scores did not differ between PA and CD groups. Awareness of performance was greater for the PA groups (r=.36) than for the CD groups (r=.08) (effect=14.73; 95% CI=2.78, 26.68; P=.01). The PA strategy was more effective than the CD strategy in attaining personal goals (effect=0.50; 95% CI=0.04, 0.96; P=.03).
Limited validity of clinical vignettes as a proxy measure of clinical practice was a limitation of the study.
Peer assessment was more effective than CD in improving adherence to CPGs. Personal feedback may have contributed to its effectiveness. Future research should address the role of the group coach.

Evidence tailored to sub-Saharan Africa on outcomes of innovations in medical education is needed to encourage and advance their implementation in this region.
To investigate preparedness for practice of students and graduates from an innovative and a conventional medical curriculum in a sub-Saharan African context.
Using mixed methods we compared junior doctors and fifth-year students from two Mozambican medical schools: one with an innovative problem- and community-based curriculum and one with a conventional lecture- and discipline-based curriculum.Read More

A questionnaire on professional competencies was administered, semi-structured interviews were conducted, and work diaries were collected. The findings were integrated in a conceptual model.
Six areas of tension between global health care ideals and local health care practice emerged from the data that challenged doctors' motivation and preparedness for practice. Four elements of the innovative curriculum equipped students and graduates with skills, attitudes and competencies to better cope with these tensions. Students and graduates from the innovative curriculum rated significantly higher levels on various competencies and expressed more satisfaction with the curriculum and its usefulness for their work.
An innovative problem- and community-based curriculum can improve sub-Saharan African doctors' motivation and preparedness to tackle the challenges of health care practice in this region.

Acad Med
Acad Med 2014 Nov;89(11):1510-9
Ms. Gingerich is a PhD candidate, School of Health Professions Education, Maastricht University, Maastricht, Netherlands, and research associate, Northern Medical Program, University of Northern British Columbia, Prince George, British Columbia, Canada. Dr. van der Vleuten is scientific director, School of Health Professions Education, Maastricht University, Maastricht, Netherlands. Dr. Eva is acting director and senior scientist, Centre for Health Education Scholarship, University of British Columbia, Vancouver, British Columbia, Canada. Dr. Regehr is associate director of research and senior scientist, Centre for Health Education Scholarship, University of British Columbia, Vancouver, British Columbia, Canada.

Social judgment research suggests that rater unreliability in performance assessments arises from raters' differing inferences about the performer and the underlying reasons for the performance observed. These varying social judgments are not entirely idiosyncratic but, rather, tend to partition into a finite number of distinct subgroups, suggesting some "signal" in the "noise" of interrater variability. The authors investigated the proportion of variance in Mini-CEX ratings attributable to such partitions of raters' social judgments about residents.Read More

In 2012 and 2013, physicians reviewed video-recorded patient encounters for seven residents, completed a Mini-CEX, and described their social judgments of the residents. Additional participants sorted these descriptions, which were analyzed using latent partition analysis (LPA). The best-fitting set of partitions for each resident served as an independent variable in a one-way ANOVA to determine the proportion of variance explained in Mini-CEX ratings.
Forty-eight physicians rated at least one resident (34 assessed all seven). The seven sets of social judgments were sorted by 14 participants. Across residents, 2 to 5 partitions (mode: 4) provided a good LPA fit, suggesting that subgroups of raters were making similar social judgments, while different causal explanations for each resident's performance existed across subgroups. The partitions accounted for 9% to 57% of the variance in Mini-CEX ratings across residents (mean = 32%).
These findings suggest that multiple "signals" do exist within the "noise" of interrater variability in performance-based assessment. It may be valuable to understand and exploit these multiple signals rather than try to eliminate them.

Abstract Background: Teachers' conceptions of learning and teaching are important for faculty development to result in enduring changes in teaching practices. Until now, studies on these conceptions have mostly focused on traditional, lecture-based curricula rather than on small-group student-centred educational formats, which are gaining ground worldwide. Aim: To explore which factors predict teachers' conceptions in student-centred curricula.Read More

Methods: In two Dutch medical schools with 10 and 40 years of student-centred education, teachers were asked to fill out the Conceptions of Learning and Teaching (COLT) Questionnaire to assess their 'teacher-centredness', 'appreciation of active learning' and 'orientation to professional practice'. Next, we quantitatively assessed the relations of teachers' conceptions with their personal and occupational characteristics and institute. Results: Overall response was 49.4% (N = 319/646). Institute was the main predictor for variance in all three scales, and discipline, gender and teaching experience significantly explained variance in two of the scales. More than 80% of the variance was not explained by these factors. Conclusion: Longer exposure to a student-centred curriculum was associated with fewer teacher-centred conceptions, greater 'appreciation of active learning' and stronger 'orientation towards professional practice'. In line with studies on lecture-based curricula, discipline, gender and teaching experience also appeared important for teachers' conceptions in student-centred curricula. More research is necessary to better understand the influence of institute on the three teachers' conceptions scales.

PLoS One 2014 13;9(11):e112805. Epub 2014 Nov 13.
Department of Educational Development and Research, University of Maastricht, Maastricht, the Netherlands.

Teamwork between clinical teachers is a challenge in postgraduate medical training. Although there are several instruments available for measuring teamwork in health care, none of them are appropriate for teaching teams. The aim of this study is to develop an instrument (TeamQ) for measuring teamwork, to investigate its psychometric properties and to explore how clinical teachers assess their teamwork.Read More

To select the items to be included in the TeamQ questionnaire, we conducted a content validation in 2011, using a Delphi procedure in which 40 experts were invited. Next, for pilot testing the preliminary tool, 1446 clinical teachers from 116 teaching teams were requested to complete the TeamQ questionnaire. For data analyses we used statistical strategies: principal component analysis, internal consistency reliability coefficient, and the number of evaluations needed to obtain reliable estimates. Lastly, the median TeamQ scores were calculated for teams to explore the levels of teamwork.
In total, 31 experts participated in the Delphi study. In total, 114 teams participated in the TeamQ pilot. The median team response was 7 evaluations per team. The principal component analysis revealed 11 factors; 8 were included. The reliability coefficients of the TeamQ scales ranged from 0.75 to 0.93. The generalizability analysis revealed that 5 to 7 evaluations were needed to obtain internal reliability coefficients of 0.70. In terms of teamwork, the clinical teachers scored residents' empowerment as the highest TeamQ scale and feedback culture as the area that would most benefit from improvement.
This study provides initial evidence of the validity of an instrument for measuring teamwork in teaching teams. The high response rates and the low number of evaluations needed for reliably measuring teamwork indicate that TeamQ is feasible for use by teaching teams. Future research could explore the effectiveness of feedback on teamwork in follow up measurements.

To determine the perceived importance of specific competencies in professional veterinary practice and education among veterinarians in several countries.
Survey-based prospective study.
1,137 veterinarians in 10 countries.Read More

Veterinarians were invited via email to participate in the study. A framework of 18 competencies grouped into 7 domains (veterinary expertise, communication, collaboration, entrepreneurship, health and welfare, scholarship, and personal development) was used. Respondents rated the importance of each competency for veterinary professional practice and for veterinary education by use of a 9-point Likert scale in an online questionnaire. Quantitative statistical analyses were performed to assess the data.
All described competencies were perceived as having importance (with overall mean ratings [all countries] ≥ 6.45/9) for professional practice and education. Competencies related to veterinary expertise had the highest ratings (overall mean, 8.33/9 for both professional practice and education). For the veterinary expertise, entrepreneurship, and scholarship domains, substantial differences (determined on the basis of statistical significance and effect size) were found in importance ratings among veterinarians in different countries.
Results indicated a general consensus regarding the importance of specific types of competencies in veterinary professional practice and education. Further research into the definition of competencies essential for veterinary professionals is needed to help inform an international dialogue on the subject.

Abstract Background: Virtual patients (VPs) are increasingly used to train clinical reasoning. So far, no validated evaluation instruments for VP design are available. Aims: We examined the validity of an instrument for assessing the perception of VP design by learners.Read More

Methods: Three sources of validity evidence were examined: (i) Content was examined based on theory of clinical reasoning and an international VP expert team. (ii) The response process was explored in think-aloud pilot studies with medical students and in content analyses of free text questions accompanying each item of the instrument. (iii) Internal structure was assessed by exploratory factor analysis (EFA) and inter-rater reliability by generalizability analysis. Results: Content analysis was reasonably supported by the theoretical foundation and the VP expert team. The think-aloud studies and analysis of free text comments supported the validity of the instrument. In the EFA, using 2547 student evaluations of a total of 78 VPs, a three-factor model showed a reasonable fit with the data. At least 200 student responses are needed to obtain a reliable evaluation of a VP on all three factors. Conclusion: The instrument has the potential to provide valid information about VP design, provided that many responses per VP are available.

BMC Med Educ
BMC Med Educ 2014 16;14:220. Epub 2014 Oct 16.
Department of Research in Education, VUmc School of Medical Sciences, P,O, Box 7057 (MF, A-114), 1007 MB Amsterdam, The Netherlands.

Teachers' conceptions of learning and teaching are partly unconscious. However, they are critical for the delivery of education and affect students' learning outcomes. Lasting changes in teaching behaviour can only be realized if conceptions of teachers have been changed accordingly.Read More

Previously we constructed a questionnaire named COLT to measure conceptions. In the present study, we investigated if different teacher profiles could be assessed which are based on the teachers' conceptions. These teacher profiles might have implications for individual teachers, for faculty development activities and for institutes. Our research questions were: (1) Can we identify teacher profiles based on the COLT? (2) If so, how are these teacher profiles associated with other teacher characteristics?
The COLT questionnaire was sent electronically to all teachers in the first three years of the undergraduate curriculum of Medicine in two medical schools in the Netherlands with student-centred education. The COLT (18 items, 5 point Likert scales) comprises three scales: 'teacher centredness', 'appreciation of active learning' and 'orientation to professional practice'. We also collected personal information about the participants and their occupational characteristics. Teacher profiles were studied using a K-means cluster analysis and calculating Chi squares.
The response rate was 49.4% (N = 319/646). A five-cluster solution fitted the data best, resulting in five teacher profiles based on their conceptions as measured by the COLT. We named the teacher profiles: Transmitters (most traditional), Organizers, Intermediates, Facilitators and Conceptual Change Agents (most modern). The teacher profiles differed from each other in personal and occupational characteristics.
Based on teachers' conceptions of learning and teaching, five teacher profiles were found in student-centred education. We offered suggestions how insight into these teacher profiles might be useful for individual teachers, for faculty development activities and for institutes and departments, especially if involved in a curriculum reform towards student-centred education.

Acad Med
Acad Med 2014 Sep;89(9):1259-66
Dr. Fokkema is a physician and PhD student, St. Lucas Andreas Hospital, Amsterdam, the Netherlands. Dr. Scheele is professor, VU University Medical Center, Amsterdam, the Netherlands, and a gynecologist and residency program director, St. Lucas Andreas Hospital, Amsterdam, the Netherlands. Dr. Westerman is a researcher, School of Medical Sciences, VU University Medical Center, Amsterdam, the Netherlands, and a resident in internal medicine, St. Lucas Andreas Hospital, Amsterdam, the Netherlands. Dr. van Exel is associate professor, Institute of Health Policy and Management, Erasmus University Rotterdam, Rotterdam, the Netherlands. Dr. Scherpbier is professor and dean, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, the Netherlands. Dr. van der Vleuten is professor, Department of Educational Development and Research, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, the Netherlands. Dr. Dörr, deceased, was professor, Department of Education and Teaching, Leiden University Medical Center, Leiden, the Netherlands, and a gynecologist, Medical Centre Haaglanden, Den Haag, the Netherlands. Dr. Teunissen is a resident in obstetrics-gynecology, VU University Medical Center, Amsterdam, the Netherlands, and associate professor, Department of Educational Development and Research, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, the Netherlands.

Anticipating users' perceptions of the effects an innovation will have in daily practice prior to implementation may lead to a more optimal innovation process. In this study, the authors aimed to identify the kinds of perceptions that exist concerning the effects of workplace-based assessment (WBA), an innovation that is widely used in medical education, among its users.
In 2012, the authors used Q methodology to ascertain the principal user perceptions of effects of WBA in practice.Read More

Participating obstetrics-gynecology residents and attending physicians (including residency program directors) at six hospitals in the Netherlands performed individual Q sorts to rank 36 statements concerning WBA and WBA tools according to their level of agreement. The authors conducted by-person factor analysis to uncover patterns in the rankings of the statements. They used the statistical results and participant comments about their sorts to interpret and describe distinct perceptions.
The analysis of 65 Q sorts (completed by 22 residents and 43 attendings) identified five distinct user perceptions regarding the effects of WBA in practice, which the authors labeled enthusiasm, compliance, effort, neutrality, and skepticism. These perceptions were characterized by differences in views on three main issues: the intended goals of the innovation, its applicability (ease of applying it to practice), and its actual impact.
User perceptions of the effects of innovations in medical education can be typified and should be anticipated. This study's insights into five principal user perceptions can support the design and implementation of innovations in medical education.

Med Educ
Med Educ 2014 Aug;48(8):761-7
Department of Educational Development and Research, Maastricht University, Maastricht, the Netherlands.

Medical education research is thriving. In recent decades, numbers of journals and publications have increased enormously, as have the number and size of medical education meetings around the world. The aim of this paper is to shed some light on the origins of this success.Read More

My central argument is that dialogue between education practice (and its teachers) and education research (and its researchers) is indispensable.
To illustrate how I have come to this perspective, I discuss two crucial developments of personal import to myself. The first is the development of assessment theory informed by both research findings and insights emerging from implementations conducted in collaboration with teachers and learners. The second is the establishment of a department of education that includes many members from the medical domain.
Medical education is thriving because it is shaped and nourished within a community of practice of collaborating teachers, practitioners and researchers. This obviates the threat of a fissure between education research and education practice. The values of this community of practice - inclusiveness, openness, supportiveness, nurture and mentorship - are key elements for its sustainability. In pacing the development of our research in a manner that maintains this synergy, we should be mindful of the zone of proximal development of our community of practice.

Med Educ
Med Educ 2014 Jul;48(7):713-23
Department of Clinical Neurological Sciences, Schulich School of Medicine and Dentistry, University of Western Ontario, London, ON, Canada.

Feedback should facilitate learning, but within medical education it often fails to deliver on its promise. To better understand why feedback is challenging, we explored the unique perspectives of doctors who had also trained extensively in sport or music, aiming to: (i) distinguish the elements of the response to feedback that are determined by the individual learner from those determined by the learning culture, and (ii) understand how these elements interact in order to make recommendations for improving feedback in medical education.
Using a constructivist grounded theory approach, we conducted semi-structured interviews with 27 doctors or medical students who had high-level training and competitive or performance experience in sport (n = 15) or music (n = 12).Read More

Data were analysed iteratively using constant comparison. Key themes were identified and their relationships critically examined to derive a conceptual understanding of feedback and its impact.
We identified three essential sources of influence on the meaning that feedback assumed: the individual learner; the characteristics of the feedback, and the learning culture. Individual learner traits, such as motivation and orientation toward feedback, appeared stable across learning contexts. Similarly, certain feedback characteristics, including specificity, credibility and actionability, were valued in sport, music and medicine alike. Learning culture influenced feedback in three ways: (i) by defining expectations for teachers and teacher-learner relationships; (ii) by establishing norms for and expectations of feedback, and (iii) by directing teachers' and learners' attention toward certain dimensions of performance. Learning culture therefore neither creates motivated learners nor defines 'good feedback'; rather, it creates the conditions and opportunities that allow good feedback to occur and learners to respond.
An adequate understanding of feedback requires an integrated approach incorporating both the individual and the learning culture. Our research offers a clear direction for medicine's learning culture: normalise feedback; promote trusting teacher-learner relationships; define clear performance goals, and ensure that the goals of learners and teachers align.

Perspect Med Educ
Perspect Med Educ 2014 Jun;3(3):190-203
Department of Educational Development and Research, Faculty of Health, Medicine and Life Sciences, Maastricht University, PO Box 616, 6200 MD, Maastricht, The Netherlands,

There is a growing need for research on culture, cultural differences and cultural effects of globalization in medical education, but these are complex phenomena to investigate. Socio-cultural activity theory seems a useful framework to study cultural complexity, because it matches current views on culture as a dynamic process situated in a social context, and has been valued in diverse fields for yielding rich understandings of complex issues and key factors involved. This paper explains how activity theory can be used in (cross-)cultural medical education research.Read More

We discuss activity theory's theoretical background and principles, and we show how these can be applied to the cultural research practice by discussing the steps involved in a cross-cultural study that we conducted, from formulating research questions to drawing conclusions. We describe how the activity system, the unit of analysis in activity theory, can serve as an organizing principle to grasp cultural complexity. We end with reflections on the theoretical and practical use of activity theory for cultural research and note that it is not a shortcut to capture cultural complexity: it is a challenge for researchers to determine the boundaries of their study and to analyze and interpret the dynamics of the activity system.

BMC Med Educ
BMC Med Educ 2014 13;14:117. Epub 2014 Jun 13.
HAN University of Applied Sciences, Department Allied Health Studies, Kapittelweg 33, 5425 EN Nijmegen, The Netherlands.

Peer Assessment (PA) in health professions education encourages students to develop a critical attitude towards their own and their peers' performance. We designed a PA task to assess students' clinical skills (including reasoning, communication, physical examination and treatment skills) in a role-play that simulated physical therapy (PT) practice. Students alternately performed in the role of PT, assessor, and patient.Read More

Oral face-to-face feedback was provided as well as written feedback and scores.This study aims to explore the impact of PA on the improvement of clinical performance of undergraduate PT students.
The PA task was analyzed and decomposed into task elements. A qualitative approach was used to explore students' perceptions of the task and the task elements. Semi-structured interviews with second year students were conducted to explore the perceived impact of these task elements on performance improvement. Students were asked to select the elements perceived valuable, to rank them from highest to lowest learning value, and to motivate their choices. Interviews were transcribed verbatim and analyzed, using a phenomenographical approach and following template analysis guidelines. A quantitative approach was used to describe the ranking results.
Quantitative analyses showed that the perceived impact on learning varied widely. Performing the clinical task in the PT role, was assigned to the first place (1), followed by receiving expert feedback (2), and observing peer performance (3). Receiving peer feedback was not perceived the most powerful task element.Qualitative analyses resulted in three emerging themes: pre-performance, true-performance, and post-performance triggers for improvement. Each theme contained three categories: learning activities, outcomes, and conditions for learning.Intended learning activities were reported, such as transferring prior learning to a new application context and unintended learning activities, such as modelling a peer's performance. Outcomes related to increased self-confidence, insight in performance standards and awareness of improvement areas. Conditions for learning referred to the quality of peer feedback.
PA may be a powerful tool to improve clinical performance, although peer feedback is not perceived the most powerful element. Peer assessors in undergraduate PT education use idiosyncratic strategies to assess their peers' performance.

Clin Anat
Clin Anat 2014 Apr 19;27(3):296-303. Epub 2013 Dec 19.
Department of Educational Development and Research, School of Health Professions Education, Maastricht University, Maastricht, The Netherlands; Department of Anatomy, Radboud University Medical Centre Nijmegen, Nijmegen, The Netherlands.

Eight factors are claimed to have a negative influence on anatomical knowledge of medical students: (1) teaching by nonmedically qualified teachers, (2) the absence of a core anatomy curriculum, (3) decreased use of dissection as a teaching tool, (4) lack of teaching anatomy in context, (5) integrated curricula (problem-based learning or systems-based curricula), (6) inadequate assessment of anatomical knowledge, (7) decreased anatomy teaching time, and (8) neglect of vertical integration of anatomy teaching. A recent review revealed a lack of evidence underpinning any of the claims owing to the poor quality of papers, and recommendations were made for education and research on teaching in context and the implementation of vertical integration and of assessment strategies. In this article, we will describe the alleged factors fully, revealing additional recommendations for improving anatomy education by promoting recognition for teaching in institutions, by enhancing the professional recognition of anatomists through the implementation of a national postgraduate training program, and by encouraging anatomists to participate in educational research.Read More

Med Educ
Med Educ 2014 Mar;48(3):234-5
Maastricht, the Netherlands.

Consistent identification and prevention of failure for at-risk medical students is challenging, failing courses is costly to all stakeholders, and there is need for further research into duration, timing and structure of interventions to help students in difficulty.
To verify the value of a new exam two weeks into medical school as a predictor of failure, and explore the requirements for a preventative intervention.
Students who failed the two-week exam were invited to a series of large-group workshops and small-group follow-up meetings.Read More

Participants' subsequent exam performance was compared with non-participants.
About 71% of students who performed poorly in the new exam subsequently failed a course. Attendance at the workshops made no difference to short- or long-term pass rates. Attendance at more than three follow-up small group sessions significantly improved pass rates two semesters later, and was influenced by teacher experience.
Close similarity between predictor task and target task is important for accurate prediction of failure. Consideration should be given to dose effect and class size in the prevention of failure of at-risk students, and we recommend a systemic approach to intervention/remediation programmes, involving a whole semester of mandatory, weekly small group meetings with experienced teachers.

Behav Res Methods
Behav Res Methods 2013 Dec;45(4):1058-72
Department of Educational Development and Research, Maastricht University, P. O. Box 616, 6200 MD, Maastricht, The Netherlands,

According to cognitive load theory, instructions can impose three types of cognitive load on the learner: intrinsic load, extraneous load, and germane load. Proper measurement of the different types of cognitive load can help us understand why the effectiveness and efficiency of learning environments may differ as a function of instructional formats and learner characteristics. In this article, we present a ten-item instrument for the measurement of the three types of cognitive load.Read More

Principal component analysis on data from a lecture in statistics for PhD students (n = 56) in psychology and health sciences revealed a three-component solution, consistent with the types of load that the different items were intended to measure. This solution was confirmed by a confirmatory factor analysis of data from three lectures in statistics for different cohorts of bachelor students in the social and health sciences (ns = 171, 136, and 148), and received further support from a randomized experiment with university freshmen in the health sciences (n = 58).

Med Educ
Med Educ 2013 Dec;47(12):1164-74
Educational Development and Research, Maastricht University, Maastricht, the Netherlands.

Although work-based assessments (WBA) may come closest to assessing habitual performance, their use for summative purposes is not undisputed. Most criticism of WBA stems from approaches to validity consistent with the quantitative psychometric framework. However, there is increasing research evidence that indicates that the assumptions underlying the predictive, deterministic framework of psychometrics may no longer hold.Read More

In this discussion paper we argue that meaningfulness and appropriateness of current validity evidence can be called into question and that we need alternative strategies to assessment and validity inquiry that build on current theories of learning and performance in complex and dynamic workplace settings.
Drawing from research in various professional fields we outline key issues within the mechanisms of learning, competence and performance in the context of complex social environments and illustrate their relevance to WBA. In reviewing recent socio-cultural learning theory and research on performance and performance interpretations in work settings, we demonstrate that learning, competence (as inferred from performance) as well as performance interpretations are to be seen as inherently contextualised, and can only be under-stood 'in situ'. Assessment in the context of work settings may, therefore, be more usefully viewed as a socially situated interpretive act.
We propose constructivist-interpretivist approaches towards WBA in order to capture and understand contextualised learning and performance in work settings. Theoretical assumptions underlying interpretivist assessment approaches call for a validity theory that provides the theoretical framework and conceptual tools to guide the validation process in the qualitative assessment inquiry. Basic principles of rigour specific to qualitative research have been established, and they can and should be used to determine validity in interpretivist assessment approaches. If used properly, these strategies generate trustworthy evidence that is needed to develop the validity argument in WBA, allowing for in-depth and meaningful information about professional competence.

E-learning resources, such as virtual patients (VPs), can be more effective when they are integrated in the curriculum. To gain insights that can inform guidelines for the curricular integration of VPs, we explored students' perceptions of scenarios with integrated and non-integrated VPs aimed at promoting clinical reasoning skills.
During their paediatric clerkship, 116 fifth-year medical students were given at least ten VPs embedded in eight integrated scenarios and as non-integrated add-ons.Read More

The scenarios differed in the sequencing and alignment of VPs and related educational activities, tutor involvement, number of VPs, relevance to assessment and involvement of real patients. We sought students' perceptions on the VP scenarios in focus group interviews with eight groups of 4-7 randomly selected students (n = 39). The interviews were recorded, transcribed and analysed qualitatively.
The analysis resulted in six themes reflecting students' perceptions of important features for effective curricular integration of VPs: (i) continuous and stable online access, (ii) increasing complexity, adapted to students' knowledge, (iii) VP-related workload offset by elimination of other activities, (iv) optimal sequencing (e.g.: lecture--1 to 2 VP(s)--tutor-led small group discussion--real patient) and (V) optimal alignment of VPs and educational activities, (vi) inclusion of VP topics in assessment.
The themes appear to offer starting points for the development of a framework to guide the curricular integration of VPs. Their impact needs to be confirmed by studies using quantitative controlled designs.


Despite acknowledgement that the Canadian Medical Educational Directives for Specialists (CanMEDS) framework covers the relevant competencies of physicians, many educators and medical professionals struggle to translate the CanMEDS roles into comprehensive training programmes for specific specialties.
To gain insight into the applicability of the CanMEDS framework to guide the design of educational programmes for specific specialties by exploring stakeholders' perceptions of specialty specific competencies and examining differences between those competencies and the CanMEDS framework.
This case study is a sequel to a study among ObsGyn specialists.Read More

It explores the perspectives of patients, midwives, nurses, general practitioners, and hospital boards on gynaecological competencies and compares these with the CanMEDS framework.
Clinical expertise, reflective practice, collaboration, a holistic view, and involvement in practice management were perceived to be important competencies for gynaecological practice. Although all the competencies were covered by the CanMEDS framework, there were some mismatches between stakeholders' perceptions of the importance of some competencies and their position in the framework.
The CanMEDS framework appears to offer relevant building blocks for specialty specific postgraduate training, which should be combined with the results of an exploration of specialty specific competencies to arrive at a postgraduate curriculum that is in alignment with professional practice.

BMC Med Educ
BMC Med Educ 2013 11;13:123. Epub 2013 Sep 11.
Quality Improvement in Veterinary Education, Faculty of Veterinary Medicine, Utrecht University, Utrecht, The Netherlands.

In competency-based medical education emphasis has shifted towards outcomes, capabilities, and learner-centeredness. Together with a focus on sustained evidence of professional competence this calls for new methods of teaching and assessment. Recently, medical educators advocated the use of a holistic, programmatic approach towards assessment.Read More

Besides maximum facilitation of learning it should improve the validity and reliability of measurements and documentation of competence development. We explored how, in a competency-based curriculum, current theories on programmatic assessment interacted with educational practice.
In a development study including evaluation, we investigated the implementation of a theory-based programme of assessment. Between April 2011 and May 2012 quantitative evaluation data were collected and used to guide group interviews that explored the experiences of students and clinical supervisors with the assessment programme. We coded the transcripts and emerging topics were organised into a list of lessons learned.
The programme mainly focuses on the integration of learning and assessment by motivating and supporting students to seek and accumulate feedback. The assessment instruments were aligned to cover predefined competencies to enable aggregation of information in a structured and meaningful way. Assessments that were designed as formative learning experiences were increasingly perceived as summative by students. Peer feedback was experienced as a valuable method for formative feedback. Social interaction and external guidance seemed to be of crucial importance to scaffold self-directed learning. Aggregating data from individual assessments into a holistic portfolio judgement required expertise and extensive training and supervision of judges.
A programme of assessment with low-stakes assessments providing simultaneously formative feedback and input for summative decisions proved not easy to implement. Careful preparation and guidance of the implementation process was crucial. Assessment for learning requires meaningful feedback with each assessment. Special attention should be paid to the quality of feedback at individual assessment moments. Comprehensive attention for faculty development and training for students is essential for the successful implementation of an assessment programme.

BMC Med Educ
BMC Med Educ 2013 27;13:132. Epub 2013 Sep 27.
Study Skills Centre, Room 202, Main Arts Library, College Road, Bangor University, Bangor, Gwynedd LL57 2DF, UK.

Previous work has shown that a programme that draws on a blend of theories makes a positive difference to outcomes for students who fail and repeat their first semester at medical school. Exploration of student and teacher perspectives revealed that remediation of struggling medical students can be achieved through a cognitive apprenticeship within a small community of inquiry. This community needs expert teachers capable of performing a unique combination of roles (facilitator, nurturing mentor, disciplinarian, diagnostician and role model), with high levels of teaching presence and practical wisdom.Read More

Yet, despite participants' convergent opinions on the elements of effective remediation, significant differences were found between outcomes of students working with experienced and inexperienced teachers. The current study explores the actual practice of teachers on this remediation course, aiming to exemplify elements of our theory of remediation and explore differences between teachers.
Since it is in the classroom context that the interactions that constitute the complex process of remediation emerge, this practice-based research has focused on direct observation of classroom teaching. Nineteen hours of small group sessions were recorded and transcribed. Drawing on ethnography and sociocultural discourse analysis, selected samples of talk-in-context demonstrate how the various elements of remediation play out in practice, highlighting aspects that are most effective, and identifying differences between experienced and novice teachers.
Long-term student outcomes are strongly correlated to teacher experience (r, 0.81). Compared to inexperienced teachers, experienced teachers provide more challenging, disruptive facilitation, and take a dialogic stance that encourages more collaborative group dynamics. They are more expert at diagnosing cognitive errors, provide frequent metacognitive time-outs and make explicit links across the curriculum.
Remediation is effective in small groups where dialogue is used for collaborative knowledge construction and social regulation. This requires facilitation by experienced teachers who attend to details of both content and process, and use timely interventions to foster curiosity and the will to learn. These teachers should actively challenge students' language use, logical inconsistencies and uncertainties, problematize their assumptions, and provide a metacognitive regulatory voice that can generate attitudinal shifts and nurture the development of independent critical thinkers.

Teach Learn Med
Teach Learn Med 2013 ;25 Suppl 1:S17-25
a National Board of Medical Examiners , Philadelphia , Pennsylvania , USA.
Adv Health Sci Educ Theory Pract
Adv Health Sci Educ Theory Pract 2013 Oct 28;18(4):659-72. Epub 2012 Sep 28.
Department of Educational Research and Development, Maastricht University, P.O. Box 616, 6200 MD, Maastricht, The Netherlands,

Although elaboration has been investigated frequently, there is little evidence for the beneficial effect of elaboration in problem-based learning. A controlled experiment tested the effect of elaboration during problem-based discussion on recall. Sixty-seven students observed a video-recorded, problem-based discussion.Read More

In one experimental condition, a tutor in the video encouraged participants to elaborate by asking elaborative questions. In a second condition, the tutor asked superficial questions. After the discussion, all participants studied a text with relevant new information. Elaborative questions had no significant effect on recall of idea units from the text, p = .39, η(2) = .01. High-ability students outperformed low-ability students, p = .04, η(2) = .07, but this effect did not interact with the experimental treatment, p = .22, η(2) = .02. Suggestions for further research are presented.

J Res Med Sci
J Res Med Sci 2013 Oct;18(10):887-91
Department of Educational Development and Research, School of Health Professions Education, Maastricht University, Maastricht, The Netherlands.

One of the methods used for standard setting is the borderline regression method (BRM). This study aims to assess the reliability of BRM when the pass-fail standard in an objective structured clinical examination (OSCE) was calculated by averaging the BRM standards obtained for each station separately.
In nine stations of the OSCE with direct observation the examiners gave each student a checklist score and a global score.Read More

Using a linear regression model for each station, we calculated the checklist score cut-off on the regression equation for the global scale cut-off set at 2. The OSCE pass-fail standard was defined as the average of all station's standard. To determine the reliability, the root mean square error (RMSE) was calculated. The R (2) coefficient and the inter-grade discrimination were calculated to assess the quality of OSCE.
The mean total test score was 60.78. The OSCE pass-fail standard and its RMSE were 47.37 and 0.55, respectively. The R (2) coefficients ranged from 0.44 to 0.79. The inter-grade discrimination score varied greatly among stations.
The RMSE of the standard was very small indicating that BRM is a reliable method of setting standard for OSCE, which has the advantage of providing data for quality assurance.

Acad Med
Acad Med 2013 Sep;88(9):1346-56
Department of Educational Development and Research, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, the Netherlands.

Because successful change implementation depends on organizational readiness for change, the authors developed and assessed the validity of a questionnaire, based on a theoretical model of organizational readiness for change, designed to measure, specifically, a medical school's organizational readiness for curriculum change (MORC).
In 2012, a panel of medical education experts judged and adapted a preliminary MORC questionnaire through a modified Delphi procedure. The authors administered the resulting questionnaire to medical school faculty involved in curriculum change and tested the psychometric properties using exploratory and confirmatory factor analysis, and generalizability analysis.Read More

The mean relevance score of the Delphi panel (n = 19) reached 4.2 on a five-point Likert-type scale (1 = not relevant and 5 = highly relevant) in the second round, meeting predefined criteria for completing the Delphi procedure. Faculty (n = 991) from 131 medical schools in 56 countries completed MORC. Exploratory factor analysis yielded three underlying factors-motivation, capability, and external pressure-in 12 subscales with 53 items. The scale structure suggested by exploratory factor analysis was confirmed by confirmatory factor analysis. Cronbach alpha ranged from 0.67 to 0.92 for the subscales. Generalizability analysis showed that the MORC results of 5 to 16 faculty members can reliably evaluate a school's organizational readiness for change.
MORC is a valid, reliable questionnaire for measuring organizational readiness for curriculum change in medical schools. It can identify which elements in a change process require special attention so as to increase the chance of successful implementation.

Med Teach
Med Teach 2013 Aug 30;35(8):661-70. Epub 2013 Apr 30.
Department of Educational Development and Research FHML, Maastricht University, POB 616, NL-6200 MD Maastricht, The Netherlands.

Earlier studies suggested national culture to be a potential barrier to curriculum reform in medical schools. In particular, Hofstede's cultural dimension 'uncertainty avoidance' had a significant negative relationship with the implementation rate of integrated curricula.
However, some schools succeeded to adopt curriculum changes despite their country's strong uncertainty avoidance.Read More

This raised the question: 'How did those schools overcome the barrier of uncertainty avoidance?'
Austria offered the combination of a high uncertainty avoidance score and integrated curricula in all its medical schools. Twenty-seven key change agents in four medical universities were interviewed and transcripts analysed using thematic cross-case analysis.
Initially, strict national laws and limited autonomy of schools inhibited innovation and fostered an 'excuse culture': 'It's not our fault. It is the ministry's'. A new law increasing university autonomy stimulated reforms. However, just this law would have been insufficient as many faculty still sought to avoid change. A strong need for change, supportive and continuous leadership, and visionary change agents were also deemed essential.
In societies with strong uncertainty avoidance strict legislation may enforce resistance to curriculum change. In those countries opposition by faculty can be overcome if national legislation encourages change, provided additional internal factors support the change process.

Med Educ
Med Educ 2013 Aug;47(8):842-50
Department of Clinic Neurological Sciences, Schulich School of Medicine and Dentistry, University of Western Ontario, London, Ontario, Canada.

Research on medical learning has tended to focus on the individual learner, but a sufficient understanding of the learning process requires that attention also be paid to the essential influence of the cultural context within which learning takes place. In this study, we undertook a comparative examination of two learning cultures - those of music and medicine - in order to unearth assumptions about learning that are taken for granted within the medical culture.
We used a constructivist grounded theory approach to explore experiences of learning within the two cultures.Read More

We conducted nine focus groups (two with medical students, three with residents, four with music students) and four individual interviews (with one clinician-educator, one music educator and two doctor-musicians), for a total of 37 participants. Analysis occurred alongside and informed data collection. Themes were identified iteratively using constant comparisons.
Cultural perspectives diverged in terms of where learning should occur, what learning outcomes are desired, and how learning is best facilitated. Whereas medicine valued learning by doing, music valued learning by lesson. Whereas medical learners aimed for competence, music students aimed instead for ever-better performance. Whereas medical learners valued their teachers for their clinical skills more than for their teaching abilities, the opposite was true in music, in which teachers' instructional skills were paramount. Self-assessment challenged learners in both cultures, but medical learners viewed self-assessment as a skill they could develop, whereas music students recognised that external feedback would always be required.
This comparative analysis reveals that medicine and music make culturally distinct assumptions about teaching and learning. The contrasts between the two cultures illuminate potential vulnerabilities in the medical learning culture, including the risks inherent in its competence-focused approach and the constraints it places on its own teachers. By highlighting these vulnerabilities, we provide a stimulus for reimagining and renewing medicine's educational practices.

There is little research into how to deliver summative assessment student feedback effectively. The main aims of this study were to clarify how students engage with feedback in this context and to explore the roles of learning-related characteristics and previous and current performance.
A website was developed to deliver feedback about the objective structural clinical examination (OSCE) in various formats: station by station or on skills across stations.Read More

In total, 138 students (in the third year out of five) completed a questionnaire about goal orientation, motivation, self-efficacy, control of learning beliefs and attitudes to feedback. Individual website usage was analysed over an 8-week period. Latent class analyses were used to identify profiles of students, based on their use of different aspects of the feedback website. Differences in learning-related student characteristics between profiles were assessed using analyses of variance (anovas). Individual website usage was related to OSCE performance.
In total, 132 students (95.7%) viewed the website. The number of pages viewed ranged from two to 377 (median 102). Fifty per cent of students engaged comprehensively with the feedback, 27% used it in a minimal manner, whereas a further 23% used it in a more selective way. Students who were comprehensive users of the website scored higher on the value of feedback scale, whereas students who were minimal users scored higher on extrinsic motivation. Higher performing students viewed significantly more web pages showing comparisons with peers than weaker students did. Students who just passed the assessment made least use of the feedback.
Higher performing students appeared to use the feedback more for positive affirmation than for diagnostic information. Those arguably most in need engaged least. We need to construct feedback after summative assessment in a way that will more effectively engage those students who need the most help.

Danish and Dutch new consultants' perceptions regarding the transition to consultant were compared to gain insight into this period, particularly the influence of contextual factors concerning the organisation of specialty training and health care therein. Preparation for medical and generic competencies, perceived intensity and burnout were compared. Additionally, effects of differences in working conditions and cultural dimensions were explored.Read More

All consultants registered in the Netherlands in 2007-2009 (n = 2643) and Denmark in 2007-2010 (n = 1336) received in June 2010 and April 2011, respectively, a survey about their preparation for medical and generic competencies, perceived intensity and burnout. Power analysis resulted in required sample sizes of 542. Descriptive statistics and independent t-tests were used for analysis.
Data were available of 792 new consultants in the Netherlands and 677 Danish new consultants. Compared to their Dutch counterparts, Danish consultants perceived specialty training and the transition less intensely, reported higher levels of preparation for generic competencies and scored lower on burnout.
The importance of contextual aspects in the transition is underscored and shows that Denmark appears to succeed better in aligning training with practice. Regulations regarding working hours and progressive independence of trainees appear to facilitate the transition.

Med Educ
Med Educ 2013 Jun;47(6):585-94
Department of Clinical Neurological Sciences, Schulich School of Medicine and Dentistry, University of Western Ontario, London, Ontario, Canada.

Although feedback is widely considered essential to learning, its actual influence on learners is variable. Research on responsivity to feedback has tended to focus on individual rather than social or cultural influences on learning. In this study, we explored how feedback is handled within different professional cultures, and how the characteristics and values of a profession shape learners' responses to feedback.Read More

Using a constructivist grounded theory approach, we conducted 12 focus groups and nine individual interviews (with a total of 50 participants) across three cultures of professional training in, respectively, music, teacher training and medicine. Constant comparative analysis for recurring themes was conducted iteratively.
Each of the three professional cultures created a distinct context for learning that influenced how feedback was handled. Despite these contextual differences, credibility and constructiveness emerged as critical constants, identified by learners across cultures as essential for feedback to be perceived as meaningful. However, the definitions of credibility and constructiveness were distinct to each professional culture and the cultures varied considerably in how effectively they supported the occurrence of feedback with these critical characteristics.
Professions define credibility and constructiveness in culturally specific ways and create contexts for learning that may either facilitate or constrain the provision of meaningful feedback. Comparison with other professional cultures may offer strategies for creating a productive feedback culture within medical education.

Anat Sci Educ
Anat Sci Educ 2013 May-Jun;6(3):149-56. Epub 2012 Oct 26.
Faculty of Medicine, Department of Anatomy, Umm Al-Qura University, Makkah, Kingdom of Saudi Arabia.

Traditionally, an anatomy practical examination is conducted using a free response format (FRF). However, this format is resource-intensive, as it requires a relatively large time investment from anatomy course faculty in preparation and grading. Thus, several interventions have been reported where the response format was changed to a selected response format (SRF).Read More

However, validity evidence from those interventions has not proved entirely adequate for the practical anatomy examination, and thus, further investigation was required. In this study, the validity evidence of SRF was examined using multiple choice questions (MCQs) constructed according to different levels of Bloom's taxonomy in comparison with the traditional free response format. A group of 100 medical students registered in a gross anatomy course volunteered to be enrolled in this study. The experimental MCQ examinations were part of graded midterm and final steeplechase practical examination. Volunteer students were instructed to complete the practical examinations twice, once in each of two separate examination rooms. The two separate examinations consisted of a traditional free response format and MCQ format. Scores from the two examinations (FRF and MCQ) displayed a strong correlation, even with higher level Bloom's taxonomy questions. In conclusion, the results of this study provide empirical evidence that the SRF (MCQ) response format is a valid method and can be used as an alternative to the traditional FRF steeplechase examination.

The supervision of specialty registrars during on-call shifts is essential to ensure the quality of both health care and medical education, but has been identified as a major novelty and stressor for new consultants in the transition from specialty training. There is a paucity of research on how consultants deal with their new supervisory roles and which factors influence this process. These issues are addressed in a prospective study designed to gather insights that can inform measures to ensure the provision of high-quality supervision and specialty training.Read More

A longitudinal qualitative study was performed in the Netherlands. Semi-structured interviews were conducted with new consultants. The study was guided by an interpretative phenomenological approach until saturation was reached. At 3-month intervals between July 2011 and March 2012, eight novice consultants in internal medicine were interviewed three times each about their supervisory role while on call. Interviews focused on their preparation for the role in training, the actions they took to master the role, and their progression over time.
Three interrelated domains of relevant factors emerged from the data: preparedness; personal characteristics, and contextual characteristics. Preparedness referred to the extent to which new consultants were prepared by training to take full responsibility for registrars' actions while supervising them from a distance. Personal characteristics, such as coping strategies and views on supervision, guided consultants' development as supervisors. Essential to this process were contextual characteristics, especially those concerning the extent to which the consultant knew the registrar, was familiar with departmental procedures, and had access to support from colleagues.
New consultants should be prepared for their supervisory role by training and by being given a proper introduction to their workplace. The former requires progressive independence and exposure to supervisory tasks during specialty training; the latter requires an induction programme to enable new consultants to familiarise themselves with the departmental environment and the registrars they will be supervising.

Insight into the transition from specialist registrar to hospital consultant is needed to better align specialty training with starting as a consultant and to facilitate this transition.
This study investigates whether preparedness regarding medical and generic competencies, perceived intensity, and social support are associated with burnout among new consultants.
A population-based study among all 2643 new consultants in the Netherlands (all specialties) was conducted in June 2010.Read More

A questionnaire covering preparedness for practice, intensity of the transition, social support, and burnout was used. Structural equation modelling was used for statistical analysis.
Data from a third of the population were available (32% n = 840) (43% male/57% female). Preparation in generic competencies received lower ratings than in medical competencies. A total of 10% met the criteria for burnout and 18% scored high on the emotional exhaustion subscale. Perceived lack of preparation in generic competencies correlated with burnout (r = 0.15, p < 0.001). No such relation was found for medical competencies. Furthermore, social support protected against burnout.
These findings illustrate the relevance of generic competencies for new hospital consultants. Furthermore, social support facilitates this intense and stressful stage within the medical career.

Med Teach
Med Teach 2013 ;35 Suppl 1:S31-8
College of Medicine, King Saud bin Abdulaziz University for Health Sciences, P.O. Box 57374, Riyadh 11574, Kingdom of Saudi Arabia.

We have performed this research to assess the effect of work-place based assessment (WBA) practice on medical students' learning approaches.
The research was conducted at the King Saud bin Abdulaziz University for Health Sciences, College of Medicine from 1 March to 31 July 2012. We conducted a qualitative, phenomenological research utilizing semi-structured individual interviews with medical students exposed to WBA.Read More

The audio-taped interviews were transcribed verbatim, analyzed, and themes were identified. We preformed investigators' triangulation, member checking with clinical supervisors and we triangulated the data with a similar research performed prior to the implementation of WBA.
WBA results in variable learning approaches. Based on several affecting factors; clinical supervisors, faculty-given feedback, and assessment function, students may swing between surface, deep and effort and achievement learning approaches. Students' and supervisors' orientations on the process of WBA, utilization of peer feedback and formative rather than summative assessment facilitate successful implementation of WBA and lead to students' deeper approaches to learning. Interestingly, students and their supervisors have contradicting perceptions to WBA.
A change in culture to unify students' and supervisors' perceptions of WBA, more accommodation of formative assessment, and feedback may result in students' deeper approach to learning.

Anat Sci Educ
Anat Sci Educ 2013 Mar-Apr;6(2):114-24. Epub 2012 Aug 15.
Department of Educational Development and Research, School of Health Professions Education, Maastricht University, Maastricht, The Netherlands.

Anatomy education often consists of a combination of lectures and laboratory sessions, the latter frequently including surface anatomy. Studying surface anatomy enables students to elaborate on their knowledge of the cadaver's static anatomy by enabling the visualization of structures, especially those of the musculoskeletal system, move and function in a living human being. A recent development in teaching methods for surface anatomy is body painting, which several studies suggest increases both student motivation and knowledge acquisition.Read More

This article focuses on a teaching approach and is a translational contribution to existing literature. In line with best evidence medical education, the aim of this article is twofold: to briefly inform teachers about constructivist learning theory and elaborate on the principles of constructive, collaborative, contextual, and self-directed learning; and to provide teachers with an example of how to implement these learning principles to change the approach to teaching surface anatomy. Student evaluations of this new approach demonstrate that the application of these learning principles leads to higher student satisfaction. However, research suggests that even better results could be achieved by further adjustments in the application of contextual and self-directed learning principles. Successful implementation and guidance of peer physical examination is crucial for the described approach, but research shows that other options, like using life models, seem to work equally well. Future research on surface anatomy should focus on increasing the students' ability to apply anatomical knowledge and defining the setting in which certain teaching methods and approaches have a positive effect.

Many studies have examined how educational innovations in postgraduate medical education (PGME) impact on teaching and learning, but little is known about effects in the clinical workplace outside the strictly education-related domain. Insights into the full scope of effects may facilitate the implementation and acceptance of innovations because expectations can be made more realistic, and difficulties and pitfalls anticipated. Using workplace-based assessment (WBA) as a reference case, this study aimed to determine which types of effect are perceived by users of innovations in PGME.Read More

Focusing on WBA as a recent instance of innovation in PGME, we conducted semi-structured interviews to explore perceptions of the effects of WBA in a purposive sample of Dutch trainees and (lead) consultants in surgical and non-surgical specialties. Interviews conducted in 2011 with 17 participants were analysed thematically using template analysis. To support the exploration of effects outside the domain of education, the study design was informed by theory on the diffusion of innovations.
Six domains of effects of WBA were identified: sentiments (affinity with the innovation and emotions); dealing with the innovation; specialty training; teaching and learning; workload and tasks, and patient care. Users' affinity with WBA partly determined its effects on teaching and learning. Organisational support and the match between the innovation and routine practice were considered important to minimise additional workload and ensure that WBA was used for relevant rather than easily assessable training activities. Dealing with WBA stimulated attention for specialty training and placed specialty training on the agenda of clinical departments.
These outcomes are in line with theoretical notions regarding innovations in general and may be helpful in the implementation of other innovations in PGME. Given the substantial effects of innovations outside the strictly education-related domain, individuals designing and implementing innovations should consider all potential effects, including those identified in this study.

Med Educ
Med Educ 2013 Mar;47(3):282-91
Quality Improvement in Veterinary Education, Faculty of Veterinary Medicine, Utrecht University, Utrecht, the Netherlands.

Why and how do students seek feedback on their performance in the clinical workplace and which factors influence this? These questions have remained largely unanswered in research into workplace learning during clinical clerkships. Research on feedback has focused mainly on feedback providers. Whether and how feedback recipients actively seek feedback are under-examined issues.Read More

Research in organisational psychology has proposed a mechanism whereby feedback seeking is influenced by motives and goal orientation mediated by the perceived costs and benefits of feedback. Building on a recently published model of resident doctors' feedback-seeking behaviour, we conducted a qualitative study to explore students' feedback-seeking behaviours in the clinical workplace.
Between April and June 2011, we conducted semi-structured face-to-face interviews with veterinary medicine students in Years 5 and 6 about their feedback-seeking behaviour during clinical clerkships. In the interviews, 14 students were asked about their goals and motives for seeking feedback, the characteristics of their feedback-seeking behaviour and factors influencing that behaviour. Using template analysis, we coded the interview transcripts and iteratively reduced and displayed the data until agreement on the final template was reached.
The students described personal and interpersonal factors to explain their reasons for seeking feedback. The factors related to intentions and the characteristics of the feedback provider, and the relationship between the feedback seeker and provider. Motives relating to image and ego, particularly when students thought that feedback might have a positive effect on image and ego, influenced feedback-seeking behaviour and could induce specific behaviours related to students' orientation towards particular sources of feedback, their orientation towards particular topics for and timing of feedback, and the frequency and method of feedback-seeking behaviour.
This study shows that during clinical clerkships, students actively seek feedback according to personal and interpersonal factors. Perceived costs and benefits influenced this active feedback-seeking behaviour. These results may contribute towards the optimising and developing of meaningful educational opportunities during clerkships.

Perspect Med Educ
Perspect Med Educ 2013 Mar 15. Epub 2013 Mar 15.
Department of Primary Care and Community Care, Radboud University Nijmegen Medical Centre, PO Box 9101, Huispostnummer 117 6500 HB, Nijmegen, the Netherlands,
Minim Invasive Ther Allied Technol
Minim Invasive Ther Allied Technol 2013 Feb 10;22(1):26-32. Epub 2012 May 10.
Catharina Hospital Eindhoven, Eindhoven, the Netherlands.

There is growing pressure from the government and the public to define proficiency standards for surgical skills. Aim of this study was to estimate the reliability of the Program for Laparoscopic Urological Skills (PLUS) assessment and to set a certification standard for second-year urological residents.
Fifty participants were assessed on performance time and performance quality to investigate the reliability of the PLUS assessment.Read More

Generalisability coefficient of 0.8, on a scale of 0 to 1.0, was considered to indicate good reliability for assessment purposes. Pass/fail standards were based on laparoscopic experience: Novices, intermediates, and experts (>100 procedures). The pass/fail standards were investigated for the PLUS performances of 33 second-year urological residents.
Fifteen novices, twenty-three intermediates and twelve experts were included. An inter-trial reliability of >0.80 was reached with two trials for each task. Inter-rater reliability of the quality measurements was 0.79 for two judges. Pass/fail scores were determined for the novice/intermediate boundary and the intermediate/expert boundary. Pass rates for second-year residents were 63.64% and 9.09%, respectively.
The PLUS assessment is reliable for setting a certification standard for second-year urological residents that serves as a starting point for residents to proceed to the next level of laparoscopic competency.


In order to reduce the inconsistencies of findings and the apparent low transfer of communication skills from training to medical practice, this narrative review identifies some main gaps in research on medical communication skills training and presents insights from theories on learning and transfer to broaden the view for future research.
Relevant literature was identified using Pubmed, GoogleScholar, Cochrane database, and Web of Science; and analyzed using an iterative procedure.
Research findings on the effectiveness of medical communication training still show inconsistencies and variability.Read More

Contemporary theories on learning based on a constructivist paradigm offer the following insights: acquisition of knowledge and skills should be viewed as an ongoing process of exchange between the learner and his environment, so called lifelong learning. This process can neither be atomized nor separated from the context in which it occurs. Four contemporary approaches are presented as examples.
The following shift in focus for future research is proposed: beyond isolated single factor effectiveness studies toward constructivist, non-reductionistic studies integrating the context.
Future research should investigate how constructivist approaches can be used in the medical context to increase effective learning and transition of communication skills.

Patient Educ Couns
Patient Educ Couns 2013 Feb 29;90(2):213-9. Epub 2012 Oct 29.
CAHPRI School for Public Health and Primary Care Research, Maastricht University, Maastricht, Netherlands.

To explore characteristics of written communication guidelines that enhance the success of training aimed at the application of the recommendations in the guidelines.
Seven mixed focus groups were held consisting of communication skill teachers and communication skill learners and three groups with only learners. Analysis was done in line with principles of grounded theory.Read More

Five key attributes of guidelines for communication skill training were identified: complexity, level of detail, format and organization, type of information, and trustworthiness/validity. The desired use of these attributes is related to specific educational purposes and learners' expertise. The low complexity of current communication guidelines is appreciated, but seems ad odds with the wish for more valid communication guidelines.
Which guideline characteristics are preferred by users depends on the expertise of the learners and the educational purpose of the guideline.
Communication guidelines can be improved by modifying the key attributes in line with specific educational functions and learner expertise. For example: the communication guidelines used in GP training in the Netherlands, seem to offer an oversimplified model of doctor patient communication. This model may be suited for undergraduate learning, but does not meet the validity demands of physicians in training.

Med Teach
Med Teach 2012 4;34(11):e732-42. Epub 2012 Jun 4.
Ross University School of Medicine, PO Box 266, Roseau, Commonwealth of Dominica.

Previous work identified complex ingredients of a remediation programme for at-risk medical students: mandatory, stable, facilitated small groups promote both cognitive and affective developments, with improved self-regulation, metacognition and reflection resulting in significant performance gains.
We explore the teachers' role in this intervention, aiming to expand and deepen understanding of remediation methods in medical education.
Extensive qualitative data from student surveys and in-depth teacher interviews, along with quantitative student performance data, produced a rich description of remediation processes.Read More

Remediation should support emotional needs and foster cognitive and metacognitive skills for self-regulation and critical thinking. Teachers of remediation need to motivate, critique, challenge and advise their learners, applying teaching and contextual expertise in a constructivist, student-centred environment that fosters curiosity and joy for learning. Teachers of remediation can mediate these processes through embodiment of five core roles: facilitator, nurturing mentor, disciplinarian, diagnostician and modeller of desired skills, attitudes and behaviours.
Remediation of struggling medical students can be achieved through a cognitive apprenticeship within a small community of inquiry that motivates and challenges the students. This community needs teachers capable of performing a unique combination of roles that demands high levels of teaching presence and practical wisdom.

BMC Med Educ
BMC Med Educ 2012 20;12:79. Epub 2012 Aug 20.
Department of Medical Education, MMC Academy, Máxima Medical Center, De Run 4600, 5500 MB, Veldhoven, the Netherlands.

Medical students in clerkship are continuously confronted with real and relevant patient problems. To support clinical problem solving skills, students perform a Critical Appraisal of a Topic (CAT) task, often resulting in a paper. Because such a paper may contain errors, students could profit from discussion with peers, leading to paper revision.Read More

Active peer discussion by a Computer Supported Collaborative Learning (CSCL) environment show positive medical students perceptions on subjective knowledge improvement. High students' activity during discussions in a CSCL environment demonstrated higher task-focussed discussion reflecting higher levels of knowledge construction. However, it remains unclear whether high discussion activity influences students' decisions revise their CAT paper. The aim of this research is to examine whether students who revise their critical appraisal papers after discussion in a CSCL environment show more task-focussed activity and discuss more intensively on critical appraisal topics than students who do not revise their papers.
Forty-seven medical students, stratified in subgroups, participated in a structured asynchronous online discussion of individual written CAT papers on self-selected clinical problems. The discussion was structured by three critical appraisal topics. After the discussion, the students could revise their paper. For analysis purposes, all students' postings were blinded and analysed by the investigator, unaware of students characteristics and whether or not the paper was revised. Postings were counted and analysed by an independent rater, Postings were assigned into outside activity, non-task-focussed activity or task-focussed activity. Additionally, postings were assigned to one of the three critical appraisal topics. Analysis results were compared by revised and unrevised papers.
Twenty-four papers (51.6%) were revised after the online discussion. The discussions of the revised papers showed significantly higher numbers of postings, more task-focussed activities, and more postings about the two critical appraisal topics: "appraisal of the selected article(s)", and "relevant conclusion regarding the clinical problem".
A CSCL environment can support medical students in the execution and critical appraisal of authentic tasks in the clinical workplace. Revision of CAT papers appears to be related to discussions activity, more specifically reflecting high task-focussed activity of critical appraisal topics.

BMC Med Educ
BMC Med Educ 2012 18;12:97. Epub 2012 Oct 18.
Department of Primary Care and Community Care, Radboud University Nijmegen Medical Centre, Postbus 9101 Huispostnummer 117, Nijmegen HB 6500, The Netherlands.

Research has shown that narrative feedback, (self) reflections and a plan to undertake and evaluate improvements are key factors for effective feedback on clinical performance. We investigated the quantity of narrative comments comprising feedback (by trainers), self-reflections (by trainees) and action plans (by trainer and trainee) entered on a mini-CEX form that was modified for use in general practice training and to encourage trainers and trainees to provide narrative comments. In view of the importance of specificity as an indicator of feedback quality, we additionally examined the specificity of the comments.Read More

We collected and analysed modified mini-CEX forms completed by GP trainers and trainees. Since each trainee has the same trainer for the duration of one year, we used trainer-trainee pairs as the unit of analysis. We determined for all forms the frequency of the different types of narrative comments and rated their specificity on a three-point scale: specific, moderately specific, not specific. Specificity was compared between trainee-trainer pairs.
We collected 485 completed modified mini-CEX forms from 54 trainees (mean of 8.8 forms per trainee; range 1-23; SD 5.6). Trainer feedback was more frequently provided than trainee self-reflections, and action plans were very rare. The comments were generally specific, but showed large differences between trainee-trainer pairs.
The frequency of self-reflection and action plans varied, all comments were generally specific and there were substantial and consistent differences between trainee-trainer pairs in the specificity of comments. We therefore conclude that feedback is not so much determined by the instrument as by the users. Interventions to improve the educational effects of the feedback procedure should therefore focus more on the users than on the instruments.

Med Educ
Med Educ 2012 Nov;46(11):1087-98
Centre for Teaching and Learning, Stellenbosch University, Stellenbosch, South Africa.

We previously developed a model of the pre-assessment learning effects of consequential assessment and started to validate it. The model comprises assessment factors, mechanism factors and learning effects. The purpose of this study was to continue the validation process.Read More

For stringency, we focused on a subset of assessment factor-learning effect associations that featured least commonly in a baseline qualitative study. Our aims were to determine whether these uncommon associations were operational in a broader but similar population to that in which the model was initially derived.
A cross-sectional survey of 361 senior medical students at one medical school was undertaken using a purpose-made questionnaire based on a grounded theory and comprising pairs of written situational tests. In each pair, the manifestation of an assessment factor was varied. The frequencies at which learning effects were selected were compared for each item pair, using an adjusted alpha to assign significance. The frequencies at which mechanism factors were selected were calculated.
There were significant differences in the learning effect selected between the two scenarios of an item pair for 13 of this subset of 21 uncommon associations, even when a p-value of < 0.00625 was considered to indicate significance. Three mechanism factors were operational in most scenarios: agency; response efficacy, and response value.
For a subset of uncommon associations in the model, the role of most assessment factor-learning effect associations and the mechanism factors involved were supported in a broader but similar population to that in which the model was derived. Although model validation is an ongoing process, these results move the model one step closer to the stage of usefully informing interventions. Results illustrate how factors not typically included in studies of the learning effects of assessment could confound the results of interventions aimed at using assessment to influence learning.

BMC Med Educ
BMC Med Educ 2012 6;12:67. Epub 2012 Aug 6.
Department of Primary and Community Care, Radboud University Nijmegen Medical Centre, Radboud, The Netherlands.

Self-regulation is essential for professional development. It involves monitoring of performance, identifying domains for improvement, undertaking learning activities, applying newly learned knowledge and skills and self-assessing performance. Since self-assessment alone is ineffective in identifying weaknesses, learners should seek external feedback too.Read More

Externally regulated educational interventions, like reflection, learning portfolios, assessments and progress meetings, are increasingly used to scaffold self-regulation.The aim of this study is to explore how postgraduate trainees regulate their learning in the workplace, how external regulation promotes self-regulation and which elements facilitate or impede self-regulation and learning.
In a qualitative study with a phenomenologic approach we interviewed first- and third-year GP trainees from two universities in the Netherlands. Twenty-one verbatim transcripts were coded. Through iterative discussion the researchers agreed on the interpretation of the data and saturation was reached.
Trainees used a short and a long self-regulation loop. The short loop took one week at most and was focused on problems that were easy to resolve and needed minor learning activities. The long loop was focused on complex or recurring problems needing multiple and planned longitudinal learning activities. External assessments and formal training affected the long but not the short loop. The supervisor had a facilitating role in both loops. Self-confidence was used to gauge competence.Elements influencing self-regulation were classified into three dimensions: personal (strong motivation to become a good doctor), interpersonal (stimulation from others) and contextual (organizational and educational features).
Trainees did purposefully self-regulate their learning. Learning in the short loop may not be visible to others. Trainees should be encouraged to actively seek and use external feedback in both loops. An important question for further research is which educational interventions might be used to scaffold learning in the short loop. Investing in supervisor quality remains important, since they are close to trainee learning in both loops.

BMC Med Educ
BMC Med Educ 2012 17;12:20. Epub 2012 Apr 17.
Department of Educational Development and Research, Maastricht University, Maastricht, The Netherlands.

An assessment programme, a purposeful mix of assessment activities, is necessary to achieve a complete picture of assessee competence. High quality assessment programmes exist, however, design requirements for such programmes are still unclear. We developed guidelines for design based on an earlier developed framework which identified areas to be covered.Read More

A fitness-for-purpose approach defining quality was adopted to develop and validate guidelines.
First, in a brainstorm, ideas were generated, followed by structured interviews with 9 international assessment experts. Then, guidelines were fine-tuned through analysis of the interviews. Finally, validation was based on expert consensus via member checking.
In total 72 guidelines were developed and in this paper the most salient guidelines are discussed. The guidelines are related and grouped per layer of the framework. Some guidelines were so generic that these are applicable in any design consideration. These are: the principle of proportionality, rationales should underpin each decisions, and requirement of expertise. Logically, many guidelines focus on practical aspects of assessment. Some guidelines were found to be clear and concrete, others were less straightforward and were phrased more as issues for contemplation.
The set of guidelines is comprehensive and not bound to a specific context or educational approach. From the fitness-for-purpose principle, guidelines are eclectic, requiring expertise judgement to use them appropriately in different contexts. Further validation studies to test practicality are required.

Adv Health Sci Educ Theory Pract
Adv Health Sci Educ Theory Pract 2012 Aug 28;17(3):339-55. Epub 2011 Jun 28.
Skillslab, Faculty of Health Medicine and Life Sciences, Maastricht University, The Netherlands.

Lack of published studies on students' practice behaviour of physical examination skills outside timetabled training sessions inspired this study into what activities medical students undertake to improve their skills and factors influencing this. Six focus groups of a total of 52 students from Years 1-3 using a pre-established interview guide. Interviews were recorded, transcribed and analyzed using qualitative methods.Read More

The interview guide was based on questionnaire results; overall response rate for Years 1-3 was 90% (n = 875). Students report a variety of activities to improve their physical examination skills. On average, students devote 20% of self-study time to skill training with Year 1 students practising significantly more than Year 3 students. Practice patterns shift from just-in-time learning to a longitudinal selfdirected approach. Factors influencing this change are assessment methods and simulated/real patients. Learning resources used include textbooks, examination guidelines, scientific articles, the Internet, videos/DVDs and scoring forms from previous OSCEs. Practising skills on fellow students happens at university rooms or at home. Also family and friends were mentioned to help. Simulated/real patients stimulated students to practise of physical examination skills, initially causing confusion and anxiety about skill performance but leading to increased feelings of competence. Difficult or enjoyable skills stimulate students to practise. The strategies students adopt to master physical examination skills outside timetabled training sessions are self-directed. OSCE assessment does have influence, but learning takes place also when there is no upcoming assessment. Simulated and real patients provide strong incentives to work on skills. Early patient contacts make students feel more prepared for clinical practice.

Postgrad Med J
Postgrad Med J 2012 Aug 17;88(1042):443-50. Epub 2012 May 17.
Departments of Intensive Care and Internal Medicine Maastricht University Medical Centre, P. Debyelaan 25, 6202 AZ Maastricht, The Netherlands.

To determine whether analysis of unsolicited healthcare complaints specifically focusing on unprofessional behaviour can provide additional information from the patients' perspective.
A qualitative study with content analysis of healthcare complaints and associated judgements using complaints filed from 2004 to 2009 at the complaints committee of a tertiary-referral centre. Subsequent comparison of the resulting categories of poor professionalism to categories perceived relevant by physicians in a previous study was performed.Read More

137 complaints (98%) yielded 46 different unprofessional behaviours grouped into 18 categories. The element 'perceived medical complications and error' occurred most commonly (n=77), followed by 'having to wait for care' and 'insufficient or unclear clarification' (n=52, n=48, respectively). The combined non-cognitive elements of professionalism (especially aspects of communication) were far more prominently discussed than cognitive issues (knowledge/skills) related to medical error. Most categories of professionalism elements were considered important by physicians but, nevertheless, were identified in patient complaints analysis. Some issues (eg, 'altruism', 'appearance', 'keeping distance/respecting boundaries with patients') were not perceived as problematic by patients and/or relatives, while mentioned by physicians. Conversely, eight categories of poor professionalism revealed from complaint analysis (eg, 'having to wait for care', 'lack of continuity of care' and 'lack of shared decision making') were not considered essential by physicians.
The vast majority of unprofessional behaviour identified related to non-cognitive, professionalism aspects of care. Complaints pertaining to unsatisfactory communication were especially noticeable. Incongruence is noted between the physicians' and the patients' perception of actual care.

Med Educ
Med Educ 2012 Aug;46(8):738-47
Department of Educational Development and Research, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, the Netherlands.

Medical schools worldwide are increasingly switching to student-centred methods such as problem-based learning (PBL) to foster lifelong self-directed learning (SDL). The cross-cultural applicability of these methods has been questioned because of their Western origins and because education contexts and learning approaches differ across cultures.
This study evaluated PBL's cross-cultural applicability by investigating how it is applied in three medical schools in regions with different cultures in, respectively, East Asia, the Middle East and Western Europe.Read More

Specifically, it investigated how students' cultural backgrounds impact on SDL in PBL and how this impact affects students.
A qualitative, cross-cultural, comparative case study was conducted in three medical schools. Data were collected through 88 semi-structured, in-depth interviews with Year 1 and 3 students, tutors and key persons involved in PBL, 32 observations of Year 1 and 3 PBL tutorials, document analysis, and contextual information. The data were thematically analysed using the template analysis method. Comparisons were made among the three medical schools and between Year 1 and 3 students across and within the schools.
The cultural factors of uncertainty and tradition posed a challenge to Middle Eastern students' SDL. Hierarchy posed a challenge to Asian students and achievement impacted on both sets of non-Western students. These factors were less applicable to European students, although the latter did experience some challenges. Several contextual factors inhibited or enhanced SDL across the cases. As students grew used to PBL, SDL skills increased across the cases, albeit to different degrees.
Although cultural factors can pose a challenge to the application of PBL in non-Western settings, it appears that PBL can be applied in different cultural contexts. However, its globalisation does not postulate uniform processes and outcomes, and culturally sensitive alternatives might be developed.

Med Teach
Med Teach 2012 ;34(9):683-97
Department of Educational Development and Research, Maastricht University, The Netherlands.

There has been increasing use and significance of progress testing in medical education. It is used in many ways and with several formats to reflect the variety of curricula and assessment purposes. These developments have occurred alongside a recognised sensitivity for error variance inherent in multiple choice tests from which challenges to its validity and reliability have arisen.Read More

This Guide presents a generic, systemic framework to help identify and explore improvements in the quality and defensibility of progress test data. The framework draws on the combined experience of the Dutch consortium, an individual medical school in the United Kingdom, and the bulk of the progress test literature to date. It embeds progress testing as a quality-controlled assessment tool for improving learning, teaching and the demonstration of educational standards. The paper describes strengths, highlights constraints and explores issues for improvement. These may assist in the establishment of potential or new progress testing in medical education programmes. They can also guide the evaluation and improvement of existing programmes.

Conceptions of medical teachers regarding learning and teaching affect their teaching practice. Therefore conceptions should be addressed in faculty development.
To facilitate this, we constructed the Conceptions Of Learning and Teaching (COLT) instrument.Read More

The COLT was adapted based on experts' comments during a meeting and interviews, followed by a Delphi procedure (Part I). It was administered to teachers from two Dutch medical schools with different traditions in student-centred education (Part II; N=646). The data were analyzed using confirmatory factor analysis and reliability analysis.
324 Teachers (50.2%) completed the questionnaire. Confirmatory factor analysis did not confirm the underlying theoretical model, but an alternative model demonstrated a good fit. This led to an instrument with eighteen items reflecting three underlying factors: 'teacher centredness', 'appreciation of active learning', and 'orientation to professional practice'. We found significant differences in COLT scores between the faculty of the two medical schools.
The COLT appears to be a construct valid tool resulting in reliable scores of teachers' conceptions of learning and teaching, in student-centred medical education. Two of the three factors are new and may be specific for student-centred medical education. The COLT may be a promising tool to improve faculty development.

BMC Med Educ
BMC Med Educ 2012 16;12. Epub 2012 Mar 16.
Centre for Teaching and Learning, Stellenbosch University, Matieland 7602, South Africa.

No validated model exists to explain the learning effects of assessment, a problem when designing and researching assessment for learning. We recently developed a model explaining the pre-assessment learning effects of summative assessment in a theory teaching context. The challenge now is to validate this model.Read More

The purpose of this study was to explore whether the model was operational in a clinical context as a first step in this process.
Given the complexity of the model, we adopted a qualitative approach. Data from in-depth interviews with eighteen medical students were subject to content analysis. We utilised a code book developed previously using grounded theory. During analysis, we remained alert to data that might not conform to the coding framework and open to the possibility of deploying inductive coding. Ethical clearance and informed consent were obtained.
The three components of the model i.e., assessment factors, mechanism factors and learning effects were all evident in the clinical context. Associations between these components could all be explained by the model. Interaction with preceptors was identified as a new subcomponent of assessment factors. The model could explain the interrelationships of the three facets of this subcomponent i.e., regular accountability, personal consequences and emotional valence of the learning environment, with previously described components of the model.
The model could be utilized to analyse and explain observations in an assessment context different to that from which it was derived. In the clinical setting, the (negative) influence of preceptors on student learning was particularly prominent. In this setting, learning effects resulted not only from the high-stakes nature of summative assessment but also from personal stakes, e.g. for esteem and agency. The results suggest that to influence student learning, consequences should accrue from assessment that are immediate, concrete and substantial. The model could have utility as a planning or diagnostic tool in practice and research settings.

Med Educ
Med Educ 2012 Jun;46(6):593-603
Department of Clinical Neurological Sciences, Schulich School of Medicine and Dentistry, University of Western Ontario, London, Ontario, Canada.

Regulatory focus theory posits the existence of two systems of self-regulation underlying human motivation: promotion focus, which is concerned with aspirations and accomplishments, and prevention focus, which is concerned with obligations and responsibilities. It has been proposed that regulatory focus theory may help to explain learners' variable responses to feedback, predicting that positive feedback is motivating under promotion focus, whereas negative feedback is motivating under prevention focus. We aimed to explore this link between regulatory focus theory and response to feedback using data collected in a naturalistic setting.Read More

In a constructivist grounded theory study, we interviewed 22 early-career academic doctors about experiences they perceived as influential in their learning. Although feedback emerged as important, responses to feedback were highly variable. To better understand how feedback becomes (or fails to become) influential, we used the theoretical framework of regulatory focus to re-examine all descriptions of experiences of receiving and responding to feedback.
Feedback could be influential or non-influential, regardless of its sign (positive or negative). In circumstances in which the individual's regulatory focus was readily determined, such as in choosing a career (promotion) or preparing for a high-stakes examination (prevention), the apparent influence of feedback was consistent with the prediction of regulatory focus theory. However, we encountered many challenges in applying regulatory focus theory to real feedback scenarios, including the frequent presence of a mixed regulatory focus, the potential for regulatory focus to change over time, and the competing influences of other factors, such as the perceived credibility of the source or content of the feedback.
Regulatory focus theory offers a useful, if limited, construct for exploring learners' responses to feedback in the clinical setting. The insights and predictions it offers must be considered in light of the motivational complexity of clinical learning tasks and of other factors influencing the impact of feedback.

Med Educ
Med Educ 2012 Jun;46(6):604-12
Department of Primary Care and Community Care, Radboud University Nijmegen Medical Centre, Nijmegen, the Netherlands.

Feedback in workplace-based clinical settings often relies on expert trainers' judgements of directly observed trainee performance. There is ample literature on effective feedback, but in practice trainees in workplace-based training are not regularly observed. We aimed to examine external conditions that impact feedback in observational workplace-based assessment (WBA).Read More

Interviews were conducted and the resulting data analysed using a qualitative, phenomenological approach. Between October 2009 and January 2010, we interviewed 22 postgraduate general practice trainees at two institutions in the Netherlands. Three researchers analysed the transcripts of the interviews.
A three-step scheme emerged from the data. Feedback as part of WBA is of greater benefit to trainees if: (i) observation and feedback are planned by the trainee and trainer; (ii) the content and delivery of the feedback are adequate, and (iii) the trainee uses the feedback to guide his or her learning by linking it to learning goals. Negative emotions reported by almost all trainees in relation to observation and feedback led to different responses. Some trainees avoided observation, whereas others overcame their apprehension and actively sought observation and feedback. Active trainers were able to help trainees overcome their fears. Four types of trainer-trainee pairs were distinguished according to their engagement in observation and feedback. External requirements set by training institutions may stimulate inactive trainers and trainees.
In line with the literature, our results emphasise the importance of the content of feedback and the way it is provided, as well as the importance of its incorporation in trainees' learning. Moreover, we highlight the step before the actual feedback itself. The way arrangements for feedback are made appears to be important to feedback in formative WBA. Finally, we outline several factors that influence the success or failure of feedback but precede the process of observation and feedback.

Studies that compare the effects of problem-based and conventional medical curricula on student performance generally show no significant differences. However, curriculum comparison studies are at best quasi-experimental, so they are prone to forms of selection bias. The purpose of this study was to reanalyze data from such comparisons by controlling for two potential biases: differential student attrition and differential exposure.Read More

The authors reanalyzed 104 previously published comparisons involving a single, problem-based medical school in the Netherlands (Maastricht University's medical school), using student attrition and study duration data from this school and the schools with which it was compared. The authors removed bias by reequalizing the comparison groups in terms of attrition and study duration.
The uncorrected data showed no differences between problem-based and conventional curricula: Mean effect sizes as expressed by Cohen d were 0.02 for medical knowledge and 0.07 for diagnostic reasoning. However, the reanalysis demonstrated medium-level effect sizes favoring the problem-based curriculum. After corrections for attrition and study duration, the mean effect size for knowledge acquisition was 0.31 and for diagnostic reasoning was 0.51.
Effects of the Maastricht problem-based curriculum were masked by differential attrition and differential exposure in the original studies. Because this school has been involved in many studies included in influential literature reviews published in the past 20 years, the authors' findings have implications for the assessment of the value of problem-based learning put forward by these reviews.

  Consultants in charge of postgraduate medical education (PGME) in hospital departments ('lead consultants') are responsible for the implementation of educational change. Although difficulties in innovating in medical education are described in the literature, little is known about how lead consultants approach educational change.
  This study was conducted to explore lead consultants' approaches to educational change in specialty training and factors influencing these approaches.Read More

  From an interpretative constructivist perspective, we conducted a qualitative exploratory study using semi-structured interviews with a purposive sample of 16 lead consultants in the Netherlands between August 2010 and February 2011. The study design was based on the research questions and notions from corporate business and social psychology about the roles of change managers. Interview transcripts were analysed thematically using template analysis.
  The lead consultants described change processes with different stages, including cause, development of content, and the execution and evaluation of change, and used individual change strategies consisting of elements such as ideas, intentions and behaviour. Communication is necessary to the forming of a strategy and the implementation of change, but the nature of communication is influenced by the strategy in use. Lead consultants differed in their degree of awareness of the strategies they used. Factors influencing approaches to change were: knowledge, ideas and beliefs about change; level of reflection; task interpretation; personal style, and department culture.
  Most lead consultants showed limited awareness of their own approaches to change. This can lead them to adopt a rigid approach, whereas the ability to adapt strategies to circumstances is considered important to effective change management. Interventions and research should be aimed at enhancing the awareness of lead consultants of approaches to change in PGME.

Adv Health Sci Educ Theory Pract
Adv Health Sci Educ Theory Pract 2012 Mar 3;17(1):39-53. Epub 2011 Apr 3.
Centre for Teaching and Learning, Stellenbosch University, Private Bag X1, Matieland, 7602, South Africa.

It has become axiomatic that assessment impacts powerfully on student learning. However, surprisingly little research has been published emanating from authentic higher education settings about the nature and mechanism of the pre-assessment learning effects of summative assessment. Less still emanates from health sciences education settings.Read More

This study explored the pre-assessment learning effects of summative assessment in theoretical modules by exploring the variables at play in a multifaceted assessment system and the relationships between them. Using a grounded theory strategy, in-depth interviews were conducted with individual medical students and analyzed qualitatively. Respondents' learning was influenced by task demands and system design. Assessment impacted on respondents' cognitive processing activities and metacognitive regulation activities. Individually, our findings confirm findings from other studies in disparate non-medical settings and identify some new factors at play in this setting. Taken together, findings from this study provide, for the first time, some insight into how a whole assessment system influences student learning over time in a medical education setting. The findings from this authentic and complex setting paint a nuanced picture of how intricate and multifaceted interactions between various factors in an assessment system interact to influence student learning. A model linking the sources, mechanism and consequences of the pre-assessment learning effects of summative assessment is proposed that could help enhance the use of summative assessment as a tool to augment learning.

Adv Health Sci Educ Theory Pract
Adv Health Sci Educ Theory Pract 2012 Mar 1;17(1):81-93. Epub 2011 May 1.
Department of Intensive Care, Maastricht University Medical Centre, P. Debyelaan 25, 6202 AZ, Maastricht, The Netherlands.

Although other web-based approaches to assessment of professional behaviour have been studied, no publications studying the potential advantages of a web-based instrument versus a classic, paper-based method have been published to date. This study has two research goals: it focuses on the quantity and quality of comments provided by students and their peers (two researchers independently scoring comments as correct and incorrect in relation to five commonly used feedback rules (and resulting in an aggregated score of the five scores) on the one, and on the feasibility, acceptability and perceived usefulness of the two approaches on the other hand (using a survey). The amount of feedback was significantly higher in the web-based group than in the paper based group for all three categories (dealing with work, others and oneself).Read More

Regarding the quality of feedback, the aggregated score for each of the three categories was not significantly different between the two groups, neither for the interim, nor for the final assessment. Some, not statistically significant, but nevertheless noteworthy trends were nevertheless noted. Feedback in the web-based group was more often unrelated to observed behaviour for several categories for both the interim and final assessment. Furthermore, most comments relating to the category 'Dealing with oneself' consisted of descriptions of a student's attendance, thereby neglecting other aspects of personal functioning. The survey identified significant differences between the groups for all questionnaire items regarding feasibility, acceptability and perceived usefulness in favour of the paper-based form. The use of a web-based instrument for professional behaviour assessment yielded a significantly higher number of comments compared to the traditional paper-based assessment. Unfortunately, the quality of the feedback obtained by the web-based instrument as measured by several generally accepted feedback criteria did not parallel this increase.

Med Teach
Med Teach 2012 ;34 Suppl 1:S42-50
College of Medicine, King Saud bin Abdulaziz University for Health Sciences, PO Box 57374, Riyadh 11574, Kingdom of Saudi Arabia.

"Assessment steers students' learning" is a statement that has been used repeatedly without solid evidence in the literature. This manuscript aims to evaluate the published literatures on the effect of teaching learning environment in particular, the implemented assessment on students' learning approaches. A literature review was performed on the effect of assessment on students learning approaches.Read More

English language literatures were searched in Pubmed, PsycINFO, and Medline without restriction to type or date of publication. Reviewing the literature, the most prominent identified theme was assessment function characterized in summative and formative assessment and general effect of assessment on students' learning approaches. The literature review has pointed clearly to the complexity of the relationship between learning environment, students' perceptions of assessment demands, and students' approaches to learning. Many factors (extrinsic and intrinsic) were theoretically proposed to mediate students' approaches to learning in response to their assessment. However, few of these factors were researched in the published literature. Formative assessment is likely to contribute to students' deep approach to learning while summative is likely to contribute to their surface approach. However, these effects are not definite and further research about the complex relationship between assessment and students' learning is required.

Perspect Med Educ
Perspect Med Educ 2012 Mar 10;1(1):24-30. Epub 2012 Mar 10.
Flinders Innovation in Clinical Education, Flinders University, Adelaide, Australia ; Department of Educational Development and Research, Maastricht University, Maastricht, the Netherlands.

Progress testing is gaining ground rapidly after having been used almost exclusively in Maastricht and Kansas City. This increased popularity is understandable considering the intuitive appeal longitudinal testing has as a way to predict future competence and performance. Yet there are also important practicalities.Read More

Progress testing is longitudinal assessment in that it is based on subsequent equivalent, yet different, tests. The results of these are combined to determine the growth of functional medical knowledge for each student, enabling more reliable and valid decision making about promotion to a next study phase. The longitudinal integrated assessment approach has a demonstrable positive effect on student learning behaviour by discouraging binge learning. Furthermore, it leads to more reliable decisions as well as good predictive validity for future competence or retention of knowledge. Also, because of its integration and independence of local curricula, it can be used in a multi-centre collaborative production and administration framework, reducing costs, increasing efficiency and allowing for constant benchmarking. Practicalities include the relative unfamiliarity of faculty with the concept, the fact that remediation for students with a series of poor results is time consuming, the need to embed the instrument carefully into the existing assessment programme and the importance of equating subsequent tests to minimize test-to-test variability in difficulty. Where it has been implemented-collaboratively-progress testing has led to satisfaction, provided the practicalities are heeded well.

Med Educ
Med Educ 2012 Feb;46(2):192-200
Department of Clinical Neurological Sciences, Schulich School of Medicine and Dentistry, University of Western Ontario, London, Ontario, Canada.

How learners interpret their clinical experiences to create meaningful learning has not been well studied. We explored experiences considered by doctors to be influential in their learning in order to better understand this process.
Using a grounded theory approach, we interviewed 22 academic doctors who had been in practice for ≤ 5 years.Read More

Participants were asked to reflect on experiences they considered to have been influential during their training. Constant comparative analysis for emerging themes was conducted iteratively with data collection.
A model of clinical learning emerged in which the clinical work itself is central. As they observe and participate in clinical work, learners can attend to a variety of sources of information that facilitate the interpretation of the experience and the construction of knowledge from it. These 'learning cues' include feedback, role models, clinical outcomes, patient or family responses, and comparisons with peers. The integration of a cue depends on the learner's judgement of its credibility. Certain cues, such as clinical outcomes or feedback from patients, are seen as innately credible, whereas other cues, particularly feedback from supervisors, are subjected to critical judgement.
Learners make complex judgements regarding the credibility of information about clinical performance. Credibility judgements influence the learning that arises from the clinical experience. Further understanding of how such judgements are made could guide educators in providing credible information to learners.

Med Educ Online
Med Educ Online 2012 17;17:11204. Epub 2012 Feb 17.
College of Medicine, King Saud bin Abdulaziz University for Health Sciences, Riyadh, Saudi Arabia.

Self-assessment is recognized as a necessary skill for lifelong learning. It is widely reported to offer numerous advantages to the learner. The research evaluated the impact of students' and supervisors' self-assessment and feedback training on students' perceptions and practices of self-assessment.Read More

Moreover, it evaluated the effect of self-assessment process on students' study strategies within a community of clinical practice.
We conducted a qualitative phenomenological study from May 2008 to December 2009. We held 37 semi-structured individual interviews with three different cohorts of undergraduate medical students until we reached data saturation. The cohorts were exposed to different contexts while experiencing their clinical years' assessment program. In the interviews, students' perceptions and interpretations of 'self-assessment practice' and 'supervisor-provided feedback' within different contexts and the resulting study strategies were explored.
The analysis of interview data with the three cohorts of students yielded three major themes: strategic practice of self-assessment, self-assessment and study strategies, and feedback and study strategies. It appears that self-assessment is not appropriate within a summative context, and its implementation requires cultural preparation. Despite education and orientation on the two major components of the self-assessment process, feedback was more effective in enhancing deeper study strategies.
This research suggests that the theoretical advantages linked to the self-assessment process are a result of its feedback component rather than the practice of self-assessment isolated from feedback. Further research exploring the effects of different contextual and personal factors on students' self-assessment is needed.

Med Teach
Med Teach 2012 ;34(3):226-31
Department of Educational Development and Research, Faculty of Health, Medicine and Life Sciences, Maastricht University, The Netherlands.

The differences of learning experiences in the workplace put challenges on the assessment: the assessment programme should be aligned with the general competency framework of the curriculum and also fit to the differences in learning contexts of the workplace. We used van der Vleuten's programmatic assessment model to develop a workplace-based assessment programme for final year clerkships. We aimed to design a programme that stimulates learning, supports assessment decision, is feasible and non-bureaucratic.Read More

The first experiences with the programme show that students think that the programme has high learning value and the assessment is sufficiently robust. Many of the commonly reported weaknesses of work-based assessment (not a good fit with the educational context, too complex, too bureaucratic and too much work) were not mentioned by the students.

Med Educ
Med Educ 2012 Jan;46(1):38-48
Flinders Innovation in Clinical Education, Flinders University, South Australia, Australia.

Programmatic assessment is a notion that implies that the strength of the assessment process results from a careful combination of various assessment instruments. Accordingly, no single instrument is superior to another, but each has its own strengths, weaknesses and purpose in a programme. Yet, in terms of psychometric methods, a one-size-fits-all approach is often used.Read More

Kane's views on validity as represented by a series of arguments provide a useful framework from which to highlight the value of different widely used approaches to improve the quality and validity of assessment procedures.
In this paper we discuss four inferences which form part of Kane's validity theory: from observations to scores; from scores to universe scores; from universe scores to target domain, and from target domain to construct. For each of these inferences, we provide examples and descriptions of approaches and arguments that may help to support the validity inference.
As well as standard psychometric methods, a programme of assessment makes use of various other arguments, such as: item review and quality control, structuring and examiner training; probabilistic methods, saturation approaches and judgement processes, and epidemiological methods, collation, triangulation and member-checking procedures. In an assessment programme each of these can be used.


Changing demands from society and the veterinary profession call for veterinary medical curricula that can deliver veterinarians who are able to integrate specific and generic competencies in their professional practice. This requires educational innovation directed by an integrative veterinary competency framework to guide curriculum development. Given the paucity of relevant information from the veterinary literature, a qualitative multi-method study was conducted to develop and validate such a framework.Read More

A competency framework was developed based on the analysis of focus group interviews with 54 recently graduated veterinarians and clients and subsequently validated in a Delphi procedure with a panel of 29 experts, representing the full range and diversity of the veterinary profession. The study resulted in an integrated competency framework for veterinary professionals, which consists of 16 competencies organized in seven domains: veterinary expertise, communication, collaboration, entrepreneurship, health and welfare, scholarship, and personal development. Training veterinarians who are able to use and integrate the seven domains in their professional practice is an important challenge for today's veterinary medical schools. The Veterinary Professional (VetPro) framework provides a sound empirical basis for the ongoing debate about the direction of veterinary education and curriculum development.

Med Educ
Med Educ 2011 Sep;45(9):927-38
Department of Medicine, Uniformed Services University of the Health Sciences (USUHS), 4301 JonesBridge Road, Bethesda, MD 20814, USA.

Prior work has found that a doctor's clinical reasoning performance varies on a case-by-case (situation) basis; this is often referred to as 'context specificity'. To explore the influence of context on diagnostic and therapeutic clinical reasoning, we constructed a series of videotapes to which doctors were asked to respond, modifying different contextual factors (patient, doctor, setting). We explored how these contextual factors, as displayed by videotape encounters, may have influenced the clinical reasoning of board-certified internists (experts).Read More

Our purpose was to clarify the influence of context on reasoning, to build upon education theory and to generate implications for education practice.
Qualitative data about experts were gathered from two sources: think-aloud protocols reflecting concurrent thought processes that occurred while board-certified internists viewed videotape encounters, and free-text responses to queries that explicitly asked these experts to comment on the influence of selected contextual factors on their clinical reasoning processes. These data sources provided both actual performance data (think-aloud responses) and opinions on reflection (free-text answers) regarding the influence of context on reasoning. Results for each data source were analysed for emergent themes and then combined into a unified theoretical model.
Several themes emerged from our data and were broadly classified as components influencing the impact of contextual factors, mechanisms for addressing contextual factors, and consequences of contextual factors for patient care. Themes from both data sources had good overlap, indicating that experts are somewhat cognisant of the potential influences of context on their reasoning processes; notable exceptions concerned the themes of missed key findings, balancing of goals and the influence of encounter setting, which emerged in the think-aloud but not the free-text analysis.
Our unified model is consistent with the tenets of cognitive load, situated cognition and ecological psychology theories. A number of potentially modifiable influences on clinical reasoning were identified. Implications for doctor training and practice are discussed.

Med Teach
Med Teach 2011 ;33(10):783-97
Flinders University, Adelaide 5001, South Australia, Australia.

There are no scientific theories that are uniquely related to assessment in medical education. There are many theories in adjacent fields, however, that can be informative for assessment in medical education, and in the recent decades they have proven their value. In this AMEE Guide we discuss theories on expertise development and psychometric theories, and the relatively young and emerging framework of assessment for learning.Read More

Expertise theories highlight the multistage processes involved. The transition from novice to expert is characterised by an increase in the aggregation of concepts from isolated facts, through semantic networks to illness scripts and instance scripts. The latter two stages enable the expert to recognise the problem quickly and form a quick and accurate representation of the problem in his/her working memory. Striking differences between experts and novices is not per se the possession of more explicit knowledge but the superior organisation of knowledge in his/her brain and pairing it with multiple real experiences, enabling not only better problem solving but also more efficient problem solving. Psychometric theories focus on the validity of the assessment - does it measure what it purports to measure and reliability - are the outcomes of the assessment reproducible. Validity is currently seen as building a train of arguments of how best observations of behaviour (answering a multiple-choice question is also a behaviour) can be translated into scores and how these can be used at the end to make inferences about the construct of interest. Reliability theories can be categorised into classical test theory, generalisability theory and item response theory. All three approaches have specific advantages and disadvantages and different areas of application. Finally in the Guide, we discuss the phenomenon of assessment for learning as opposed to assessment of learning and its implications for current and future development and research.

Eval Program Plann
Eval Program Plann 2011 Aug 5;34(3):206-16. Epub 2011 Mar 5.
Utrecht University, The Netherlands.

The goal of this article is to contribute to the validation of a self-evaluation method, which can be used by schools to evaluate the quality of their Competence Assessment Program (CAP). The outcomes of the self-evaluations of two schools are systematically compared: a novice school with little experience in competence-based education and assessment, and an innovative school with extensive experience. The self-evaluation was based on 12 quality criteria for CAPs, including both validity and reliability, and criteria stressing the importance of the formative function of assessment, such as meaningfulness and educational consequences.Read More

In each school, teachers, management and examination board participated. Results show that the two schools use different approaches to assure assessment quality. The innovative school seems to be more aware of its own strengths and weaknesses, to have a more positive attitude towards teachers, students, and educational innovations, and to explicitly involve stakeholders (i.e., teachers, students, and the work field) in their assessments. This school also had a more explicit vision of the goal of competence-based education and could design its assessments in accordance with these goals.

Med Educ
Med Educ 2011 Aug;45(8):807-17
Department of Medicine, Faculty of School of Medicine, Uniformed Services University of the Health Sciences, 4301 Jones Bridge Road, Bethesda, MD 20814, USA.

This study aimed to investigate the relationship between the authenticity of instructional formats and outcome measures within a pre-clerkship clinical reasoning course.
We conducted a randomised, prospective, crossover study with Year 2 medical students taking a pre-clerkship clinical reasoning course. Students were randomised to small groups and exposed to three formats of differing instructional authenticity (paper case, DVD presentation, standardised patient [SP] presentation) across three subject areas (abdominal pain, anaemia, polyuria).Read More

Three student cohorts were taught using one instructional format per subject area so that each cohort received a different instructional format for each of the three subject areas. Outcome measures (objective structured clinical examination, video quiz, written examination) were selected to determine the effect of each instructional format on the clinical reasoning of students.
Increasingly authentic instructional formats did not significantly improve clinical reasoning performance across all outcome measures and subject areas. However, the results of the video quiz showed significant differences in the anaemia subject area between students who had been instructed using the paper case and live SP-based formats (scores of 47.4 and 57.6, respectively; p = 0.01) and in the abdominal pain subject area, in which students instructed using the DVD format scored higher than students instructed using either the paper case or SP-based formats (scores of 41.6, 34.9 and 31.2, respectively; p=0.002).
Increasing the authenticity of instructional formats does not appear to significantly improve clinical reasoning performance in a pre-clerkship course. Medical educators should balance increases in authenticity with factors such as cognitive load, subject area and learner experience when designing new instructional formats.

Med Educ
Med Educ 2011 Jun;45(6):578-84
EMGO Institute for Health and Care Research, Vrije Universiteit University Medical Centre, Amsterdam, The Netherlands.

Training in and assessment of consultation skills are high on the agenda of vocational training institutes for postgraduate training. There is a need to establish valid and reliable instruments to assess consultation skills in authentic settings. We investigated the number of assessors and observations needed to achieve reliable assessments of the consultation skills of general practice trainees (GPTs) using a communication instrument (MAAS-Global) and either standardised patient (SP) encounters or videotaped real patient (RP) encounters.Read More

Eight teachers at the Vrije Universiteit (VU) University Medical Centre in Amsterdam attended a training course on the use of the MAAS-Global instrument, which they subsequently used to assess the consultation skills of 53 GPTs in 176 videotaped consultations (102 with SPs, 74 with RPs). All consultations were randomly allocated and assessed by two teachers independently. The reliability of the ratings was estimated using generalisability theory.
It was easier to obtain acceptable reliability using RP consultations than SP consultations. Two assessors and five consultations were required to achieve minimal reliability (generalisability coefficient 0.7) with RPs, whereas three assessors and 30 consultations were needed to achieve minimal reliability with SPs.
Inter-observer and context variability in the assessment of the consultation skills of GPTs remains high. To achieve acceptable levels of reliability, large samples of observations are required in both formats, but, interestingly, RP encounters require a smaller sample than SP encounters.

Med Educ
Med Educ 2011 Jun;45(6):636-47
Office of Continuing Medical Education, Faculty of Medicine, Dalhousie University, Halifax, Nova Scotia, Canada.

Conceptualisations of self-assessment are changing as its role in professional development comes to be viewed more broadly as needing to be both externally and internally informed through activities that enable access to and the interpretation and integration of data from external sources. Education programmes use various activities to promote learners' reflection and self-direction, yet we know little about how effective these activities are in 'informing' learners' self-assessments.
This study aimed to increase understanding of the specific ways in which undergraduate and postgraduate learners used learning and assessment activities to inform self-assessments of their clinical performance.Read More

We conducted an international qualitative study using focus groups and drawing on principles of grounded theory. We recruited volunteer participants from three undergraduate and two postgraduate programmes using structured self-assessment activities (e.g. portfolios). We asked learners to describe their perceptions of and experiences with formal and informal activities intended to inform self-assessment. We conducted analysis as a team using a constant comparative process.
Eighty-five learners (53 undergraduate, 32 postgraduate) participated in 10 focus groups. Two main findings emerged. Firstly, the perceived effectiveness of formal and informal assessment activities in informing self-assessment appeared to be both person- and context-specific. No curricular activities were considered to be generally effective or ineffective. However, the availability of high-quality performance data and standards was thought to increase the effectiveness of an activity in informing self-assessment. Secondly, the fostering and informing of self-assessment was believed to require credible and engaged supervisors.
Several contextual and personal conditions consistently influenced learners' perceptions of the extent to which assessment activities were useful in informing self-assessments of performance. Although learners are not guaranteed to be accurate in their perceptions of which factors influence their efforts to improve performance, their perceptions must be taken into account; assessment strategies that are perceived as providing untrustworthy information can be anticipated to have negligible impact.

Despite its ubiquitous use over the past 4 decades, there is no structured, formal method with which to assess polypectomy.
To develop and validate a new method with which to assess competency in polypectomy.
Polypectomy underwent task deconstruction, and a structured checklist and global assessment scale were developed (direct observation of polypectomy skills [DOPyS]).Read More

Sixty bowel cancer screening polypectomy videos were randomly chosen for analysis and were scored independently by 7 expert assessors by using DOPyS. Each parameter and the global rating were scored from 1 to 4 (scores ≥3 = competency). The scores were analyzed by using generalizability theory (G theory).
Fifty-nine of the 60 videos were assessable and scored. The majority of the assessors agreed across the pass/fail divide for the global assessment scale in 58 of 59 (98%) polyps. For G-theory analysis, 47 of the 60 videos were analyzed. G-theory analysis suggested that DOPyS is a reliable assessment tool, provided that it is used by 2 assessors to score 5 polypectomy videos all performed by 1 endoscopist. DOPyS scores obtained in this format would reflect the endoscopist's competence.
Small sample and polyp size.
This study is the first attempt to develop and validate a tool designed specifically for the assessment of technical skills in performing polypectomy. G-theory analysis suggests that DOPyS could reliably reflect an endoscopist's competence in performing polypectomy provided a requisite number of assessors and cases were used.

Medical curricula should focus on the future of health care. Contemporary competency frameworks for curriculum design such as Canadian Medical Education Directions for Specialists (CanMEDS), ACGME and Tomorrow's Doctors share this vision by stressing generic competencies.
The objective of this study was to investigate how well a contemporary competency framework fits in with clinicians' perspectives on future health care.Read More

Using a strategic planning approach, a semi-structured open-ended questionnaire on the future of their profession was sent to 102 Dutch gynecologists. Through inductive analysis, a future perspective and its needed competencies were identified and compared to the CanMEDS framework.
The 62 responses showed content validity for the CanMEDS roles. Additionally, two roles were identified: advanced technology user and entrepreneur. Within the role Communicator, the focus will change through more active patient participation. The roles Collaborator and Manager are predicted to change in focus because of an increase of complex interdisciplinary teamwork and leadership roles.
By studying the Dutch gynecologists' perspective of the future in a strategic planning approach, two additional roles and focus areas within a contemporary competency framework were identified. The perspective of clinicians on future health care provides valuable messages on how to design future-proof curricula.

Med Educ
Med Educ 2011 May;45(5):438-9
Department of Educational Research and Development, Faculty of Health, Medicine and Life Sciences, Maastricht University, The Netherlands.
Med Teach
Med Teach 2011 ;33(6):478-85
Department of Educational Development and Research, Maastricht University, The Netherlands.

In assessment a considerable shift in thinking has occurred from assessment of learning to assessment for learning. This has important implications for the conceptual framework from which to approach the issue of assessment, but also with respect to the research agenda. The main conceptual changes pertain to programmes of assessment.Read More

This has led to a broadened perspective on the types of construct assessment tries to capture, the way information from various sources is collected and collated, the role of human judgement and the variety of psychometric methods to determine the quality of the assessment. Research into the quality of assessment programmes, how assessment influences learning and teaching, new psychometric models and the role of human judgement is much needed.

Med Educ
Med Educ 2011 Apr;45(4):329-38
Department of Neurology and Neurosurgery, McGill University, Montreal, Quebec, Canada.

Script concordance test (SCT) scores are intended to reflect respondents' competence in interpreting clinical data under conditions of uncertainty. The validity of inferences based on SCT scores has not been rigorously established.
This study was conducted in order to develop a structured validity argument for the interpretation of test scores derived through use of the script concordance method.Read More

We searched the PubMed, EMBASE and PsycINFO databases for articles pertaining to script concordance testing. We then reviewed these articles to evaluate the construct validity of the script concordance method, following an established approach for analysing validity data from five categories: content; response process; internal structure; relations to other variables, and consequences.
Content evidence derives from clear guidelines for the creation of authentic, ill-defined scenarios. High internal consistency reliability supports the internal structure of SCT scores. As might be expected, SCT scores correlate poorly with assessments of pure factual knowledge, in which correlations for more advanced learners are lower. The validity of SCT scores is weakly supported by evidence pertaining to examinee response processes and educational consequences.
Published research generally supports the use of SCT to assess the interpretation of clinical data under conditions of uncertainty, although specifics of the validity argument vary and require verification in different contexts and for particular SCTs. Our review identifies potential areas of further validity inquiry in all five categories of evidence. In particular, future SCT research might explore the impact of the script concordance method on teaching and learning, and examine how SCTs integrate with other assessment methods within comprehensive assessment programmes.

BMC Med Educ
BMC Med Educ 2011 6;11:101. Epub 2011 Dec 6.
Skillslab, Faculty of Health Medicine and Life Sciences, Maastricht University, Maastricht, The Netherlands.

The role of deliberate practice in medical students' development from novice to expert was examined for preclinical skill training.
Students in years 1-3 completed 34 Likert type items, adapted from a questionnaire about the use of deliberate practice in cognitive learning. Exploratory factor analysis and reliability analysis were used to validate the questionnaire.Read More

Analysis of variance examined differences between years and regression analysis the relationship between deliberate practice and skill test results.
875 students participated (90%). Factor analysis yielded four factors: planning, concentration/dedication, repetition/revision, study style/self reflection. Student scores on 'Planning' increased over time, score on sub-scale 'repetition/revision' decreased. Student results on the clinical skill test correlated positively with scores on subscales 'planning' and 'concentration/dedication' in years 1 and 3, and with scores on subscale 'repetition/revision' in year 1.
The positive effects on test results suggest that the role of deliberate practice in medical education merits further study. The cross-sectional design is a limitation, the large representative sample a strength of the study. The vanishing effect of repetition/revision may be attributable to inadequate feedback. Deliberate practice advocates sustained practice to address weaknesses, identified by (self-)assessment and stimulated by feedback. Further studies should use a longitudinal prospective design and extend the scope to expertise development during residency and beyond.