The August 2023 supplemental issue of Academic Medicine featured the outstanding work of the 50 individuals who participated in the 2022 Macy Conference on Ensuring Fairness in Medical Education Assessment. Through commissioned papers, case studies, and the consensus recommendations that emanated from the December 2022 conference, we hope to offer readers resources and a potential framework for moving toward a more equitable medical education assessment system.
In this month’s Macy Notes, I invited five of our conferees to share their reactions to the following prompt:
Please provide a concrete example of how the Ensuring Fairness in Medical Education Assessment Conference recommendations have been or can be operationalized at your institution or in your daily practice.
We hope that the following comments will serve as a springboard for exploration and transformation at your own institutions and organizations. I welcome your feedback, input, and ideas at president@macyfoundation.org. We are eager to hear from and support those who are working in the area of equitable assessment, as it is a crucial facet of improving health and health care for everyone.
Hannah Anderson, MBA
Instructor of Pediatrics, University of Pennsylvania Perelman School of Medicine
Research Associate, Medical Education, Children’s Hospital of Philadelphia
The conference supplement poses the question: “How might our programs of assessment be transformed if we took up tools to dismantle any inequitable practices, rebuilding with equity as a core design feature rather than an afterthought?”
In our study group, we are operationalizing that question by undertaking a stakeholder-participatory design project that will redesign the assessment systems at five pediatric residency programs across the U.S. Our project, called Designing for Equity in Medical Education, is guided by the principles of design justice, which seeks to advance equity in design projects by centering underrepresented and marginalized stakeholders in the redesign; adopting an accountable, accessible, and collaborative process rather than a checklist approach to equity; and working towards sustainable, stakeholder-led and -controlled outcomes.
At each project site, we are co-creating assessment systems with stakeholders (e.g., resident learners, program administrative staff, assessors, program leaders) by gathering their experiences and perspectives about local needs and context and giving them direct input into the design of the assessment system at each site. We will test and refine the assessment systems alongside stakeholders. Using a mix of qualitative and quantitative methods, we will measure the systems’ equity, validity, utility, feasibility, and acceptability in both processes and outcomes. We are thankful for the support of the Josiah Macy Jr. Foundation, which funds this important work.
Michael Barone, MD, MPH
Vice President of Competency-Based Assessment
National Board of Medical Examiners
In fall 2022, as the Macy Conference on Ensuring Fairness in Medical Education Assessment approached, several activities were underway at the National Board of Medical Examiners (NBME) related to equity in assessment. During the Conference, I placed these existing and emerging activities into the context of the developing consensus recommendations.
Programs at NBME have ongoing activities focused on ensuring fairness. Inclusive of action step 2.5, NBME embeds equity practices throughout the test-development process, from content development to exam assembly to scoring and reporting. NBME has also established a Patient Characteristics Advisory Panel, which provides guidance regarding how patient characteristics are used in NBME’s products and services.
NBME recently partnered with the Association of American Medical Colleges (AAMC), the American Board of Internal Medicine (ABIM), and the American Board of Medical Specialties (ABMS) to host the inaugural Equity in Measurement and Assessment Conference (EMAC). This galvanized a commitment to advance fairness and equity, especially those action steps under Macy Conference Recommendation 5. For a summary of EMAC take-home points, please visit the NBME website.
NBME provides grants to educators and scientists to advance assessment, specifically equity in education and assessment, which is consistent with action step 5.8. Through our contributions, NBME provides grants to organizations focused on building equity and accountability frameworks for medical education. In existing grant programs such as the Stemmler Fund, we are exploring how grantees can further demonstrate their commitment to principles of diversity, equity, inclusion, and justice (DEIJ). Lastly, this August we launched the Emerging Innovators Grant Program, which focuses on research and program evaluation of assessment in medical education, with a specific focus on DEIJ.
There’s much more happening at NBME. Feel free to contact me for further information.
Tasha Wyatt, PhD
Associate Director of the Center for Health Professions Education
Uniformed Services University of the Health Sciences
Research has shown that the language used in narrative evaluations for women and underrepresented minorities in medicine refers to personal attributes more than competency-related behaviors, raising the concern for ingrained implicit bias in evaluators. Few evidence-based interventions exist to mitigate implicit bias in narrative evaluations. Dr. Kristin Mangalindan, an internal medicine physician and master’s-level student at the Center for Health Professions Education and her advising team will explore this topic through a study on anti-bias assessment.
Our goal is to synthesize knowledge on actionable solutions to improve implicit bias in narrative evaluations. We hope to address this in two phases: first, we plan to conduct a scoping review of the existing literature on evidence-based interventions to address implicit bias in narrative evaluations within medical education. Second, we will conduct qualitative, semi-structured interviews with key stakeholders: the learners and the assessors. Depending on the available studies, we would narrow our inclusion and exclusion criteria to women, underrepresented minorities in medicine, or the intersection of both. In these interviews, we will focus on the effects of implicit bias in narrative evaluations, the efficacy of evidence-based solutions, and proposed solutions for wider implementation in different institutions.
Justin Bullock, MD, MPH
Nephrology Fellow
University of Washington School of Medicine
The University of Washington School of Medicine (UWSOM) recently underwent a series of Clinical Assessment Workgroup meetings, led by Associate Dean for Curriculum Dr. Heather McPhillips. The Clinical Assessment Workgroup was charged by the Curriculum Committee to revise the clinical assessment system due to concerns of learners and educators around inequities in assessment and the implications of assessment changes. The workgroup was made of diverse stakeholders (trainees, clerkship directors, program directors, administrators, etc.), and it invited assessment experts from across the nation to share cutting-edge data as it pertained to equity in assessment and drew upon literature published from within the institution.
The first meeting decided on boundaries of the workgroup, gave background on current assessment strategies in use nationally, looked at our local assessment data, and set desired outcomes. The second meeting was a visioning session where small groups worked through models of assessment, refined them for UWSOM, and presented them to the larger group for input. The third meeting involved refining/revising the shared assessment model, discussing how clerkship grades and medical student performance evaluation comments should be determined, and discussing the grading system. The final meeting revisited the group insights and allowed for discussion of the benefits and drawbacks of various assessment paradigms before a vote, which informed the Dean of Curriculum and final recommendation to be presented to the Curriculum Committee.
Benjamin Kinnear, MD, MEd
Associate Professor of Internal Medicine and Pediatrics; Associate Program Director, Medicine-Pediatrics and Internal Medicine Programs
Cincinnati Children’s Hospital Medical Center / University of Cincinnati Medical Center
I arrived back in Cincinnati on a wave of enthusiasm and conviction to improve the fairness and equity of assessment at my institution. I am part of two residency programs at University of Cincinnati (Internal Medicine and Med-Peds), and I am encouraged to see areas where we align with the conference recommendations. We have committed to striving toward recruitment practices that are ethical, equitable, and transparent, recognizing that we still have much work to do. We use a holistic application review process that focuses on how applicants spend their time (i.e., activities), distance travelled, and career interests, while we blind interviewers from biased metrics such as class rank, test scores, and honor society status. We also provide interviewers with training on concrete strategies to attempt to mitigate implicit bias. Our residency programs are foregrounding equity and fairness in assessment and recruitment much more than in years past; the progress is encouraging.
However, the Macy Conference recommendations crystallized ways in which our programs can do much more. As we wrote in one of the conference papers, our assessment system adheres to many competency-based practices that promote fairness, but we must take a more critical look at the foundational assumptions and principles undergirding our assessment strategies to truly find how equity can be improved. As someone with quality improvement background, action step 2.5 (develop a system to monitor assessment data to identify harmful bias) particularly resonates with me. I hope these recommendations will catalyze my institution to develop new metrics for evaluating assessment for the potential presence of harmful bias, share these metrics with relevant stakeholders, and link such evaluations to program improvement initiatives that are co-constructed with our learners.