News and Commentary Ensuring Fairness in Medical Education Assessment: Insights from Conferees

Head shots of the conferees: Hannah Anderson, MBA, Michael Barone, MD, MPH, Tasha Wyatt. PhD, Justin Bullock, MD, MPH, Benjamin Kinnear, MD, MEd
Hannah Anderson, MBA (top left), Michael Barone, MD, MPH (top middle), Tasha Wyatt, PhD (top right), Justin Bullock, MD, MPH (bottom left), Benjamin Kinnear, MD, MEd (bottom right)

The August 2023 supplemental issue of Academic Medicine  featured the outstanding work of the 50 individuals who participated in the 2022 Macy Conference on Ensuring Fairness in Medical Education Assessment. Through commissioned papers, case studies, and the consensus recommendations that emanated from the December 2022 conference, we hope to offer readers resources and a potential framework for moving toward a more equitable medical education assessment system.

In this month’s Macy Notes, I invited five of our conferees to share their reactions to the following prompt:

Please provide a concrete example of how the Ensuring Fairness in Medical Education Assessment Conference recommendations have been or can be operationalized at your institution or in your daily practice.

We hope that the following comments will serve as a springboard for exploration and transformation at your own institutions and organizations. I welcome your feedback, input, and ideas at We are eager to hear from and support those who are working in the area of equitable assessment, as it is a crucial facet of improving health and health care for everyone.

Hannah Anderson, MBA

Instructor of Pediatrics, University of Pennsylvania Perelman School of Medicine 

Research Associate, Medical Education, Children’s Hospital of Philadelphia 

The conference supplement poses the question: “How might our programs of assessment be transformed if we took up tools to dismantle any inequitable practices, rebuilding with equity as a core design feature rather than an afterthought?”

In our study group, we are operationalizing that question by undertaking a stakeholder-participatory design project that will redesign the assessment systems at five pediatric residency programs across the U.S. Our project, called Designing for Equity in Medical Education, is guided by the principles of design justice, which seeks to advance equity in design projects by centering underrepresented and marginalized stakeholders in the redesign; adopting an accountable, accessible, and collaborative process rather than a checklist approach to equity; and working towards sustainable, stakeholder-led and -controlled outcomes.

At each project site, we are co-creating assessment systems with stakeholders (e.g., resident learners, program administrative staff, assessors, program leaders) by gathering their experiences and perspectives about local needs and context and giving them direct input into the design of the assessment system at each site. We will test and refine the assessment systems alongside stakeholders. Using a mix of qualitative and quantitative methods, we will measure the systems’ equity, validity, utility, feasibility, and acceptability in both processes and outcomes. We are thankful for the support of the Josiah Macy Jr. Foundation, which funds this important work.

Michael Barone, MD, MPH

Vice President of Competency-Based Assessment

National Board of Medical Examiners

In fall 2022, as the Macy Conference on Ensuring Fairness in Medical Education Assessment approached, several activities were underway at the National Board of Medical Examiners (NBME) related to equity in assessment. During the Conference, I placed these existing and emerging activities into the context of the developing consensus recommendations.

Programs at NBME have ongoing activities focused on ensuring fairness. Inclusive of action step 2.5, NBME embeds equity practices throughout the test-development process, from content development to exam assembly to scoring and reporting. NBME has also established a Patient Characteristics Advisory Panel, which provides guidance regarding how patient characteristics are used in NBME’s products and services.

NBME recently partnered with the Association of American Medical Colleges (AAMC), the American Board of Internal Medicine (ABIM), and the American Board of Medical Specialties (ABMS) to host the inaugural Equity in Measurement and Assessment Conference (EMAC). This galvanized a commitment to advance fairness and equity, especially those action steps under Macy Conference Recommendation 5. For a summary of EMAC take-home points, please visit the NBME website.

NBME provides grants to educators and scientists to advance assessment, specifically equity in education and assessment, which is consistent with action step 5.8. Through our contributions, NBME provides grants to organizations focused on building equity and accountability frameworks for medical education. In existing grant programs such as the Stemmler Fund, we are exploring how grantees can further demonstrate their commitment to principles of diversity, equity, inclusion, and justice (DEIJ). Lastly, this August we launched the Emerging Innovators Grant Program, which focuses on research and program evaluation of assessment in medical education, with a specific focus on DEIJ. 

There’s much more happening at NBME.  Feel free to contact me for further information. 

Tasha Wyatt, PhD

Associate Director of the Center for Health Professions Education

Uniformed Services University of the Health Sciences

Research has shown that the language used in narrative evaluations for women and underrepresented minorities in medicine refers to personal attributes more than competency-related behaviors, raising the concern for ingrained implicit bias in evaluators. Few evidence-based interventions exist to mitigate implicit bias in narrative evaluations. Dr. Kristin Mangalindan, an internal medicine physician and master’s-level student at the Center for Health Professions Education and her advising team will explore this topic through a study on anti-bias assessment.

Our goal is to synthesize knowledge on actionable solutions to improve implicit bias in narrative evaluations. We hope to address this in two phases: first, we plan to conduct a scoping review of the existing literature on evidence-based interventions to address implicit bias in narrative evaluations within medical education. Second, we will conduct qualitative, semi-structured interviews with key stakeholders: the learners and the assessors. Depending on the available studies, we would narrow our inclusion and exclusion criteria to women, underrepresented minorities in medicine, or the intersection of both. In these interviews, we will focus on the effects of implicit bias in narrative evaluations, the efficacy of evidence-based solutions, and proposed solutions for wider implementation in different institutions.

Justin Bullock, MD, MPH

Nephrology Fellow

University of Washington School of Medicine

The University of Washington School of Medicine (UWSOM) recently underwent a series of Clinical Assessment Workgroup meetings, led by Associate Dean for Curriculum Dr. Heather McPhillips. The Clinical Assessment Workgroup was charged by the Curriculum Committee to revise the clinical assessment system due to concerns of learners and educators around inequities in assessment and the implications of assessment changes. The workgroup was made of diverse stakeholders (trainees, clerkship directors, program directors, administrators, etc.), and it invited assessment experts from across the nation to share cutting-edge data as it pertained to equity in assessment and drew upon literature published from within the institution.

The first meeting decided on boundaries of the workgroup, gave background on current assessment strategies in use nationally, looked at our local assessment data, and set desired outcomes. The second meeting was a visioning session where small groups worked through models of assessment, refined them for UWSOM, and presented them to the larger group for input. The third meeting involved refining/revising the shared assessment model, discussing how clerkship grades and medical student performance evaluation comments should be determined, and discussing the grading system. The final meeting revisited the group insights and allowed for discussion of the benefits and drawbacks of various assessment paradigms before a vote, which informed the Dean of Curriculum and final recommendation to be presented to the Curriculum Committee.

Benjamin Kinnear, MD, MEd

Associate Professor of Internal Medicine and Pediatrics; Associate Program Director, Medicine-Pediatrics and Internal Medicine Programs

Cincinnati Children’s Hospital Medical Center / University of Cincinnati Medical Center

I arrived back in Cincinnati on a wave of enthusiasm and conviction to improve the fairness and equity of assessment at my institution. I am part of two residency programs at University of Cincinnati (Internal Medicine and Med-Peds), and I am encouraged to see areas where we align with the conference recommendations. We have committed to striving toward recruitment practices that are ethical, equitable, and transparent, recognizing that we still have much work to do. We use a holistic application review process that focuses on how applicants spend their time (i.e., activities), distance travelled, and career interests, while we blind interviewers from biased metrics such as class rank, test scores, and honor society status. We also provide interviewers with training on concrete strategies to attempt to mitigate implicit bias. Our residency programs are foregrounding equity and fairness in assessment and recruitment much more than in years past; the progress is encouraging.

However, the Macy Conference recommendations crystallized ways in which our programs can do much more. As we wrote in one of the conference papers, our assessment system adheres to many competency-based practices that promote fairness, but we must take a more critical look at the foundational assumptions and principles undergirding our assessment strategies to truly find how equity can be improved. As someone with quality improvement background, action step 2.5 (develop a system to monitor assessment data to identify harmful bias) particularly resonates with me. I hope these recommendations will catalyze my institution to develop new metrics for evaluating assessment for the potential presence of harmful bias, share these metrics with relevant stakeholders, and link such evaluations to program improvement initiatives that are co-constructed with our learners.

More News and Commentary


Bridging the Chasm of Inexperience through Virtual Gaming Simulation

Healthcare educator programs face the constant challenge of providing high-quality experiential learning for students. Within nursing, intense competition for clinical placements, recruitment of clinical faculty to...

Webinar: Macy Conference Recommendations

The Josiah Macy Jr. Foundation hosted a webinar to discuss conference recommendations on ensuring fairness in medical education assessment.

Practices that Promote Inclusion in the Clinical Learning Environment

Promoting inclusion in the clinical learning environment requires a deliberate and individualized approach from both faculty and institutional leaders, and it starts by building trust, respecting, and authentically...

Disability Pride: Peril or Possibilities in Healthcare?

July is Disability Pride Month, an opportunity to celebrate the historic signing of the Americans with Disabilities Act (ADA) into law; yet, 33 years later, we still have a long way to go.