When it comes to improvements and innovations in Health Professions Education (HPE), one of the most important questions is always, what is working and what isn’t? Measurement is everything, but it can be difficult to assess how the clinical learning environment (CLE) best supports learners and other stakeholders. At times, measurement can even be an afterthought, despite the reality that measuring the quality of learning environments is an integral part of all HPE.
2012 Macy Faculty Scholar Sandrijn M. van Schaik, PhD, MD, of the University of California San Francisco (UCSF), has a history of improving communication between health professionals. In 2018, she co-authored a vision paper for a special Macy Foundation Conference that explored what exemplary clinical learning environments look like in HPE. Several of the outlined principles actually became cornerstones of the most recent Macy Foundation strategic plan. But the paper sparked a bigger idea for Dr. van Schaik, and shortly after she started a project to help tackle one of the most difficult challenges in HPE: measuring the quality of the CLE. In 2021, Dr. van Schaik launched A Score Card for Clinical Learning Environments to Promote Continuous Improvement with funding from the Josiah Macy Jr. Foundation and created a team and advisory boards to help her develop a constructive and usable framework for measuring the effectiveness of the CLE.
What Makes a Scorecard?
At UCSF, Dr. van Schaik and her team take existing data sources and compile information to create a “one-stop-shop” for CLE measurement and analysis. Most of the data needed for a full cross-section evaluation has already been collected through other forms of surveys. But answers across different groups typically are not compared.
“The problem with most educational evaluations is that they occur in silos for different groups of learners and stakeholders, meaning their review occurs in silos as well,” said Dr. van Schaik. “Common CLE problems are often overlooked as a result. The scorecard aims to provide a holistic view of the CLE, and is then shared with teams of people who have shared accountability to work on improvements.”
Once collected, the scorecard data is then grouped into the following domains:
- Equity, inclusion, and belonging
- Collaboration and teamwork
- Alignment between education and health systems goals and continuous processes
- Continuous improvement of individuals, teams, and systems
Dr. van Schaik and her team look at the experiences of the main stakeholders in the environment: students, residents, fellows, staff, and faculty. They establish benchmarks and targets on differing scales across categories, and color-code domains based on their scores. The data is then put into summary tables and identified by stakeholders for easier reference.
“We create a standard based on the performance of that specific department so we’re measuring against realistic benchmarks,” said Dr. van Schaik. “These benchmarks can be aspirational standards and we need to hold learning environments to high standards, but we also want the measurement to be useful to foster real improvements.”
The Scorecard in Practice
The scorecard itself is a work in progress, as it is being constantly tweaked and improved by Dr. van Schaik and her team based on the needs of different departments. It is currently being adapted for nine departments across UCSF, and work has started to adapt it at Georgetown University in the Department of Medicine.
The scorecard pushes departments to look beyond the “checkbox” that often appears on evaluations. It helps departments understand how trainees, faculty, and staff are actually experiencing these learning environments to identify issues and promote improvements moving forward.
For instance, the scorecard helped the pediatrics department at UCSF identify that trainees across all levels of training felt like they weren’t receiving adequate feedback from the faculty. Without the structure of the scorecard, it may not have been obvious that this problem was not unique to one learner group. The scorecard made it easy for the team to extrapolate these results and find that trainees across UCSF had the same feeling, creating the need for a broader institutional response. In response, Dr. van Schaik and her team created and launched a feedback campaign across the institution to help clinical supervisors develop skills in feedback.
“When something is seen as a problem for just one program, it’s easy for people in charge to brush things off when really there is a bigger issue at play,” said Dr. van Schaik. “But when you can get more people across an institution involved and bought into a mission of improvement, amazing change can happen.”
At the end of the three-year grant, Dr. van Schaik and her team hope to have developed a refined scorecard template that can be easily adapted across different departments. The template will include a processing map for how the scorecard can be implemented with recommended benchmarks, a self assessment guide for identifying areas for improvement, and data collection tips. The UCSF team is also working to make the scorecard available to other institutions. It’s currently being implemented at Georgetown University, and the Mayo Clinic and Hofstra University have also expressed interest in adapting it.
“Measurement is such a complex problem to tackle, but it’s so important to investigate and address issues happening in the CLE,” said Dr. van Schaik. “As educators, we need to make sure that learners and other stakeholders are getting the best experience possible. The mission is critical to improving education and ultimately patient care as a whole.”
Interested in becoming a Macy Foundation grantee? Check out our grantee page for upcoming opportunities.