The programs in RWJF’s HST portfolio aim to catalyze adoption of new health care models and systems changes that center the interrelated physical, mental health, and social needs of people who are eligible for Medicaid. The HST invested in three programs: Advancing Health Equity (AHE), Advancing Integrated Models (AIM), and Health Systems Transformation Research Collaborative (HSTRC). As the evaluator, Equal Measure sought to learn about each program’s approach to advancing health equity.

The following interview reflects a conversation between the Equal Measure evaluation team, including Eve Weiss, Siobhan Costanzo, Kristin Giordano, and Victoria Worthen Lang, and Jacquelynn Orr, a program officer in Research Evaluation Learning at Robert Wood Johnson Foundation.

Q: In the first half of 2020 the global COVID-19 pandemic arrived just as the HST portfolio evaluation was getting underway. The pandemic had intense and disparate impact on healthcare and hospital workers as well as individuals and families in communities of color. The summer of that year also saw the national racial justice uprisings in the wake of George Floyd’s murder. How did the dramatic events of 2020 influence the foundation’s approach to the evaluation?

RWJF: We really had to recognize our privilege. RWJF was able to transition to remote work in March 2020, but we couldn’t assume everyone, including grantees, had that level of flexibility, or were able to prioritize some of the defined goals in their funding proposals. I remember one of our first discussions with Equal Measure was whether we would even be able to schedule interviews with grantees. As a result of that discussion, we delayed the start of the evaluation from spring 2020 to summer 2020. We also engaged our network and talked to RWJF colleagues about what they were doing to improve processes to better serve the community through these projects as well as creative methods they were using to bring the community-level context into evaluation work. Some internal changes included funding general operating support grants, removing annual narrative reporting, and increasing the involvement of grantees with the development of the evaluation.

Q: On the evaluation side, how did Equal Measure respond to the events of the first half of 2020, and how did they influence the design phase of the evaluation?

Equal Measure: We were all living through and adapting to the pandemic and societal events together, which may have allowed both parties to be more vulnerable in our discussions, planting the seeds for shifts in the power dynamics between funder and evaluator. The unusual dynamic strengthened the authenticity of our conversations and built trust between the team, the foundation, and the grantees. As evaluators, we felt seen as partners and collaborators, rather than just contractors; we felt we could be more forthright with RWJF about prioritizing equitable grantmaking approaches, designing our evaluation approach using principles of equitable evaluation and Culturally Responsive and Equitable Evaluation (CREE). Meanwhile, RWJF was also undergoing deep strategic changes, adopting more trust-based philanthropic practices.

RWJF agreed to include us in early conversations with program officers and grantees about evaluation design. At the foundation, we worked closely with each HST program officer to better understand the history and nuances of the initiative, strengthening our relationships with the program officers and allowing us to write evaluation questions and design an evaluation plan that reflected the needs and priorities of each initiative. To co-design with grantees, we asked grantee leads to review the evaluation plan and questions to ensure our data collection plan and resulting learnings would meet their learning and capacity needs.

Then, as we were developing the theory of change for the portfolio of projects, we invited all involved program officers at RWJF and grantee leads for each initiative to attend design meetings. Often when we work with a large foundation, we work mostly with one program officer who facilitates communication between other foundation staff and our team. Involving multiple RWJF program officers elevated more diverse perspectives and feedback, united them around a common understanding of the work, supported relationship building, and signaled the importance of evaluation to this work. Including grantee leads in these sessions as well was essential to creating a theory of change grounded in grantees’ experiences, perspectives, and their in-depth knowledge of their local health systems and the changes needed to advance health equity.

Q: In your position at the foundation, what was it like to bring all the stakeholders into the room? Would you say there is something unique about RWJF culture that allowed engagement of multiple stakeholders?

RWJF: It was intimidating! I started three months before this project began and was still considered one of the “new kids on the block.” On top of that, I’m aware that some may look at this Research Evaluation Learning role as representing a field historically known for being created in a particular moment and informed by the values, experiences, and needs of a small group of middle-aged white heterosexual males. I was interested in reimagining how we support our colleagues, collaborators, and the discipline in a more evolved way than what most consider to be many foundations’ typical practice. Since there had been a number of internal culture shifts and more promotion of incorporating equitable evaluation and grantmaking practices from the Equitable Evaluation Framework^TM, it only made sense to me to bring stakeholders like RWJF foundation staff, grantee partners, our evaluators, etc., together. Having them all involved in discussing the project would ensure that all learning would be relevant to them and wouldn’t just sit on a shelf at the end of the project.

Q: As you mention, there was a significant call at this time for philanthropy to shift toward more equitable practices. We observed RWJF making a lot of changes in how they talked about the work and that showed up in this project. How did you change your ways of talking and working?

RWJF: We really focused on shifting language, in alignment with best practices at the time, to explicitly name structural racism as a root cause of health inequities. We were also working to shift practices with grantees to really listen to them and truly understand more about the details of their experiences. This helped us better meet their needs in a more timely fashion instead of waiting for our monthly meeting updates. We treated them as the experts in their work and took a step back to support them. One major change we supported was that relevant interview questions for the evaluations started to come more often from grantees, and we prioritized those questions over what the grantees may have thought RWJF would need to complete the evaluation or incorporate into renewal discussions.

Q: At this time, there were new and urgent calls for evaluation’s data collection and learnings to be useful for grantees. Simultaneously, grantees were experiencing even more limited capacity and competing priorities due to the pressures of the pandemic. As the evaluator, what equitable practices did Equal Measure use when working with grantees? 

Equal Measure: It was essential to us that the grantees be included. Once we began our evaluation work, we practiced flexibility, adapting our data collection as needed, which reduced grantee burden and helped our learnings be useful and relevant for grantees. For example, we used insights from our conversations with grantee leads to draft interview protocols aligned with their experiences. We asked questions about the impact of COVID and the racial justice movements on their work and made room for discussions about how their priorities or activities may have shifted from their original plans. We conducted virtual focus groups and interviews and accepted written responses.

Trust is relational. Building trust with the grantee leads was essential for having honest conversations about their needs. We held regular meetings with initiative leads to be transparent about our process, to get to know each other, and to learn more about their approaches. We were reliable, doing what we said we would do, and we practiced curiosity to demonstrate an openness to understanding their priorities.

As a final step in the co-creation process, we shared memos with the funders, initiative leads, and their grantees. As we finalized each memo, we shared the drafted memo with initiative leads and scheduled calls with each one to hold a conversation about our findings and the resultant considerations and recommendations. Each initiative lead could also provide written feedback. If we used specific examples from grantees, we asked the grantees to review them, ensuring we represented their experience accurately. Once each memo was almost final, it was sent to the funder, initiative leads, and their grantees for additional feedback or input. The multistep process was time and labor intensive, but resulted in a higher quality, more inclusive, and actionable product.

Q: The social sector is increasingly seeking systems-level change to better address structural racism and make progress toward a more racially equitable society. What are some of the ways that RWJF shifted its approach toward equitable systems change?

RWJF: We were looking for ways to build on known best practices, innovative practices, and new learnings so we could generate progress on important social change issues, rather than reinventing the wheel by continually funding the same processes. We were listening and learning directly from grantees, which pushed us to make changes to how we were working. For example, we were hearing from one program lead that grantees did not feel like they were getting as much face time with RWJF as they had hoped. RWJF listened and provided office hours, as well as no-cost extensions and increased flexibility as a result of additional feedback received from the grantees.

Most importantly, we were acknowledging important societal challenges that the philanthropic sector can address, and putting real support behind the stakeholders that are addressing those challenges in communities.

Q: What are some specific practices Equal Measure used in their evaluation to elevate systems-level work from the grantees?

Equal Measure: We analyzed interview and focus group data to understand systems-level changes, such as relationship and partnership building, to create wider support for the work or identifying indicators of momentum toward policy change. We noted examples of practices or different ways of working grantees implemented to center equity, especially practices that might have broad impact and replicability. For example, using root cause analysis to understand the causes of the inequities in their systems and designing solutions to address them; training physicians in culturally responsive, anti-racist, and trauma-informed care approaches; and implementing a human-centered design approach to collaborate with patients and families to surface patient and family priorities and redesign health provider visits. Many grantees also began to name redressing structural racism and working toward racial equity as a goal of their work. We highlighted these findings, the challenges of doing this work in a health care setting, and drafted recommendations to the foundation about how a funder can tactically support the advancement of racial equity in health systems.

The unexpected pressures of the turbulent times—beginning with the pandemic in 2020—supported a more vulnerable and authentic partnership between Equal Measure as the evaluator and RWFJ as the funder. The context created an unusual opportunity to advance equitable evaluation and grantmaking practices. Additional infrastructure for strong funder-evaluator relationships in evaluation projects and relationship building may be an important step toward accelerated implementation of equitable practices among both evaluators and philanthropies.