Advancing Racial Equity in Evaluation: Lessons from Philadelphia

By Matthew Closter, Katie Mosher, Cynthia J. Román Cabrera, Giselle Saleet, and Eve Weiss, in partnership with Research for Action, Evident Change, and Cities United

An Innovative Grant to Reduce Gun Violence

In 2022, in response to a deadly outbreak of gun violence in Philadelphia, the City of Philadelphia’s Office of Policy and Strategic Initiatives for Criminal Justice and Public Safety took a new approach to an ongoing and systemic problem by creating the Community Expansion Grant. The City distributed $13.5 million in grant funding to 28 small- to mid-sized community-based organizations deeply embedded in Philadelphia neighborhoods. These nonprofits provided services in their own communities for the residents most likely to be impacted by violence: Black and Brown boys and men between the ages of 16 and 34. The grantees focused on recruiting participants for mentoring opportunities, human services, and workforce development programs, with the goal of reducing gun violence.

Co-designing an Evaluation with Racial Equity at the Center

While many of these organizations lacked traditional nonprofit infrastructure, they were rich in deep, authentic roots and relationships in the affected neighborhoods, and many benefitted from Black or Latine leaders drawn from the same communities.  

Because many of the organizations, often overlooked by traditional philanthropy, had limited staff and were charged with standing up new programs or recruiting participants very quickly, demonstrating the impact of these programs would take more than a traditional evaluation. The City turned to Equal Measure to lead a team of partners that included Research for Action, Evident Change, and Cities United. 

Centering racial equity in our approach, we designed a year-long collaboration with community-based organizations, evaluation partners, program participants, and the City of Philadelphia. The objective of our mixed-methods evaluation was to understand and communicate the value of these place-based programs in reducing violence. We hope through this blog post to share some of the equitable practices we applied in the evaluation and what we learned as a team along the way.  

Our Approach and Learnings

Throughout the project, we used a developmental approach, continuously gathering and providing feedback to our partners in the Managing Director’s office and to grantees in real time. Approaching the work with curiosity and flexibility allowed us to adapt our evaluation and programmatic activities to the evolving circumstances in the city. We aimed to understand the accomplishments and challenges faced by grantees, provide necessary support, adjust evaluation activities to meet their needs, and uncover how their experiences could influence broader efforts to address gun violence in communities. 

Over the nine-month grant period, the organizations launched or expanded programs in their communities that incorporated research-based practices for reducing gun violence. In the full report, we outline emergent but substantive learnings that have informed the city’s program and offer insights for other cities’ anti-violence initiatives. 

KEY EQUITY-DRIVEN PRACTICES

It was essential that our vision for the evaluation and our approach, as well as our methods and ongoing collaboration, were rooted in racial equity. To that end, we implemented equity-focused practices at each phase of the work: setting up the team, implementing data collection and analysis, developing findings, and managing the process 

Establishing a Mindset: How We Set Our Team Up for Success

To ensure our team was prepared to manage the complexities of the evaluation and center racial equity in the work, we:  

  • Prioritized the well-being of communities: We agreed our goal was to prioritize the lived experiences and needs of community residents, and to develop trust and accountability with the grantees. 
  • Co-developed values, protocols, and processes. As a team, we:  
    • Reflected on individuals’ lived experience and the racial dynamics, current and historic, that exist between evaluators and Black and Brown communities. We returned to these reflections throughout the project.
    • Examined power dynamics and positionality within our own team, reflecting on how our intersecting identities like gender, race, and class affect how we engage with the world around us.
    • Framed gun violence as a symptom of a larger systemic issue of historic and current disinvestment in communities. Root cause analysis and literature reviews offered great starting points to engage with the issue. The literature helped us build a rubric that identified evidence-based practices for reducing gun violence used by grantees.
    • Diffused power structures within the team to create shared ownership and accountability for our work, for example, rotating leadership of meetings and agenda development. 
    • Practiced transparency, humility, and vulnerability to build trust and support as teammates. 
  • Trained in trauma-informed practices: The team participated in a “PIES” training offered by Network of Neighbors, a program of the City’s Department of Behavioral Health and Intellectual disAbility Services. PIES is a structure to facilitate conversations in stressful circumstances, encouraging participants to check in on how they are feeling P-Physically (Body), I-Intellectually (Thoughts), E-Emotionally (Feelings), and S-Safety (Support, Spiritual, Specific) to stay grounded in the present and manage the stress of discussing traumatic, violent events. 

Implementing the Work: Centering Expertise in Community, Illuminating Power Structures

The evaluation involved multiple audiences and partners: our evaluation partners, the 28 grantees, program participants, and the City of Philadelphia. To ensure representation of the many points of view, and keeping the needs and interests of the communities at the center, we implemented several practices:  

  • Tapped into the expertise of partners: Our strategic partners—Research for Action, Evident Change, and Cities United—brought their expertise to the project. We worked together to understand each other’s priorities and progress toward shared goals.  
  • Centered the grantees as experts and an essential audience for the work: In collaboration with the grantees, we co-developed data collection processes and deliverables. We shared draft materials with all the grantees for feedback and provided final individualized reports of their own data with our analysis for each grantee organization. 
  • Designed for learning, not “measurement”: We structured the evaluation to focus on understanding the implementation of evidence-based practices to reduce gun violence that was appropriate for each grantee, rather than “measuring” impact against an arbitrary standard that would have been impossible to assess in the timeframe for the grant.  

Doing the Work: Managing the Process with Intention

As a team committed to racial equity, we prioritized several ways of working together. We: 

  • Showed up—emotionally, physically—beyond typical scheduling or reminder emails. Our team responded to any and all questions and concerns raised by grantees and participants. We scheduled interviews and focus groups around their availability, not ours, even when it meant evenings, weekends, or last-minute planning.  
  • Named tensions: We made time to discuss and understand tensions that arose—among partner organizations and with the client—and reflected on the complex power dynamics across the multiple partners. Among many competing priorities and viewpoints, we found it helpful to hold to a North Star of remembering who and what we were doing the work for: the communities and in the service of racial equity.  
  • Developed new ways of working, leaning on each other to get work done together by dividing the workload equitably with shared ownership of workstreams and recognizing time to pause, reflect, and connect with each other’s emotional well-being.

Conclusion

Our experience in the year-long evaluation of the City of Philadelphia’s Community Expansion Grant taught us the importance of leading with compassion and care. As evaluators, we are committed to interrogating the role of racial equity in social issues and centering Black and Brown communities in our work as partners, as co-creators, and as the most essential beneficiary of our evaluations and other services. We aim to show up with fortitude, determination, flexibility, and compassion, and advocate authentically and realistically, keeping our eyes focused on the goal of a more racially equitable future for communities in Philadelphia and around the country.

Interested in continuing the conversation about equity in evaluation, our organizational journey, or our project team experience? We’d love to hear from you.