Process Evaluation

Thanks to innovations in outcomes-focused contracting “because that’s the way we’ve always done it” is no longer an acceptable reason to continue to operate social programs the same way every year hoping for better results. By linking payments to measurable social outcomes, Pay for Success (PFS) contracting has made data and results integral aspects of any social service. Through multi-year randomized control trials (RCTs), which test the outcomes of a PFS project, social service providers and government funders can finally quantify the impact that their programs are making in the lives of their most vulnerable residents and they can use that information to improve future iterations of the program. While this data-driven feedback loop created by PFS’s use of RCTs is critical to achieving Third Sector’s mission of “accelerating America’s transition to a more performance-driven social sector,” exclusive reliance on RCTs keeps that acceleration at a slow crawl. Enter: Process Evaluation.

Process Evaluation is an approach to gauging outcomes and verifying program assumptions while the project is in process; the RCT is underway, but the formal evaluation is still multiple years from completion. It serves as a temperature check to see if the project is on track to achieving its desired outcomes and what changes should be made to ensure success. It identifies whether the program as envisioned is the program being implemented. It feeds the feedback loop with valuable data before the RCT concludes and while there is still time to adjust programming.

As part of the Cuyahoga Partnering for Family Success Program, the first county-level PFS project in the nation, the Center on Urban Poverty and Community Development at Case Western Reserve University (“CWRU”) recently completed a Process Evaluation to test whether or not the program was on its way to reaching its target outcome of safely reducing the number of days that homeless children spend in out-of-home placement away from their families. The Process Evaluation sought not only to test progress towards this key outcome, but also to paint a fuller picture of the service delivery and the service recipients’ paths to family reunification and stability. CWRU used four data sources for the Process Evaluation:

1. Client-level data from the Homeless Management Information System (HMIS) provided detailed demographics on service recipients and enabled comparisons of recipients’ shelter stays pre- and post-program.
2. A review of anonymized FrontLine Service case management and client progress notes revealed trends in the type, frequency, length, and theme of services provided.
3. Interviews of Cuyahoga County Division of Children and Family Services, Cuyahoga Metropolitan Housing Authority, and FrontLine staff added the service provider perspective on collaboration and service consistency across program partners.
4. An analysis of benefits and child welfare data from the Ohio Department of Jobs and Family Services and the Cuyahoga County Division of Children and Family Services demonstrated changes in TANF and SNAP benefits and child welfare involvement since participants started the program.

Overall, the Process Evaluation yielded positive results that indicate increased family reunification and improved family stability after program participation. Compared to the control group that did not receive services, the treatment group demonstrated:

• 12% greater use of SNAP assistance
• 6% fewer substantiated maltreatment cases
• 9% more children exiting out-of-home placement after nine months
• Decreased emergency shelter usage

The Process Evaluation data not only suggest positive changes and increased family stability for the treatment group, but also provided key insights to continue to improve service delivery, especially for the most challenging cases. For example, the study revealed that families affected by domestic violence were often the most difficult to reunify and stabilize, especially when the domestic violence incidence is hidden or leads to recidivism. With more than three-quarters of the treatment group reporting to be domestic violence victims, the Process Evaluation revealed a need to improve responses in this area.

The quantitative and qualitative data from the Process Evaluation provided valuable feedback to the many partners involved in the Partnering for Family Success Program, and changes are already underway to implement recommendations from the study. We are confident that insights gleaned from this evaluation will improve the services provided to the children and families of Cuyahoga County and will promote stronger outcomes when the program’s RCT concludes in 2020. We would also recommend using a Process Evaluation at an intermediate point in an outcomes contract whenever possible and are currently conducting one in Salt Lake County. After all, more data and additional feedback can only fuel our performance-driven acceleration!