Rigor in Workforce Development Evaluation: New Resources Available

By Celeste Richie and Katherine Shamraj

Rigorous evaluation is a cornerstone of Third Sector’s approach to Pay for Success (PFS). The success of a PFS project hinges on quality evaluation methods to determine whether an intervention achieves desired outcomes. In our engagements, we advocate for the most rigorous method appropriate--including well designed Randomized Control Trials (RCTs) and high quality Quasi Experimental Designs (QEDs) studies--for a given intervention.

Third Sector’s insistence on rigor is a differentiator. Only four PFS programs launched in the US to date have used RCTs; three of which were constructed by Third Sector.* Yet our support of rigorous evaluations still faces the central challenge of evidence-based policy: relatively few social programs have demonstrated statistically significant results through RCTs. The faster policymakers shift to a paradigm of rigorous evaluation, the sooner social service providers will be able to improve programs and focus resources on evidence-based practices.

To preempt this challenge in preparation for Third Sector’s Pay for Success Workforce Development Competition funded by the Corporation for National and Community Service’s Social Innovation Fund, we commissioned a literature review to identify youth workforce programs that have deployed rigorous evaluation methodologies. Though it is promising that several programs demonstrated effectiveness through statistically significant impacts, the outcomes were not always game changing. For example, in one RCT of a workforce development program targeting at-risk youth, the two- to three-year earnings of youth who had participated in the program’s skill-building and training efforts exceeded those of the control group, but by year six this earnings gain had disappeared. Without a rigorous evaluation, this long-term outcome would have been missed.

While only a handful of youth workforce development programs have deployed RCTs to measure effectiveness, every new study is an opportunity to grow the field of rigorous social service evaluation. As we embrace a new cohort of SIF-funded projects this spring, it is important to place evaluation in its wider context. Our objective in the coming years is to experiment, iterate, and improve. With each new generation of rigorous evaluations, social service innovators will be better able to calibrate programs for effectiveness.

There is no better way to improve upon the previous and current generations of programs than studying and learning from them, particularly by deploying the most rigorous scientific methods in social science research. In that spirit, we are excited to share the following resources with our competition applicants, as well as with anyone curious about experimental and quasi-experimental design of workforce development program evaluations:

• Review of Evidence-Based Youth and Young Adult Workforce Initiatives (ABT Associates)
• Research Synthesis on Opportunities for Youth (Department of Labor)
• Opportunities for Youth Protocol (Department of Labor)
• Evidence on the Effectiveness of the National Guard Youth ChalleNGe Program (Department of Labor)

In the long run, the benefits of rigorous evaluation far outweigh the costs. The next generation of quality evaluations will help direct limited financial resources to workforce development programs that increase incomes and improve the employment trajectory of a significant number of participants. These studies will not only rigorously evaluate whether a program generates a worthwhile return on investment, but will also provide robust evidence of a program’s ability to improve individual lives.

For more information about our evaluation methods or our SIF-funded PFS Workforce Development Competition, please contact Celeste Richie.