Bottom Line
A nonprofit service agency creates an ongoing evaluation system that is operated in-house, and uses the data to improve programs and raise money.
“You don’t do it to put numbers in a box that no one will ever look at,” said Isaac Castillo, director of LAYC’s Learning and Evaluation Division (LED). “You do it to serve your clients better. …
“Reframing it that way … is the single biggest reason we’ve been able to change the culture of an organization that was very hesitant to do evaluation and data collection beforehand, and now very much embraces it in all of the programming we do.”
The system is expensive, but the agency believes it has paid for itself. “We did an estimate of the funding for [LED] for the first three years, and it was about $500,000,” Castillo said. “But we easily raised about $1 million during that same time using our output data.”
Background
The Latin American Youth Center, founded in Washington, D.C., in 1968 to serve immigrant Latino youth, now serves low-income youth and families of all ethnicities in the city and two adjacent Maryland counties. In fiscal 2008, the center provided 58 multilingual programs in five areas: educational enhancement, workforce investment, social services, art and media, and advocacy.
It tracks detailed demographic, process and outcome information across programs as diverse as after-school tutoring, summer youth employment, HIV/AIDS prevention, an online teen radio station and coalition-building.
“The implicit idea is that we’re doing something that feels good and sounds like a good idea, so it must be leading to positive outcomes. Well, how do you know if you’re not measuring outcomes?” Castillo asked.
Methodology
LAYC created its in-house Learning and Evaluation Division in 2005 to develop and maintain a data tracking system, conduct ongoing evaluations of its programs, and inform programmatic decision-making and organizational strategic planning.
The nonprofit uses Efforts to Outcomes (ETO) – an Internet-based data collection system designed by Social Solutions – to accomplish those goals. (See Castillo discuss ETO at http://www.youtube.com/watch?v=6qH7w2YFuEo.)
During fiscal 2008, a total of 155 employees, most of whom provide direct services to clients, received training in data collection and entry techniques. Program managers received additional training on troubleshooting and how to customize output reports.
The three LED staff members help each program at the center use ETO to measure progress toward its respective outcome objectives. The staffers also serve as technical resources, supporting each program’s data collection and evaluation activities, resolving questions, and designing and modifying evaluation tools.
Castillo said program and LED staff together select and/or adapt existing instruments for assessing risk behavior, self-esteem, mental health and other indicators. Most of the center’s programs also administer pre- and post-participation tests to measure knowledge and behavioral changes. Finally, partnerships with local schools provide access to school records for some participants.
The data that LAYC program staffers typically input include: client attendance, hours of programming provided and topic areas covered; narrative case notes; pre- and post-test data; school-supplied report card grades, attendance, test scores and behavior ratings; contacts among staff members and participants, teachers and employers; and employment history and job retention rates.
The center says each program evaluation conducted by LED seeks to answer three basic questions:
• Whom does the program serve?
• What does the program do and what are the outcomes?
• Why did the outcomes occur?
(Outside evaluators are sometimes hired to confirm LAYC data on certain programs. For example, Public/Private Ventures is reviewing one program now.)
What They Were Looking For
All programs must fit at least one of their outcome goals into one of LAYC’s three major mission areas: increased academic success; successful transition to work or completion of vocational/technical school; or improved healthy behaviors.
Those are measured by 16 indicators, including:
• For in-school youth (ages 6 to 18): A 90 percent school attendance rate, promotion to the next grade level, graduation and completion of at least two years of post-secondary education.
• For out-of-school youth: Successful transition back to school, receipt of a GED or completion of alternative school.
• For older youth: Demonstrated improvements in employment readiness skills, working at least 20 hours per week and retaining a job for at least 12 months.
• For various clients: Increases in knowledge about safe sex practices, stable housing, reductions in or elimination of substance abuse, and demonstrated skills in accessing health care, legal services and government benefits.
What They Found
Selected key findings from LAYC’s fiscal 2008 evaluation report include:
• AmeriCorps Program: AmeriCorps staffers that the center placed in Washington elementary and middle schools worked with 177 low-performing students for an average of 49.5 hours per student. An analysis of report cards from 45 of the students showed that more than half improved at least one letter grade in language arts, and 44 percent improved at least one letter grade in math.
• Upward Bound: LAYC’s Upward Bound program prepared 75 high school students for post-secondary education during fiscal 2008. All 16 seniors enrolled in the program graduated and enrolled in post-secondary education.
• Evening Reporting Center: From July 2005 to the end of fiscal 2008, LAYC provided alternative juvenile detention services to 322 youths, who reported to the center Monday through Saturday. During that period, less than 1 percent of LAYC’s participants were rearrested while in the program, and just 3 percent failed to make all of their scheduled court appearances.
• Counseling and Treatment Services: Some 221 youths (ages 12 to 21) received mental health and substance abuse treatment services in fiscal 2008. Using the Ohio Mental Health Consumer Outcomes System to track treatment progress, 71.9 percent of those youths showed decreases in problem behaviors and 28 percent showed improvements in coping skills.
• Residential Services: In fiscal 2008, several LAYC programs collectively provided housing and support services to 55 youths (ages 16 to 21). Life skills inventories that were administered upon entering the program, and at three- and six-month intervals, found that participants demonstrated significant improvements in housing, anger management, food management and money management skills.
• Changes: Castillo said negative outcomes have prompted LAYC to change components of a program, but very rarely an entire program. For example: Data from a staff-led parenting program that included a domestic violence component actually showed a strengthening in belief among participants, after the program, that “hitting could be a sign of love.”
After consulting experts about the possible reasons for that outcome, LAYC hired a local domestic violence organization to deliver its own curriculum in gender-segregated classes. The next “post-test” showed a decrease in that belief about hitting.
