Philadelphia—Like everyone who runs a youth-serving agency, Simran Sidhu believes her YouthBuild program here improves lives. But like most program managers and front-line youth workers, she’s been frustrated that the evidence “continues to be anecdotal,” built mainly on updates from alumni who call or visit.
But unlike most agency mangers, Sidhu recently got what she wanted – two studies that demonstrate the long-term impact of YouthBuild and show a way that other programs might track their own impact as well.
The studies, released last month, give operators at the nation’s 200 YouthBuild programs the kind of information that youth agencies covet: confirmation that they have systematically improved the lives of at-risk kids, feedback on what seems to be effective and indications about what they should improve.
The studies are unusual because they do what “every consultant who works with a program” tells its administrators to do, said researcher Andrew Hahn: “Track your graduates” to see how they fare after the program ends. Hahn, a professor at the Heller School for Social Policy and Management at Brandeis University, teamed with program evaluator Thomas D. Leavitt at Brandeis to survey 882 YouthBuild graduates. The other study, conducted by two professors at Temple University, consisted of in-depth interviews with 57 graduates.
YouthBuild helps at-risk young people ages 16 to 24 work toward high school diplomas or GEDs through a youth development program in which they learn job skills through building and renovating affordable housing. The programs are funded largely through the U.S. Department of Housing and Urban Development (at $65 million in fiscal 2004).
The studies were commissioned by YouthBuild USA, a nonprofit that provides training and other support toYouthBuild programs, with $300,800 in grants from the W.K. Kellogg Foundation and an anonymous supporter. “I wanted to get some systematic answer to the question of, ‘How are our graduates doing?’”said Dorothy Stoneman, president of YouthBuild USA.
The biggest challenge was to find people who had graduated from YouthBuild as far back as 1988.
Tough to Find
Eleven YouthBuild agencies that agreed to take part in the study set out to find their alumni, while the researchers sought graduates from other sites by tapping an alumni database from YouthBuild USA. The process was helped by the fact that YouthBuild programs keep in touch with many alumni through such means as newsletters, conferences and career development assistance.
The researchers were trying to find a particularly elusive population: young people, most of them poor, who are highly mobile. Those who were found then chose whether to fill out a 15-page survey.
Although the effort was far beyond what the vast majority of programs attempt, the response rate for 3,900 graduates was 22.5 percent – “less than ideal,” acknowledges the final report, “Life After YouthBuild.”
This points to limitations that would face anyone trying to study graduates of such youth programs, and raises questions about whether the sample is scientifically representative. Not only was the response rate low, but the 11 sites that make up the base of the study, including Sidhu’s YouthBuild Philadelphia Charter School, are among the stronger YouthBuild programs. Also, it seems likely that most of those who took the time to complete the survey had positive feelings about their YouthBuild experiences and were doing relatively well (which might mean they’re more stable and easier to find)
Hahn acknowledged the concerns, but said several factors convince him that he had a representative sample. First, the survey responses come from graduates of 73 sites. Second, 882 is a high enough number of responses that, after a certain point, more replies don’t make much statistical difference. Finally, one site (in Bloomington, Ill.) had an astounding 98 percent response rate. Its results essentially mirrored the results of the entire study, the report says
The second part of the project – the interviews – involved 57 graduates from eight sites. One of the two principal researchers, Erin McNamara Horvat, said that in order to balance a potential bias – that those who were most inspired to talk about their YouthBuild days were the ones who most liked it and were doing well – the researchers made sure to interview two people from each site who were out of school and out of work.
It must be noted that YouthBuild USA sought to assess the program’s impact on people who completed it; it did not include those who dropped out, who might be more likely to being doing poorly or to have negative assessments of the program. Stoneman and Hahn stressed that they did not set out to scientifically evaluate YouthBuild as a whole, although both said a next step could be a study that includes dropouts and a control group.
“If I was sitting in a congressional budget office, would I rely solely on this” to decide on YouthBuild funding, Hahn said. “No. … This is a building block.”
Impact on Jobs, Drugs and Crime
Sidhu probably reflects the views of other YouthBuild managers when she says that what she got from the studies was primarily “a lot of confirmation” that good YouthBuild programs improve kids’ lives.
By and large, the YouthBuild graduates were doing well and attributed much of their success to what they learned in the program. That means not only construction skills, but what they learned about “mental toughness,” responsibility, a work ethic and leadership.
But don’t dismiss the value of the construction experience. YouthBuild is exploring alternative job tracks, such as computer technology, but the Temple University researchers said they “caution against it” after being taken aback by the value that the graduates placed on physical labor. The report says those who were interviewed routinely spoke of “immense pride” in seeing the fruits of that labor and of walking home in their dirty work clothes, with their neighbors seeing that “they had been working at a legitimate job.”
Among the key statistical findings from the survey of graduates:
• 74.6 percent were “successful,” meaning they were working, or in school or job training.
• 76.2 percent were not getting food stamps, welfare or unemployment benefits.
• 60 percent rated YouthBuild “excellent;” 31 percent rated it “good.”
• 65 percent expected to live longer than they had expected before their YouthBuild experience – by an average of 32 years.
• 80 percent had engaged in none of three negative behaviors since leaving YouthBuild: selling drugs, getting convicted of a felony or spending time in prison.
For those behaviors and several others, researchers examined differences before and after YouthBuild. For example, 32 percent of those surveyed said they had sold hard drugs before entering the program, and 30 percent reported having used them. After YouthBuild, 8 percent said they’ve sold hard drugs and 6 percent said they’ve used them. The survey found similar before-and-after differences for outcomes such as convictions, homelessness and being a victim of violence.
Criticisms? While 73 percent said they’d received post-graduate help from YouthBuild (such as small grants or career guidance), one common theme was that the graduates needed more help. Many expressed frustration at trying to find steady jobs with opportunities for advancement, having had to settle for low-level service industry jobs.
“For some students, one ten-month tour in YouthBuild is not nearly enough to counter the weight of these problems,” said the Temple report, referring to those who slipped back into drugs, crime and abusive relationships.
YouthBuild is looking for ways to improve graduate services. But the central benefit of the studies, Stoneman said, is that “we got great reassurance that our work is worthwhile.”
Contact: YouthBuild USA (617) 623-9900, www.youthbuild.org. To view the studies, click on “graduate research report.”
