The Final Report of the White House Task Force on Disadvantaged Youth, long awaited by youth policy mavens, was recently released with all the fanfare of a CIA dossier. But if you’re at a youth-serving agency that gets federal money, you might want to track down a copy; it calls for significant changes in how Washington evaluates and funds youth programs.
At a time when youth-serving agencies feel more pressure than ever to produce evaluations, the task force found little scientific evidence that most federally funded youth programs work. It says programs should be measured by more stringent scientific methods or risk losing their funding.
And, while many youth programs seek to provide comprehensive services for both kids “at risk” and those who are better off, the task force says there’s not enough money for that; it says funds should be targeted more toward those most at risk.
Those are among the major findings of a report that has leading advocates for the youth field impressed with its knowing assessment of the field’s strengths and problems, cautiously optimistic about its impact – and a little worried.
The report could form the basis of legislative and administrative changes to improve youth services, or it could be used to put more burdens on youth-serving agencies to prove their effectiveness or lose funding. It could also suffer the fate of many a Washington report and simply die on bureaucrats’ shelves.
Even that would be better than what many had feared. When the task force was formed in late 2002, “people were on pins and needles as to whether the task force was just an excuse for putting a report around a hit list” of programs that should lose federal funding because they hadn’t proven their worth, says Karen Pittman, executive director of the Washington-based Forum for Youth Investment. “That didn’t happen.”
Instead, the report speaks the language of positive youth development, and it “reflects the needs of the field,” says Shay Bilchik, president of the Child Welfare League of America. It focuses not on community-based programs but on the federal bureaucracy, finding that federal youth programs are too fragmented among agencies, with frequent overlaps and almost no coordination of objectives, approaches and methods to assess effectiveness.
The report’s recommendations include the creation of a Disadvantaged Youth Initiative to steer efforts to streamline federal youth policy and determine what approaches work best.
The feelings of promise and worry around the report are reflected in the fact that four people interviewed for this story used the phrase “the devil is in the details” to assess what impact it might eventually have.
Mission Creep
The task force was like a group of building inspectors examining their own house. It was composed largely of mid-level political appointees and senior civil servants from federal agencies, including the Departments of Health and Human Services, Justice, Education, Labor, and Housing and Urban Development (HUD), with contributions from nongovernmental experts such as Gary Walker, president of Public/Private Ventures (P/PV), and researcher Doug Besharov of the American Enterprise Institute.
Their mission was to examine the federal government’s role in helping disadvantaged youth. To no one’s surprise, they found a tangled mess: 339 youth programs in 12 departments, spending $223.5 billion in fiscal year 2003. Most of this money went to nonprofits and local government agencies. (A “program” for the task force was a federal program, such as Job Corps, not the thousands of grantees funded by such programs.)
“The current federal response to disadvantaged youth,” the task force found, “is a perfect example of ‘mission fragmentation.’” It says the 339 programs waste money and manpower through “overlap and duplication” as well as “mission creep,” and that agencies have “considerable discretion in the activities they conduct and populations they serve. … Agencies exercise that discretion aggressively and widely. As time goes on, agencies often expand the program to add in the issue du jour.”
That might be OK if the programs produced results, but the task force found that more than half of the federal programs hadn’t been evaluated in the past five years. Of those that had, only 27 used evaluations based on random assignment studies, which the report calls the most reliable tool. Of 28 programs rated under the Performance Assessment Rating Tool process by the Office of Management and Budget, only one was rated “effective,” and three were “moderately effective.”
This might surprise managers at youth-serving agencies, which in recent years have been increasingly pressed by government and foundation funders to demonstrate their impact on youth. “Every after-school program in the world has an evaluation,” says Walker of P/PV.
In 2001, the prestigious National Academies released a 388-page study, “Community Programs to Promote Youth Development,” which concluded that scientific-based evidence shows that certain youth development programs and practices help adolescents in specific ways. (“Scientists Dissect Youth Development,” December/January 2002.) That report is not cited among the federal report’s 151 footnotes.
Now the task force says almost none of the federal youth programs are conducting evaluations, and most of the evaluations that are done aren’t worth much. Why the gap?
Go for the Gold
The new report reflects a gap between the evaluations most often done in the field and evaluations that meet scientific standards.
Most youth agency evaluations are either process evaluations, tabulations of outputs (such as how many kids were served and how many hours of community service they logged) or measurements of how youth are doing after a certain amount of time in a program (such as how many have raised their grades or haven’t been rearrested).
“There are folks reporting on outcomes, but they’re far from scientific,” Pittman says. “Coming from the perspective of the White House task force, there are very few experimental or even quasi-experimental evaluations of youth programs.”
Besharov, who advised the task force on evaluations, noted that while some grants come with small set-asides for evaluations, those set-asides are typically too small to cover a scientifically valid study. “They have somebody with a Ph.D. come in and look around,” he says.
Besharov wants to see more of what the report calls the “gold standard”: evaluations that use large samples, with youth randomly assigned to program and control groups.
“Those are really expensive,” says Phillip Lovell, director of public policy for Camp Fire USA. “When you have limited resources, are you going to spend your time and money on evaluations, or are you going to spend your time and money on helping kids?”
Setting up control groups that don’t get services is anathema to many youth program leaders. “We’ve tried to avoid highly expensive evaluations or evaluations which force you to exclude young people from ever being served,” says Dorothy Stoneman, president of YouthBuild USA, an independent association of the HUD-funded YouthBuild agencies. “Given that we do have long waiting lists, you don’t really want to say to those young people, ‘Not only are you not in this year, but you can’t be in next year.’ ”
The task force doesn’t seem to think most youth programs should run such studies, either. “They should not be worried when we talk about the gold standard,” says Karen A. Morison, a consultant who served as task force staff director.
So what does the task force want?
Evaluations
Anyone who runs a youth program with federal money might get worried reading this from the report: “With a federal investment in youth-serving programs of hundreds of billions of dollars annually, we needed to firmly hold programs accountable for results showing that they actually achieve what they were designed to accomplish. This means that we need well-designed evaluations of current programs so that those not achieving their goals can be quickly discontinued and their resources diverted to other priority needs.”
The report recommends that the government develop a “unified protocol” to determine what works and build a “rigorous and unified” research agenda for youth programs. Those programs would be held accountable for results, with the government developing standards for measuring grantee performance.
While random assignment experiments are preferred, the report also notes the value of nonexperimental evaluations, such as those that compare youth before and after participation in a program.
Although the youth field has been inching toward more and better evaluations for years, these recommendations would push things much further along. That raises at least two concerns for youth-serving agencies.
One is who will conduct the evaluations and with what money. “The ‘gold standard’ evaluation is a concern in that it’s unrealistic” to carry it out with current government funding, says Irv Katz, president of the D.C.-based National Collaboration for Youth.
Heather Weiss, director of the Harvard Family Research Project, notes that the report doesn’t call for more federal spending on research. “The commitment to research and evaluation is important only if some money follows it,” she says.
Several task force members and advisers, such as Morison and Besharov, say they’d expect the federal government to fund and conduct the studies. They envision studies determining what practices seem to produce the desired results, with government funders holding grantees responsible for implementing those practices.
Several federal agencies do something like this, such as the Substance Abuse and Mental Health Services Administration’s National Registry of Effective Programs. But these best practice summaries generally serve as guidelines for grant applicants rather than requirements that they follow certain models.
Under the approach envisioned by the task force, government agencies would demand fidelity to certain program models while allowing some flexibility for local conditions.
“What you will increasingly see is that, in order to get federal dollars for an after-school program, recipients will have to adopt a program that has evidence of effectiveness associated with it,” or be part of something innovative that they can show will work, says task force education committee member Grover Whitehurst, director of the Department of Education’s Institute for Education Sciences.
Whitehurst says that if he were running a youth-serving agency, “I would be trying to think about how I could align my practices with the best evidence of what works. Or if I’m doing something innovative, how I can show that my unique services add value.”
The second major cause of concern is that the evaluations could be used to cut programs. Last year the Bush administration used a federally funded evaluation to justify deep reductions in the 21st Century Community Learning Centers after-school program. After-school advocates were furious, saying the study, which showed weak effects, was flawed and the programs were evaluated too soon for the measurements to be fair.
“You really can see the potential for the following scenario,” says Bilchik, conjuring the image of federal officials grilling a youth agency: “We challenge the field to do more evaluation. We did some set-asides in the budget. Have you been able to show the effectiveness of your programs? No? We’re going to be slashing your funding.”
That’s why it would be important to develop “decision rules about what a fair evaluation is,” says Weiss at Harvard. The rules would cover such matters as how many years a program should be operating before it is judged under a rigorous evaluation.
Getting that far depends on how the report’s findings are implemented, if at all.
Now What?
The task force largely steered clear of specifying how its ideas would be carried out. The report’s cover letter notes that the mission was to develop “a framework for federal youth policy” within “existing authorities and programs.” There are no proposals for more spending, legislation or agencies.
It does recommend the creation of a Disadvantaged Youth Initiative to guide implementation of the recommendations. It also makes two specific administrative changes of note: Moving YouthBuild from the Department of Housing and Urban Development to the Department of Labor, saying that it is primarily a job training program; and moving the Gang Resistance Education and Training Program from the Bureau of Alcohol, Tobacco, Firearms and Explosives, which is in the Department of Justice, to the department’s Office of Juvenile Justice and Delinquency Prevention.
Stoneman at YouthBuild USA says she needs to talk more with federal officials to determine whether the move would be good for YouthBuild programs.
Whether any of these ideas go anywhere depends on who takes the initiative to make them happen. No one in Washington appears to “own” this report – that is, to be responsible for making sure any part of it is implemented or even to advocate for it.
The report was released so quietly to federal agencies that word of its existence only got around informally last month among policy leaders in the youth field, some of whom had trouble tracking it down. Several federal officials declined to talk about it because they had received no direction from the White House. John Bridgeland, the USA Freedom Corps director who served as task force vice chairman and was seen as a champion of its work, resigned last month to spend more time with his family. The task force has disbanded, now that its work is done.
“It’s not just that it isn’t clear whether the recommendations are going to be used,” Pittman says. “It’s whether anybody’s going to read it.”
Morison notes that enacting many of the changes, which involve collaboration among offices in different federal departments, will require something rather complicated: “leadership across agencies.” She’s optimistic about that happening, however, because the task force was made up of key executives from those agencies.
Some people outside government aren’t waiting. Groups such as the National Assembly, Campfire USA, the National Youth Employment Coalition and the Forum for Youth Investment are talking about how to carry out some of the recommendations.
A few of those recommendations sound a lot like elements of legislation that has already been proposed: the Younger Americans Act. But that legislation hasn’t even picked up enough support for a congressional hearing, and Lovell of Camp Fire says that “moving that piece of legislation in this Congress is not politically viable.”
He and others envision pushing for smaller pieces of legislation and for administrative changes within agencies to accomplish some of the report’s objectives.
Says Morison, “People in the youth field, the more they talk about it, the more they write about it, the more they express their opinions to the White House about it,” the better the chances that the study will become something more than another dust collector.
Patrick Boyle can be reached at pboyle@youthtoday.org.
Resources
Final Report of the White House
Task Force on Disadvantaged Youth
www.ncfy.com/disadvantaged/report.htm
Karen A. Morison, Staff Director
kam9t@alumni.virginia.edu
Douglas J. Besharov, Director
Social and Individual Responsibility Project
American Enterprise Institute
1150 17th St. N.W.
Washington, DC 20036
(202) 862-7160
dbesharov@aei.org
Karen Pittman, Executive Director
Forum for Youth Investment
7064 Eastern Ave. NW
Washington, DC 20012
(202) 207-3333
www.forumforyouthinvestment.org
Gary Walker, President
Public/Private Ventures
2000 Market St., Suite 600
Philadelphia, PA 19103
(215) 557-4400
www.ppv.org
The Task Force
Chairman
• Margaret Spellings, assistant to the president for domestic policy
Vice chairman
• John Bridgeland, director, USA Freedom Corps
Staff
• Karen A. Morison, staff director
• Sonia Chessen, senior policy adviser
• Stan Chappell, senior adviser
• Mary Beth Luna, counsel
• Robert Patterson
Members
• Eric Andell, deputy under secretary, Department of Education
• Dr. Andrea Barthwell, deputy director for drug demand reduction, Office of National Drug Control Policy
• Roy Bernardi, assistant secretary, Office of Community Planning and Development, Department of Housing and Urban
Development
• Mason Bishop, deputy assistant secretary of labor for employment and training, Department of Labor
• Jim Capretta, associate director for health and personnel, Office of Management and Budget
• Deborah Daniels, assistant attorney general, Office of Justice Programs, Department of Justice
• Stephen Goldsmith, chairman, Corporation for National and Community Service
• Roy Grizzard, assistant secretary for disability employment policy, Department of Labor
• David Kuo, deputy director of the White House Office of Faith-Based and Community Initiatives
• Larry Matlack, deputy associate director, education and human resource programs, Office of Management and Budget
• Juliet McCarthy, director of the USDA Center for Faith- Based and Community Initiatives, Department of Agriculture
• Susan Sclafani, counselor to the secretary of education, Department of Education
• David Reingold, director of research and policy, Corporation for National and Community Service
• Don Winstead, deputy assistant secretary for human service policy, Department of Health and Human Services
More Focus on Kids at Risk?
“Target, Target, Target.”
That was the first recommendation Gary Walker, president of Public/Private Ventures, made in a letter last March to Margaret Spellings, assistant to the president for domestic policy and chairwoman of the White House Task Force on Disadvantaged Youth.
“Spreading limited funds across many youth may appear to be preventive, but in fact, it typically underserves or never engages the youth who most need help – and who become our most serious social problems,” Walker wrote. “Public money should be spent on public problems – not on grand schemes to benefit all youth.”
Spellings passed the letter to task force staff director Karen A. Morison, who says that of the 30 to 50 letters the task force received, “His was the most on-target in terms of what direction that I understood we needed to go in.” Walker accepted an invitation to speak to the task force, and Morison says his ideas helped to shape the report’s focus.
The report says the federal government should focus first on youth who are in public institutions and create public expense – namely, those in foster care and the juvenile justice systems. It then identifies other groups who are at risk and in need of special attention, such as immigrant youth and children of incarcerated parents. It calls for more and better education, work force training and mentoring for such youth.
While those youth need help, Shay Bilchik, president of the Child Welfare League of America, worries that such a focus “misses the maximization of opportunities for youth more broadly who are also vulnerable. You don’t have to be in the foster care system to need the full range of support.”
Blueprint for an Overhaul
Key findings in the Final Report of the White House Task Force on Disadvantaged Youth
Aspirations for Disadvantaged Youth
To grow up:
Healthy and safe
Ready for work, college and
military service
Ready for marriage, family and parenting
Ready for civic engagement and service
Recommendations
Better Management
Streamlining
• Create a Disadvantaged Youth Initiative to eliminate duplica- tion, make messages more consistent across agencies, bring programs into the agency with whose mission they are most closely aligned, have a unified definition of best practices.
Mission Alignment
• Move YouthBuild to the Department of Labor and the Gang Resistance Education and Training program to the Office of Juvenile Justice and Delinquency Prevention.
Interagency Coordination
• Improve coordination of mentoring programs.
• Support state and local community planning process.
Improve the Federal Grants System
• Modernize the Catalog of Federal Domestic Assistance.
• Create a central and improved grants database.
• Improve the database of the Federal Assistant Awards Data System.
• Create a resource mapping function for the database.
• Research eligibility of faith- based grant applicants.
Better Accountability
Holding Programs Accountable for Results
• Develop standards for measur- ing grantee performance.
• Implement grantee-level perfor- mance measurement guidelines.
• Conduct rigorous oversight of earmarked grantees.
• Implement No Child Left Behind in Defense Department schools.
Understanding What Works
• Developed a unified protocol for federal “what works” clearinghouses.
• Build a rigorous and unified disadvantaged youth research agenda.
• Improve data collected on the well-being of families.
Better Connections
• Increase parent involvement in federal youth programs.
• Design a youth service initiative.
• Recruit youth for federal grant review panels.
Give Priority to the Neediest Youth
• Target youth in public care.
• Target kids at high risk.
• Create demonstration program for education of foster youth.
• Create interagency committee to focus on education needs of foster youth.
• Create interagency program of work force training and education for migrant youth.
• Expand mentoring programs to foster and migrant youth.
