Throughout the year, first lady Laura Bush has traveled the country touting programs that the White House says do everything from steering kids away from drugs and violence to increasing reading skills and building character. The visits are part of Mrs. Bush’s Helping America’s Youth initiative, which seeks to promote programs that have been demonstrated to be effective.
But as the first lady prepares to convene a national Helping America’s Youth summit in Washington next month, a look at some of the programs she has visited shows that, by and large, they are based on promising ideas, but have little scientific evidence of effectiveness.
The adviser who helped find model programs for the first lady describes facing the same challenge that confronts the operators, advocates and funders of youth development programs: “When you look out there, the number of programs that meet strong standards [for evaluations] are just not there,” says G. Reid Lyon, a well-known education researcher.
Mrs. Bush has said in speeches and interviews that the programs she’s visiting are “very effective,” “successful” and “have some track record.” But while some of the programs have been examined by independent evaluators, most of their evidence consists of self-evaluations, incomplete preliminary data, anecdotes or a belief that certain activities help kids – the kinds of evidence that youth development advocates have been citing for decades, but are now often told is not good enough.
There is no doubt that the programs do wonderful things for youth and that the first lady’s campaign has brought valuable attention to youth work. But her visits illustrate a continuing struggle for the youth work field at a time when government and foundation funders are demanding better evaluations.
Many if not most of the programs spotlighted by Mrs. Bush might not meet the evaluation standards increasingly demanded by her husband’s administration, which has cited a lack of evidence of effectiveness in proposing to eliminate or trim youth programs. They almost certainly don’t meet the standards for scientific evidence called for two years ago in a report by the White House Task Force on Disadvantaged Youth.
Yet for anyone looking for insight into what kinds of youth programs the White House prefers, the programs that Mrs. Bush chose to visit are a good place to start.
First Lady’s Agenda
Much of the news media have described the first lady’s initiative as “anti-gang,” but it is not.
The label comes from the president’s State of the Union speech in February, when he proposed “a three-year initiative to help organizations keep young people out of gangs, and show young men an ideal of manhood that respects women and rejects violence. Taking on gang life will be one part of a broader outreach to at-risk youth.”
The communications-savvy Bush administration might have figured that describing the initiative as anti-gang would make more of an impact on Congress, the media and the public than pitching, say, a “youth development” effort. Congress is considering several pieces of tough anti-gang legislation.
However, Helping America’s Youth is actually a broad umbrella for youth development programs involving responsible fatherhood, healthy marriages, reading skills, character education, mentoring children of prisoners and teen sexual abstinence, as well as gang prevention.
Although those are mostly non-school issues, the initiative’s roots lie in the first lady’s interest in education, especially literacy. During her travels around the country last year, Lyon says, Mrs. Bush, a former teacher and librarian, grew more concerned that “even with the best reading program … a lot of the kids were having difficulties focusing on academics, for a wide variety of reasons,” such as living in poverty or having dysfunctional families. “When they get to school, they’re just not ready for learning.”
Mrs. Bush wanted “to take a look at programs that developed good academic skills through better behavior,” he says.
She was especially concerned about at-risk boys. Mrs. Bush has said that concern was inspired by a New York Times Magazine article last year about Kenneth Thigpen, a troubled young man in Milwaukee who turned his life around and became a responsible father.
Lyon says Mrs. Bush asked him to find programs that helped youth both academically and behaviorally. The idea was that “we would visit sites across the country that … combined good reading instruction – evidence-based reading instruction – with good behavioral support programs that had data with them.”
Lyon, a sometimes controversial advocate of scientifically based reading instruction methods, was then a top official at the National Institute of Child Health and Human Development and had advised the administration before. He is now overseeing the launch of a worldwide chain of for-profit teachers’ colleges for a Texas company, Best Associates.
Officials in federal agencies such as the Justice, Labor, and Health and Human Services departments sent suggestions for programs to highlight, Lyon says. But he ran into the reality that claims about the effectiveness of many youth development programs “are based on anecdotes, philosophical beliefs and untested assumptions.” Lyon saw that few programs aimed at changing youth behavior had been scientifically tested, and almost none could meet the “gold standard” for evaluations, which would include random samples and control groups tracked over a significant period.
Mrs. Bush “was very aware that the evidence was not the same as in other fields where we had been working,” such as education, Lyon says.
It is a continuing dilemma: Except for programs focused primarily on academics, youth-serving programs have not traditionally been geared toward showing the kinds of outcomes that can be measured, say, over the course of a school year or a summer break.
The Final Report of the White House Task Force on Disadvantaged Youth, released in late 2003, said there was little scientific evidence that most federally funded youth programs work. It said programs should be measured by more strict scientific standards or lose federal funding.
“We need to firmly hold programs accountable for results showing that they actually achieve what they were designed to achieve,” the report said. (See “Federal Youth Programs and Policy Taken to Task,” February 2004.)
The Bush administration cited poor evaluations when it proposed cutting the 21st Century Community Learning Centers program by 40 percent in 2004 (which failed). It cited a lack of proven effectiveness to justify cutting TRIO (an educational outreach program for disadvantaged youth) by more than half in its 2006 budget proposal and to justify eliminating such programs as Even Start and GEAR UP (college prep) in the Education Department, and Juvenile Mentoring, Safe Schools and the Juvenile Accountability Block Grant in the Justice Department.
There are, however, scientific studies that demonstrate the effectiveness of certain elements of youth work, like providing close relationships with caring adults (such as mentors) over a long period of time. These findings were analyzed in a 2001 study by the National Academies, “Community Programs to Promote Youth Development.”
The first lady’s office chose to highlight programs that “were based upon behavior principles and social development principles and family interaction principles that were known to be effective,” Lyon says.
In talking about her initiative with “The PBS NewsHour” early this year, Mrs. Bush said she is “visiting these programs around the country that are already successful, that we know they’re successful, that they have some track record.” Her role on each visit, she said, is “to highlight a very effective program.”
Youth Today selected a random sample of those programs, looking especially for those that appear to have some evidence of effectiveness. Following are the results.
Helping Youth with … More Funding?
Programs that Laura Bush has touted under the Helping America’s Youth initiative might get a financial boost, but not necessarily from the federal government.
The initiative has a funding aspect, although it is modest by Washington standards, and the first lady has not been publicly involved in pushing it. The president’s 2006 budget proposal includes a number of ideas tied to Helping America’s Youth: $40 million for fatherhood programs, $100 million for healthy marriage programs, $24 million for character education and $150 million over three years for an anti-gang “role model” effort. Early indications are that the Republican-controlled Congress might not even fund that much.
The first lady’s office says that Helping America’s Youth is not primarily about federal money but about calling attention to worthwhile models that can be replicated and expanded with support from state and local governments, foundations, corporations and others.
Agencies say the visits by Mrs. Bush have brought them more attention, including attention from potential donors. For example, a visit in March to Rosalie Manor Community & Family Services in Milwaukee prompted a $17,000 challenge grant from the Milwaukee-based Harley Davidson Co., says agency Director Al Castro.
Good Behavior Game (GBG)
Baltimore, Md.
Site visited: George Washington Elementary School
When: February
The program: GBG is a classroom behavior modification strategy developed at the University of Kansas in 1968 and used in schools throughout the country. Teachers divide students into teams, define a set of disruptive behaviors and give demerits to teams whose members display those behaviors.
The game’s start is initially announced once or twice a week and played for short periods. Later, the game is begun with no warning and can be played for rest of the day. At the end of each game period, the teacher awards a prize to the teams who have not exceeded the maximum number of demerits.
Mrs. Bush said: It’s a “simple, inexpensive intervention that has a dramatic impact on a child’s behavioral and academic development.”
The Research: Nearly 20 studies since 1969 have shown that GBG has positive results, some of them lasting into adulthood.
Early studies found that classroom disruptions – including talking out of turn, leaving one’s seat and displaying aggressive behavior – fell by between 10 and 99 percent each time the game was played. During game times, teachers were able to cover 25 percent more material.
After one year of GBG intervention in the 1985-86 school year, Baltimore first-graders (especially those rated “highly aggressive” by teachers and peers) showed significantly reduced aggressive behavior and increased on-task performance in the classroom. A later study found that the positive effects waned in third and fourth grades, but re-emerged later.
Sheppard Kellam, director of the Center for Integrating Education and Prevention Research in Schools at the American Institutes of Research (AIR), is researching the program’s impact at schools in Baltimore. AIR has tracked youths by collecting three waves of data since 1985, Kellam says. The AIR studies are randomized field tests, which are considered to be rigorous evaluations. The studies are funded by the National Institutes of Health and the National Institute on Drug Abuse.
Kellam says the most recent research findings, which have not been published, indicate that children who participate in the game in first and second grade show surprisingly long-term effects – even into young adulthood. He says they graduate at higher rates, have fewer personality disorders, have less involvement in drug, alcohol and tobacco abuse, and engage in less high-risk sex.
Researchers have identified peer pressure, competition and social recognition as key components of the game’s success. As for the long-lasting impact, one theory is that the game helps to instill disciplined behavior at a young age.
Kellam and his colleagues have produced a GBG manual, which is available free at www.bpp.jhu.edu/publish/Manuals/gbg.html.
Boys & Girls Clubs
Philadelphia, Pa.
(215) 844-7771
www.bgcphila.org/clubcorner/germantown
Site visited: Germantown Boys & Girls Club
When: February
The program: This is the first program Mrs. Bush visited after Helping America’s Youth was announced. She sat in on and talked about two Boys & Girls Clubs of America (BGCA) programs that are run here: Passport to Manhood and SMART Moves.
Passport to Manhood is designed to teach responsibility and character development to boys, pre-teen and older. Started in 1999, the program operates in more than 100 of the 3,700 Boys & Girls Clubs, says BGCA Vice President Jim Cox. Passport offers 10 to 14 weekly meetings at which boys interact with a male role model and talk about “what it means to be a man in society, with a positive code of ethics,” Cox says.
SMART Moves teaches youth to resist pressures to use drugs and engage in sex.
Mrs. Bush said: Research shows that in neighborhoods with Boys & Girls Clubs, “there’s generally a reduction in vandalism, drug trafficking and youth crime.”
The Research: The BGCA says it has no evaluations of Passport’s impact.
Several studies show that BGCA clubs have positive impacts on youth and their communities, including improving some refusal skills. It appears there is no study that finds “a reduction in vandalism, drug trafficking and youth crime” in communities with BGCA clubs.
The findings of a study at the Columbia University School of Social Work, dated 1991, seem to come closest to what Mrs. Bush said. The study looked at 15 public housing sites, with and without BGCA clubs, in several cities.
A BGCA summary of the study says public housing sites with clubs had 25 percent less “crack presence” than those without clubs, along with 22 percent less drug activity and 13 percent fewer juvenile crimes. Among the sites with clubs, the presence of crack and drug dealing was lowest in sites with SMART Moves.
The study did not measure whether any of these activities declined at the sites with BGCA clubs. Nevertheless a BGCA brochure cites the Columbia University study and says, in large type, “Clubs in public housing areas dramatically reduce crime and drug presence,” and says the housing sites with clubs saw a “13 percent reduction in juvenile crime,” etc.
“It’s arguable that the word ‘reduction’ should not have been used” in the brochure, says BGCA spokesman Evan McElroy. He notes that the study says clubs “appear to be associated with an overall reduction in substance abuse, drug trafficking, and other drug-related criminal activity,” but that seems to be the researchers’ inference.
Computer Assisted Debate (CAD)
Atlanta, Ga.
(404) 727-6189
www.cadatl.org
Site visited: Benjamin S. Carson Honors Preparatory School
When: March
The program: For more than a decade, advocates of education reform (such as the Open Society Institute) have been promoting high school debate as a powerful tool for youth, sparking what is called the “urban debate” movement. There are numerous high school debate programs in cities around the country, but Mrs. Bush visited a program that is too new to have been significantly evaluated and takes on the challenge of teaching debate to at-risk middle schoolers.
A few years ago, an Atlanta Housing Authority official noticed a report about a school debate in Baltimore, and thought a structured series of debates in local schools would be good for the youths there.
The after-school CAD takes place at the Carson school, where the Boys & Girls Club of Metro Atlanta funded a computer lab for youth to do research. The middle schoolers volunteer for the program, which is run by school faculty and staff and by graduate students from local universities. The program began in the 2004-05 school year; a second middle school was recently added.
The program includes practice and instruction after school and in summer camp, as well as competition in debate tournaments.
Mrs. Bush said: Participants “learn to see both sides of an issue, which helps them defuse potentially dangerous situations. Debaters learn how to identify a good argument and reject a bad one, so they’re better equipped to deal with the hazards of negative peer pressure. … CAD encourages students to value their own opinions, which leads to increased self-awareness and higher self-esteem.”
The Research: The program is undergoing its first evaluation, which has indicated mixed results so far.
Lyon says the fledgling Atlanta program was chosen because it had elements shared by programs that have been shown to be effective. Various studies have shown that debate seems to improve grades and self-confidence, and there are indicators that it might improve behavior.
There appears to be no research that says debate experience helps youth “defuse potentially dangerous situations,” as Mrs. Bush said.
Studies dating back to the 1980s have found that debate improves critical thinking among college students. But one recent study notes, “Most claims about the value of debate for high school, let alone at-risk, students, are based on anecdotal testimonies … or extremely limited empirical data.”
That study, “Argument for Success: A Study of Academic Debate in the Urban High Schools of Chicago, Kansas City, New York, St. Louis and Seattle,” measured debaters’ and non-debaters’ levels of self-esteem, locus of control (degree of control over events in their lives), risk-taking propensity and self-efficacy (sense of confidence in their abilities). The study measured 421 youths from the five cities, using randomly selected debaters and control groups of non-debaters.
Over about one academic year, the reading scores of debaters went up 25 percent more than did those of the control group, the study says. Debaters also had higher self-esteem and scored significantly lower on the Adolescence Risk Taking Scale than did non-debaters, although that might reflect more on the youth who enter debate programs than on the impact of the programs. Changes in those two characteristics were not measured over time, as grades were.
The researcher, Linda Collier, says she found evidence of self-empowerment among debaters, and that feelings of self-empowerment “help kids resist negative peer pressure.”
Numerous studies link high self-esteem to increased grades and decreased levels of destructive behaviors such as drug abuse and violence, although some researchers have said the value of self-esteem has been significantly overblown. (See “Getting Down on Self-Esteem,” April 2004.) Studies have also found high academic achievement to be a “protective factor” for at-risk youth.
The study is being reviewed for publication and was provided by Collier, who is an associate professor of communication studies at the University of Missouri-Kansas City.
The Atlanta program is in the early stages of a study by researchers at Georgia State University. Preliminary, unpublished results showed “noticeable increases” in grades, according to a summary provided by Carol Winkler, chairman of the university’s communications department.
The study stresses that this was a pilot program. “After the first year, protocols were tightened” to better help the youth with public speaking, Winkler says, and to better measure changes in areas such as reading skills, class engagement and violence.
Think Detroit
Detroit, Mich.
(313) 833-1660
www.thinkdetroit.org
Site visited: Wigle Community Center
When: February
The program: In 1997, local lawyers Michael Tenbusch and Daniel Varner decided that they could help at-risk boys stay out of trouble by creating a sports program that would feature mentoring by coaches. What they dubbed Think Detroit now has a $1.2 million annual budget and 650 volunteer coaches who each year serve 5,000 children as young as 4 years old.
Think Detroit, founded at the community center, is open to all youth. Co-founder Varner says that many of its participants live below the poverty line and “come from challenging environments.” Varner is convinced that “youth sports programs have a positive impact on kids.”
Mrs. Bush said: The program “empowers young people, both on and off the playing fields.”
The Research: There has been no evaluation. A three-year study is under way.
There is anecdotal evidence of effectiveness for some youth. Varner tells the story of Willie Reid, 12, whose father was murdered in the boy’s presence last year. Ever since, Willie had been in daily fights at summer day camp and at school. He never had played baseball, but joined a team through Think Detroit this summer, became a starting player and stopped fighting. The boy attributed the change to his coach’s mentoring.
Researchers have studied what makes for a successful sports program in terms of youth development. One key is “not just throwing the ball out on the court, but connections with caring adults for significant periods,” says Dan Gould of the Institute for the Study of Youth Sports at Michigan State University, which is evaluating Think Detroit.
Today’s Dads
Milwaukee, Wis.
(414) 449-2868
www.rosaliemanor.org/todays_dads
Site visited: Rosalie Manor Community & Family Services
When: March
The program: In 2001, this nonprofit social service agency started inviting men to one-on-one counseling for such issues as anger management, finding jobs, advancing their education and improving their parenting skills. The agency often recruits men at hospitals after their children are born, offering them services “to get their lives straightened out,” says Al Castro, Rosalie Manor’s director. “Many of them can’t read and write, and need help getting jobs, housing and legal help.”
Figures provided by the agency for fiscal 2003-04 showed that of 134 fathers served, 36 were teens. Among the teens, 79 percent were not living with the mothers of their babies.
The two staff members of Today’s Dads have an average total caseload of 100. The program hopes to put together an evidence-based curriculum of “survival skills” for fathers who don’t have child custody.
The program operates on annual funding of about $80,000 from state and federal grants, including grants from the Office of Adolescent Pregnancy Prevention and Special Projects of Regional and National Significance (SPRANS), both under the U.S. Department of Health and Human Services.
Mrs. Bush said: Men in the program “are making responsible decisions to stay involved in the lives of their children. Their children will benefit greatly from having a relationship with a loving and responsible father.”
The Research: Today’s Dads provided in-house data – compiled from case managers’ interviews with teen fathers and an external evaluation completed as a requirement of its SPRANS funding – indicating the men are doing much of what they’re supposed to do while they’re in the program. It is not yet clear if there are any long-term benefits.
Of 36 teen fathers served in 2003-04, five exited the program because they felt they had achieved their goals, such as finishing high school, obtaining employment and gaining visitation rights with their children. Among the remaining fathers, nearly three-quarters continued participating. Castro believes this is a credible record, given the “mobile population” of the agency’s clientele.
Among the measurements provided by Today’s Dads: Nearly three-quarters of teen girls who had babies with the fathers in the program reported being in contact with the father at least weekly, 57 percent said the father is “a significant help in raising the baby,” and 82 percent reported that “the program helped in the relationship with the baby’s father.”
Except for the last item, the measurements do not directly gauge the impact of the program; the benefits are implied. Because the men join voluntarily, they appear to be motivated to stay involved in their babies’ lives. The program appears to help motivated teen dads stay involved and achieve such goals as getting jobs or going back to school.
No measurement is available about the fathers’ behavior after they leave the program. Men stay in the program for an average of seven months, according to Rosalie Manor.
CeaseFire Chicago
Chicago, Ill.
(866) 862-3273
www.ceasefirechicago.org
Site visited: Alliance of Logan Square Organizations
When: June
The program: A noted anti-crime project in Boston under the CeaseFire name focused on law enforcement, namely by confronting street gangs. The Chicago version is more of a grassroots collaborative that is “about changing behavior, changing norms,” says Gary Slutkin, director of the Chicago Project for Violence Prevention, which launched the initiative in 1995. It operates in 16 Chicago neighborhoods and five sites elsewhere in Illinois.
The program networks with community-based organizations, faith leaders and law enforcement to intervene in conflicts, promote alternatives to violence and instill a community norm that violence is unacceptable. A paid staff of former convicts spreads the message “that shooting and violence don’t make sense,” Slutkin says. “They are the most credible messengers.” They make contact with “high-risk” kids at meetings and in informal settings – such as parks, street corners and other places young people gather – and distribute flyers and posters advertising the program.
The program runs on $3.9 million annually. Funders include the U.S. Department of Justice and the MacArthur and Robert Wood Johnson foundations, according to the Chicago Project’s 10th anniversary report.
Mrs. Bush said: “It’s so important for everybody in the community to be involved in something like CeaseFire. And that’s why you’re effective, because it isn’t just the police by themselves, but also the clergy join, community people join, doctors join, with a clear plan, with a clear goal, with clear objectives that everybody understands, so, at the end, you have a clear outcome. And you know if you’re successful, because the homicide rates decrease.”
The Research: The only evaluation was done in part by Slutkin and highlights some correlations that may or may not have anything to do with the program. In a paper prepared for this fall’s American Public Health Association annual meeting, Slutkin and colleagues from the University of Illinois at Chicago’s School of Public Health say that Chicago’s West Garfield Park neighborhood had a 67 percent drop in shootings in 2000, which correlates with the beginning of the CeaseFire program there. Seven communities using the CeaseFire approach reported an average 48 percent reduction in shootings last year, much better than the record of other Chicago neighborhoods.
According to the program’s anniversary report, combined average reductions in shootings in five principal CeaseFire zones from the year the program was implemented (as far back as 2000) through 2004 is 71 percent. Shootings in all of Chicago went down 44.5 percent from 1999 to 2004.
The National Institute of Justice has commissioned an independent evaluation this fall by Wesley Skogan, a professor of political science and faculty fellow at the Institute for Policy Research at Northwestern University in Evanston, Ill.
