Yakima, Wash.—For youth worker Darrel Armstead, the challenge of implementing one of the nation’s most expensive and highly touted youth programs was epitomized by the rapper from Yakima.
After repeated nudging got the teen to drag himself to a few sessions of Armstead’s Quantum Opportunity Program (QOP), the boy’s family moved 100 miles away, near Seattle. He didn’t go to school there, didn’t want a job, didn’t want help from QOP. He wanted to cut a rap CD.
That put Armstead in a quandary: Should he try to work with the boy long distance – thus adhering to the fundamental “once in QOP, always in QOP” philosophy – or invest his precious time in youths who showed interest in the program?
After repeated phone conversations with the family, Armstead stopped.
That wasn’t in the plan. But the experiences of agencies trying to follow QOP’s plan offer sobering lessons for all youth agencies about the real-world challenges of taking to scale practices that research says should work. And in a field where practitioners recite “there are no cookie cutter solutions” as a mantra, the QOP experience shows how the routine practice of adapting program models to fit local conditions can undercut the model.
Ever since QOP’s strong impact on grades and behavior grabbed the attention of the youth work field in the early 1990s, foundations and the federal government have spent millions trying to replicate it. While the program has had impressive effects on many kids, the latest research suggests that QOP is so expensive and demanding that agencies usually fail to follow the model, producing diluted impacts at a cost that can approach $10,000 a year per youth.
“We found it incredibly valuable,” says Lisa Moore Willis, vice president of programs for Bridges, the nonprofit which ran the QOP demonstration in Memphis, Tenn. Nevertheless, “We’re just not able to sustain it. … It is very difficult to replicate.”
Such talk makes Robert Taggert mad. QOP’s co-creator says the problem with a recent five-year demonstration run by the U.S. Department of Labor (DOL) and the Ford Foundation is that the replicating agencies, such as Bridges, didn’t stick to the model – because the DOL didn’t force them to. “No one was there kicking their butts,” says Taggert, director of DOL’s youth programs during the Carter administration. The DOL “had no control over any of the sites.”
Taggert’s prescription to insure model fidelity might sound like the opposite of what agencies generally want: funders riding herd on operational details throughout the project, even dictating how grantees use staff. “It’s not fun to be the ass-kicker,” says Taggert. But “our biggest weakness in youth programs is that they’re crappily managed and crappily replicated.”
Willis understands Taggert’s passion about sticking to the program design, but says, “There’s this whole thing called reality.”
Thus QOP takes the youth field back to one of its most persistent and hard-to-answer questions: How does promising research apply to on-the-ground youth work?
Breaking Rules
QOP is like some kids you know: full of potential but difficult to manage, because it’s different.
Taggert, president of the Remediation and Training Institute (RTI) in Alexandria, Va., developed QOP in 1988 with Gordon Berlin, then of the Ford Foundation, and Benjamin Lattimore of Opportunities Industrialization Centers of America (known as OIC), based in Philadelphia. They built it in part from the Comprehensive Competencies Program, a computer-assisted math and reading program that Taggert had developed earlier with Ford grants. QOP added mentoring, community service and youth development.
The objectives were ambitious: Increase the chances that youths would stay in school and go on to postsecondary education or job training, increase academic achievement and reduce risky behavior.
The threesome designed a program with several key elements that were unique, at least in combination:
• Intense Participation: Youth were to spend 750 hours a year on program activities.
• Intense Relationships: Youth stayed in QOP for four years. (A less intense follow-up year has been added to help after high school.) Each case manager would be responsible for 15 to 25 youths and be available around the clock.
• Intense Outreach: The programs enrolled youth from selected schools regardless of whether they wanted to participate. Once in QOP, they remained in QOP – meaning staff had to find and engage those who quit, moved or never participated.
• Financial incentives: Agencies paid the youth stipends (about $1.25 an hour in the most recent demonstration) for time spent on most program activities; deposited a matching amount in an accrual account, which the youths could claim after reaching an objective such as graduating from high school and enrolling in college; and paid bonuses for reaching milestones, such as getting certain grades.
QOP “breaks the rules that we live by in terms of youth intervention and programming,” says Mary Beth Bartholomew, who directed the QOP demonstration in Cleveland. It “makes people uncomfortable because it’s difficult, it’s long-term, it’s expensive.”
The Ford Foundation put up $1.3 million to test the idea in five cities from 1989 to 1993, with OIC affiliates running the programs. Each site had 25 kids and an equivalent control group.
An evaluation by Andrew Hahn at Brandeis University’s Center for Human Resources (now the Center for Youth and Communities) found some impressive results. (The Milwaukee site was dropped because it essentially didn’t implement QOP.) Of about 100 QOP youths at the four remaining sites, 63 percent graduated from high school, compared with 42 percent in the control group; 42 percent went to post-secondary education, compared with 16 percent in the control group; 23 percent dropped out of high school, compared with 50 percent in the control group; and 24 percent had children, compared with 38 percent in the control group.
Hahn calculated that when increased earnings and lower public expenditures on youths in trouble were factored in, QOP produced $3.68 in benefits for every dollar spent.
“The QOP impacts dwarf those of any program we have evaluated,” Taggert said at a briefing of the Washington-based American Youth Policy Forum (AYPF).
The program was widely hailed as model, landing on several best practice lists. Among those publishing glowing summaries of QOP in the 1990s: Del Elliott’s Center for the Study and Prevention of Violence, in Boulder, Colo., which chose it as one of 10 programs in its “Blueprints for Violence Prevention” series; the U.S. Office of Juvenile Justice and Delinquency Prevention, which paid to publish and disseminate a book, Quantum Opportunities Program, by Lattimore; the AYPF (“Promoting Success Youth Development in Urban Communities – Unprecedented Success for the Quantum Opportunities Program”); the U.S. Department of Health and Human Services (“Preventing Teen Pregnancies: Promoting Positive Strategies”); and Youth Today, with a page 1 story in 1995 saying QOP could be “the harbinger of a whole new approach to job training for disadvantaged youth.”
But while each of the sites showed positive results for key indicators (such as high school graduation), the most stunning results were in Philadelphia – headquarters of OIC, which spent more than any site ($15,000 per youth over four years, compared with an average of $10,000 for all the pilot sites) and whose staff showed more commitment than anyone to make QOP work.
Looking back at that pilot in its QOP evaluations this year, Mathematica contends that “few significant impacts appeared in the other four sites.”
For instance, 18 of the 25 Philadelphia QOP youths went to college. In San Antonio, there was no boost in high school graduation rates or post-secondary enrollment, according to Lattimore’s 1998 book.
That book noted “large differences in the ‘QOP effect’ among the four sites. Philadelphia stood apart.” In its “blueprints” summary, the center for violence prevention cautioned that “the program’s success is heavily dependent on successful replication.”
QOP’s creators acknowledged that a bigger demonstration was needed to see if the results could be replicated on a larger scale. Taggert called on his old agency, the DOL, to kick in $1 billion a year.
Calling Bill Gates
Well, it doesn’t hurt to ask. The DOL did devote about $5 million in Job Training Partnership Act (JTPA) money for a five-year QOP demonstration, in partnership with the Ford Foundation. Community-based agencies in five cities each got $200,000 a year from DOL, and were to find local matching funds. Ford gave OIC $4.1 million to run two more sites and provide technical assistance to all of them. Taggert’s RTI also provided technical assistance.
Each DOL site (Cleveland, Memphis, Houston, Forth Worth and Washington) served about 100 youths, while the two Ford sites (Philadelphia and Yakima) served 50 apiece. There were also control groups for each site. All youth were randomly assigned.
Taggert saw trouble. Months before the programs began in July 1995, he wrote to DOL Secretary Robert Reich that the demonstration would fail because the design differed significantly from the QOP pilot. He asked the department to appoint OIC as intermediary over the sites to reduce bureaucratic hassles for the agencies (like managing the stipends) and insure that they followed the model. “To paraphrase Lloyd Bentsen,” he wrote, “ ‘I knew the Quantum Opportunity Program, and this is not QOP.’”
Eight years later, Taggert’s fears were confirmed.
Although the programs ended by late 2001, a lot was at stake in the two Mathematica evaluations that came out this year. More than a dozen youth agencies now run QOPs or versions of them. The Milton S. Eisenhower Foundation has started six new QOPs in about the past year. (See “QOP Lite” sidebar.)
They might learn a lot from Mathematica’s impact and implementation findings. The researchers said the seven programs boosted high school completion rates and attendance in post-secondary education or training. The programs did not improve grades or reduce risky behaviors, or have significant positive, overall impacts on anything else. (See “Impacts” sidebar.)
Most striking is the impact of the programs individually. The only sites to show broad positive results, Mathematica said, were those funded by Ford and run by OIC. And between those two, Philadelphia (operated this time by the national OIC) again stood out as largely responsible for the strong findings.
Except for Philadelphia, Mathematica said, community based organizations “found implementing QOP difficult, primarily because QOP was substantially more comprehensive, intensive and complex than their traditional programs.”
Even here at the Yakima site, run by the local OIC affiliate, then-JTPA coordinator Sandra Pieti says, “The difficulty was going from short-term training – work experience, a few credits, GED – to five years of TLC, big time.”
Once again, OIC in Philadelphia understood QOP better than the other agencies and was more devoted to the program that it had helped to create. It stuck closest to the model and spent more: an average of $49,000 per youth for the five years, compared with an average of $25,000 for all the demonstration sites.
“Philadelphia’s success, more than anything, had to do with the fact that we couldn’t let it fail,” says Deborah Scott, who coordinated the Philadelphia program and is now national program manger for OIC. “It was not an option for us.”
Managers at the other sites say they didn’t fail, either, despite the evaluations. They saw troubled kids blossom. They helped kids improve in school, deal with crises and set firm courses for their futures.
“Enough of them still come back to us and say, ‘You know what? This program saved my life,’ ” Bartholomew says.
They believe the measured results were dragged down by several factors, such as including enrollees who didn’t participate.
Most sites did show significant results in certain areas. The Yakima youth, for instance, achieved big gains in math scores.
But with the only clear overall success (Philadelphia) costing $9,800 a year per youth, can anyone afford it? “It’d be tough to take this thing to scale in a city unless you had Bill Gates” paying for it, Bartholomew says. Even in Philadelphia, there is no stand-alone QOP now because there is no funding for it.
Taggert says the long-term savings are worth the cost, but someone must make sure the agencies implement the model. He says the DOL (which declined to comment on the studies) let the agencies go so far off course that the model was not accurately tested. “It’s interesting that some of these programs can be called QOP when they are doing their own thing,” says RTI Vice President Barbara Dunn.
Just about any youth-serving agency can take lessons from QOP’s struggles and success, which involve universal issues such as getting kids to keep coming as they age through high school, transporting youth to programs, establishing lasting mentor relationships and avoiding staff burnout.
Following are summaries of the major issues raised by QOP:
Never Give Up On a Kid
It sounds nice: “Once in QOP, always in QOP.” The idea is that programs should not only served motivated participants, but also make it a mission to find and cajole those who show no interest or drift away.
Dropped out of school? Got locked up? Told the youth worker to bug off? Didn’t matter. To leave QOP, you had to die. In Yakima, QOP staffers even kept working with a diabetic girl who slipped into a coma three times over the five years.
But following this credo often imposed an immense burden on staff, with questionable payoff.
Most of the youths had never heard of QOP until staffers came to their homes or schools and told them they’d been chosen. Most went along, but some weren’t interested. There were, however, no substitutions or dropouts; each youth would be factored into the evaluation of the agency’s performance.
“We dogged those kids” to participate, says Armstead, the QOP coordinator in Yakima. He admits that “without the requirement” to meet with and involve every youth on the list, “we wouldn’t have done it.” Thus QOP designers got part of what they wanted: the involvement of youths who otherwise wouldn’t have participated.
The question is, at what point does this laudable theory become a waste of precious time? Youth workers “reported wasting many hours contacting individuals who had never participated in any QOP activities,” Mathematica reports.
“You had kids who had already given up on school, who we couldn’t persuade to come in here,” Armstead says. “There are still a couple of kids we never met.”
In Philadelphia, Scott says, OIC “sent out bounty hunters,” paying some QOP youths $50 to get uninvolved kids to come for two weeks. But “there’s a point at which you do cut your losses” and just make an occasional call or mail a “thinking about you” note.
Hunting kids isn’t always efficient, but that wasn’t the point of the demonstration. The point was to see what happens when agencies dedicate themselves to bringing in hard-to-reach kids – rather than the routine practice of picking the “low-hanging fruit,” that is, kids who are motivated to participate.
“You’ve got this recalcitrant group that’s never been invited in [to programs] and is suspicious,” says Hahn, who evaluated the QOP pilot. “You’ve got to work and work and work on that.”
Causing even more trouble were those who moved – and as is typical of programs serving low-income youth, moving was routine. By the fourth year of the demonstration, the proportion of participants who had moved from the school at which their QOP program was originally based ranged from 14 to 57 percent. This set off a logistical and manpower nightmare.
“By the third year we had kids in over 17 high schools,” says Bartholomew in Cleveland. Some case managers struggled to figure out how to get their ever-dispersing group to the program site each day, or spent their days visiting kids at various schools, trying to deliver services to them separately.
Staffers even visited hangouts and places of employment to track down kids who moved without notice.
Transportation costs soared, as agencies gave the kids bus fare or used vans to shuttle them to the programs. “We spent a lot of money on gas and bus tickets,” Bartholomew says.
Even when youths moved out of state, the QOP model called for the agencies to keep providing services. Some tried:
Mathematica reports that one caseworker drove hundreds of miles to visit a youth. But most staffers just tried to keep in touch by mail and phone, if at all. They asked about report cards and encouraged the kids to get active in local youth programs.
“The model says we follow the kids” wherever they go, Bartholomew says. But “there was too much work in our own
backyard.”
Handing Out Cash
Should an agency pay kids to let adults help them? How about paying the adults extra for doing a good job? QOP said “yes.”
The hourly stipends “induced newly enrolled youth to attend program activities,” Mathematica found, giving youth workers a chance to develop relationships that might keep the teens coming in.
Although the impact of the modest stipends generally faded with time – “It’s not as much as they’d make on a corner,” Bartholomew notes – the money encouraged some youths, especially with matching funds available at the end of the line.
“That kind of pushed me to get more hours and more money,” says Alondra Garivay, who was in the Yakima program.
The matching-fund accounts were intended to help the youths with college or vocational school expenses, and teach them about saving and investing. By the end of the demonstration the accounts ranged from a few hundred dollars to nearly $10,000, averaging less than $2,000.
But with the push to get kids to attend program activities, Bartholomew says, sometimes “it seemed like we were almost giving kids stipends for breathing next to us.” She recalls a day in the computer lab when a youth put a ream of paper in the printer and half-jokingly said, “How much do I get paid for that?”
The programs also offered bonuses to staff when their youths met objectives, such as a certain number of hours of participation. But providing bonuses, along with letting QOP case managers work on flex time, sometimes created resentment among other staffers. “It was a sensitive issue here,” says Willis in Memphis, and it remains so as her agency continues with its versions of QOP.
Those versions do not include matching funds accounts, and some don’t include stipends. Willis says those incentives add a “huge” cost to the program, which the agency cannot afford.
Time
QOP’s incredible ambition and challenge is best illustrated by the stipulation to provide 750 hours of activities annually for each youth, evenly divided among education, community service and youth development. That’s more than 14 hours a week, year-round.
Only two agencies provided that many hours, Mathematica said, and for the vast majority of youth, actual participation fell far short of that goal. For all the sites combined, youths averaged 174 hours a year through the first four years – 23 percent of the amount of time stipulated by the design. Two patterns stand out:
• Program participation dropped drastically as the youths aged, as is typical for youth programs. The average number of hours fell from 247 the first year to 89 in the fourth. The number of youth spending no time on QOP rose from 1 percent the first year to 36 percent in the fourth.
• The Ford-funded sites averaged more than twice as many hours (294) as the DOL-funded sites.
The youths who participated the least (less than 100 hours total for all the years) said they were uninterested in the activities or were committed to other after-school activities, such as sports, jobs or caring for family members.
Community Service
Nothing got less attention than this. All seven sites gave community services less emphasis than the education and developmental components, and none of the sites offered 250 hours a year, as the plan stipulated.
The most common reason was lack of interest among kids and a feeling among youth workers that the kids needed other services more. “Most sites decided to reallocate their resources away from community service to mentoring, case management and educational activities,” Mathematica said.
“With the population we’re working with, we really work hard to just make sure they get to school,” says Willis in Memphis.
Armstead saw that as kids aged, they had less interest in or time for community service. Once again, jobs and family obligations took precedence. Several agency managers noted that for many kids, “community service” is what someone does to work off a court sentence.
Education
In this day and age, the requirement seems ridiculously easy: Provide computers for the kids. “My God, it seems like such a simple thing, doesn’t it?” Bartholomew says. “It’s what gave me gray hairs the first three years of the program.”
Staff-assisted computer instruction was one segment of QOP’s education component that no site completely fulfilled. Some sites did do well in other educational areas, and many kids showed academic progress. The youth workers often called parents when a youth was absent from school and visited the homes of enrollees who missed school for several days. “These efforts were not sustainable in the long run,” Mathematica reported, because youths eventually spread out to so many schools or repeatedly “reneged on promises” to attend school.
“Few sites” found time to conduct regular tests to assess academic performance, Mathematica found, and only two developed education plans for each youth. The plans were generally shallow, listing generic activities such as “after-school tutoring, teacher conferences.”
Perhaps most instructive was the difficulty with computer labs and tutoring.
Computer Labs
When the demonstration program started, only two of the seven sites had functioning computer labs – Philadelphia and Yakima, where Ford paid for the labs. One windfall: Here in the middle of economically depressed Yakima sits a room at OIC with an impressive collection of computers, which the agency uses for other youth programs now that QOP has ended.
The five DOL-funded sites had no money set aside to create computer labs and scrambled to find computers and provide instruction. One had a state of the art facility by the start of the second year, Mathematica said, while the other four “spent a great deal of staff time arranging for access to computer facilities” run by other organizations, such as schools, with mixed success.
Consider Cleveland. A local high school had a spanking new lab. QOP co-creator Lattimore saw it and “was just blown away,” Bartholomew says. But her agency couldn’t use it. “That was mostly school politics,” she says.
It took three years to get access, three days a week. “By then, our kids are 17, they’re working after school,” Bartholomew says. “We kind of lost that opportunity, when they were younger and not working, to have that captive after-school audience.”
The lab “served just a very few kids.”
Memphis never found a lab. With the QOP youths spreading out among different schools, Willis realized that any place she found would be too inconvenient for most of the kids. “Even if you had the lab, who would come?” she says.
Ironically, the computers could draw kids to the program, thus increasing their hours of participation. In Yakima, Garivay says, “The reason I went” to QOP so often “was because we had access to the Internet” there.
Tutors
Can you find AB if trapezoid ABCD is isosceles and AB = 5x-3 when CD = 3x + 3? Neither could most QOP workers.
“The staff were great mentors,” says Bartholomew in Cleveland. But “of the five staff we had, we probably only had one who could really tutor math. A couple of them got better.”
In Houston, the youth workers “occasionally helped with homework and encouraged students to take advantage of free tutoring services provided by the school,” Mathematica found. In Washington, “case managers provided homework help when students requested it.”
At least three sites – Philadelphia, Memphis and Cleveland – addressed the problem by hiring outside tutors.
Youth Development
It’s no surprise that overall, the agencies were best at what they already did most: youth development. Mathematica found that “all sites successfully implemented developmental activities,” which included recreation, culture, life skills and employment-readiness training.
The most popular and memorable activities were often things that middle class youth might take for granted. In Yakima and Philadelphia, for instance, the kids earned their way onto out-of-town trips to visit colleges. The Yakima OIC chartered a bus to take youths to Southern California, about 800 miles away.
“A lot of our kids had never been more than 45 minutes from Yakima,” Armstead says.
The California visit had a big impact on Garivay, who had assumed until then that she wouldn’t go to college, especially after
having had a baby at 15. “It opened up doors,” she says. “When you went to U.C.L.A., that was just amazing. My [QOP] counselor told me, “ ‘You could go here.’ ”
Garivay studies nursing at Washington State University, having decided to stay close to home.
The recreational activities increased program participation and built relationships between the youth and youth workers, the evaluations found. But while certain activities, such as pregnancy prevention discussions and anger management classes, served specific, measurable objectives, the youth workers appreciated that QOP freed them to pursue fundamental youth development with no outcome strings attached.
“We could take them to a movie with no occupational skill attainment,” says Pieti in Yakima. “The biggest events that made a difference in the kids’ lives was being able to experience things they’d never experienced.”
That included golfing (“If the kids didn’t have money, we’d take up a collection” among the staff, Armstead says) and eating at tony restaurants. The latter made an impression on Garivay; she recalls being struck by place settings with three forks. The kids didn’t just eat; they got an education in formal dining, covering such matters as what each fork is for and the proper placement of water glasses and coffee cups.
Intensity
This is where QOP stood out. Just as the design intended, QOP created significantly more intense services and youth/adult relationships than the average youth program. “Many of the school administrators, faculty and CBO managers … stated that QOP was the most intensive program they had ever encountered,” Mathematica said.
One principal told evaluators that QOP was the most successful youth program he’d seen, saying, “They pick youth up at home. They do whatever is needed, inside or outside of school. They provide a stable and caring environment; they provide consistency.”
Staffers reported forming “close mentoring relationships” with 40 to 60 percent of the youth assigned to them. All the sites successfully implemented mentoring, Mathematica found. Most youths had the same case managers for all five years.
But the expectation for staff to be available on nights and weekends created a workload that was heavy even by youth work standards. “It isn’t a job, it’s a life,” Armstead says.
Staffers were told “they will not be working normal lives, and they will be giving up a lot of their free time,” says Scott in Philadelphia.
Staffers at several sites routinely worked off the clock. Some worked in teams, so they could share responsibilities during off hours. Some worked on flex time.
“Boy, that makes people nervous,” Bartholomew says of flex time. “You have to trust people” to do their jobs under less supervision, and hear gripes from resentful staff: “No one else does that. We’ve never done that before. What do you mean they’re not coming in at 8 and leaving at 4:30? I was always fighting political battles.”
Most of the off-hour calls from kids were handled over the phone, Mathematica said. But case managers routinely got calls from youths asking to borrow money for family emergencies or to bail them out of jail. “You’ve got somebody who calls and says, ‘I’m being kicked out’ ” of home, Willis says, and soon thereafter, “they’re sleeping on my couch.”
Mathematica found that staff turnover was not a significant problem. Nevertheless, Scott in Philadelphia recalls, “The burnout factor was real, absolutely real.”
Resources
“The Quantum Opportunity Program Demonstration: Implementation and
Short-Term Impacts,” December 2002
“The Quantum Opportunity Program Demonstration: Implementation Findings,” December 2002
Mathematica Policy Research
P.O. Box 2393
Princeton, NJ 08543-2393
(609) 799-3535
www.mathematica-mpr.com
Darrel Armstead
Yakima Valley OIC
815 Fruitvale Blvd.
Yakima, WA 98902
(509) 454-4617
Mary Beth Bartholomew
QOP Director
Youth Opportunities Unlimited
2000 E. 9th St.
Cleveland, OH 44115
(216) 566-5445
Deborah Scott
National Program Manager
Opportunities Industrialization Centers of America
1415 N. Broad St.
Philadelphia, PA 19122
(215) 236-4500
Robert Taggert, President
Remediation and Training Institute
604 Wolfe St.
Alexandria, VA 22314
(703) 683-7795
Lisa Moore Willis
Vice President of Programs
Bridges
314 S. Goodlett St.
Memphis, TN 38124
(901) 452-5600
QOP’s Impact – And Some Caveats
According to recent evaluations of the QOP demonstration programs by Mathematica Policy Research, when taken together the programs:
• Increased the likelihood of high school graduation by 7percentage points. The impact of the QOP pilot that began in 1989 was 19 percentage points.
• Increased the likelihood of youths going on to postsecondary education or training by about 6 percentage points, when the measurement included college, vocational or technical school attendance, apprenticeship enrollment or military enlistment.
• Did not improve grades.
• Did not reduce risky behavior.
The “statistically significant” short-term impacts on youth at specific sites, as defined by Mathematica:
• Yakima: Higher math scores, less likely to have a baby; less likely to attend high school, postsecondary education or training, or to work.
• Philadelphia: More likely to attend postsecondary education or training, and to work; less likely to have a baby; more binge drinking.
• Washington: Lower math scores.
• Cleveland: More likely to complete high school and attend college; less binge drinking.
• Houston: Lower grades.
• Fort Worth: No significant impacts.
• Memphis: More binge drinking.
Several factors probably helped to keep down the impacts:
• The QOP pilot that began in 1989 served youth in families on public assistance, but their academic achievement varied. The recent demonstration served youth who had low grades and were entering high schools with dropout rates of at least 40 percent. Thus the youth in the demonstration may have been more challenging to serve.
• The youths were also more challenging to serve than are the youth in many other programs, because enrollees are randomly selected. They do not seek out QOP, nor do they have to complete it to satisfy an academic or court requirement.
• Including enrollees who participated very little or not at all probably watered down the impacts. The results might have been stronger if only active youth had been included in the calculations.
• Sixteen percent of the research sample was still attending high school at the time of the evaluation, so follow-up data might change the results. Deborah Scott, who coordinated the QOP in Philadelphia, says some measurable impacts won’t be clear for years, as the youths apply their QOP experiences to their lives as young adults.
• QOP program managers believe, as Lisa Moore Willis from the Memphis program says, that “a lot of the benefits to these kids aren’t found in a Mathematica report. … There were immeasurable impacts for these young people.”
QOP Lite: Less Money, Works Great?
The Quantum Opportunities Program (QOP) holds so much promise that youth-serving agencies continue to take on the challenge to replicate its demanding model. Most are trying to do it by dropping or trimming some of the more intense and expensive elements of the design.
Their efforts may show the extent to which an agency can use portions of a successful model and still get good results.
Everyone agrees on step one: Find a big, outside funder. QOP is too expensive for most nonprofits to take on within their standard budgets.
Among the efforts:
• Probably no one has more interest in immediately learning lessons from the recent QOP evaluations than Johnny Gage: He oversees six QOPs started within about the past year by the Milton S. Eisenhower Foundation, based in Washington, D.C.
Unlike other QOPs, these are at public housing developments. That’s because Eisenhower saw that in its Safe Haven projects in public housing, older teens “were starting to sort of stray,” says Gage. “There just wasn’t a lot that kept them involved.”
The programs – in Washington; Herndon, Va.; Dover, N.H.; Portland, Ore.; and at two sites in Columbia, S.C. – will each get about $90,000 a year, Gage says. Five serve 20 youths, and one serves 30. All are run by nonprofits except for Dover, which is run by the city housing authority.
Eisenhower is trying to replicate QOP on $4,500 per youth annually, slightly less than the average spent by the seven sites in the recent demonstration run by the U.S. Department of Labor (DOL) and the Ford Foundation, and less than half of what was spent by the one that was most clearly successful.
It’s not enough. Gage says the sites are expected to find local funding as well, and to develop a system of regular volunteer youth workers.
Gage is using the recent QOP evaluations to talk with his grantees about the areas of danger. For instance, he’ll pay close attention to the number of activity hours planned for the youth, and set plans for corrective action if the hours fall off pace.
• In Memphis, Bridges liked QOP so much that after its DOL demonstration program ended in 2001, it pieced together funding from disparate sources – including the city government, the DOL’s Youth Opportunity (YO) grants and a local corporation – to set up nine QOPs, each serving 25 youth.
But those programs don’t follow the QOP model to the letter. Depending on the funding sources, says Bridges Vice President Lisa Moore Willis, some of the programs provide stipends to the youths and some don’t. None features matching-fund accrual accounts. And, Willis says, “we don’t have money to hire a tutor.”
That has helped to keep costs down to an average of $3,200 per student, Willis says.
After the funding periods end for those programs, no new ones are on the horizon, she says, because no one has put up the money.
• Another QOP demonstration site, Youth Opportunities Unlimited, in Cleveland, subcontracts to run a version of QOP for the city schools, also with federal YO funds. That program, serving 260 youth, has kept most of the QOP elements, says QOP Director Mary Beth Bartholomew. But it does not require 750 hours of activities each year, and youths may move in and out of the program. The budget: about $600,000 a year, or about $2,300 per youth.
• The York, Pa., YWCA runs four QOP groups (of about 30 youth each, teamed by grade), and reports no trouble planning 750 hours of activities a year or adhering to any of the other QOP elements. One big difference: Youths volunteer for the program after being told they’re eligible because they’re on public assistance. Most of the youth come from one high school to the YWCA.
The program is run on a four-year, $1.2 million grant from the Pennsylvania Commission on Crime and Delinquency, using funds from the U.S. Office of Justice Programs.
• About 80 miles away, in Philadelphia, home of the most successful QOP to date, there’s no money to run a full QOP right now. “What we’ve done is tweak the QOP model,” says Deborah Scott, national program manager for OIC of America. The agency has blended portions of QOP into school-based programs for older youth, mostly ages 18 to 21. Not all of those programs include student stipends, and none requires 750 hours a year of program activities.
All of which raises the question: Can portions of QOP be grafted onto other programs and still be effective? QOP co-creator Robert Taggert says agencies have to stick to the model or their results will plummet, and the recent Mathematica reports back him up. The further an agency drifted from the model, the weaker its overall results.
But it appears that few agencies can successfully run a QOP as it now stands. The two most common suggestions for change, from the Mathematica reports and agency operators: Reduce the number of hours each year (from 750 to 450, Mathematica says), and don’t make the agencies keep trying to haul in kids who definitely are not interested or who move far away.
Taggert is open to some adjustments, like dropping a completely uninvolved youth after a certain time. “The most important thing,” he says, “is that you have a four-year commitment to the individual. The principle of making that effort is very important.”
