The Bill and Melinda Gates Foundation launched the Community Access to Technology (CAT) program in 1999 to help provide local communities in the state of Washington with greater accessibility to digital technology through after-school programs run by nonprofit organizations. Over the years, the program was renamed the Community Connect Network and came to focus more narrowly on at-risk youth, people with disabilities, homeless people, immigrant populations, Native Americans and people in rural communities.
In 2003, MGS Consulting began a multi-year evaluation of the program’s efficacy, sustainability and outcomes for priority populations. During year two of the study (2004), MGS began noticing demographic and outcome trends in programs serving at-risk youth.
Intrigued, the researchers decided to use the Search Institute’s newly released Developmental Assets Profile (DAP) assessment tool to evaluate youths in CAT programs during year three of the study (2005). Development assets are experiences and qualities that research has shown to be essential to healthy psychological and social development in adolescents. The DAP survey enumerates the developmental assets found in the lives of youth in five areas – personal, social, family, school and community. The quantitative results of the DAP place youth in one of four asset categories: “low,” “fair,” “good” or “excellent.” (Learn more about the Search Institute’s 40 Developmental Assets here: www.search-institute.org/developmental-assets.)
“The key findings [of the year three study] were centered on the Search Institute’s DAP scores,” said Maureen MacCarthy, MGS’ principal consultant for the evaluation. “It wasn’t just that the kids increased in assets, but how much shifting out of categories the kids experienced.”
“The biggest gains were from the kids at the lower end, the kids who started out [in the]‘poor’ [category] and did a single jump or double jump in asset categories,” she said.
Methodology
Thirty-nine sites in Washington (representing 10 organizations), participated in the year three evaluation. All of the sites were serving youth between the ages of 10 and 21. (Not all of the sites were included in the final sample.) The sites included both “high-touch” sites – defined by the researchers as sites having a high level of structure, rules and learning requirements – and “low-touch” sites – defined as sites that operated more informally, often on a drop-in basis, and with few or no requirements for participation.
MGS researchers or program staff members administered youth surveys on three occasions between May 2005 and June 2006.
Part one of the youth survey collected data on the youths’ own perceptions of their confidence levels, grades and future employability. Part two consisted of the online version of the DAP – a series of 40 statements such as “I stand up for what I believe in,” “I feel in control of my life and future” and “I do my homework” – to which the youths responded on a four-point scale ranging from “Not at All or Rarely” to “Extremely or Almost Always.”
In addition, MGS researchers conducted face-to-face interviews with 32 key program managers, led a small group discussion and administered two written surveys to program staff members. In each staff setting, the researchers asked about the characteristics of a high-quality technology program, staff members’ perceptions of the youths’ academic and job readiness outcomes, and which social supports staff members saw as necessary components of a successful technology program.
What They Were Looking For
The researchers looked at three measures tied scientifically to academic, employment and social success:
• DAP scores – a measure of positive adolescent experiences and qualities essential to success in adulthood, including feelings of empowerment, constructive use of time, commitment to learning, social competency and positive identity.
• Technical fluency – defined by the National Research Council as a youth’s ability to understand, explain and discuss technical tools and concepts.
• Outlook for the future – a measure of youths’ hopes and aspirations for themselves.
In looking at what factors successful CAT programs shared, the researchers initially expected to divide their findings between two well-defined groups.
“When we first started out, we said, ‘We’re going to have a high structure group and a drop-in group,’ ” MacCarthy said. “Clearly these were two different kinds of animals.”
But the researchers soon found that something else was going on.
“At sites that didn’t have a policy, structure and curriculum, the kids were advancing and doing challenging work. So it became not so much about whether there was a curriculum, but what the experience of the children was, if there were behavioral outcomes and expectations, and whether there was an investment not in just the technology, but in how people behave,” MacCarthy said.
What They Found
By the end of the evaluation period, nearly two-thirds (62 percent) of CAT youth who scored in the “low” DAP category had moved into the “fair” or “good” categories, and one-third who scored in the “fair” category had moved into the “good” or “excellent” categories.
According to earlier Search Institute research, those shifts significantly reduced participants’ likelihood of engaging in problem behaviors and increased their likelihood of engaging in positive behaviors, MacCarthy explained.
For example, researchers calculated that CAT participants who moved from the “low” to “good” DAP category lowered their risk of problem alcohol use from 45 percent to 11 percent, their risk of engaging in violence from 62 percent to 18 percent, and their risk of illicit drug use from 38 percent to 6 percent. Likewise, those youth who double-jumped from “low” to “good” increased their potential for succeeding in school from 9 percent to 34 percent, their potential for exhibiting leadership from 48 percent to 78 percent and their potential for maintaining good health from 27 percent to 69 percent.
In addition, technical fluency increased with time spent in a CAT program, with high-touch program participants, males and teens exhibiting more fluency than low-touch participants, females and pre-teens. Fluency scores seemed to peak between one and three years in the program, indicating a need to refresh and extend program content to keep long-time enrolled youth engaged, MacCarthy said.
Sites that showed the best overall outcomes for youth included both high- and low-touch locations, a finding that MacCarthy said may seem contrary to expectations, but that simply reflects the programs’ “different values.”
“The kids in the low-touch programs [which generally focused on using computers for homework help] reported that what they were doing was actually helping them immediately in school (67 percent) – and they reported that more than the kids in the high-touch programs (62 percent),” MacCarthy said. More youth in the high-touch programs (70 percent) than in the low-touch programs (60 percent) reported that they believed their CAT participation would help them go to college.
Despite programmatic differences, the most successful programs shared several traits.
“They either had really high structure – significant behavior consequences, or attendance requirements, or a curriculum, or an admittance test – or they had adults who were themselves consistent and caring and who … imposed that same standard of behavior and conduct” as did those with a higher level of structure, MacCarthy said.
Successful programs also had high levels of parent participation, safe facility space that “felt good” to the youths, and adults who taught “soft skills” such as teamwork, cooperation and professional presentation.
