If you make cars or deodorant, it’s relatively easy to see the effect of your advertisements: Look at sales. But if you produce a message for kids – to get more physically active, for instance – it’s harder to tell if you’re getting through.
The federal government’s VERB campaign figured out how to get some answers, and how to make adjustments in response.
Launched by the U.S. Centers for Disease Control and Prevention (CDC) in 2001, the “VERB: It’s What You Do” campaign encouraged 9- to 13-year-olds to “find their VERB” – an action they enjoy, such as jumping rope, running, riding bikes or playing an organized team sport – and to do it every day. The campaign did this through paid television, radio and print advertising, Internet activities, and school and community promotions.
CDC and researchers from Westat, the Rockville-based consulting firm, published their first-year evaluation findings for VERB in the August issue of Pediatric. What they found exceeded everyone’s expectations.
After one year of programming, 74 percent of the more than 3,000 youth in VERB’s national longitudinal study sample said they were aware of the campaign. The campaign’s goal was 50 percent.
Among other findings: Nine in 10 children who were aware of the campaign demonstrated an understanding of VERB’s message: Physical activity is fun, cool and socially appealing. Nine- and 10-year olds who were familiar with VERB engaged in more free-time physical activity than those who weren’t. The more they were aware of VERB, the higher their activity levels.
But despite VERB’s timeliness, rigorous evaluation and stellar outcomes, the U.S. Department of Health and Human Services’ fiscal 2006 budget proposal does not include funds to continue the program past the five-year demonstration, which ends this year. (See story, page 29.)
Although the campaign’s funding ($125 million in the first year) and national scope place it in the Cadillac class of evaluated programs, VERB offers lessons for all youth-serving organizations about the value of early logistical planning and ongoing formative and process evaluation.
“It is important to think like an evaluator from the beginning, whether your budget is 100 bucks or $125 million,” said Faye L. Wong, director of the campaign. “If we didn’t, we would have wasted a whole lot of money.”
Groundwork
From the beginning, research and evaluation were recognized as campaign priorities and were built into the “logic model” of the VERB campaign. The model is a visual diagram of the sequence and relationships of program inputs, activities, impacts and outcomes.
“A logic model allows you to really focus on what outcomes you’re trying to achieve and how to get there,” Wong said.
“A lot of times people jump right into intervention without that pre-thinking, and that’s where I think programs fail, or don’t quite get to where they’re trying to go.”
VERB’s model used input from campaign partners to create marketing, promotional, public relations and community outreach activities.
The impact of those activities was measured by a variety of short-, mid- and long-range outcomes. Short- and mid-range outcomes included the number of adults and “tweens” (9- to 13-year-olds) exposed to the campaign and the positive “buzz” generated about physical activity. Long-term outcomes included tweens engaged in physical activity and reductions in chronic diseases and unhealthy behaviors.
VERB’s pre-thinking didn’t end with the design of its model. Research conducted at the beginning of each phase of the campaign explored the influences of cultural, ethnic and economic factors on tweens, including motivators and barriers to physical activity. A randomized baseline survey administered to 3,120 sets of children and their parents captured a pre-campaign snapshot of tweens’ participation in, and attitudes about, physical activity.
Discipline
A key to success is specifying what the intervention is going to produce and sticking to the plan, said Lance Potter, project director of the Youth Media Campaign Evaluation at Westat, the CDC’s evaluation partner for VERB.
Lack of focus “is part of the problem with interventions being effective, and it’s part of the problem of demonstrating that they’re effective,” Potter said. Targeting a program to a fairly narrow focus allows a program to target its evaluation, “which always yields better results.”
Potter said that initially, CDC hoped to target not only physical activity, but pro-social behaviors. Doing so would have broadened the target audience too much, he said, and would have blurred the advertising message by including ads on such things as the value of piano lessons, family time and volunteering.
“The advertisers … realized, ‘We can’t create anything coherent that will accomplish all of those things,’ ” Potter said. “And [Westat], as the evaluators, said, ‘Yeah, and we can’t measure it.’ So … everyone had to take a long time to struggle with narrowing our vision to something that people thought they could change.”
Ongoing Evaluation
The campaign teamed with some of the country’s leading advertising firms, including firms that specialize in ethnic audiences, to deliver sleek, sophisticated ads on the Nickelodeon, Disney, BET, Telemundo and other networks. Once the campaign was in full swing, media buyers estimated that 85 percent of 9- to 13-year-olds had an opportunity to see a VERB ad 8.8 times each month. (That doesn’t mean they actually saw them.)
Throughout the campaign, researchers analyzed the conditions under which the efforts were taking place, and whether those efforts were being conducted as planned. This “process evaluation” involved documenting the channels, reach and frequency of communications and measuring the reactions of the target audience. The latter was done through randomly dialed monthly telephone surveys of families with tweens.
Evaluators say they were constantly engaged in “formative research,” collecting the thoughts, feelings and perceptions of VERB’s target audience. They observed and interviewed tweens at VERB-sponsored community events, and conducted follow-up interviews to see if short-term outcome goals were being met.
Testing advertising concepts with youth and parents helped advertisers select communication strategies that were acceptable, understandable, culturally appropriate and motivating, Wong said. Testing messages after an ad’s release helped to identify messages the target audience found confusing or controversial.
The researchers’ ability to scan all that data for trends and patterns helped CDC improve VERB’s message delivery. For instance, researchers noted that at the events in the first year, “lots and lots of 9- and 10-year-olds were showing up; very few 12- and 13-year-olds were showing up,” Potter said.
Sure enough, he said, “When we got the outcome data [from the longitudinal sample] at the end of 2003, … the subgroup that showed effects consistently was the 9- and 10- year- olds.”
At the same time, feedback collected in focus groups and telephone surveys indicated that VERB’s focus on the fun and sociability of physical activity was less appealing to boys than it was to girls.
“The way we applied that data is that boys care more about mastery of skills,” Wong said. So in the second year, “we introduced how-to videos on our website that showed boys how to hit a hockey puck, how to throw a football.” The focus on mastery also “aged up” the campaign for older tweens.
“But the important thing is that this is a for-kids, by-kids campaign. We ask, and we listen to the kids,” Wong said.
According to Potter and Wong, that strategy led to second-year results that show even more widespread awareness and understanding of the campaign and even greater positive effects on the levels of physical activity for ethnic, age and gender subgroups of tweens. Those results are being reviewed for publication.
It’s unclear whether, by that time, VERB will still exist or, if so, in what form.
