Despite all the political rhetoric about the importance of a good education, you don’t have to look far to find evidence of the poor results achieved by U.S. students in comparison to their international peers.
For instance, the 2009 Programme for International Student Assessment (PISA), a program of the Organisation for Economic Co-operation and Development (OECD), found that American 15-year-olds ranked 25th out of 34 countries in math achievement, and that they fell further behind their peers in other countries as they progressed through the school. Lest the 15-year-olds bear too much of the blame, the 2012 OECD Education at a Glance report found the United States to be a country lagging behind many developed countries on everything from enrollment in preschool to growth in higher education enrollment.
Not everyone thinks international comparisons are important, of course, but let’s assume for a minute they are telling us something meaningful about the educational achievement of U.S. students. What are the best ways to invest our dollars to bring about improvement in student outcomes?
Although everyone seems to have an opinion on this matter, and a collection of anecdotes to support their point of view, such opinions aren’t really useful if we are serious about answering this question. After all, if you were ill, you’d probably prefer to receive medical treatment backed up by scientific research, rather than relying on whatever your relatives or neighbors had to say on the subject.
To make this question more manageable, let’s look at one type of intervention often posited to improve student results: after-school programs. (See today’s story by our New York Bureau and reporting last week from Baltimore by our correspondent Kaukab Jhumra Smith.) Because there is such variety among the types of programs and the students they serve, it’s difficult to process all the information available — and if you want to find evidence, based on a single example, that afterschool programs are or are not effective in achieving their goals, you can probably find it.
A more useful approach in examining the question can be found through a technique called meta-analysis. The basic idea behind meta-analysis is to combine the results from a number of related studies in order to find patterns among them, and perhaps also to reach a conclusion about one or more broad questions of practical interest — in our case, a question like “do after-school programs work?”
This may be clearer if we look at an example. One such study, which should be available through academic and research libraries, is “A Meta-Analysis of After-School Programs That Seek to Promote Personal and Social Skills in Children and Adolescents,” by Joseph A. Durlak, Robert P. Weissberg and Molly Pachan, and published in the American Journal of Community Psychology in 2010. It offers a good illustration of the steps of a meta-analysis, including defining what kinds of studies will be included in the study, conducting literature searches to identify relevant studies, and calculating an index of effect for each study — a single number summarizing the results of the study.
They found 68 relevant studies that included sufficient information to be included in the meta-analysis. About half of the programs studied (46 percent) served elementary school children, more than one-third (37 percent) junior high school students, 9 percent high school students and the remainder did not specify the age of the participants. The studies were conducted between 1979 and 2008, with two-thirds (67.6 percent) conducted after 2000.
Although not all programs studied had positive effects, overall they found a significant positive effect for after-school programs, i.e., a positive effect unlikely to have occurred by chance. They found that, overall, after-school programs had significant positive effects on six outcomes (child self-perceptions, school bonding, positive social behaviors, problem behaviors, achievement test scores and school grades), but did not have significant effects on drug use or school attendance. However, programs following the SAFE paradigm (sequenced, active, focused and explicit) for skill training found significant positive results for all eight outcomes, suggesting that following this paradigm is associated with more effective after-school programs.
To help rule out alternative explanations (i.e., that something other than the after-school programs produced the observed results), Durlak and colleagues also conducted a series of analyses looking at the effects of many variables, including program level (elementary, junior high or high school), gender of participants and presence of an academic component in the program. Overall, these variables were found to have no more than a chance relationship with the outcomes, lending strength to the assertion that the observed differences were due to the after-school programs.
How is this type of study useful to someone interested in creating an after-school program, or who must make the decision whether or not a program should receive funding? First of all, it provides good evidence that after-school programs can produce desired outcomes. Second, it helps identify some characteristics that are associated with the most successful programs, as well as some that don’t seem to make a difference. Of course, the success or failure of any program depends on a number of factors, but looking at the overall picture provided by studying a large number of similar programs can help provide a useful context in which to evaluate any particular program.
Photo by Joao.Trindade | Flickr.com
