As Ann Durham and her team at Providence After School Alliance, or PASA, crunched the data to see if afterschool programs in middle schools were in fact supporting the needs of youth in their Rhode Island community, the findings began to point to a hole in services.
“We had youth graduating out of our middle school programs without enough options at the high school level,” said Durham, the executive director of PASA, an intermediary organization that connects the area’s afterschool programs with the community. When “only a fraction” can continue to access afterschool activities, then “we’re not meeting the community’s needs,” she added.
The power of using data to drive decision-making has become a mainstay in most industries, and in recent years has begun to take hold in the afterschool programming space, where data can help decision makers evaluate the quality of a program, as well as shed light on efforts to meet a community’s broader goals to serve youth. Regularly measuring youth outcomes becomes especially critical, advocates say, when it comes to ensuring there is equitable access to programming and that the programs are supporting underrepresented youth.
Without a data-driven approach to assessing youth outcomes, “you can inadvertently end up in a situation in which your current practices are not the best practices,” Durham said. “It may have worked really well in the past, but over time, things changed so subtly that you didn’t even recognize the slight decline.”
While collecting and analyzing data has become an essential process for intermediaries like PASA to support afterschool programming, many communities across the country lack the resources and experience to successfully implement a data-driven framework.
“Given the course of the pandemic and the racial reckoning in this country, it’s a really important time for communities to have more resources and tools for data to think about racial equity in light of data collection and how they can support more equitable communities,” said Jessica Donner, executive director of Every Hour Counts.
Donner said when communities don’t take advantage of data to measure their programs at a system-wide level, the ones who suffer are youth, especially those who are already impacted by structural inequalities.
That is why Every Hour Counts, a national coalition of intermediaries advocating for afterschool programs, recently released a framework for how to use and measure data. The document outlines how to identify outcomes and indicators. A companion guidebook walks organizations through the entire process.
Donner said the publications — funded by the Charles Stewart Mott Foundation and the Wallace Foundation in partnership with the RAND Corporation –– aim to help “demystify data collection, help communities think holistically about youth outcomes, systems change, the importance of program quality and the importance of a varied and wide range of youth outcomes that capture things like social-emotional skills, participation and youth engagement.”
“When you’re trying to collaborate with one another and take a set of systems-wide or a city-wide approach, the data becomes especially important,” said Erik Skold.
Every Hour Counts’ resources are geared toward helping intermediates implement a data-driven approach and offer guidance on how to onboard reluctant nonprofit youth providers. In some cases, youth organizations may be concerned about what the data is being used for or worry that collaborating and sharing outcomes will put them at a competitive disadvantage when it comes to winning grants and raising funds.
According to the guidebook, over the years, nonprofit youth organizations, which historically operated in isolation of one another, began to see the power of collaborating with each other and system organizations, like school districts and city agencies. Intermediary organizations were tasked with bridging those worlds, but to do that effectively, they need data.
“When you’re trying to collaborate with one another and take a set of systems-wide or a city-wide approach, the data becomes especially important,” said Erik Skold, director of Sprockets, a network of afterschool programs in St. Paul, Minnesota. “That’s because you’re working together across organizations, across the shared vision, and you really want to be sure that you’re achieving some of the goals that you’ve set out to do.”
Skold, who helped craft the Every Hour Counts framework document and guidebook, said having data is beneficial in several ways. Organizations can look at a map of what locations are underserved to see where they could start their next program. The data can also help investors decide where they should invest and ensure investments are going to programs addressing their goals. Further, the data can be used to help get grants. Youth in the programs also get to look at the data and weigh in on whether it matched their experiences, Skold said.
“It’s not data collection for data collection’s sake,” Skold added. “It’s really about data utilization.”
That means Skold uses the framework to help programs and the community figure out what information will help improve programs, how to get that information, and then helping the programs reflect upon outcome measurements and think about different ways to improve on them.
Once there are data insights, that’s when the real work begins. Data alone won’t plug any holes.
Back at PASA, the organization followed the Every Hour Counts framework to drill deeper into the data they had collected, revealing a more comprehensive picture as to why some high schoolers were dropping out of afterschool programs.
The data showed that the youth most likely to stop participating were those living in low-income homes. Those youth said they needed to spend out-of-school time earning money, and the programs in their community didn’t offer enough paid internship options.
Ultimately, having paid opportunities requires more money, and even with a windfall of federal funding set for afterschool programs, most public monies are prohibited from being used to directly pay youth, Durham added.
“We have to create and figure out how we’re going to increase the number of paid opportunities for young people,” Durham said. “And that is going to require a change from our traditional funding model.”
