Guide Makes Measuring Data Easier For After-school Systems

Figuring out what to measure and how to measure it is the 21st century question. Technology allows for vast data gathering and number crunching, but what data, which numbers and why?

The question is particularly perplexing in the youth development field, which is ill-defined and has poor name recognition but serves millions of young people in innovative after-school and summer programs around the country — many in underserved communities.

After-school leaders see data as a way to show the impact of what they are doing — as well as to guide and improve their work.

“Having good data is helpful in making the case the field is essential,” said Hillary Salmons, executive director of Providence After School Alliance (PASA), the after-school coordinating agency in Providence, R.I.

“We’re trying to make sure we can demonstrate and prove that we’re making a contribution to the healthy development of children,” she said.

Data is also vital in communicating with youth and their families, Salmons said: “You matter and we’re going to make sure you have a high-quality experience and we know how to measure that. Just because you’re poor, we don’t water [the program] down.”

Data is also needed to design and manage a system that is effective and financially sustainable.

Enter the organization Every Hour Counts. It’s a network of citywide after-school systems from Providence to the San Francisco Bay Area. The systems, often called intermediaries, coordinate the after-school programs in a city and pull them together into a cohesive whole.

All about outcomes

In 2014, Every Hour Counts created a blueprint for measuring outcomes for youth, for after-school programs and for a city system as a whole.

Outcomes for youth might be academic or social-emotional. Desired outcomes for programs could range from having a positive emotional climate to being well-aligned with the school. Outcomes at the system level might range from having diverse financial support to having a broad reach in underserved neighborhoods.

“So many organizations think just at the youth level,” said Jessica Donner, executive director of Every Hour Counts. But it’s important to look at the program and system level, too, she said.

The variety of data, however, can be overwhelming.

“We knew there were a plethora of outcomes that intermediaries could focus on,” she said.

Every Hour Counts engaged the RAND Corp., a research institute, to look at the data-crunching experiences of three after-school intermediaries, PASA in Providence; Sprockets in St. Paul, Minn.; and Boston After School & Beyond in Boston.

The resulting guide, “Putting Data to Work for Young People: A Ten-Step Guide for Expanded Learning Intermediaries,” draws from their experience. It was funded by The Wallace Foundation and the Charles Stewart Mott Foundation.

The guide recommends a systematic approach beginning with clearly defined goals.

“Understanding what your goals are is key,” said Anamarie A. Whitaker, a policy researcher at the RAND Corp. and a co-author of the report.

The goals determine which data will need to be collected.

According to the guide, the reasons to collect data are to:

• guide the system’s policies and practices
• support the after-school programs
• impact student outcomes
• communicate with stakeholders.

Different city, different data

After-school systems are different in each city and have different needs and goals.

“As a community, you need to figure out what’s the right fit for you,” said Ann Durham, PASA deputy director.

In one city, a goal might be to find the geographic areas where young people are underserved and fix that. The pertinent data then would be number of students served and their ZIP codes.

In Providence, where many after-school programs are located at school sites and kids come from all over the city to the school of their choice, the issue is not about targeting geographic areas, Durham said.

PASA instead decided to focus on social and emotional outcomes for students. Because PASA is a collection of 70 small to medium community organizations, it made sense to survey those organizations to determine which social-emotional outcomes to measure, Salmons said.

Another after-school system might choose to measure academic outcomes.

After-school coordinating agencies should pick only two or three things to tackle, Salmons said.

“We’re not going to focus on everything,” she said.

It’s important to be strategic and look at the right variables rather than overcollect data, she said.

The guide also keeps cities and their after-school systems from putting the cart before the horse.

A mayor or funders may have big goals such as improving kids’ reading and math scores or showing kids’ social and emotional gains, Salmons said.

But it’s too ambitious to try to move complex youth outcomes without a good system in place, she said. “Start with making sure you have a quality improvement plan and the capacity to deliver it,” she said.

The report helps everyone realize that a series of steps must be taken at the system, program and student levels, she said. “There are stages of this,” Salmons said.

Going step-by-step may take longer than a community initially envisions, but the community will get better outcomes as a result, Durham said.

“The 10 steps provide a reality check,” she said. 

Make a sketch

After step 1, which is to define your goals, the guide spells out the following steps:

• Sketch out how the data will be visually presented.

“This provides a thorough understanding of what data are needed and how an organization wants to present it,” Whitaker of RAND Corp. said.

A simple chart or graph of attendance may be the most useful to program providers. Show them a sketch and get their feedback, the guide recommends.

• Hire a full-time experienced data coordinator. 
• Adopt policies and technology for securing and organizing data. 

List all the procedures used in gaining consent and collecting data. This might include a memorandum of understanding with the school system that spells out how to securely transmit and store data. Choose a management information system with adequate security features.

• Create an inventory of data to be used and a “dictionary” of definitions and rules.

List each piece of data to be collected, along with its description, its location and who is responsible for it. Each student and program site needs to have a unique identifier to allow the combining of data sets from different sites and times. Carefully define the information you are collecting.

 “Keeping it up to date is very important, too,” Whitaker said.

• Establish clear-cut data collection procedures.

A written set of instructions is necessary for each data-collecting activity, whether it’s a survey of students or a program observation. This is important to ensure data accuracy.

• Train staff in how to work with data.
• Develop a process and a timeline for analyzing and reporting data.

Determine when and what kinds of reports are needed.

• Create a visual aid to track the flow of data.

“It gives you an understanding of where data are within the system,” Whitaker said. It makes the gaps visible, she said.

• Regularly evaluate and troubleshoot the data system.

Working with data is not easy, said Donner, of Every Hour Counts.

“There’s not a lot of resources [in the after-school field]. We lack staff capacity and funding resources to help with the data efforts,” she said.

But the guide can fill the gap for many communities.

“They don’t need to reinvent the wheel,” she said.