Photos courtesy of Christopher House
.
A colorful mosaic of smiling children brightens the wall outside Christopher House near Logan Square in Chicago.
Inside the building is a preschool noted for its success in graduating kindergarten-ready children, despite poverty, housing insecurity and unsafe neighborhoods. Christopher House works to close the opportunity gap for low-income kids through its preschools, programs for school-age kids and a strong support of parents.
Traci Stanley
But it also has led an unusual collaboration of Chicago youth-serving nonprofits. For 10 years, the group known as the Chicago Benchmarking Collaborative has gathered data together and measured effectiveness together.
“It is a unique collaborative that uses data to inform processes to increase impact,” said Lori Baas, CEO of Christopher House.
The collaborative of seven nonprofits is sharing what it’s learned in a guide written by Christopher House director of quality assurance Traci Stanley:
Who they are
All seven partners in the collaboration work with children, youth and adults.
Some of the partnering organizations evolved from settlement houses and serve specific neighborhoods.
- Chicago Commons Association was founded in 1894 to serve immigrants on Chicago’s north side. Today it provides early childhood education, family support and senior services to about 3,000 people per year.
- Erie Neighborhood House reaches a Latinx community, offering education, access to critical services and advocacy,
- Gads Hill Center has four sites on Chicago’s west and southwest side and includes mental health programs, out-of-school time programs as well as preschool.
- The Chinese American Service League is a social service agency that eases cultural transitions and strengthens families.
- Chicago Youth Centers serves 1,400 youth in early childhood and out-of-school time programs at seven centers and some partner sites.
- St. Vincent Marillac provides social services including early childhood and youth programs
The partner organizations vary in size, with budgets ranging from $5 million to $50 million, Baas said.
At the outset they had different ways to measure their impact on children and families. Among social service agencies, measures of impact vary widely as agencies try to create their own solutions to the problems they see around them.
“Prior to forming the Chicago Benchmarking Collaborative 10 years ago, we were limited to measuring impact solely against our own agency outcomes,” said Kirstin Chernawsky, executive director of Erie Neighborhood House, in a statement.
What they chose to measure
After the Chicago Benchmarking Collaborative formed, the partners agreed on two early childhood outcomes to measure — increased school readiness and parental engagement.
The group decided to measure four outcomes for kids in after-school programs; improved school performance, improved social skills and interpersonal behavior, graduation from high school, and enrollment in post-secondary education.
Lori Baas
Outcomes for adult education were; improved level of education and English literacy, increased earnings, and increased sustainable employment and retention.
The decisions were not easy.
A facilitator is necessary when deciding what outcomes to track, Christopher House’s Stanley said. The facilitator can present research on outcomes, discuss various tracking tools and help to merge the different points of view.
Getting a group of seven youth-serving professionals to agree on the best metric and assessment “takes time and the building of trust,” Baas said.
“One of the things I’m most proud of is that … despite some differences folks made a decision to move forward,” she said.
Lessons learned
Well before starting to determine the metrics, the leadership must articulate a clear vision, Baas said.
She underscores the need for developing trust among member organizations.
“You need to develop a group of peer organizations that have trust. It’s a very vulnerable process where … you’re going to see [how your organization’s program] compares with peers.”
Stanley said that once a clear vision is set, it’s crucial for the leadership of each organization to champion the initiative. It’s a culture change for the organizations.
“It’s important that staff see the value of the initiative and the vision. That comes from having a clear vision shared by leadership,” she said.
The Chicago Benchmarking Collaborative found certain practices to be crucial:
- Identify a project facilitator
- Adopt a common management information system
- Offer training about outcomes measurement to staff across all the agencies so that they can speak a common language.
- Keep the data measures simple in the beginning.
- Direct-service staff enter most of their own data, especially notes having to do with case management.
- Create reports often and share them with the staff, so they don’t feel like their data is going into a black hole.
- Staff should discuss strategies they think are contributing to the data results. Since control groups aren’t used, there is no way to prove a strategy is linked to a result.
- Create improvement plans in the fall and align them to SMART (specific, measurable, achievable, relevant and timebound) criteria.
- Success is tied to group cohesion and trust.
“One of the things that made the process smoother was that we had flexibility in our software,” Stanley said. The collaborative could track the agreed-upon outcomes, but one of the agencies could also track additional outcomes, she said.
“The software was very customizable, which benefited our group,” she said.
It helped to create an annual schedule of work for all the organizations, Stanley said. Staff gave input on what they thought are reasonable deadlines to set for having data entered into the system, she said.
Data showed the way to help kids
Mosaic outside Christopher House, Logan Square, Chicago.
Once the system was in place, the collaborative could begin to learn from it.
The data on student grades soon showed two member organizations with higher academic gains among their students. For example, 59 percent of the kids in one program were raising their grade point average, compared with 47 percent among all the organizations. In one organization, 90 percent of students maintained a B average, compared with 51 percent among all the organizations.
It turned out that the two high-performing organizations offered one-on-one tutoring and mentoring, compared with the group tutoring and mentoring offered at the others. After considering the data, some of the organizations decided to adopt one-on-one tutoring.
“It took a couple of years to see the change in results and scores over time,” Stanley said. “For the organizations that restructured themselves, they did see over time an increase in their GPAs.”
Over time, other gains became clear.
In 2011, the data showed that pre-school math scores across all the organizations were low. As a result, teachers were offered professional development. From 2011 to 2018, preschooler math scores rose 28 percent.
To promote the outcome of kids graduating from high school and entering post-secondary education, the collaborative assessed the motivation and goal-directed behavior of school-age kids using the Survey for Afterschool Youth Outcomes (SAYO). In one instance, the collaborative developed an improvement plan that called for staff to create a set number of activities based on students’ interests in order to build student engagement and confidence. Measures on SAYO rose.
Baas sees the collaborative as an important way nonprofits can increase their impact.
“Data shows and we firmly believe that this sort of shared learning and data is helping close opportunity gaps and change the trajectory for children and families,” she said.
She wants to get the word out about lessons learned in the Chicago Benchmarking Collaborative.
Her hope is that nonprofits in other communities will choose to collaborate similarly.
