While it may take a village to raise a child, it seems to take even more to assess the quality of children’s after-school programs – including a statewide partnership of public and private agencies, a federal shift in oversight responsibilities and the financial support of a mega-funder.
Such a convergence in New York recently produced the first after-school program assessment tool in the nation that is mandatedfor use by 21st Century Community Learning Centers (CLCs).
The result of a multiyear collaboration by more than 200 stakeholders, the Program Quality Self-Assessment Tool from the New York State After-School Network (NYSAN) is designed to help after-school practitioners measure their quality against a common set of standards, identify where they need technical assistance and training, and prioritize and plan improvements.
Several months ago, the state Department of Education began requiring all after-school programs that receive CLC funds in New York to conduct the NYSAN evaluation twice a year in conjunction with their federally mandated annual performance reports.
While several after-school assessment tools have already been developed – by citywide programs such as Boston’s, for-profit vendors like Foundation Inc., evaluation research organizations like Mathematica and regional education labs – few states have gotten that far. Priscilla Little, associate director of the Harvard Family Research Project and program manager of its Out-of-School Time Learning and Development Initiative, points to an assessment tool developed by PlusTime New Hampshire that’s “actually very similar to NYSAN’s.”
NYSAN Program Director Sheri Gruber, who regularly meets with statewide after-school networks funded by the Charles Stewart Mott Foundation, says that other states are working on self-assessment, but “in terms of having an actual tool, New York is way out ahead.”
How NYSAN got there is as instructive as the result.
Building the tool
Back in 2001, the education department formed a statewide advisory group of after-school stakeholders to help navigate the federal government’s handover of CLC administration to the states. That group became NYSAN when it organized as a separate entity in 2003. The education department joined NYSAN’s steering committee, along with The After-School Corporation (TASC), the Children’s Aid Society, the Partnership for After School Education and about a dozen other groups.
By then, TASC, which sponsors more than 200 after-school programs, already had years of experience in assessing its sites and looking for consistency in other evaluation tools across the county.
“TASC has always tried to integrate or implement evaluation findings from one year to the next,” says Bill Casey, vice president of TASC and a member of NYSAN’s education and quality assurance committee.
In 2004, NYSAN won a $225,000 grant from the Mott Foundation, which is supporting the development of statewide after-school networks. Mott says its network grantees must develop “a statewide system of evaluation that includes self-assessment and continuous improvement strategies and rigorous impact evaluation.”
The foundation has awarded $3.8 million in three-year grants to 18 states (nine each in 2003 and 2004), and has chosen a third group of seven states to begin receiving funds this year.
NYSAN began by conducting a nationwide review of evaluation instruments and research. The group ultimately credited 17 resources as key to the development of its assessment tool, including LA’s Best, Baltimore Safe and Sound, the National School-Age Care Alliance and the National Youth Employment Coalition.
The wide array of best practices and standards collected in the review became the framework for discussions among NYSAN’s partners as they sought to define high-quality after-school programming. “Everyone was able to come together and say, ‘We agree. This is what represents quality in an after-school program,’ ” Gruber says. “That was a huge step for us in New York State. … That was a really unique moment.”
NYSAN piloted its first draft of the evaluation in about 40 TASC programs. Thus began a lengthy period of “work in progress.”
“We had a ‘draft’ label on this document for almost two years,” Casey says.
After a final round of post-pilot discussions with NYSAN partners, the Program Quality Self-Assessment Tool was publicly released in February.
What it Measures
The assessment consists of nearly 100 statements, called “quality indicators,” that are organized under 10 “essential elements” of a high-quality after-school program: environment/climate; administrative/organization; relationships; staffing/professional development; programming/activities; linkages between day and after-school; youth participation/engagement; program sustainability/growth; parent/family/community partnerships; and measuring outcomes/evaluation.
For each of the seven to 12 indicators listed under each category, programs rate their performance using a numeric scale that reflects the one New York uses to rate students’ academic performance: 4 – “Excellent/Exceeds Standards;” 3 – “Satisfactory/Meets Standards;” 2 – “Some Progress Made/Approaching Standard;” and 1 – “Must Address and Improve/Standard Not Met.”
To address indicators that need improvement, programs check boxes to indicate that they plan to improve “Right Now,” “This Year” or “Next Year.”
The timeline represents a culture shift in the after-school field, Casey says. “The state education department is actually saying to 21st Century grantees, ‘We don’t expect you to fix everything at once,’ ” he says.
At the same time, he says, the requirement to use the tool tells programs, “You have to get a sense of where you are at this point, but you also have to have a good sense of where you want to be, and you have to have some very concrete steps that you’re planning to take in order to get you there.”
The indicators include both nuts-and-bolts practical measures, such as “Conducts all required fire/safety drills,” and subjective statements, such as “Has a culture that allows participants to take initiative and explore their interests.”
Other indicators may have little to do with a program’s mission, or seem beyond the program’s control or resources, such as “Is represented on the school’s curriculum planning committee,” or “Provides opportunities for literacy and related educational experiences for the families of the participants.”
Casey says that “each one of those was important to a number of our partners, or it would not have survived its existence on the page.”
While he expects some resistance to a few of the indicators, Casey emphasizes that the tool “doesn’t say that you must” perform highly on all of the indicators, but that the indicators are intended to “generate conversation that might have never taken place before.”
Expectations
NYSAN and the education department expect programs using the assessment to work “toward achieving, at minimum, a Satisfactory level in all of the quality indicators within each of the 10 elements,” according to the tool’s printed instructions. “Programs must continue to strive to achieve a level of Excellence.”
What’s to stop a program from rating itself “excellent” on all of the indicators? “That’s a continuing challenge,” Casey admits, “because one of the criticisms of [the tool] is that some people don’t even know what ‘good’ looks like.”
NYSAN has a strategy to curtail inflated scoring while also promoting continuous improvement: Programs that excel will be enlisted to train programs that need help. “Where programs have labeled themselves as exceeding the standards in certain areas, what they’re also saying is that they’re willing to help others in that area,” Casey says.
“We all felt it was rather unfortunate that so much time and so much money, in some instances, is spent on evaluation, but frequently, the practitioners just don’t benefit from it.”
The beneficiaries can include any after-school programs, which may use the tool to assess themselves even though they don’t get CLC money through New York state. The tool is available at the bottom of NYSAN’s Web page.
Contact: NYSAN (212) 547-6908, www.nysan.org.
