PALERMO89/Shutterstock
.
What is a high-quality after-school program? What do you expect to see in high-quality programs? As an experienced practitioner, you have likely thought about these questions and have begun the self-assessment process to enhance your practices.
Measures of program quality often focus on two aspects: process quality and structural quality. Process quality, sometimes called point-of-service quality, refers to the experiences that young people have at the program, including their interactions with staff and peers, and their interest and engagement in program activities.
Yangyang Liu
Structural quality refers to aspects of the program, such as organization of resources at the program site, safety of the environment, adult-child ratio and staff background characteristics such as education and training. Both structural quality and process quality have been shown to predict youth outcomes within and beyond the program setting.
Process quality is found to predict a broad range of youth outcomes. Because of its importance, measures have been developed to specifically assess youth experiences in the program settings.
The Youth Program Quality Assessment scale developed by the David P. Weikart Center is a widely used tool focusing on the quality of youth experiences in the program. The YPQA consists of two overall rating scales, 103 items that assess seven domains of program quality: engagement, interaction, supportive environment, safe environment, high expectations, youth-centered policies and practices, and access.
The YPQA scale can be used by program staff for both self-assessment and program improvement purposes. Because of its focus on process quality, the majority of items on the YPQA scale assess multiple domains of process quality, including interactions between youth and adults, interactions among peers, the extent to which youth are engaged in the program and the nature of their engagement.
Deborah Vandell
Process quality is measured via observations. At least one hour of observation of multiple activities should be done, with observation divided among different observers (e.g., three people each observe 20 minutes, four people each observe 15 minutes). To help observers score the program quality, the YPQA offers concrete descriptions of a phenomenon illustrating what a score of 1, 3 or 5 looks like. The YPQA produces scores at the item, scale and domain level. Scores above the items level are created using average scores. Scales are averages of items, and domains are averages of scales.
The YPQA was first designed for youth in grades four through 12. A developmentally appropriate version of the scale, named the School-Age PQA, is also available to use with children in grades K to six.
Only holistic score
Another prominent measure on process quality is the Promising Practices Rating System developed by the Wisconsin Center for Education Research & Policy Studies Associates. The PPRS is also an observation scale assessing eight domains of youth experiences in the program, supportive relations with adult, supportive relations with peers, level of engagement, opportunities for cognitive growth, appropriate structure, overcontrol, chaos and mastery orientation.
Different from the YPQA tool, which has multiple items on each scale, the PPRS offers only a holistic score on each domain of youth experience. Though specific exemplars (positive and negative) with detailed indicators are provided for each domain, the PPRS places more burden on the observer such that they need to make inferences from their observations to assign a score at the overall practice level. Part of the difficulty of using the PPRS results from the fact that this measure was first developed with a research audience in mind and the tool has not been tailored for practitioner use.
Structural features of a program indicate both availability and organization of available resources in the setting. Structural quality is thought to lay the foundation of a high quality program, and it predicts process quality. While most existing measures assess process quality, not all measures examine structural quality at the program level.
Quality measures developed by national or state after-school networks often measure various aspects of structural quality. For example, the program quality self-assessment tools developed by both the New York State Network for Youth Success and the California Afterschool Network include in their measures items on safety and organization of space, available materials, administration of daily operation, staff recruitment and professional development. Both measures are administered through observation.
However, these measures differ in minor way on what aspects they assess. For example, while the QSA tool developed by both New York and California includes a section on family/community collaboration, the QSA tool developed by the California Afterschool Program also has a separate section on reaching out to culturally diverse families and youth. When deciding which tool to use, practitioners need to select the tools that best serve their interest and goal.
If you are interested in learning about more measures, you can refer to a review article by Nicole Yohalem and colleagues that review nine program quality measures. Similarly, the California Department of Education and the California Afterschool Network collaborated to provide a crosswalk of seven program quality measures, which overlaps to some extent with the Yohalem review.
Yangyang Liu is a Ph.D. candidate in the School of Education at the University of California, Irvine. Her research examines how after-school programs and extracurricular activities promote positive development for diverse youth, with a focus on program quality.
Deborah Lowe Vandell is a professor of education at the University of California, Irvine, where she was the founding dean of the School of Education. She studies the effects of after-school programs and extracurricular activities on children’s and adolescents’ academic and social outcomes across K-12 schooling and the development of assessment tools to measure program quality.
