Opinion

OST: Learn Differences Between Tools to Measure Your Program Quality

PALERMO89/Shutterstock

.

What is a high-quality after-school program? What do you expect to see in high-quality programs? As an experienced practitioner, you have likely thought about these questions and have begun the self-assessment process to enhance your practices.

Measures of program quality often focus on two aspects: process quality and structural quality. Process quality, sometimes called point-of-service quality, refers to the experiences that young people have at the program, including their interactions with staff and peers, and their interest and engagement in program activities.

rating systems: Yangyang Liu (headshot), Ph.D. candidate in the School of Education at the University of California, Irvine, smiling young woman with long black hair, blue top.

Yangyang Liu

Structural quality refers to aspects of the program, such as organization of resources at the program site, safety of the environment, adult-child ratio and staff background characteristics such as education and training. Both structural quality and process quality have been shown to predict youth outcomes within and beyond the program setting.

Process quality is found to predict a broad range of youth outcomes. Because of its importance, measures have been developed to specifically assess youth experiences in the program settings.

The Youth Program Quality Assessment scale developed by the David P. Weikart Center is a widely used tool focusing on the quality of youth experiences in the program. The YPQA consists of two overall rating scales, 103 items that assess seven domains of program quality: engagement, interaction, supportive environment, safe environment, high expectations, youth-centered policies and practices, and access.

The YPQA scale can be used by program staff for both self-assessment and program improvement purposes. Because of its focus on process quality, the majority of items on the YPQA scale assess multiple domains of process quality, including interactions between youth and adults, interactions among peers, the extent to which youth are engaged in the program and the nature of their engagement.

rating systems: Deborah Vandell (headshot), professor of education in the School of Education at the University of California, Irvine, smiling woman with short blond hair, necklace, striped turtleneck, black sweater.

Deborah Vandell

Process quality is measured via observations. At least one hour of observation of multiple activities should be done, with observation divided among different observers (e.g., three people each observe 20 minutes, four people each observe 15 minutes). To help observers score the program quality, the YPQA offers concrete descriptions of a phenomenon illustrating what a score of 1, 3 or 5 looks like. The YPQA produces scores at the item, scale and domain level. Scores above the items level are created using average scores. Scales are averages of items, and domains are averages of scales.

The YPQA was first designed for youth in grades four through 12. A developmentally appropriate version of the scale, named the School-Age PQA, is also available to use with children in grades K to six.

Only holistic score

Another prominent measure on process quality is the Promising Practices Rating System developed by the Wisconsin Center for Education Research & Policy Studies Associates. The PPRS is also an observation scale assessing eight domains of youth experiences in the program, supportive relations with adult, supportive relations with peers, level of engagement, opportunities for cognitive growth, appropriate structure, overcontrol, chaos and mastery orientation.

Different from the YPQA tool, which has multiple items on each scale, the PPRS offers only a holistic score on each domain of youth experience. Though specific exemplars (positive and negative) with detailed indicators are provided for each domain, the PPRS places more burden on the observer such that they need to make inferences from their observations to assign a score at the overall practice level. Part of the difficulty of using the PPRS results from the fact that this measure was first developed with a research audience in mind and the tool has not been tailored for practitioner use.

Structural features of a program indicate both availability and organization of available resources in the setting. Structural quality is thought to lay the foundation of a high quality program, and it predicts process quality. While most existing measures assess process quality, not all measures examine structural quality at the program level.

Quality measures developed by national or state after-school networks often measure various aspects of structural quality. For example, the program quality self-assessment tools developed by both the New York State Network for Youth Success and the California Afterschool Network include in their measures items on safety and organization of space, available materials, administration of daily operation, staff recruitment and professional development. Both measures are administered through observation.

However, these measures differ in minor way on what aspects they assess. For example, while the QSA tool developed by both New York and California includes a section on family/community collaboration, the QSA tool developed by the California Afterschool Program also has a separate section on reaching out to culturally diverse families and youth. When deciding which tool to use, practitioners need to select the tools that best serve their interest and goal.

If you are interested in learning about more measures, you can refer to a review article by Nicole Yohalem and colleagues that review nine program quality measures. Similarly, the California Department of Education and the California Afterschool Network collaborated to provide a crosswalk of seven program quality measures, which overlaps to some extent with the Yohalem review.

Yangyang Liu is a Ph.D. candidate in the School of Education at the University of California, Irvine. Her research examines how after-school programs and extracurricular activities promote positive development for diverse youth, with a focus on program quality.

Deborah Lowe Vandell is a professor of education at the University of California, Irvine, where she was the founding dean of the School of Education. She studies the effects of after-school programs and extracurricular activities on children’s and adolescents’ academic and social outcomes across K-12 schooling and the development of assessment tools to measure program quality.

Comments

Youth Today is the only independent, internationally distributed digital media publication that is read by thousands of professionals in the youth service field.

Youth Today adheres to high-quality journalistic standards, providing readers with professional news coverage dedicated to examining a wide spectrum of complex issues in the youth services industry from legislation to community-based youth work.

EDITORIAL INDEPENDENCE

Our organization retains full authority over editorial content to protect the best journalistic and business interests of our organization. We maintain a firewall between news coverage decisions and sources of all revenue.

DONORS & DONOR TRANSPARENCY

We are committed to transparency in every aspect of funding our organization. Donors may be quoted, mentioned or featured in our stories. Our news judgments are made independently – not based on or influenced by donors. Accepting financial support does not mean we endorse donors or their products, services or opinions…(read more)

Recent Comments

Archives

Categories

Search

LOGO Institute for Nonprofit News 3 turquoise boxes stacked in "J" shape
Logo Grant professional Association Business Alliance

Copyright © 2019 Youth Today and MVP Themes --- Published by Center for Sustainable Journalism,
Kennesaw State University, 1200 Chastain Blvd. Suite 310, Kennesaw GA 30144

To Top