This blog explores an evaluation conducted by the Office of Community College Research and Leadership (OCCRL) into the strategies that the Illinois Community College Board (ICCB) is leveraging as part of its internal program of study review process.
Overview
The ICCB is the governing body for Illinois’ 48 community colleges and works in concert with the Illinois State Board of Education to administer the federal Carl D. Perkins Career and Technical Education (Perkins V*) program. They have applied a hyper-local lens to the program review process to support learner-centered, equity-minded, and evidence-driven change across the Illinois community college system.
ICCB’s Program Review Manual lists the requirements for institutions completing a program review and instructions for how data should be used to inform program design if gaps are identified. Colleges are asked to evaluate the quality and cost-effectiveness of all student and academic support services, including Career Techincal Education (CTE) programs.
Some of the criteria outlined by the ICCB to complete this program of study review are:
- At a minimum, systematically examine the need, cost and quality of individual programs and services, ensuring that programs continue to be responsive to local needs, cost-efficient, and integrate quality components into programming;
- This data is provided by faculty and appropriate administrators who are directly responsible for the areas being reviewed as well as faculty, academic support professionals and administrators from across the campus;
- Examine assessment results appropriate to the unit, as well as comparative data on enrollments, completions, and costs using the most recent audited state-level data;
- Be responsive to identified areas of weakness by developing and implementing feasible and measurable action steps;
- Include the use of disaggregated data to uncover and address equity gaps in programming, performance or service;
- Assure that the process is well documented and use the results to inform campus planning initiatives, quality improvement efforts and budget allocation decisions;
- Report results and actions resulting from reviews to local boards, advisory committees, and other stakeholders as appropriate;
- Adhere to a minimum review cycle of once every five years for all programs listed in the Program Review Schedule.
With significant and recent changes taking place to ICCB’s Program Review Manual, the OCCRL’s study, Advancing Program Review Evaluating and Envisioning the Future of Program Review at Illinois Community Colleges, sought to provide feedback that can improve the efficacy of the manual and supporting practices. The OCCRL framed the study around the concept of program review process as a tool for institutions to support learner-centered, equity-guided and evidence-driven improvements. The intent of the review process is to support colleges in making campus-level planning and decision-making related to the quality, cost-effectiveness, assessment and improvement of programs. While these goals are similar to the process that local entities go through to complete their Comprehensive Local Needs Assessment (CLNA), this approach offers an opportunity to facilitate greater consistency and communication across institutions to improve outcomes for every learner.
Methods
Through the study, OCCRL attempted to answer three evaluation questions:
- What institutional and environmental factors within the context of program review affect institutions’ culture of evidence (using data to inform program design) and how do these factors vary across institutions in the state?
- What substantive challenges influence the design, implementation and impact of the program review in Illinois and how might these challenges be remedied?
- What are critical design considerations for advancing the future of program review processes at community colleges in Illinois?
In order to complete their evaluation, OCCRL noted that their approach had to reflect the high level of variation that exists among Illinois community colleges and the programs nested within them. To meet this need, they focused on working closely with individuals within the field who were currently engaged in the work and who had developed expertise working within the system, allowing them to suggest improvements and envision its future. Evaluation data was collected through a series of focus groups composed of representatives from the 49 community colleges, including community college practitioners, faculty, institutional researchers and chief information officers. The topics of the focus groups covered environmental and institutional factors influencing program review, colleges’ procedures for managing and using evidence produced from program reviews, and finally, challenges related to program reviews.
Evaluation
In their report, OCCRL identified seven factors that affect an institution’s ability to successfully foster a culture of evidence and support equity-minded use of the resulting findings:
- Leadership: A culture of evidence is championed throughout the college through informal and formal leadership.
- Strategic investments: The college strategically invests resources in learner-centered improvements directed toward programs and pathways.
- Equity-guided: A commitment is shared throughout the college to systemic and localized changes to improve equity for underserved learners.
- Engagement: Diverse viewpoints and roles are encouraged and sought after and contribute to ongoing improvement processes.
- Statewide guidance and governance: The state provides clear and supportive standards and policies that support both accountability and data-driven change.
- Data literacy: Institutionally and individually, the college supports access to data and the data literacy necessary to frame inquiries and to effectively collect, operationalize, analyze, and interpret data.
- Infrastructure and information technologies: Systems are available to support the collecting, storing, accessing, analyzing, and disseminating of data throughout the college.
Resulting Supports
As a result of requests from study participants, OCCRL developed an equity-centered rubric as a standardized tool that could be used to advance learning and support for institutions across Illinois in their CTE program review process. This self-assessment tool is intended to support practitioners in completing high-quality reviews and enables them to more closely examine the demographic groups within CTE programs.
In addition to the rubric, participants shared high-level reflections on the solutions to common program review challenges that were discussed during the third focus group:
- Identify strategies to make the program review process user-friendly, as “Concise is Nice”.
- Support the statewide solutions sharing.
- Protect the necessary time for accurate, proactive planning.
- Implement a peer review process to leverage the depth of available expertise.
Additionally, the authors of the study suggest strategies that state leaders might consider incorporating into their own CLNA processes.
- Program review process should lead to action steps reflective of equity-guided continuous improvement models.
- Program reviews should include learner input data to inform practice and stakeholder commitment to strategic planning and resource sharing.
Visit Advance CTE’s Learning that Works Resource Center to access additional resources for state leaders looking to integrate equity-guided and data-driven strategies into their state plans.
- Making Good on the Promise: Improving Equity and Access to Quality CTE Programs for English Learners
Amy Hodge, Membership & Policy Associate
*As amended by the Strengthening Career and Technical Education for the 21st Century Act
Tags: Access and Equity, Carl D. Perkins Act, CLNA, community colleges, Illinois, Perkins V, Program Quality, Research