Academic Program Review Guidelines

▼   Purpose

Academic program review (APR) is an opportunity for programs without discipline specific accreditation or approval to engage in:

  • Synthesis of program-level data to include both operational and student learning.
  • Evaluation of the degree to which the program meets expectations regarding operational and student learning outcomes.
  • Strategic planning to guide continuous improvement of the program.
  • Peer review to gather constructive feedback from the field regarding the quality of the program.
▼   APR Coordination

APR is coordinated by the Office of Institutional Effectiveness every seven years for programs without periodic, discipline specific accreditation or approval.  New programs undergo program review in their 4th year in preparation for post-implementation review by the Alabama Commission on Higher Education (ACHE) and every seven years thereafter. 

OIE meets with program faculty participating in APR during the Fall semester.  The guidelines are reviewed and data packets are provided.  Throughout the APR process, OIE staff provide support to faculty, as needed.

▼   Time Frame

The process generally takes about 12 months to complete; however, the maximum amount of time allotted to complete the self-study is three semesters (not including summer). On the next page, a recommended schedule is presented. The major milestones for the APR are listed below.

Year 1

September (first half) - Send external reviewers names to the Dean

April – Upload completed self-study into TracDat

May – Send report to External Reviewer

Year 2

September – External Reviewer’s visit

September – Incorporate the findings of the self-study into program assessment

October – Upload final APR into TracDat

APR Timeline

Month/Date

Activity

July

Offices of Institutional Effectiveness (IE) and Institutional Research (IR) prepare program review materials and program specific data. 

September (first half)

Program coordinator and/or chair meets with (IE) to review the program review process and schedule.

Chair and/or program coordinator meets with faculty and staff.

Chair and/or program coordinator submits list of potential external reviewers to the Dean for approval.

September (first half)

Dean submits list of approved reviewers back to chair and/or program coordinator.

September (last half)

Chair and/or program coordinator secures external reviewer. 

October – March

Program staff and faculty prepare self-study.

April

Program staff and faculty upload the self-study into TracDat for preliminary approval by the Dean, AVPIE, and Provost/SVPAA.

May

Chair and/or program coordinator sends the report to the external reviewer.

Month/Date

No Campus Visit

With Campus Visit

August (last half)

External reviewer’s report submitted to Dean; Review external reviewer’s comments with the dean, chair/program coordinator, and faculty.

Prepare for campus visit.

September
(last business day)

Revise assessment plan, update results and develop actions based on findings of the self-study and the external reviewer’s recommendations.  Report in TracDat.

External reviewer campus visit completed.

External reviewer’s report submitted to Dean; Review external reviewer’s comments with the dean, chair/program coordinator, and faculty.

October
(last business day)

Upload the final self-study into TracDat for Dean, AVPIE, and Provost/SVPAA for approval.

Revise assessment plan, update results and develop actions based on findings of the self-study and the external reviewer’s recommendations.  Report in TracDat. Submit the final self-study to Dean, AVPIE, and Provost/SVPAA for approval.

November
(last business day)

Notify OIE that the self-study has been approved by the Dean.

Notify OIE that the self-study has been approved by the Dean.

▼   APR Reporting

APR documentation is reported in TracDat.  The instructions are included in a folder labeled APR in the TracDat Document Repository as well as some of the data needed for this report.  Recommended length of the narrative is 10-12 pages, excluding supporting tables and other relevant documentation. 

▼   Suggestions for Organizing the Process
  1. Appoint a committee of three or more faculty from the unit under review.
  2. Include both junior and senior faculty and students, if possible.
  3. Create a detailed timeline for completing the self-study.
  4. Contact OIE with questions or data requests.
▼   Components of the APR
  1. Title indicating the program’s official name and CIP code
  1. Program Overview
  • Short statement about the administrative home of the unit (school, college, etc.)
  • Description of the program to include the mission statement, its role within the College/School and University, and how it supports the University’s mission and strategic plan.
  • Discuss strengths and challenges of the program.
  • List at least three peer institutions.
  1. History of the program
  • A brief history of the program.
  • Discuss any changes that have occurred in the unit since the last APR (first-time reviews do not complete).
  • Provide a summary of the recommendations of the previous academic program review and changes made in response to the recommendation (first-time reviews do not complete).
  1. Students
  • Compared to peer institutions, discuss whether the student enrollment is adequate and/or appropriate to support the program?  TracDat>Documents>Document Repository>Program-Name>APR>Program Review Packet>Year Tables 1 and 2; for programs submitting ACHE post-implementation reviews also include Table 9
  • Processes and activities related to recruitment and retention of students.
  1. Faculty
  • List of faculty by rank
  • Discuss quality of instruction based on student evaluations of teaching. TracDat>Documents>Document Repository>Program-Name>APR>Year>SET summary report for three years
    • Describe faculty involvement in professional development activities and as it relates to instruction.
  • Discuss faculty involvement in scholarly and/or research activities to include publications, grant funding.
  • Compared to peer institutions, discuss whether faculty qualifications are sufficient to support the program. Create Table 3 based on information in TracDat.  TracDat>Documents>Document Repository>Program-Name>APR>Program Review Packet>6 semester trend report 9.20.16
  • Compared to peer institutions, discuss whether the program is adequately staffed for faculty to offer the curriculum.
  1. Curriculum and Overview of Assessment
  • A sample degree plan
  • In TracDat, create a curriculum map that reflects connections between student learning outcomes, assessment processes, and required courses
  • Discuss the current assessment plan for the program.
  • If applicable, discuss the adequacy of the general education curriculum for preparing students for upper level courses.
  • Compared to peer institutions, evaluate the currency of the curriculum including inclusion or exclusion of program requirements.
  • For undergraduate programs, to what extent do students have the opportunity to engage in high impact practices? Create Table 4.
  1. Student Learning Outcomes Assessment
  • Based on this current plan, what can faculty conclude about the quality of student learning? Please consider data you have collected for each of the last three years on the program’s student learning outcomes individually and as a unit.   Create Table 5, see Appendix A. 
  • Questions to prompt inquiry:
    • Is the standard for achievement set at an appropriate level?
    • Is the standard understood and agreed upon by all faculty?
    • Are there student learning outcomes where students perform particularly poorly or particularly well?
    • Are there student learning outcomes where the standard for achievement should be raised or lowered?
    • Where data are available, how do student compare to students in other USA programs or nationally?
  • What can faculty conclude about the ability of their students to progress through and graduate from the program?
    • Questions to prompt inquiry:
    • Do students exhibit difficulty progressing through the major at any point? If no, what are the things that faculty do that support student progression?  If yes, at what points and/or why? 
      • Do students exhibit difficulty completing the program? If no, what are the things that faculty do that support student completion?  If yes, at what points and/or why?   

TracDat>Documents>Document Repository>Program-Name>APR>Year>Tables 6 and 7; for programs submitting ACHE post-implementation reviews also include Table10

  • If there are areas of student learning identified as needing improvement (improving outcomes or raising standards), is modification/improvement of instruction identified as a possible means to improvement?
  • What can faculty conclude about the preparedness of their graduates to enter the workforce and/or continued post-baccalaureate education?
  • Questions to prompt inquiry: Consider the program’s student learning outcomes and how they translate to knowledge, skills, and/or abilities that make your students attractive to potential employers and/or graduate school TracDat>Documents>Document Repository>Program-Name>APR>Year>Table 8
  1. Program Resources
    • To what extent is the program utilizing technology to improve course/program delivery and/or enhance instruction
    • To what extent are facilities, space, technology, and other resources sufficient to support the program?
    • To what extent do students and faculty have access to the library resources and services that support the program? Also, please identify what resources and services may be lacking, if any.
    • Other important resource areas as identified by program faculty.
  1. Conclusion
  • Summary of significant conclusions drawn from the program review process to include identification of the program’s strengths, areas in need of improvement, and proposed goals and actions that will inform future program development.
  • Summary may be presented in the form of a SWOT analysis to include strengths, weaknesses, opportunities, and threats. Data/evidence, which should be detailed in the full document, must be used to justify conclusions. 
  1. Approval from Chair, Dean, and/or Provost/Sr. VPAA indicating self-study is ready for external review
  1. External review
  • Minimum qualifications for the external reviewer

Certified reviewer from a professional association or a distinguished colleague in the discipline with regard to teaching and scholarship.

Must not have a conflict of interest (e.g., current/former USA faculty, family member or business partner with any current departmental faculty, etc.).

  • Process for choosing an external reviewer
  • Prior to identifying potential reviewers, the Dean and Department chair should meet to determine whether the external reviewer will conduct a paper review only or also conduct a site visit. The decision to host a site visit is made by the Dean. At this meeting, the Dean and Department Chair also should clarify expectations for qualifications of the reviewer, expectations for the self-study, and expectations for the site-visit (if applicable).
  • Department chair and/or faculty identify three potential reviewers and submit curriculum vitas for the three individuals to the Dean.
  • The Dean makes the selection and notifies the department chair.
  • External reviewer report (See Appendix B for required outline and rubric)­
  • Additional areas of review and evaluation (beyond those covered by the rubric) are encouraged and welcomed as appropriate.
  • A sample itinerary is included as Appendix C.
  1. A plan of improvement based on the internal self-study and external review.
  1. Required Appendices
  • Chair and program coordinator/director curriculum vitae
  • Course syllabi or a sample of course syllabi for courses included in the curriculum map
  • Curriculum
  • Student/Program handbooks, if applicable
  • Assessment report for last three years
  • Other data, evidence, or other supporting documentation as applicable or needed to justify conclusions about programs effectiveness
▼   Data for Self-Study

The department and program’s annual assessment and data reporting should be sufficient to identify the three-year trend data needed to complete the self-study.  Please note that enrollment data is reviewed over a five year period. 

Outcomes and conclusions identified in the self-study and/or the external review should be used to develop subsequent annual assessment plans for programs, departments, and colleges/schools, as appropriate.  The components of the program review are uploaded into TracDat.

The program review document is entered into TracDat.  Instructions are in the Document Repository in TracDat.

▼   FAQ

What is the purpose of Academic Program Review?

  • Academic Program Review provides the University with an ongoing process for systematic assessment, planning, and improvements across academic departments. Academic Program Review contributes to overall planning efforts at the department, school, and university levels.

How is a department selected for review?

  • OIE sets a seven-year schedule of review for each department or program. The process is cyclical, with each department engaging in a review every five to seven years.

How long will a review take?

  • The review process will extend over three academic terms. The review process begins in the Fall term and complete the process the following Fall.

What does the Academic Program Review process involve?

  • The department will prepare a Self-Study Report, and a team of External Reviewers will visit the department and also prepare a report. These two reports will be used to produce a common set of recommendations and an action plan.

How do we carry out the self-study?

  • The Academic Program Review Guidelines provide an outline for departments to follow, but the review process does not rely upon specific or standardized criteria. Departments are expected to adhere to the program review timeline and general framework, but each department or program has the latitude to approach the self-study in its own way. However, because each review is unique, it is essential that the department, the Dean, and the Provost’s Office agree on the department’s plan for the self-study at the beginning of the process.

Are there any tips for the structure of the final report?

  • It is not necessary to answer every single question in the self-study guides. The questions are intended to prompt reflection and discussion among those conducting the self-study. Not all of the guiding questions will be relevant to each program. Follow the basic format (i.e., the main section headings), and use the questions as guidelines only. Remember that an Academic Program Review examines a program as it currently exists. Keep the report and appendices as succinct as possible and include a list of documents that are available on request or may be provided to the external reviewers on-site. 

Do I have to complete my annual assessment report for the program under review?

  • You do not have to submit an annual assessment report for year two of the review process.                                         
▼   Appendix A List of Required Data/Evidence for APR

All required data/evidence is available either 1) through your annual TracDat reports or 2) from the Office of Institutional Research or Office of Institutional Effectiveness.  Program faculty should feel free to include other data in addition to the required elements to support their decisions and/or conclusions in the self-study.  The University’s Instant Data Tool can be found at http://usadev.southalabama.edu/idt/.  In addition, if double majors are a component of the program enrollment and resource utilization, please run BANNER report ZSGR0152 for additional information.

Table 1: Student Credit Hours, Academic Years

Table 2: Student Headcount Enrollment Academic Years

Table 3: Number and Percent of Student Credit Hours Taught by Full-Time Faculty

Table 4: Participation in high impact practices.  Table may include high-impact practices and numbers of students participating or identification of courses in which the practice was used (last two years).  For courses, please list course name/number and the semester in which it was taught.  High Impact Practices for student success/engagement in undergraduate programs (not exhaustive). 

  • Service-Learning
  • Undergraduate Research (UCUR)
  • Learning Communities
  • First-Year Experience
  • Capstone Experiences
  • Internships
  • Team-Based Learning
  • Intrusive advisement
  • Study Abroad

Table 5: Program-Level Aggregation of Student Mastery of Student Learning Outcomes for the Last Three Years (also needed for ACHE post-implementation review).

Table 6: Cumulative course Success Rates

Table 7:  Retention and Graduation Rates, Program, College, and University

Table 8:  Program Level Aggregation of Post-Graduation Outcomes Assessment for the Last Three Years.  This may include one or more of the following (but not limited to) exit surveys, alumni surveys, employment rates, graduate school acceptance rates, etc.  (Also needed for ACHE post-implementation review)

Special Note:  For programs submitting ACHE post-implementation reviews in the year following program review, two other conditions also must be addressed; however, data for these two conditions is provided by the Office of Institutional Research.

Table 9:   Annual Average New Enrollment Headcount

Table 10: Annual Average Number of Graduates (Degree Completion)

▼   Appendix B Outline of External Reviewer’s Report and Student Learning Outcome’s Rubric
  1. Executive Summary
    1. Overall program quality with specific attention to fulfilling its academic and research mission
    2. Program quality (see 1.a) particularly as compared to peer institutions;
    3. Specific strengths or areas of excellence;
    4. Specific weaknesses or areas for improvement;
    5. Reviewer recommendations (if any).
  2. Overview of information, external to the self-study, used in the evaluation such as meetings, interviews, document reviews, etc.
  • Compared to similar programs (both in terms of numbers of majors and credit hour production), is the program adequately staffed for faculty to offer the curriculum and engage in service and scholarly/creative activity (see Faculty Workload Faculty Handbook section 4.3.1)?
  1. Compared to similar programs, are physical facilities (e.g., classrooms, office space, labs) adequate to support the academic and/or research mission of the program?
  2. As applicable, please highlight any other resource needs.
  3. Compared to peer programs and/or to expectations from the professional/disciplinary association, please evaluate the currency of the curriculum including the inclusion or exclusion of program requirements such as capstone/internship experience, portfolio, thesis, etc.
  • Compared to peer programs and/or to expectations from the professional/disciplinary association, please evaluate the program’s integration of online education.
  • (For Graduate Programs) Compared to peer programs/and or to expectations among professional/disciplinary associations, please evaluate the extent and quality of research activity among faculty and students.
  1. Please evaluate the quality of student learning outcomes (SLOs) and methods of assessment used by program faculty. Your narrative should address the following:
    1. Do the SLOs clearly state what student outcomes are expected?
    2. Do the SLOs clearly map back to the curriculum (e.g., are the SLOs that are not taught or assessed anywhere in the program’s curriculum)?
    3. Are the SLO’s current/relevant as compared to similar programs’ student expectations?
    4. Are the SLO’s directly assessed (e.g., test, paper, etc. versus student’s perception of what learning occurred)?
    5. Are the assessments aligned with expected student outcomes with regard to relevancy, rigor, and timeliness?
    6. (For Graduate Programs with a Related Undergraduate Degree ) Do the SLO’s demonstrate post-baccalaureate rigor (i.e., SACSCOC Comprehensive Standard 3.6.1)?
  2. Based on your review of information and your evaluation, please provide recommendations for future actions such as might be included in a departmental or program strategic plan. Suggested “categories” for recommendations include but are not limited to student success (e.g., retention, graduation), student learning outcomes, extramural funding, curricular design, and teaching pedagogy.
  3. Based on your review of information and your evaluation, please provide recommendations for future action that the administration (departmental, college, or university) should consider. Suggested “categories” for recommendations include but are not limited to physical facilities, faculty/support staffing, and professional development.
▼   Appendix C Sample Itinerary for the External Reviewer

30 minutes with the Provost

30 minutes with the Dean

30 minutes with OIE

Recommend scheduling time with

  • Faculty (individually or in a group setting)
  • Students (individually or in a group setting)