Program Review at the University of Ottawa

The strategies, resources, and tools used at the University of Ottawa in the context of cyclical program review and curriculum development have evolved quite a bit over the last 5-6 years. The basis of the pedagogical support we use today at the Centre for Innovative Pedagogies and Digital Learning (CIPDL) was initially developed by Bob Parson. Since 2012, many practices including the SWOC/SOAR analysis focus groups (for faculty and students) and curriculum analysis data collection now have an online option. As such, we have developed surveys, templates and guides to assist programs with the collection and analysis of data which will help them describe the structure and function of their program. We outline a brief snapshot of our process and tools below.

Each academic program at the University of Ottawa must follow the protocol for cyclical review every 7-8 years. Consisting of an initial self-evaluation of the program, internal recommendations by a Senate committee, external review and a final recommendation report, the timeline is structured as follows:

Figure 1: University of Ottawa timeline for the cyclical review of undergraduate programs

What is Curriculum Mapping?

Curriculum mapping is the visual representation of the underlying logic of curricular design for a given program (Maki, 2004). In higher education, it is most frequently used for two main purposes:

1. To ensure the alignment and sequencing of learning outcomes and assessments across courses when developing a new program; and
2. To evaluate the current alignment and look for any gaps, redundancies and inconsistencies in order to enhance an existing program (Uchiyama & Radin, 2009; Kopera-Frye, Mahaffy & Svare, 2008).

   Curriculum mapping is “a deliberate process of curriculum deconstruction in order to understand better how the sum of the parts relates to the whole” (Jackson, 2000, p.144).

Curriculum Mapping in Five Stages

An evidence-based approach to collaborative curricular enhancement can commonly be characterised by five stages:

Figure 2: Five stages of curriculum enhancement

Stage 1 – Data Collection and Analysis

Using a Canadian based internet survey tool (Fluidsurvey), instructors identify which program learning outcomes are covered by their courses, at which level these outcomes are presented (introductory, intermediate, advanced), how the outcome is covered (taught, practiced, assessed), and if assessed, by what means.

Figure 3: Screen capture of the learning outcome section of the survey

Figure 4: Screen capture of the evaluation methods section of the survey

Stage 2 – Interpretation and Discussion

Once the course information is collected, tables can be generated within a spreadsheet to better visualize the program’s underlying framework. Current tables include the integration of learning outcomes across program courses, assessment methods used throughout the program, grade weight and distribution of assessments across the program, favoured instructional approaches, etc.

Figure 5: Sample learning outcome distribution table

Figure 6: Sample assessment distribution table

Stage 3 – Identification of Areas for Improvement

As a catalyst for reflection and interdepartmental discussion, a series of questions are provided (Kenny, 2014). These include:

• What learning outcomes are most/least emphasized?
• Where are the strengths and gaps in teaching and assessment across the program?
• Do the instructional and assessment methods used best align with the intended learning outcomes?
• What instructional/assessment strategies are most/least used?
• Are the instructional and assessment methods used in the courses congruent with the discipline and the program’s/Institution’s mission/vision?
• In terms of supporting student learning, how well are the selected/utilized instructional and assessment methods actually working?

Stages 4 and 5 - Development of an Action Plan and Implementation

An action plan is most effective when it is faculty driven, data informed and supported by curriculum design specialists, and includes specific responsibilities and timelines as well as mechanisms to sustain the curriculum review and enhancement process by systematically revisiting the five stage cycle (Wolf, 2007).

Resources / Curriculum Analysis Templates

Undergraduate programs

Graduate Programs

________________________________________________________________

Seamless Data Collection: uoSyllabus

As these practices evolve at the Centre for Innovative Pedagogies and Digital Learning (CIPDL), the questionnaire continues to grow and become more robust. Of particular interest is how to collect information regarding student achievement of program learning outcomes and track progress on a continuous basis.

A tool called uoSyllabus being tested in a pilot phase may help standardise the collection of this information. uoSyllabus is an online module allowing instructors to produce syllabi that conform with the rules outlined by Senate and offer the possibility of adding content which aligns with best practices. Once the information is entered, the module produces a syllabus in a PDF format.

The module offers the possibility to specify which learning outcomes, accreditation standards or attributes are covered by each course within a program, thus facilitating data collection for various accreditation reports which many programs (including programs in professional schools) are subject to. The module equally enables curriculum design specialists to continuously collect information, thus facilitating the analysis of curricular alignment for the purpose of program evaluation.

uoSyllabus is now being tested by four professional Masters programs from the School of Rehabilitation Sciences: Audiology, Speech-Language Pathology, Occupational Therapy and Physiotherapy. All programs have accreditation standards that are specified in uoSyllabus, thus making it possible to identify how these standards are met by program courses.

List of uoSyllabus Advantages

List of Pre-Formatted Data Accessible Through the Control Panel

Upcoming Development of uoSyllabus

• All information regarding PLOs will be centralised under one tab (Curriculum Tab) allowing for the collection of more refined data about how they are taught and evaluated.
• Ability for faculty to enter the average grade for each course assessment.
• Faculty members will be able to enter the average result obtained for each evaluation (giving us a first look at which PLOs are most or least achieved by students).
• Through a central system, we will be able to link the mean course grade to PLOs
• Manager access for administrative staff in departments will allow one to:
• Add user / prof
• Ability to see and edit all course syllabi for their programs
• Add program learning outcomes
• Add instructional approaches
• Add assessment methods
• Two types of access to uoSyllabus: Regular mode and Course Design mode.       

Resources / uoSyllabus 

Web Page

User Guide

________________________________________________________________

Online SWOC Analysis

In this two-hour session, participants identify the elements that are most important to their stakeholders and take part in an enriching dialogue amongst peers, discussing the program in a constructive fashion. As well, participants produce concrete suggestions on what could or should be done to improve the program.

In previous SWOC workshops for programs under review, participants indicated that the SWOC :

• Created a concise vision of the program
• Brought new ideas to the discussion
• Generated active participation
• Encouraged recognition of what was positive about the program
• Confirmed common goals and aims

By invitation, the CIPDL will facilitate a SWOC analysis discussions with full time/part time faculty or students and provide a written report of findings. A meeting space may also be provided by the Centre for Innovative Pedagogies and Digital Learning (CIPDL).

For certain programs, reaching students/faculty can be challenging.  Increasingly, programs are choosing to move this process online and thus gather even more input from students and Faculty members, using an online SWOC to ask questions that normally are not part of a SWOC analysis.

Resources SWOC / Templates and Examples

Generic SWOC survey

Modified SWOC for a Master's Program

________________________________________________________________

References

Jackson, N. (2000). Programme specification and its role in promoting an outcomes model of learning. Active Learning in Higher Education, 1(2), 132-151.

Kenny, N. (2014). Curriculum Mapping. Retrieved from http://www.uoguelph.ca/vpacademic/avpa/outcomes/curriculummap.php

Kopera-Frye, K., Mahaffy, J. & Svare, G.M. (2008). The map to curriculum alignment and improvement. Collected Essays on Teaching and Learning, 1, 8-14.

Maki, P. (2004). Assessing for learning: Building a sustainable commitment across the institution. Sterling, VA: Stylus.

Uchiyama, K.P. & Radin, J.L. (2009). Curriculum mapping in higher education: A vehicle for collaboration. Innovative Higher Education, 33, 271-280.

Wolf, P. (2007). A model for facilitating curriculum development in higher education. New Directions for Teaching and Learning, 112, 15-20.