The TLSS Evaluation Framework
The Framework Development Process
A participatory approach was adopted from the work of Cousins and Earl (1995) to ensure meaningful engagement across service-level stakeholders. This process allowed for a more comprehensive mapping of the services being tracked along with the development of ownership in the process by members of the TLSS team. To guide stakeholder engagement, a reflective worksheet was created (Click here to view a copy). The structure of questions borrowed from both the Theory of Change (Hart et al., 2009) and Guskey’s Education Impact Framework (Guskey, 2002).
Development work drew on the contributions of evaluation scholars such as: Amundsen and Wilson (2012), Chalmers (2015), Guskey (2002), Hines (2017), Kirkpatrick (1994), Kreber and Brook (2001).
Types of Data Sources
A series of data collection sources, instruments, and processes were put in place to collect evidence associated with each indicator. These included the following during the first data collection phase in winter/spring 2018. For multiple data sources, various informants (including instructors, students, administrators, and TLSS team members) provided feedback in terms of expected, perceived, and experienced impact in an effort to aid in data triangulation.
Next to many of the data sources below are examples of data collection instruments that we have used. Please feel free to use and adapt these and to let us know how it goes. We would be happy to send these to you directly and collaborate on projects regarding their use and enhancement (Contact: Jovan Groen at ).
- Descriptive statistics regarding use of services
- General consultation statistics and tracking of insights (Consultation Tracking Strategy)
- Online project and resource development exit interviews (Sample Questions)
- Targeted professor/student surveys (regarding a particular pedagogical practice, use of a specific instructional tool or a specific learning space)
- Targeted Course Initiative Survey Templates
- Learning Management System Survey Template
- Lecture Capture Survey Template
- Videoconferencing Survey Template
- Active Learning Classroom Survey Template
- New professor orientation program follow-up survey (Sample Survey)
- TA Orientation program follow-up survey (Sample Survey)
- TLSS grant recipient follow-up survey (Sample Survey)
- Online course build student surveys
- Class observations (Sample Observation Form 1 and Form 2)
- Anecdotal feedback