The TLSS Evaluation Framework
In addition to implementing a systematic data collection, analysis and reporting process, a standing goal of the Teaching and Learning Support Service (TLSS)’ evaluative framework is to examine and enhance varied facets of its work. Based on what these data indicate regarding trends in campus teaching and learning, opportunities for future innovation, visioning and strategic planning may emerge.
Note: By placing the cursor over the symbols above, each set of indicators will appear.
All while integrating existing data tracked by individual sectors within the service (predominantly data related to participation or use of service); the framework has provided direction for the collection of data from multiple sources to be used in complementarity to report on a series of indicators. Using an adaptation of the Theory of Change (Hart et al., 2009) and Guskey’s (2002) Model of Teacher Development, the figure below outlines different levels of indicators in three interdependent groupings.
The first grouping, in blue, represents an increasing level of sophistication in the examination of impact. Beginning with transformation of use and satisfaction with specific services, followed by transformation of thinking and conceptualisation (in terms of knowledge, skills, attitudes and values) of those using TLSS services and resources, and thirdly transformation of practice as exemplified by changes in instructional approaches and engagement in the examination of their effectiveness.
The second grouping, in orange, represents the subsequent impact of the multiple individual levels of transformation (in blue). For instance, how a particular instructor’s transformation in thinking and practice might have an impact on the learning of their students or the thinking and practice of some of their colleagues.
The third grouping, in red, indicates transformation in the culture of teaching and learning at the meso-level (department/faculty) and macro-level (institutional). This transformation may be influenced by the direct input and advocacy of the TLSS or indirectly through the collective impact of an increasing number of individuals who have transformed their thinking and practices (individual level transformation in blue).
Excerpts from Data Collected Winter/Spring 2018
Coming soon ...
E.g. The initial grant given to us by the TLSS gave us the opportunity to rethink how we deliver content and engage students in classes (large or small). Is has opened new teaching and grant opportunities and allowed us to develop new ideas on how to engage our students or deliver content that could benefit our students. It stimulated us and gave us the opportunity to think differently about our classrooms, our students, and reflect on our teaching skills and experience in ways we would not have otherwise
Systematic Data Collection
The development of an evaluation framework is the first part of an effort by the TLSS to systematically evidence the nature and extent of the impact that its services and initiatives have on instructional practices, on student learning, and on the culture of advancement and innovation in teaching and learning. It is intended that this practice become a habit as it increasingly becomes embedded in the educational and media development work of the service.
The Evaluation Process
During this first data analysis and interpretation process, methods to streamline the format used to collect and store data, along with methods to more accurately and efficiently sort and analyse data, are being noted for further development. This first iteration is already proving to be helpful in terms of determining what criteria will we used to prioritise the indicators that we chose to track and the data sources that we use to evidence them.