A framework for the evaluation of educational development programs in higher education in Chile

This paper outlines the experience of evaluating the impact of educational development in Chilean higher education drawing on the example of the Universidad Católica de Temuco. The aim is to demonstrate the importance of a implementing a robust and flexible evaluation and impact framework to identify the effectiveness of education development programs. The rationale and processes that informed the development of the evaluation and impact framework are described and then illustrated with one example, the Faculty Learning Communities (FLC) program. The example shows how the overall framework is contextualised in a specific program, drawing on indicators and outcomes to demonstrate its flexibility and robustness. The rich evidence gathered has been used to inform the educational developers on the effectiveness of their work, and the faculty participants on their knowledge and practice. Just as importantly, it has informed the institution about the impact of the programs and student engagement. The evaluation framework provides a Chilean example informed by international best practice.


Introduction
A developing trend in Higher Education Institutions is to seek measures of impact.This is seen in the assessment of research, such as the extensive research assessment reviews carried out in The UK and Australia, but is becoming apparent in educational development.Yet there is no consensus about what constitutes quality (Harvey & Mason, 1995), or the nature of impact (Land, 2004).However, a strong argument has been made that "impact" needs to be understood as "evidencing value" (Bamber & Stefani, 2016), integrating measurement and experience when defining the object of evaluation and the methodology.
A related issue is the use of institutional level performance indicators of effectiveness and impact.The significance of educational development work risks becoming invisible because broad level indicators may not provide evidence of institutional enhancement.Bamber and Stefani (2016) suggest the use of a situated and nuanced approach.Educational development itself is an example of the interplay between individuals and their environments (Knight, Tait & York, 2007;cited in Hoessler, Godden & Hoessler, 2015).
The task of measuring the effectiveness and impact of educational development has long been recognised (Kreber & Brook, 2001;Kirkpatrick, 1998;Chalmers et al., 2012;Stes et al., 2007), though traditional approaches to the evaluation of the diversity and flexibility of educational development have been considered weak.Hoessler et al. (2015) have argued for a holistic approach that embraces analyses that are messy, however, it is unlikely to meet the corporate needs of institutional reporting.

Chilean higher education context
The Chilean Ministry of Education has provided funds to improve higher education institutions" capacity to implement curricular reform.One of its main strategies was the creation of centres with a focus on Teaching, Innovation and Technology (CINDA, 2009).
In 2007, The Universidad Católica de Temuco (UCT) became pioneer with the establishment of the Teaching Development and Innovation Center -Centro de Desarrollo e Inovación de la Docencia, (CeDID) (Pey & Chauriye, 2011).UCT is located in the south of Chile and serves one of the poorest regions of Chile, including a significant Mapuche population.
In common with the few centres that were also established at this time, CeDID"s functions were to support the development of the new educational curriculum model and train faculty members in student-centred teaching strategies (Universidad Católica de Temuco, 2007).

Rationale for developing an evaluation and impact framework
The Chilean Higher Education Presidential Advisory Committee (Reich, et al., 2011) established a need for measurable targets.However, as a new Centre, a comprehensive framework to evaluate the impact of these programs had yet to be established.CeDID"s measures used between 2007-2010 were centred on numbers of staff and students participating, satisfaction ratings of students and staff, and other largely anecdotal sources (Chalmers et.al., 2012;Kreber & Brooke, 2001) The need for a robust evaluation framework for educational development programs was recognised, but the lack of Chilean examples led CeDID to develop a framework informed by international best practice.By 2013, CeDID there were five Faculty Learning Communities (FLC) with the common purpose of transforming the teaching and learning process.Each FLC used a different approach and this diversity presented challenges to identifying meaningful ways to evaluate impact.The traditional quantitative performance indicators were insufficiently nuanced to capture the qualitative changes taking place.
The development of the CeDID evaluation framework began formally by the end of 2013, building on the work of an Australian project (Chalmers et al., 2012) which had already established the theoretical underpinnings of a framework.Over a period of four years, CeDID extended Chalmers" work (Chalmers & Gardiner, 2015) to ensure that it applied to all CeDID"s educational development activities and disciplines (Turra & Moya, 2016).
At UCT, FLC processes start by analysing the situational factors of their courses (Fink, 2003), searching for, implementing, and evaluating new suitable teaching and learning strategies.In collaboration with CeDID consultants, faculty members developed teaching practices and resources and trialled these with their students.
As the implementation concludes, FLCs demonstrate their concern about evaluating the implementation of the teaching initiatives through research, which aligns with one of the goals of FLCs identified by Arthur (2016), such as the production of knowledge, in this case, oriented towards the teaching and learning enhancement in university settings.
CEDID extended its educational development evaluation framework over the following four years to ensure that it could be flexibly applied to all of its programs, and particularly those located in the disciplines (Turra & Moya, 2016).

CeDID's methodology to develop its evaluation and impact framework
The flexible yet robust evaluation framework approaches the diverse needs of:  Faculty to make judgments about their teaching;  CeDID to evidence the impact of their programs;  The University to inform its attainment of its planned goals; and finally  The Ministry of Education on the effectiveness and impact of the funded programs The CeDID Evaluation Framework drew on Guskey"s five-level model, which identifies where educational development programs can demonstrate impact (Chalmers & Gardiner, 2015).These are 1) Teachers" reaction to the development program; 2) Conceptual changes in teachers" thinking; 3) Behavioural changes in the way teachers use the knowledge, skills and techniques learners; 4) Changes in organisational culture, practices, and support and; 5) Changes in student learning, engagement, perception, study approaches.
There are four types of quality indicators for each level.These indicators have been designed to evaluate the effectiveness of a specific programs" practices and processes.These also identify any changes or outcomes (Chalmers & Gardiner, 2015).Together, they provide a "comprehensive picture of the quality of teaching and learning activities" (Chalmers & Gardiner, 2015).
The different types of indicators (input, process, output, and outcome) (Chalmers, 2008) and sources of evidence (Chalmers & Hunt, 2016) were organised into CeDID"s evaluation matrix.This matrix also guides data analysis and evaluation, and frame the outcomes and mechanisms (Bamber & Stefani, 2016).
The logic and principles used to develop the overall framework have been applied to the FLC program to demonstrate how it can be used to accommodate the specific goals and contexts of different programs.(Contact Author 1 for full details of the CeDID evaluation framework matrix of indicators of quality and impact).

Impact evaluation of the Faculty Learning Communities (FLC) program
The evaluation framework for the FLC identified that changes in Faculty reaction and changes in practice were critical areas to evaluate and so finding meaningful indicators of these was important to identify.

Change in Faculty reaction
Input, process, output, and results indicators were identified, including suitable instruments and data that would need to be collected (see Table 1).The information collected in this category helps the institution and individuals understand how faculty responds to the different transformation initiatives.The number of people involved in learning communities serves as a general indicator for UCT authorities and MECESUP to account for the efficient use of the resources.
Indicators such as the results from the Approaches to Teaching Inventory (ATI) and the procedural ones contribute to the improvement of the programs (FLCs and certificate program).As an example, to assess disposition to student focused-teaching, FLC faculty completed the validated ATI (Trigwell & Prosser, 2004).This instrument served as a diagnostic tool (input) that allowed CeDID"s professionals to analyze possible difficulties.
The results showed that the teachers" approaches were skewed towards the Information Transfer / Teacher-focused approach.This data was necessary for the consultants and the teachers themselves to be aware of when approaching the course re-design process.
Working with teachers who assume students learn by "being told about things" (Boore & Deeny, 2012, p.127) provide the consultants with courses of action that need to be undertaken.Examples of these are the reflection of the teaching practice, analysis of beliefs towards teaching, and mediation towards a student-centred approach to teaching.
The value of using the ATI extended beyond its initial use in informing the professional and curriculum development programs.When re-administered, it helped consultants identify whether there have been changes to the teaching approaches.

Change in Faculty practice
Changing teacher-centred practices to student-centred teaching has proven to be difficult to achieve.Teachers may change their understanding to being more student-centred, however, changing their teaching practices is not an automatic progression (Chalmers & Gardiner, 2015).Evaluation frameworks, therefore, need to include indicators that will identify and track changes in teaching practices (see Table 2 below).The Academic Performance survey is an instrument applied to students at the end of every semester and has a focus on faculty performance.Although this is students" perception, it is an indicative record of past teaching performance and provides a general sense of common teaching, before and after course transformation.Similarly, the ATI results show whether faculty implement student-centred learning strategies.
The indicators target the dissemination of the innovation through academic work and how the current programs impact on courses outside the scope of FLCs.Faculty Learning Communities have been a critical educational development strategy for building faculty teaching capacity and curriculum transformation at UCT.The monitoring of these indicators has allowed the identification of an incipient FLC practice cycle.In this sequence, the results obtained by the first FLCs have been instrumental in generating interest in other teacher groups.Also, experienced FLCs have obtained access to institutional funding, thus facilitating the permanence and scaling of the initiatives.

Conclusion
In this paper, we have described the rationale and processes of developing an evaluation and impact framework on the effectiveness of education development programs that inform participants, institutions and the sector, contextualised in one Chilean university and illustrated using one example of a complex educational development program.The example shows how the overall evaluation framework can be flexibly and robustly contextualised through identifying specific indicators and outcomes.