Outcome assessment of the online English learning and management system : A project on educational technology application to English learning in Taiwanese polytechnic higher education

This paper presents the result of a three-year action research plan involving the innovative application of “LiveDVD”, a web-based English learning system loaded with VODs and embedded into the English learning program of a polytechnic university in Taiwan. Over the implementation period, the LiveDVD action plan essentially involves use requirement on students along with creation of learning and assessment materials for the target VODs and outcome management. Outcome assessment indicators include (1) the association between use activeness of LiveDVD and pass status of English proficiency certificates and (2) the causal relations between LiveDVDrelated variables and English learning outcome. A survey questionnaire on LiveDVD and the management platform were employed to collect relevant data. The chi-squared test of group independence and an SEM-based path analysis were performed to address the outcome indicators respectively. The findings of the project include: (1) use of LiveDVD is positively and significantly associated with the pass status of English proficiency certificates and, (2) use motivation has direct effect on English proficiency and create the significant mediation effect on English proficiency certificates. This project yields significant implications especially for vocational higher education in Taiwan in the innovative application of educational technology to English education coupled with an effective outcome management model.


Introduction
Vocational higher education has hardly been the government's top priority, and schools in this system have long been criticized by policymakers and EFL teachers for having undesirable English learning standards.In a formal school English learning setting, two critical but long neglected elements conducive to successful English education at polytechnic universities are, arguably, an effective learning tool or program and outcome management.Lack of learning motivation of students, especially those of vocational and technological universities, has long been claimed to cause undesirable outcomes of English education in Taiwan.In a technology-friendly age, language learning enhanced and aided by technology has been a vital issue in ESL/EFL teaching and learning.How to best utilize education technology to enhance English learning becomes particularly significant in terms of increasing, on a large scale, the learning motivation of low achievers in the Taiwanese vocational higher education.Teaching approaches geared towards initiating self-directed learning via technology-mediated support with a view to enhancing learning motivation have hence been gaining ascendancy in English learning settings across different educational levels in Taiwan.The purpose of this paper is therefore to document the process, method, management and result of embedding the technology enhanced learning aid to the English education of Southern Taiwan University of Science and Technology (STUST) as a long-term action research project.

LiveDVD Online Film-based English Learning and Management System
LiveDVD is characterized by a series of learner-friendly and helpful features along with a powerful management module.The system platform can accommodate 600 concurrent users to access as many as around two hundred VODs on the LiveDVD server.Students connect to the system platform to watch the uploaded films, learning English by using a series of functions, including (1) display of multi-mode film subtitles (e.g.English-Chinese subtitles), ( 2) key word search and playback of movie clips, (3) the on-line dictionary, (4) collection of movie lines and words, (5) vocabulary coverage based on the TOEIC word lists, among others.
The LiveDVD system comes with a management platform, or module, known as WEBMEN, specifically for school administrators and teachers to collect and analyze data of students' learning by LiveDVD.The main function of the platform is to establish students' learning profiles and track their learning hours, movies watched, practice tests completed, and specific features practiced.All types of learning data can be exported for statistical analyses and assessment for administrative, teaching and research purposes.
The strengths of LiveDVD essentially relate to employing multimedia to intrigue learners and engage them in learning English.Multimedia provides variety and excitement to a computer-supported teaching and learning environment, adapting instruction to diverse learning preferences of students (Zaidel & Luo, 2010).Multimedia materials such as DVDs and VODs as effective self-directed learning aids enhance students' comprehension and memory, increase their motivation, and promote their concentration on the content in a near natural environment (Astleitner & Wiesner, 2004;Deimann & Keller, 2006;Guariento & Morley, 2001).Multimedia has been widely investigated for language learning and teaching (Astleitner & Wiesner, 2004;Mayer, 1997;Synder, 1988;Weyers, 1999).Several studies have found that a combination of audio and visual media aids and sequential or simultaneous use of a variety of media formats in a given presentation or self-study program can facilitate language learning (Edasawa, Takeuchi & Nishizaki,1990).As noted by Willberschied & Berman (2004), learners achieve a better understanding of a given combination of audio with texts, which is dynamic, immediate and accessible, and helps learners gain broad access to oral communications audiovisually.Moreover, interactive video or audio materials have changed students from passive observers to active participants (Chavez, 1998;Goh, 2002).Technology aided teaching featured by both authenticity and uniqueness is capable of creating sufficient information to provide learners maximum linguistic and cultural input, and increasing learners' motivation (Stempleski & Arcario, 1992).The different aspects of authenticity created by multimedia technology are facilitating to language learning (Rost, 2002), for they provide rich input for the EFL learning environment by integrating phonetic, syntactic, semantic, pragmatic, and sociocultural features.

LiveDVD Learning Action Plan
The LiveDVD action plan involves implementation of the learning regulations embedded into the required English course and creation of extended learning materials for selected VOD films.As far as learning regulations are concerned, a minimum learning-hour-persemester requirement is set for every participant of the required English course to ensure that students use LiveDVD as expected by the school.Students are also required to view at least one VOD film on LiveDVD every semester, and are tested at the semester end on the designated film as an estimate of their learning achievement of the film.As a graduation requirement, students must also pass an English ability threshold indicated by a variety of English proficiency certificates (e.g.TOEIC, BULATS, etc.) recognized by the university.Additional learning materials are also created for 20 VODs (e.g.A Christmas Carol, Slumdog Millionaire, etc.) uploaded on the LiveDVD platform.The developed materials include (1) film-based English word glosses, (2) English and Chinese learning guides, and (3) self-assessment materials.The management platform, furthermore, can track the Outcome assessment of the online English learning and management system learning status of students and alert them to catch up with their learning whenever it falls short of expectations.

Methodology
To evaluate students' learning outcomes and processes associated with LiveDVD, two indicators are focused: (1) association between usage hours and pass status of English proficiency certificates and (2) relationship among the variables underlying the LiveDVD learning.
For the first outcome indicator, a Chi-squared group independence test was performed using R package v.3.1.0(R Core Team, 2014) to test the independence of the variables of LiveDVD use level and the certificate pass status as an indication of whether the degree of LiveDVD use is statistically associated with whether students eventually acquire the English proficiency certificates.Two categorical variables are posited, Use Level (5 levels) and Pass Status (2 levels), and a five-by-two contingency table is generated (v.Table 1) for the Chi-square analysis to test the independence of the two variables.
Another outcome indicator is the relationship between the observed variables on LiveDVD (i.e.use motivation, learning features, and learning regulations) and outcome variables (i.e.English proficiency, and pass status of English proficiency certificates) as an indication of the effect of LiveDVD on the English learning.For this purpose, the observed variable and SEM-based path model was put forward, which consists of six exogenous and independent variables, namely, Group, Year, Learning Features (LF), Learning Regulations (LR), Use Motivation (UM), and English Proficiency (EP), and five endogenous and dependent variables, namely, LF, LR, UM, and Certificate Attainment (CA).To test the proposed model, a valid sample from 1739 students was obtained through the administration of the questionnaire QLDVDL with 7-point Likert items, and the OV-based path analysis was conducted using LISREL 9.1 to yield the results.

Chi-squared test of group independence
Table 1 presents a five by two contingency table comprising the two categorical variables.A Pearson's chi-squared test for group independence was performed using R 3.1.0,returning a chi-squared value of 90.110 (df = 4) and an extremely small p-value (p = 2.2e-16), rejecting the null hypothesis of independence of the two variables, and suggesting that whether the students obtained English proficiency certificates is significantly associated with how actively they use LiveDVD in English learning.Figure 1 reveals furthermore the Huang, Da-Fu interaction between the two variables, indicating the tendency of the ever-increasing passing rate of English proficiency certificates with the increasingly active use of LiveDVD, or when the use of LiveDVD gets more active, the passing rate of English proficiency certificates would get higher and the no-pass rate gets lower.

SEM-based path model
The test results of the proposed path model are shown in Table 2, with the model fit statistics found to meet the commonly accepted criteria (Bagozzi andYi, 1988；Hair, Black, Babin, Anderson andTatham, 2006), indicating that the proposed model is not significantly different from the expected model, and suggesting that the proposed model is fairly fit for the input data.Outcome assessment of the online English learning and management system 2, in which the solid line indicates a statistical, or statistically significant, relationship, while a dotted line refers to a non-statistical, or statistically non-significant, relationship between two observed variables.

Conclusion and Implication
This paper presents the result of the action research project featuring the LiveDVD online English learning system embedded into the English education program of STUST aiming to promote students' learning motivation and upgrading their English ability.Central to the project is the LiveDVD action plan embracing learning regulations and learning of supplementary materials.To assess the 3-year learning outcome, analyses were made to understand association of use activeness and pass status of English proficiency certificates, and relationships between LiveDVD-relevant and learning outcome variables.
The activeness of using LiveDVD was found to be statistically associated with the pass status of English proficiency certificates, suggesting that the more actively LiveDVD was used, the more likely students would be able to pass the English proficiency certificate tests.The testing of the proposed SEM-based path model identified Use Motivation as a full mediator variable between Learning Regulations and Learning Features and English Proficiency, indicating not only the direct effect of Use Motivation on English proficiency, but a significant indirect effect of Learning Regulations and Learning Features on English Proficiency, which, in turn, has the full mediation effect on Certificate Attainment.This result has significant implication for the importance of perfecting learning features and improving learning regulations to motivate students to enhance their English ability by using LiveDVD, and ultimately to obtain English proficiency certificates.
This project sets a successful example of applying LiveDVD for English learning and teaching purposes in a vocational higher education context typically characterized by a large student population with overall mid or low English entry ability and learning motivation.The outcome of this project shows that the technology enhanced teaching or learning approach via the learning instrument such as LiveDVD can make a positive change in students' learning motivation and improve their learning outcome.These positive outcomes create avenues for further development on the EFL teaching and research fronts.More significantly, this study engenders a valuable empirical model demonstrating how a better English education can be achieved in vocational higher education through innovative

Figure 1 .
Figure 1.Association between LiveDVD use activeness and English Certificate pass status

Figure 2 .
Figure 2. Path coefficients of the proposed model The path coefficients of the path model are presented in Figure LR, UM, EP and CA was performed, and two regressions were conducted using EP as a dependent variable, and UM/LR and UM/LF as predictor variables.With the introduction of Motivation into the path model, the statistical correlations between Motivation and Feature and Proficiency are established, but the statistical correlation originally existing between Feature and Proficiency becomes not statistical, demonstrating that UM functions as a full mediator variable for LF and EP.The same role of a full mediator variable is true of UM in the path model where LR and EP are fully mediated by UM.