An empirical evaluation of e-learning usage in the higher education context

E-learning has been adopted for several years in Greece and abroad, and it is considered an integral part of blended learning. E-learning systems accumulate a vast amount of data which may be very valuable. The educational organizations may exploit the power provided by e-learning, if they analyze the usage and the content of the courses. An early assessment of the of e-courses use may provide useful information to the educators, in order to make educational interventions in their teaching material. This study suggests that the evaluation of e-learning usage may be carried out with the assesment of variables and metrics related to teacher training material and student trafficking. We propose three metrics which are combined efficiently, in order to quantify the quality characteristics of the courses and offer useful insights about the educational material and e-learning usage. This case study was implemented in the e-class platform of a Greek Higher Education educational institute. This platform created by the Greek Universities Network (GUNET) is very popular in Greece, since the majority of the Greek universities have adopted it. The results of our study confirmed the validity of our suggested approach, and highlighted the need for a more learnercentered focus and active participation of the students.


Introduction
E-learning systems can be used and adopted by academia, as well as by private and public organisations. Nowadays, there are many different types of e-learning systems which may support various styles of learning such as: individual, collaborative, content management, activity management, formal, informal and workplace learning.
An important goal of e-learning is that it should be equivalent or better to the learning provided through other methods, such as the traditional face-to-face and classroom-based teaching. E-learning has various advantages, since it allows learners to access the teaching material on a 24/7 basis at any place, facilitates an interactive and student-centered learning (Martínez-Caro, 2015) and enables the instructor to monitor the learners' progress continuously (DeLima, 1999). However, e-learning may have various disadvantages such as limit to the role of instructors, to allow inappropriate use from the learners (e.g. cheating, plagiarism), etc. (Arkorful & Abaidoo, 2015).
For several years, the educational institutions have adopted e-learning systems since they may: i) provide their services with a low cost and a high quality standard (Selim, 2007), ii) provide asynchronous learning to those who have working or family commitments and iii) enhance their presence and become more popular (Shen et al., 2008).
E-learning systems must be usable in order to be accepted by academic users, regardless of their background and experience (Harrati et al., 2016). Some researchers Hogo (2010), Hassanzadeh et al. (2012) have suggested metrics and statistical methods in order to assess the usability of the e-learnig systems.
The aim of this study is to evaluate the usability of an e-learning system, in order to understand the users' level of satisfaction and suggest the ways to improve it.
We analyze the log data from the eClass platform used in the Accounting and Finance department of the Eastern Macedonia and Thrace Institute of Technology (TEI EMT). In order to evaluate the usability level of the platform, we estimate various usability variables and metrics suggested by the authors in their previous research (Valsamidis et al., 2014).

Approach
This paper evaluates the e-learning platform usage at the Accounting and Finance department of TEI EMT. The evaluation method that was adopted was based on indexes and metrics that are analysed over the next sections. This data includes all the online courses regardless their type (Mandatory or Optional) and their category (Theory, Laboratory or Mixed). In addition some other attributes are used in this research such as the course code and semester, the course files size, number and the duration of students visits.
The next two steps of the adopted approach are the calculation of the indexes and the computation of the proposed metrics. These stages are also the basis of the e-learning system evaluation, which is the goal of this work. In particular, we describe the Average Files Size, Visits Per Duration, Course Usage and User Perception metrics. After the calculation of these metrics we can conclude on specific outcomes regarding the usage of the e-platform from the students.
For each course we define ten attributes which are presented in Table 1. The above attributes are selected due to certain educational goals. The main goal is to examine the factors that may affect learners' activity and are divided to three main categories (i) course primitive such as Semester, Type, Category, (ii) course content such as Files and Size and (iii) course usage by learners such as Visits and Duration.
Based on the attributes of table 2, we can compute the following metrics that reflect users' behaviour related to the educational material.
Average Files Size (AFS) which reflects the contents of the courses in the e-learning platform. It is a metric that expresses the average size of files related to the number of files per course. The results of this metric reflect how the volume of uploaded files in each course is shaped. Usually, the larger the volume of data, the higher the traffic, but this is not the only criterion.

AFS = Size / Files (S/F) (1)
Visits Per Duration (VPD) is another metric that expresses the number of visits on the course pages per duration. This metric is directly related to students' interest, as the lower the prices are, the greater the attachment of the students to the posted instructional material, with no rapid interchanges between the posts.
The thirs proposed metric is related to the course quality and is called smooth or distinct Course Utilization and user Perception metric (CUP). This metric expresses how smooth or abruptly-randomly visit time duration of users per course is distributed over the academic semester. In order to calculate the CUP metric measurements of daily or at least weekly total visit time per course are required. Then for each one of the evaluating e-learning courses a ratio estimate is calculated according to the following formula: Where D(x) is the daily or weekly total duration of visits, {x} is the weekly or daily dataset and D(xi) the the daily or weekly value of total visit time for xi day or week accordingly. If CUP->0 then we have smooth course utilization over time while if CUP->1 we have only distinct weeks of course utilization. In cases of CUP>1, we assume that such courses maintain either a one time utilization or abnormal utilization of very low and very high. We consider such courses as flapping or abandoned courses.
Considering the aforementioned indexes and metrics we can group them in three distinct categories.
The first category of attributes is related to the course. The attributes of these categories most of the times remain stable over the time. However, usefull outcomes can be reached if they are analysed in combination with the attributes of the other metrics. In particular, the semester that course is taught gives an estimation of attendees' experience in higher education. It would be interesting to examine whether novice learners of first years have similar performance with their fellows of higher years who are more expert in the skills of academia. The course category allows the researches to study whether it could affect learners' activity since compulsory courses are typically more challenging than optional courses. Similarly, the type of the course is used to examine if learners behavior is different in practical lab courses or theory courses.
The second category of attributes is related to the courses' online educational content. More specifically, the number of files and their corresponding sizes give an estimation of the content quantity, which is a crucial factor of online educational content. On one hand, if the number of files and their size are small, this might be due to the weakness of the educator to upload enough educational content into the online platform. On the other hand, if the course has a lot of files with big sizes this could lead learners to face the cognitive overload problem and not study the course effectively.
The third category of attributes helps researchers to discover learners' activity and follow up in a course. The variables number of visits and duration may indicate whether learners find course useful and like to visit its pages. For example, if learners visit more pages of a specific course for a long time, this means that course content is interesting and useful for them. This could reflect the course quality. Consequently a good course in terms of quality may help learners at their study.
The last step of the proposed approach is to apply a regression analysis over all the attributes of table 1, in order to find dependencies that may affect the usage of the online courses.
This step exploits the available data in order to find, through regression analysis, course attributes that affect learners' e-learning usage.
Regression analysis can be used to explore the relationship among different characteristics of each performance metrics and to predict the upcoming evaluation o both courseseducators and students based on past knowledge (Kotsiantis and Pintelas, 2005;Feng et al., 2005;Myller et al., 2002;Mcdonald, 2004). Some regression techniques have been used to predict student's performance from log and test scores in web-based instruction using a multivariable regression model (Yu et al., 1999) to identify variables that could predict success in colleges courses using multiple regression (Golding and Donalson, 2006).
Some of the goals of the present research uncover how the learners might improve their performance and in which way they will visit more often the educational content in the platform, as more visits means greater interest and more updated content for each course. It is investigated whether the attributes such as the number of visits and the duration are affected by the educational content. Educational content is expressed as the number of files and their corresponding sizes. Other attributes such as the academic semester in which each course corresponds, the type (compulsory/non-compulsory) and the category (theory, laboratory, mixed) of each course are also investigated to validate whether they affect the output attributes.

An empirical evaluation of e-learning usage in the higher education context
Editorial Universitat Politècnica de València

Case Study
In this section a case study is presented based on the Open eClass E-Learning dataset, collected from the Department of Accounting and Finance, of TEI EMT. The collected dataset is of the winter semester of 2017-2018. It includes E-class logged data for 38 department courses. 10 of these courses are 1st semester courses, 10 of these are 3rd semester courses, 7 of these are 5th semester courses and 11 are 7th semester courses.
29/38 courses are compulsory (Type: C), while 9/38 of optional subject (Type: O). 5/38 are theoretical courses that include assesment exercises (Category:M), 5/38 are optional courses with Laboratory exercises (Category: L) while the others are theoretical courses with oral presentation only (Category: T). Due to lack of space a representative data sample of 8 out of 38 courses has been evaluated, indicativelly taken from all types and categories. Table 4 shows the metric measurements of this set taken out from the E-Learning platform. expresses the total course session time as indicated by platform logins and logouts-session expirations. In table 2, the metrics VPD and AFS mentioned in the previous section are indicators of each course visits' rate and mean delivered content size accordingly. The mean number of files per course is 34.1 for all study courses while the mean filesize is 22483 KB. The mean number of total visits for all courses is 9353 while the mean course total duration time for all courses is 16353,4 min.
Subsequently we try to define the factors that affect the number of visits for each course. We initially test one by one using appropriate tests, which variables affect the number of visits. The results are shown in Table 3. We notice that only the variables "Semester", "Total Files" and "Duration of visits" affect the "Number of visits".
More specifically:  First semester students have significantly more visits than those of the 7 th semester (p-value=0.038 and corresponding 95% Confidence Interval (283.35, 11444.67)).  The more files we have the more visits we have (Pearson Correlation Coefficient= 0.303, p-value=0.065indicatively significant).  The length of visits is affected by the number of visits, which is self-evident (Pearson Correlation Coefficient= 0.907, p-value<0.001).
Initially we tried to investigate whether the number of visits depends on other data which were recorded, using multiple linear regression. The variables that included in the final model as significant are "Number of Files", "Type" and "Semester". Moreover satisfied the assumption of linearity/multicolinearity (all VIF test are between 1.05 and 1.61) and this of lack of autocorrelation (Durbin Watson = 1.88). Finally F testis 4.79 with six degrees of freedom, p-value=0.002 and adjusted R2=37.8%, so the following model can not be used to predict the mean number of visits because of its relatively low adjusted R2.
The values of the variables of the coefficient, VIF and p-valueare represented in Table 4.  For each one file added to the platform, we have an average of 97 more visits.  The web-pages of the lab-courses have an average of 7367 more visits than the corresponding web-pages of theory-courses.  The course of the fifth and seventh semesters have an average of 4792 and 4803 respectively less visits than those of the first semester's courses.

Discussion and Conclusions
The aim of this study was to analyze and evaluate the e-learning system eClass which runs in the Department of Accounting and Finance of Technological Educational Institute of EMTh. Our assessment was based on the use of the indices method. The online has many features, the so-called tools, such as documents, videoconferencing, wiki, questionnaires, etc., which are distinguished in active and inactive. It is clear that the availability of as many active tools as possible contributes to make the most of the platform's potential.
The processing and evaluation of data leads to the conclusion that not all of the platform's capabilities are used as there are several lessons that do not contain files. There are also several inactive tools. Therefore, the eClass of the department of Accounting and Finance of TEI EMTh operates less satisfactorily compared to the corresponding platform of other universities and Institutions.
From the learners' point of view, by activating more tools on the platform will be an incentive for more frequent use of the platform, it will increase their interest, and will increase e-learning traffic. In general, familiarizing learners with computer use and technology which is constantly increasing, will make platform use easier and more demanding. Modern tools such as video, videoconferencing, etc. attract their interest.
From the teachers' point of view, it is possible to inform them, through the statistics provided by the platform, about its use by the learners on a daily, weekly, or monthly basis. Updating through statistics is an important aid for teachers to take care of feedback on their course. Comparing with the pages of their colleagues of the department as well as of other universities would be an incentive to take full advantage of the platform's potential.
The use of the platform offers both learners and teachers the opportunity to contribute more to the department's courses, each with its own role. However, the maximum performance of the platform will be achieved if it is more learner-centered, giving the learner the possibility of more active participation in it and not just as a passive consumer. Essentially, by participating in problem solving and creative discussions with educators and fellow learners, s/he will get a full picture of the subject of his/her studies and will want to participate in expanding the platform's actions.
The optimal operation of the digital platform of the department and the e-courses provided by it, is an important aid, but it can not substitute for traditional teaching. Instead, these two areas, e-learning and traditional teaching, are interdependent, as there is no e-learning available without the second and experienced staff involved.