Assessment of oral and written communication competences in the European Higher Education Area: a proposal of evaluation methodologies

The international accreditation for the Master and Bachelor degrees offered at our university, together with the demands of the employers, have made it clear that the students’ curricula should specify not only what they have studied, but also what they are actually able to do. Although the competence based curricula approach has been used in the development of the new programmes for the Master and Bachelor degrees within the European Higher Education Area in recent years, the assessment of these competences is still a pending task. This work presents an ‘outcomes’ approach for the assessment of the oral and written communication skills within subjects related to mechanical and materials engineering. In particular, this paper proposes some rubrics developed in order to quantify the level of achievement. These rubrics are based on the evaluation of some learning outcomes that can be observed by using different strategies during the course. Conclusions about preliminary results and the difficulties found in order to create these tools are also described here.


Introduction
The Bachelor and Master degree programmes developed at our university within the frame of the European Higher Education Area follow a competence based approach (A. Sursock, 2010;Murias, de Miguel, & Rodríguez, 2007;Rieckmann, 2012). These programmes clearly define the specific and generic competences to be worked in each degree, and also, the particular subjects along the degree. The assessment of the specific competences continues to be reflected on the students' curricula by using numerical qualifications, but the assessment of the generic competences has been passed over somehow. It has been understood that the students would have acquired these capacities and skills at the end of the studies. The international accreditation of these programmes, together with the requirements of the employers to have better information on the students' competences, have put the university to work on this issue (Andrews & Higson, 2008;Entwistle & Peterson, 2004). This paper presents some results obtained in the frame of an innovative project (PIME program) on the evaluation of three generic competences that have been traditionally worked within subjects of mechanical and materials engineering: capacity for problem analysis; capacity for applying knowledge in practice; and communication skills, using name convention from Tuning (2014) (http://www.unideusto.org/tuningeu/). In particular, in this work some methodologies for the analysis of the skills in oral and written communication have been developed (Sparks, Song, Brantley, & Liu, 2014;Dunbar, Brooks, & Kibicka-Miller, 2006). The aim is to obtain reliable information in order to evaluate strengths and weaknesses of the students in the communication competences (Jonsson & Svingby, 2007). Furthermore, these methodologies of evaluation try to encourage students to participate and to be focused during lessons, as they will have to explain the key information in public to their classmates, in case of oral competence, and to develop a technical report dealing with the solution of a problem, in case of the written competence. Some quantitative results are presented using data collected from two subjects: lab works on Materials Science, from the Bachelor degree in Chemistry Engineering, and Mechanical Vibrations, from the Master degree in Aeronautical Engineering, both offered at the Technical University of Valencia (UPV).
In the case of the oral competence, students were divided into work groups of 4-5 people for the Bachelor degree, while for the Master degree they worked individually. Students had to work during the time between lessons (1-2 weeks) to maximize their selflearning, so that they could expose the lesson in the best way to their classmates. Students were expected to interact, sharing concepts and strategies assuming responsibility for their self-learning and that of the other members of the class that were evaluating their work. It sought not only to develop public communication skills but also, to improve autonomous learning skills, critical thinking, synthesis capacity and responsibility to search for, analyse and verify main information to explain it to their mates. In the case of the written competence, students worked individually. Students were proposed to develop a pair of open case studies related to a specific subject. They were expected to be involved in a real problem in which, with all the acquired knowledge during the lessons, they had to decide a strategy, a method to apply, and to take decisions to solve the problem by explaining the plan and steps followed to achieve the solution. The method tries not only to develop the written communication skills, also to offer the students an active learning method as a challenge and an opportunity to deal with a real problem, that is, they have to plan hypotheses and make diagnosis of the chosen situation to find the best solution using their individual knowledge.

Methods
For the evaluation of the oral communication skills, students were asked to prepare oral presentations ranging from 10 to 15 minutes to explain some topics taught in the previous lesson to the class. The oral presentation was individual, but for the Bachelor students the work was team-based developed (4-5 people). To ensure that each member participates in a significant manner, at the beginning of each lesson the teacher would randomly choose a student to make the oral presentation of the group. The rest of the students and two teachers evaluated the presentation using the rubric below ( Table 1; Rubric 1). The students' presentations were evaluated considering eleven different items. The first four items were mainly related to the organization, structure and content of the whole presentation, that is, they correspond to the same dimension (content). The fifth item concerned the effectiveness into the interpretation and justification of the information presented and the sixth item evaluated how clearly the student structured and communicated the topic to the audience. The seventh item evaluated the technical language usage, and the eighth item was related to the use of additional resources (board, objects, examples…) that can ensure better effectiveness on delivery the information to the audience. The ninth and the tenth items were related to the temporal planning, and to how well the speaker explained to the audience in terms of diction, corporal position, tone of voice and visual contact, respectively. Finally, to evaluate the analytical capacity, the audience (teachers or students) made questions to value the eleventh item.
Written communication skills were only evaluated for Master students. They had to formulate and solve two different problems explaining all the solution process in a written report to demonstrate and apply the acquired knowledge during the lessons.  Table 2) was developed to be applied on the students' reports with a view to evaluating the level of achievement on the written communication competence.

Rubric 2 (
Similarly to the oral communication evaluation, the students' written reports were evaluated also along ten different items. The first four items were about the content (introduction, objectives, results and conclusions). The fifth item dealt with the structure and formal coherence of the information, while the sixth item evaluated the quality in terms of style and format of the report. The seventh item was related to the presence of grammar and orthography mistakes, and the eighth item valued the appropriated use of technical language. The ninth and the tenth items evaluated the achievement of a professional and high quality report through the argumentation and justification of the information presented, as well as through the use of graphical resources to emphasize the most relevant results.
For both oral and written skills evaluation, each of the items was assigned a scale ranging from 0 to 5, representing an increasing level of development. As described before, first to fourth items corresponded to the same dimension, while for the rest, fifth to eleventh (tenth in case of written skills), one item corresponded to one dimension. It means that, for oral communication, there were eleven items to quantify and eight different dimensions of the competence, while for written communication there were ten items to quantify and seven dimensions. The overall score was assigned by simply adding the marks corresponding to all the items. Finally, to evaluate the level of achievement of the competences, an assessment scale was developed. The total value range obtained from the rubrics (0 to 55 points for oral communication and 0 to 50 points for written communication) were split up into six different ranges ( Table 3) from low to master level. These scales allowed to obtain comparable values of the overall communication competence. Ranges were delimited as logical as it was possible, taking into account that most of the students would have to achieve a medium level in the development of the evaluated competence. That is, lower and higher levels value ranges were shorter than medium. Applying the same logic, higher levels of achievement were the most difficult to reach, and so they had the shortest range.
This evaluation methodology (rubric based), helped the authors not only to evaluate the oral, written and the whole communication skills of the students, but also to know to what extent the lessons had been understood (critical thinking), providing a valuable feedback about students' knowledge.

Results and Discussion
Among the objectives of the presented work, one is to evaluate the differences between data collected by the different teachers and the data collected by the teachers and students. These results are directly related to the efficacy and reliability of the evaluation methods developed. Figure 1 and Figure 2 show a comparison of the average data collected on Bachelor and Master courses, respectively, to evaluate oral communication competence . Figures 1a/2a compare the average marks from studentsteachers evaluations and Figures 1c/2c compare marks from teacher-teacher evaluations. Radial direction represents the marks obtained from the evaluation using the rubric 1, the minimum value being zero and the maximum 55. Each point in the graph represents one single evaluation act. Figures 1b/2b show the students' standard deviations of each evaluation act, and Figures 1d/2d are the teachers' standard deviations. These data has been ordered from lower to higher values for better understanding.
As it was expected, teachers' scoring was slightly lower to that of the students; however, small differences between students and teachers evaluations occur resulting in a relatively good correlation (Figures 1a and 2a). Regarding the standard deviation between the students marks for each presentation (Figure 1b), variation in Bachelor values is also low, 42% of the values having SD ≤ ±3, while 63% yielding SD ≤ ±5. Comparing teachers' evaluations (Figure 1d), a still lower dispersion of the data was obtained, 74% of the values having SD ≤ ±3, while 95% of the data delivering SD ≤ ±5. In case of the Master values, a similar picture is obtained. Taking into account teachers and students' evaluations (Figure 2b), 60% and 80% of the data provide SD ≤ ±3 and SD ≤ ±5, respectively, while for teachers' evaluation (Figure 2d) 100% of the values present SD ≤ ±5. Thus, considering the influence of standard deviation in the final marks of the students, these only vary 5% or 9% taking into account SD ≤ ±3 or SD ≤ ±5 respectively. The Figures 1a/1c and Figures 2a/2c show a fine correlation between evaluations made by different evaluators, this has been confirmed by means of statistic analysis based on t-Student's test. Commonly, it is used to decide if it could consider means difference statistically significant. In teacher's evaluations for Bachelor and Master, p-value obtained 0.33 and 0.83 respectively, are higher than 0.05 which indicates lack of difference. In case of students versus teacher marks in Bachelor degree, 53% of evaluation acts showed no mean differences between students and teachers. This value became higher for Master degree, reaching up to 80% of agreement. The increase may be caused by further extended formation of Master students and also by more objective student evaluations.   The use of rubrics has proved to be very beneficial for the students' learning and motivation. Figure 3 shows the assessment of the written communication skills (Master students) from the beginning to the end of the course. The general trend in the global final marks (Figure 3a) obtained from Rubric 2, is an improvement along the course. Analyzing each individual item, it also results in higher marks for all the students at the end of educational program, pointing to a better development of the command task with the time. With the purpose to evaluate the flexibility of the methodology proposed, Figure 4 shows a comparison between the data obtained from oral communication evaluation of Bachelor degree and Master students. As it was expected, maximum total results are higher for Master students compared with Bachelor degree students. Distributing the data according to the developed scale of achievement (Table 3; Oral communication scale), 37% of the Bachelor degree students' marks are in a medium level, while most of the students (53%) are in a medium-high level and only 10% of the marks achieve high level in the development of this competence. In case of Master students' marks, 40% are in a medium-high level, 50% in a high level and 10% of students achieve Master level. Results are in accordance with the students' level, reflecting good connection between the method developed and the feedback obtained.

Conclusions
The aim of this paper is to present a rubric-based method developed for the evaluation of the oral and written communication skills of Bachelor and Master students. The results obtained were analysed to test the reliability and validity of the methods proposed. Low differences between students judgments were obtained, meaning a good understanding of the criteria used for evaluation. That is, a good correlation between data collected by students and teachers indicates that the criteria proposed are easy to understand, clear and transparent for different kinds of people (teachers, Master and Bachelor students). Thus, the proposed methodology can be used with independence of the framework of the subject (in technical subjects) and/or the degree of the students. This methodology presents a good correlation between teachers' evaluations and between teachers and Master students. Differences between Bachelor students' and teachers' evaluations could be caused by students inexperience in peer review evaluation and in some cases due to a doubtful objectivity. The reason for the assessment of the competences to improve in time is that, through this evaluation method, students know what they have to do, what it is important and why they are doing it, making the method meaningful for them and so promoting their learning. Furthermore, by applying the final scales developed to qualify the students in the achievement level of the competence, the flexibility of the method is demonstrated, as results obtained from students of different technical subjects and levels can be successfully "standardise" and compared. Based on the evidences found, rubrics seem to promote learning and to improve teachers' instruction as their use provides teachersstudents feedback in both directions. They also provide self-assessment to the students, what makes them an effective and reliable tool to evaluate students' performance.