Assessment of problem-solving skills in subjects related to mechanical and materials engineering

The international accreditation for the Master and Bachelor degrees offered at our university, together with the demands of the employers, have made it clear that students’ curricula should specify not only what they have studied, but also what they are actually able to do. Although the competence based curricula approach has been used in the development of the new programmes for Master and Bachelor degrees within the European Higher Education Area in recent years, the assessment of generic competences is still a pending task. This work presents an ‘outcomes’ approach for the assessment of the problem-solving capacity in subjects related to mechanical and materials engineering. In particular, this paper proposes a scale in order to quantify the level of achievement and shows some tools developed for this purpose. These tools are based on the evaluation of some learning outcomes that can be observed by using different strategies during the course. Conclusions about preliminary results and the difficulties found in order to create these tools and the scale are also described here.


Introduction
Bachelor and Master degree programmes developed at our university within the frame of the European Higher Education Area follow a competence based approach (A. Sursock, 2010;Murias, de Miguel, & Rodríguez, 2007;Rieckmann, 2012).These programmes clearly define the specific and generic competences to be worked in each degree, and also, the particular subjects along the degree.The assessment of the specific competences continues to be reflected on the students' curricula by using numerical qualifications, but the assessment of the generic competences has been passed over somehow, and it is understood that the students would have acquired these capacities and skills at the end of the studies.The international accreditation of these programmes, together with the requirements of the employers to have better information on the students' competences, have put the university to work on this issue (Andrews & Higson, 2008;Entwistle & Peterson, 2004).This paper presents some results obtained in the frame of an innovative project (PIME program) on the evaluation of three generic competences that have been traditionally worked in subjects of mechanical and materials engineering: capacity for problem solving; capacity for applying knowledge in practice; and communication skills, using name convention from Tuning (2014).In particular, in this work a methodology for the assessment of the competence on problem solving has been developed.
The definition used in this work for the competence on problem solving refers to the capacity to analyse and solve a problem in an effective way, identifying and determining the most relevant parts of it.The main aim in developing this competence is that the students increase their self-confidence and promote their own capabilities and skills to learn, understand and apply their knowledge.It is an important competence that contributes to the lifelong self-learning of the student, and helps in the developing of some other competences as team working, creativity, critical analysis and leadership.
It should be also noticed that problems are not exercises, i.e., problems refer to new open situations that encourage individuals to use new approaches (Pozo Municio, J.I., & Pérez Echevarría, M.P 2009).These situations can be solved by using different strategies, and they do not usually have only one solution.Solving a problem implies using thinking skills, and not only repeating a known procedure to obtain the solution.
This competence can be decomposed in different learning outcomes that can be summarised as follows (see also Table 1): • To identify a real problem and define precisely the most relevant facts.
• To apply the methods learned to analyze a problem, gather relevant information and propose different alternative solutions.• Using the experience and judgment to generate an efficient and effective solution.
First in this paper, the methodology proposed for the assessment of the competence on problem solving is presented.This methodology is based on the use of real complex problems that force the students to put the aforementioned skills into practice.The solution process is then evaluated by using a checklist that has been developed from the learning outcomes.This checklist allows the authors to quantify the level of achievement of the students in order to give a numerical evaluation.
Some preliminary results regarding a first attempt to use this checklist in two subjects of Bachelor and Master degrees are also described here.Finally, some conclusions are derived from this experience in order to improve the methodology in the future.

Methods
To carry out the evaluation of the competence on problem solving in Bachelor and Master degree students, they are asked to solve an individual problem.This problem and its evaluation are designed to make clear if the learning outcomes have been achieved: from problem identification to critical analysis of solution, together with the information gathering, the proposed methodology as well as the reasoning to choose it.Proceeding in this way, the student is not only encourage to problem solving, but also self-learning, based problems learning, critical thinking and written communication competence are promoted in an indirect manner.
In Table 1, the learning outcomes are developed into increasing levels of achievement: "low", "low-medium", "medium", "medium-high", "high" and "master", and some evidences regarding these outcomes are proposed.From these evidences, the checklist detailed in Table 2 has been considered as the evaluation tool.It is structured in nine items in order to evaluate the different key aspects of problem solving.Items 1 and 2 refer to the first learning outcome; items 3-6 to the second one; and items 7 to 9 to the last one.
1st International Conference on Higher Education Advances, HEAd´15

High
Master EVIDENCES

To identify a real problem and define precisely its most important facts
The student is able to identify with difficulty a real problem and define the most important facts The student is able to identify a real problem, but discriminating with difficulties proven facts from speculations The student is able to identify a real problem and its causes, but provides very limited assumptions about its origin The student is able to identify a real problem and evaluate the origin and impact of the different causes The student is able to identify complicated problems, to evaluate its causes and split them into manageable parts The  The problem identification, and whether it is up into smaller and easier units to solve, respectively, are considered in the first two items.The information gathering is reflected in item 3. Regarding the proposed methodology, items 4-6 involve its description taking into account if several proposals are used to validate the resolution and if the choice of the followed method is justified, respectively.Item 7 refers to the estimation of the data for problem resolution and its justification, while item 8 comprises the analysis done of the problem solution guarantying the expected order of magnitude.Finally, the efficacy in problem resolution is captured in item 9.
According to Table 2, each of these checkpoints is assigned with a score from 0 to 3 when the level of achievement is "inappropriate", "sufficient" and "appropriate", respectively.A value range is ascribed to each level, detailed in Table 3.Consequently, depending on skills reflected in the total mark obtained from the checklist, the student will achieve one competence level or another.These competence levels have been defined so that all the Bachelor degree students can be included in the first four levels, while Master students should be in the two latter.

Results and Discussion
The methodology described above and used to evaluate the competence of problem solving, consists of the activity and assessment tool and the decomposition of the competence in learning outcomes and achievement levels.This methodology has been implemented in both Bachelor degree and Master subjects.The results obtained are analyzed and discussed below.
Figure 1 shows the score of the activity assessment, which has consisted of solving problems independently and individually.This evaluation has been carried out by the Checklist, where each item is scored on a 0-3 range, and the maximum score for activity is 27 points.In Figure 1a) the final score for each evaluated evidence is detailed, for Bachelor degree and Master students.In all evidence collected during the course, a higher final score is observed for Master students compared to Bachelor degree students.If the analysis is performed for each item defined in the Checklist (except item 5 that has been discarded because it could not be assessed in the Bachelor degree problems), as shown in Figure 1b), the average score on each checkpoint for Master students is again higher than the one corresponding to Bachelor degree students.It is important to note that the evaluations of items 7 and 8 show the lowest scores, associated with a poor justification of the used data and insufficient analysis of results, respectively.
Figure 2 is obtained if the items are grouped into learning outcomes, as discussed in the previous section.The score is reflected in terms of percentage of the total score.It should be noted that in the first learning outcome there is little discrepancy (3.5%) between Bachelor degree and Master students, that is, both are able to identify the objective of the problem and split into simpler parts.Regarding the second learning outcome, which is based on the collection of information, description and justification of the methodology followed to solve the problem, a notable discrepancy (19%) is observed, the Master students showing higher skills.Finally, in the third learning outcome, there is a large discrepancy between Master and Bachelor degree students, 52% approximately.Therefore, Master students have greater skills to estimate and justify problem data, critically analyse the solution and do it effectively and efficiently.Taking into account that the aim of the work is to develop a methodology able to evaluate the competence on problem solving, Figure 3 details the assessment of such competence for Bachelor degree and Master students by applying the methodology described before.Regarding Bachelor degree students, it is observed that 80% of them reach levels of competence framed in the first 4 levels associated with Bachelor degree courses.As it can be seen, the approximate distribution yields 10% "low-medium" level, 20% "medium" level and 50% "medium-high" level, the remaining being 10% "high" level, reserved for Master skills.On the other hand, it is observed that the level achieved by Master students is higher than that for Bachelor degree students, as expected.In this case, 80% of Master students are in areas associated with Master courses (last two levels), 30% being "high" level and 50% reaching "master" level.The remaining 20% has a "medium-high" level associated with the last course of Bachelor degree.After analysing the different results, that is, the individual items' assessment, the total scores of evidences and the achievement levels, it is observed in all of them that Master students have more skills to solve problems, as expected.Therefore it can be stated that the methodology described for carrying out the assessment of the problem solving competence is effective and reliable, from activity and assessment tool to the definition of its learning outcomes and achievement levels.

Conclusions
A methodology has been defined to carry out the assessment of the problem solving competence achieved by Bachelor degree and Master students.For this purpose, an individual and complex real problem has been adopted as the evaluation activity.The student must identify the problem and gather the necessary information, propose and justify solving methodologies in order to obtain an efficient and effective solution and critically analyze it.
To evaluate such activity, a Checklist has been defined as an evaluation tool.It is split into 9 items that collect the most relevant aspects of problem solving and are scored at a range of 0-3 points, the maximum score for evidences being 27 points.In order to identify the achievement level, once the evidence is evaluated, the competence is decomposed in three learning outcomes, each one structured into six achievement levels, with a score range assigned.Consequently, the methodology allows to identify the problem solving skills of the students and to assign them an achievement level.
The proposed methodology has been proved in Bachelor degree and Master courses, and the results have been analysed.It is concluded from the analysis that Master students have higher skills than Bachelor degree students in all checkpoints assessed for problem solving.However, it is important to note that the lowest score is found in items 7 and 8 for Bachelor degree and item 7 for Master students.That is, the students show deficiencies when justifying the estimates data for problem solving and perform a very limited review of the solution obtained.On the other hand, the students show a good ability to split the problem into simpler parts, to collect relevant information and to describe schematically the resolution procedure followed to obtain the solution, items 2-4 respectively.
In general terms, after comparing the achievement levels, the Master students show competence levels mostly "high" and "master", while Bachelor degree students are located on "medium" and "medium-high" levels.As expected, Master students have more skills than Bachelor degree students, so it can be concluded that the proposed methodology is valid and reliable to perform the assessment of the competence on problem solving.

1stFigure 1 .Figure 2 .
Figure 1.Marks obtained on the different evidences collected during the course for Bachelor degree and Master students.a) Total marks from Checklist.b) Average marks for each item of Checklist on Higher Education Advances, HEAd´15 This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License (CC BY-NC-ND 4.0).

Figure 3 .
Figure 3.Comparison of competence achievement levels for Bachelor degree and Master students on Higher Education Advances, HEAd´15 This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License (CC BY-NC-ND 4.0).

Table 1 .
Learning outcomes and achievement levels for the competence on problem solving

Table 2 .
Checklist as assessment strategy to the competence ** The answer "No" gives a score of 0 points.It is not necessary to complete the field "To what extent?"

Table 3 .
Range values to obtain the achievement level of the competence 1st International Conference on Higher Education Advances, HEAd´15 This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License (CC BY-NC-ND 4.0).