Enhancing the formative value of self-and peer-marking through ‘ Test Workshops ’

This paper reports on the development of ‘Test Workshops’ in a first year chemical and metallurgical engineering course in a South African university. The workshops were designed to enhance student learning through a combination of selfassessment, reflection and group interaction. The development began with an investigation into the summative and formative value of selfand peer-assessment in the context of the course and how the students responded to these unfamiliar forms of assessment. While the accuracy of both selfand peer-marking was found to be inadequate for summative purposes, their formative value was found to be considerable. Based on these findings, the Test Workshop format was designed as a sequence of activities that consisted of writing a short problem-based test under summative test conditions, followed immediately by a group discussion of the solution to the problem, followed by self-marking of the individual solutions against a model answer and marking rubric, followed by a written reflection. Analysis of the students’ experiences found that the effectiveness of this format as a learning methodology was very encouraging and that it had applicability in a wide range of contexts.


Introduction
The formative value of self/peer-assessment and its potential to enhance student learning in a course is widely recognized (Dochy et al., 1999;Race, 2001;van Zundert et al., 2010).Many benefits for student learning have been reported, such as developing reflective skills and a greater sense of a student's responsibilities (Dochy et al., 1999); becoming better informed about the nature of the assessment systems being used (Bloxham & West, 2004); and the development of critical thinking and higher-order cognitive skills (Snowball & Mostert, 2013).This paper describes the development of a formative assessment methodology -'Test Workshops' -that is based on self-and peer-marking.It was developed in the context of a first year process engineering course in a South African university.The course was an introduction to process engineering but had as one of its objectives the development and reinforcement of skills needed for academic success at university.The methodology developed through two phases over the period 2007 to 2013.The first investigated the efficacy of self/peer-marking in the context of the process engineering course.This is described in Part 1 of the paper.The second was the design and evolution of the 'Test Workshop' format and a study of its efficacy.This is described in Part 2 of the paper.

Part 1: Self-and peer-marking in the context of the engineering course
The introduction of a self/peer assessment component into the course was motivated by four objectives: to enhance student learning; to familiarize students with how their work was marked; to improve the students' skill and self-awareness with regard to engineering problem-solving and test-taking practice; and to reduce the marking load.In order to establish the extent to which such a component was likely to achieve these objectives in the context of the course, a pilot exercise was conducted in 2007 over a six week period.The exercise involved a series of three short class tests that were both peer-and self-marked followed by a class survey to solicit student reaction to the assessment process.The first two tests were treated as training exercises to familiarize the students with the system because it was clear from the literature cited earlier that such training was essential for the success of any self/peer assessment strategy.Prior to the first test, the self/peer marking exercise and the rationale behind it was explained and students were made aware that the third test would contribute to their final mark.
The tests were written bi-weekly and copies of the scripts were self/peer marked by the students the day after a test.To make the peer marking blind, students were allocated predefined script numbers and wrote these and not their names or student numbers on their answer scripts.
Each test involved a single question and consisted of a numerically based engineering problem that was typical of questions students would encounter.To guide their marking, students were given marking instructions, an example of a marked script, and the marking memo.The memo had two sections: a 'technical section' that was essentially the marking rubric a course coordinator would use when grading; and a 'quality of practice' section intended to focus the students' attention on issues of good practice with respect to categories labelled 'perception', 'thinking', and 'communication'.Each of these issues were assessed according to the sub-categories listed in Table 1 and required the students to make a judgement about each sub-category on a seven point scale: i.e. perfect, excellent, good, OK, needs work, poor, hopeless.The scripts from the third test were marked by the course coordinator for summative purposes and as a basis for evaluating the accuracy of the students' marking.In both cases, attention was given only to the 'technical' marks.The results are summarized in Figure 1 and show that neither self-marking nor peer-marking were sufficiently reliable to be used for summative purposes.Figure 1A shows the extent to which self-marking diverged from marks given by the coordinator.In some cases, the divergence was as high as 50% in absolute terms with weaker-performing students deviating more than the stronger.
With regard to peer-marking, Figure 1B shows that the accuracy of peer-marking was not much better than that of self-marking.It was less biased but significant deviations were still evident.No correlation was evident between the extent of deviation and the apparent academic strength of the students.It was however apparent that although students' tended to over-mark their own work (by about 6.4% on average) this tendency fell away when marking the work of peers.

Students Reactions to the self / peer-marking exercise
The class survey conducted a week after the last test consisted of six fixed-response questions using a five point Lichert scale.It also solicited open comments.With regard to the students' reactions to the self/peer-marking exercises as a whole, the survey found that the majority of students responded positively to the self/peer marking exercise.The responses indicated that 22% regarded the exercise very favourably and 69% reasonably to very favourably while 20% of students were unsure.However, 12% were negative towards it but no students were very negative.More significantly, an overwhelming majority (95% of the students) saw benefits in the system: 41% saw major benefits while 5% saw no benefits.The open ended comments in the survey returns indicated the following general reasons why students endorsed the system.
• Seeing a variety of ways of solving the problem improved their problem solving skills and understanding of the associated concepts.• Becoming more accurately aware of their own mistakes had a similar effect.
• Acquiring a better understanding of the workings of the assessment process itself helped students to know what to expect and how to answer questions more effectively.
The negative comments made by the students focused almost exclusively on the issue of the marking standard of peers.In addition, a number of concerns were expressed about details of the marking process (such as how to award part-marks) and a small number of students indicated that they found the process to have little value and to be tedious and time consuming.
With regard to the students' perspectives on the educational value of the self/peer-marking exercises the survey indicated the following.Fully 67% of the students believed their understanding of the course material improved reasonably or substantially, while 8% believed it was not helpful in this regard.When asked a more general question ("To what extent has your learning improved as a result of the self/peer marking system?"), 88% of the students indicated some, good or very good improvement while 46% indicated a good improvement, and 8% no improvement.With regard to the extent to which the self/peer marking exercises were seen to be helpful for understanding the marking process and, consequently, for improving their understanding of what was required of them in tests and exams, the indications from the survey were again positive: 71% indicated it had been reasonably to very helpful; 37% indicated it had been very helpful; while only 2% indicated it had not been helpful at all.With regard to gaining a better understanding of what was expected of them in tests and exams, 99% of the students indicated that it had been helpful to some degree: 27% indicated that it had been very helpful; 57% that it had been quite helpful and 15% that it had helped a little.

Summary of findings
The survey suggested that many of the intended outcomes of the self/peer-marking process had been achieved.A majority of students reported gaining greater familiarity with the assessment processes and greater clarity about what examiners expected and so benefited accordingly in the tests.Many reported improved learning at a number of levels as was expected from the literature (Falchikov & Boud, 1989;Falchikov & Goldfinch, 2000).In general, the students responded positively to the system and its formative value seemed to be considerable.
More negatively, it was apparent that student marking was not reliable enough for summative purposes.Also, the complexities associated with making peer-marking blind in order to promote marking accuracy were somewhat detrimental to the formative value of the assessment system; for some students the complexity created confusion and for others tedium.While there were many positive aspects of the self/peer marking that was piloted, it was clear that its format needed to be modified if it was to be implemented on a more ongoing basis.

Part 2: Test Workshops
Based on the encouraging findings just described, a different kind of formative assessment system was developed over several years and evolved into what became known as 'Test Workshops'.The format that emerged was based on self-marking but also included guided reflection and group interaction that were intended to enhance the formative impact of selfassessment.It was designed to be conducted during the three hour period allocated for an afternoon tutorial and consisted of a series of linked activities as follows.
1) Test: Under summative test conditions students write a short test (20 to 40 minutes) that consists of a single problem requiring the calculation of one or more numerical answers.2) Transfer: Students copy their answers onto an answer sheet that is handed in for grading.
The scripts themselves are kept by the students for the subsequent activities.3) Group interaction: Students form groups of three or four to discuss their solutions, to compare solution methods, and to reflect on their performance.4) Self-marking: A model solution and a marking rubric similar to the one described in the previous section are handed out and students mark their own work.5) Reflective assignment: Students reflect on various aspects of their performance and write individual reflections which they hand in for grading along with their self-marked scripts.6) Portfolio and review assignment: Within a week of a Test Workshop, the scripts and reflective assignments are returned to the students.Students are required to build a portfolio of these along with other work from the course.The portfolio constitutes the data for a review assignment which the students are required to hand in (with their portfolios) at the end of the semester for grading.In this assignment the students are required to reflect on their learning experience in the course and the combined impact of its various components on their learning and development, including the impact of the test workshops.
The rationale behind some of the features of the Test Workshop format requires explanation.
The first to consider is that a summative test environment is retained even though the primary focus of the workshops is formative.The rationale here is to exploit the intensity with which students engage with set problems and tasks in a summative environment -in the students words, the work is 'for marks'.The result of this intense engagement is a degree of familiarity with the particularities of the task or problem that is deeper than is likely to be achieved by a student under any other circumstances given the same amount of time commitment.This means that all the subsequent activities in the workshop are conducted from the perspective of that depth of awareness and engagement thus enhancing their formative value.Because all but one of these activities follow immediately after the test, the formative impact is maximized.
A second feature of the Test Workshop to consider is the inclusion of group interaction.In effect, group interactions, together with the self-marking activity, replace the peer-marking aspect of the original format described in Part 1 and provide a broader range of formative benefits than is normally associated with peer-marking.For example, in the group interaction phase of a test workshop, each student is able to view scripts from two or three peers and is also able to engage with those peers on any issues that might arise.
A third feature to discuss is the inclusion of reflective exercises in the Test Workshop format.
The rationale here derives from the broad consensus that good reflective skills are important for quality learning and student development.In the same way as the prompts in Table 1 in the original format were designed to help students to make judgements about aspects of the 'quality of performance' evident in a script, so a list of reflective prompts are provided in a Test Workshop to guide students in their reflections and, in this way, to help foster the further development of this important graduate attribute.

The impact of the Test Workshops
The work which the students collected in their portfolios and their review assignments provided the most meaningful data available for an evaluation of the impact of the workshops on the students' learning and development.Ten submissions from the 2012 cohort (319 students) were selected for the evaluation.(More are currently being analysed to increase the representivity of the sample.)However, what is currently indicated by an analysis of the sample of student work is the following.
The detail given by the students in the post-test reflections and in their portfolio assignments indicated that the test workshop had been successful in fostering a high level of student engagement with the problems set in the tests and in the reflective and group interactions associated with it.With regard to the latter, several students commented on insights they had gained from their peers in the group interactions.One of them commented as follows.
My second problem during this test was pointed out to me by my partner when comparing answers.He highlighted the fact that although I am strict with how I answer the question, I forgot to include a key, nomenclature, units in calculations and consistency with significant figures.I will endeavour to remedy these problems from now on.
It was also clear that the test workshops had achieved the objective of familiarising students with the assessment systems used in the course.One student wrote … There was definitely an improvement in (my) marks as I … gained a better understanding of how to study and tackle (process engineering) problems.This can be seen in the general improvement in marks from … test to … test as I got more familiar with how the questions were asked and what information to look for.
What was particularly evident was growth in the students' understanding, application and appreciation of reflective practices.All students in the sample reported this.In summary, they explained that the reflective activities in the test workshop had lead them to recognize shortcomings in their conceptual grasp, in problem solving and test-taking skills, and/or in the quality of their performance in the tests.Further, the format had enabled them to make appropriate and timely adjustments in order to overcome these shortcomings.Four examples from the student portfolios illustrate this.I (became) much better at critically assessing myself and communicating my point of view.I think this was because I had had more practice in writing those necessary (reflective) reports.I also noticed this improvement when reading through the reflections we did after every test.
From my reflection I realized that the inability to understand the question had pulled me down.In the following week when we wrote the (test) I managed to get the solution presented in a manner that was more like that of the model solution.… I had managed to work on my previous week's weakness through reflection.Reflection is a learning practice that has helped me to understand this course better and other courses as well.
From last week I learned that writing (out the) given information helps me solve the problem so that this week I did exactly that.… By doing this I could see where I could find my required calculation from.This was the first tut test (i.e.test workshop) in which I panicked but instead of concentrating on the panic, I focused on what I was given and verified everything I did.In so doing I made sure that my panic wasn't making (me) short-sighted (sic).
I found the … reflective exercises after each tut test (i.e.test workshop) were of particular importance to me.Not only did they assist me in identifying where my shortcomings were, but they also helped me discover new learning guidelines that were especially beneficial for me.
A further point that is illustrated by these examples is the personal growth that had occurred in the students apparently as a result of their experiences in the test workshops and the associated reflective activities.Some students reported understanding concepts better, gaining new insights, developing new problem solving techniques or "new learning guidelines" or an enhanced ability to "critically assess" oneself.Others mentioned a greater ability to overcome the effects of panic in tests; a greater awareness of their "strengths and weaknesses" and of the link between doing well in tests and concentrating in lectures, and completing as many tutorial problems as possible.A student who described himself as a "rather private person" predisposed 'to shyly working on his own' (his words) reported gaining confidence to engage with class mates in the group discussions and also more generally.One student remarked that their improved reflective skill was transferring to their work in other courses.
The overall conclusion from analysing the work of the 2012 cohort was that all the features built into the test workshop format to enhance its formative value had been sufficiently efficacious to justify a continuation of the implementation of the format on an ongoing basis.
Two additional features of the Test Workshop format are considered to be important.These are the reflective environment created in a Test Workshop and the use of an expanded rubric in the self-marking activities so that the attention of the students is drawn not only to the disciplinary concepts and issues raised by the test but also to more generic issues of skill, individual performance and 'quality of practice'.As such the methodology is considered to have relevance and applicability for a wide range of learning contexts.

1st
International Conference on Higher Education Advances, HEAd´15 This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License (CC BY-NC-ND 4.0).

Figure 1 :
Figure 1: Relative Accuracy of Self and Peer Marking on Higher Education Advances, HEAd´15 This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License (CC BY-NC-ND 4.0).

1st
International Conference on Higher Education Advances, HEAd´15 This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License (CC BY-NC-ND 4.0).

1st
International Conference on Higher Education Advances, HEAd´15 This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License (CC BY-NC-ND 4.0).

Table 1 :
Sub-categories of 'Quality of Practice' used in the marking rubric