‘ Don ’ t feedback in anger ’ : enhancing student experience of feedback

This research explores three iterations of the delivery of audio feedback in relation to formative assessments at the School of Law, University of Sheffield. The evidence base includes similar practice at Edge Hill University and collaboration on good practice between the two institutions. This paper will set out the context for the implementation of audio feedback, namely to help address the difficult issues experienced with feedback from non-engagement by the student in the whole feedback process, to a lack of utilization of formative feedback for 'feedforward' purposes. Qualitative comments from both students and staff experiencing this model of feedback will be drawn upon, which include references to the perceived benefits and challenges of this mode of feedback by both sets of stakeholders. This paper will then take participants through the methods addressed to engage student with feedback on formative assessments, in order to create and encourage proper 'feedforward' to summative assessments, and to provide effective, focused, consistent and constructive feedback. This paper in particular aims to show how the provision of audio feedback has the potential to greatly enhance the student learning experience, and can provide a more positive attitude generally to the giving, and receiving of feedback from both staff and students alike.


Introduction
The complexity of learning and assessment makes a prescription for effective feedback problematic and even where feedback has demonstrated utility in one circumstance this may not be transferrable between groups (Sutton & Gill 2010). Even successful feedback may not be sustainable in terms of tutor time and effortand the withdrawal of such feedback has a demotivating impact upon students (Kluger & DeNisi 1996). Nevertheless there is consensus in the literature that quantity, type and method of feedback differentially impact its effectiveness. Understanding these phenomena may provide guidance to tutors making decisions in their local circumstances (Sadler 1998;Hattie & Timperley 2007). The idea of audio feedback is not new, for example see Cryer (1987). However, screencasting includes both audio comments and visual cues such as the assessor highlighting text, moving their cursor and accessing resources. The student sees what the tutor sees and hears what they think, screencasting exposes the tutor's thoughts, presenting a number of potential advantages:-1. It could increase the amount of useful, specific feedback provided.
2. The mixed modality of situated visual cue and audio comment may focus attention more effectively than written annotation.
3. It may increase the time students spend engaging with feedback.
4. Vocal cues (pace and intonation) provide additional information, helping students understand the tutor's intention.
5. It potentially models how assessment criteria are applied.
Here we focus upon how screencast and audio feedback might impact upon three aspects of feedback -clarity of language, student and tutor expectations and the crucial affective component of receiving feedback.
Clarity of assessment feedback is often problematic because students find the language used in assessment comments difficult to understand and decode (Macfarlane-Dick & Nicol, 2004, Carless et al., 2010. Boud & Falchikov, (2006) note the "systematisation and formality found within educational institutions" can in itself obscure meaning. In terms of screencast feedback literature, verbal comments appear to offer some communication gains. Feedback is reported to be easier to decode (Cullen 2011) and more personal (Marriott & Teoh 2012). Voice intonation helps students to focus and ameliorates problems with tutor handwriting, students also felt tutor voice conveyed authentic emotion such as enthusiasm (Marriott & Teoh 2012). Students experiencing written, audio and screencast feedback expressed a preference to receive feedback in screencast form (71%) citing the situational visual cue as useful additional information (Marriott & Teoh 2012). There may be a mismatch between tutors' and students' expectations of assessment criteria, with students emphasizing low-level task oriented skills (Gibbs & Simpson 2004). This may lead to them expending effort which has a low-level impact upon their learning. It is therefore likely that feedback which explicitly references assessment criteria may improve feedback (ibid). Macgregor et al., (2009) suggests audio feedback may be more closely aligned with these pedagogical concerns than written feedback. Specific improvements included audio that established a personal relationship, clarified expectations, was detailed and easy to decode. Several authors draw direct parallels with tutorials, suggesting audio feedback could emulate these meetings, with the additional benefit of re-playability (Brearley & Cullen 2012;Macgregor et al. 2009;Marriott & Teoh 2012). What is less clear is the effect on learning outcomes which in some cases appears to match control groups receiving written feedback.
In contrast (Ice & Richardson 2009) found that audio feedback increased both student involvement in the assessment process and retention of learning content.
Assessment is an emotionally charged process. Falchikov and Boud consider that in the worst cases assessment can have an impact that inhibits learning and lasts for many years. Student perceptions of "fairness" appear to be directly related with levels of student engagement with feedback (Sutton & Gill 2010). Therefore, it is likely that feedback which clearly communicates justification of comments or marks may be particularly effective. Students' feelings about feedback are mediated by their relationship with the tutor (Sutton & Gill 2010). Feedback produced by tutors perceived as "uncaring" may be perceived as a "lack of interest and dismissed" (ibid, p.9). Interestingly the perception of care (or lack of) may originate from the feedback comments themselves, with generic feedback perceived as a powerful signifier of an uncaring tutor and conversely specific individual feedback equated with notions of care (ibid). Dialogic feedback can help to make the intention of feedback clear and reduce some of the emotional charge inherent in assessment (ibid). This hypothesis is borne out in the screencast literature, where students were more likely to collect and use audio feedback and to request more tutorials than with written feedback (Macgregor et al. 2009;Lunt & Curran 2010). Students also report replaying such feedback on multiple occasions (Brearley & Cullen 2012). Recorded feedback therefore provides a mechanism to operationalize Macfarlane-Dick & Nicol's view of effective feedback, where "...comments should indicate to the student how the reader experienced the essay as it was read -'playing back' to the students how the essay worked -rather than offering judgemental comments" (2004). Students' report that audio feedback was personalised to them and discretea factor likely to engender a sense of "tutor care" (Marriott & Teoh 2012).
Overall, the literature suggests that the majority of students preferred audio/screencast feedback to written, referencing clarity most frequently (Lunt & Curran 2010;Brearley & Cullen 2012;Marriott & Teoh 2012;Cullen 2011). For example in one study 85% felt that audio feedback was high quality, detailed and useful in identifying missing elements and improving subsequent work; 75% felt this was better than in written comments (Lunt & Curran 2010).

Methodology
Prior to this study the three researchers had conducted individual pilots of various modalities of audio and screencast feedback, subsequently collecting a range of evaluative data. In general terms this involved producing formative assessment against a module rubric by recording a screencast or audio file summarising the main points of feedback and 'feedforward'. The feedback provided was semi-structured, focusing on achievements and improvements to take forward to the summative assessment. The audio files were attached to the electronic submission and returned to students via a VLE. Written comments were also provided on the scripts where appropriate.
In this research the authors maintained a range of screencast practices, whilst standardising the student perception data collection tool through a standard survey mechanism. Sutton & Gill's (2010) conceptual framework was adopted to focus on feedback's relationship to students at practical, epistemological and ontological levels. This framework informed the following categories of questions:- Student reaction and action when receiving feedback.
 Frequency that students revisit feedback.
 Whether feedback informs and affects the tutor-student relationship.
 Affective componentsemotional reaction to specific feedback they have received and why this might differ between assignments.
The respondents' (N=35) comments were then analysed for common themes related to these aspects. Specifically, a thematic analysis was undertaken, based on a theoretical approach to qualitative analysis (Braun and Clarke 2007), the common themes sought in the data were informed by the relevant literature on audio feedback. Given this approach, there was an increased need for all three of the academics to remain independent throughout and not to generalise the data collected. In order to tackle this recognised risk of bias, a multiple analyst approach was adopted for the data analysis, in that all three of the academics engaged in the process of qualitative analysis. This allowed for robust triangulation.

Results and Discussion
From the qualitative data collected and analysed, a number of common recurring themes emerged which illuminated differences between text and screencast feedback modalities. The most prominent of these were -the perceived advantages of screencast feedback contrasted against deficits of traditional feedback, students' subsequent use of feedback and student's emotional response to feedback (often related to their perceptions of tutor "care").

Students comparing Screencast versus traditional text feedback
Many of the qualitative comments focussed on the clarity of screencast feedback comments when contrasted with traditional text comments.
"With traditional feedback there can be a feeling that it is rushed if the comments are short / grammatically difficult to follow [this is] definitely a positive step. Avoids the misinterpretation of the written word on a feedback sheet. You can also obtain a lot more feedback in terms of volume and is not subject to difficulties in reading handwriting." "The audio feedback was exceptionally useful. It not only discussed generic advice, but also tailored specific advice to my paper and focused on areas that I had not quite answered correctly. This will be useful for the actual exam, as I will be able to focus my study on these aforementioned areas to strengthen my understanding." In addition, some students commented on the dialogic nature of screencast feedback and the efficiency of conveying a greater volume of information in the same timeframe.
"I think it is easier to digest, sometimes written comments lack meaning and need clarification by having a conversation with the marker, whereas this is a more colloquial conversation type feedback which can be more explanatory." "….it takes far less time to explain a point. What can be said in a few seconds would usually need a paragraph if written. Therefore, it is a more efficient way of giving feedback and also allows for more in depth feedback." It appears that voice intonation provides useful cues and reassurance as well as pacing information into manageable chunks.
"I prefer it as hearing the feedback in my tutors voice in a conversational style leaves little room for misunderstandings." "Excellent way of giving feedback as the comments were more detailed and I liked the way you were walked through your work and areas that could be improved were highlighted." Students clearly preferred the audio feedback. This seems to be due to a number of factors. Students perceive that audio comments have greater clarity than written comments both due to clarity of language and an amelioration of any difficulties with reading handwriting. The comments seem to indicate that the students receiving audio feedback would have little need to seek clarification from the tutor on the feedback given. It would be interesting to explore whether audio feedback has the potential to be a time saving exercise in terms of follow up student appointments following the provision of feedback.
Increased efficiency in the audio feedback process allows time to be focused on more indepth feedback. It is far quicker to speak than write so by providing audio feedback the tutor is able to spend the same amount of time producing the feedback, but has more time to convey more information. Students value the voice intonation in an audio recording, which cannot be replicated in the written word. This was not anticipated by the tutors prior to the research. It seems likely that greater clarity in the feedback would result in higher rates of engagement with the feedback in preparation for summative assessment. This can only serve to increase the effectiveness of such feedback.

Students' use of feedback
The mean number of occasions students reported listening to the screencast feedback was 3.0 (SD 1.8 occasions). There was wide variation, one student viewed the feedback more than ten times. Overall 88% of students reported viewing the feedback more than once. Unfortunately the authors do not have comparative quantitative data for other similar modules where students received traditional feedback. However students have reported anecdotally that they rarely view written feedback more than once: "The screencast feedback seems to be more personal [...] again it is paper based then it is only checked, once received, then filed away." This project did not make use of analytics which would show the number of times content was viewed, the sections viewed, time of day and platform -this would prove useful data for any future research and would be an approach recommended for other researchers to adopt.

Students' emotional response to feedback
Student reactions were overwhelmingly positive though caution is needed around novelty or halo effects. Students positively evaluate this new method and report faith in its authenticity. It is important to note the student perception that the feedback is personal to them. All feedback should be personalised for that individual student, but in terms of written feedback there was a general perception that it was more generalized or boilerplate. The comments that audio feedback is more personal, may positively benefit the tutor-student relationship. Interestingly students were equally positive about negative and positive feedback messages:-"Excellent, I could actually hear your disappointment in my work rather than just reading about it!" Many comments referred to the time taken by tutors to produce the feedback, equating this with notions of "care" about the work and for the student. Interestingly, the production of this feedback took approximately the same time as written feedback.
"It made me certain that the tutor had in fact looked at all of my work in a detailed manner as they made reference to key points chronologically." "It was also nice to know that the tutor has actually taken the time to record feedback to each student." "[I would prefer] Audio/screencast for sure -I think a lot of written feedback is copied and pasted and generic -it doesn't feel tailored whereas this definitely was." Some students made direct reference to the conversational elements of the feedback, which they saw as valuable:-"It's amazing how the throw away comments sometimes give you the best understanding of where you went wrong." Overall 82% of students expressed a strong preference for screencast feedback in their future assignments, 12% expressed no preference and 6% would prefer written feedback. The latter group explained that they preferred to be able to skim feedback to reference a specific point which is certainly a limitation of screencast feedback. The data above also reinforces the desire by students and tutors that the feedback actually be fed forward to aid learning progression and improvement, specifically for any summative assessment. It is significant that the perceived emotional engagement of the tutor has been taken on board by students. The data above shows how audio feedback has developed student emotional investment in the feedback process leading to students spending more time reviewing their work.

Conclusion
Students expressed a strong preference for screencast feedback. They value the quantity of detailed feedback, its sequential narrative of situated comments and the voice intonation cues it provides. Students often felt screencasts represented greater tutor "effort" and equated this with notions of "care". In reality tutors expend equal effort, but screencasting exposed the assessment process directly. Once exposed to screencasting students expressed negative feelings towards written feedback. It seems likely that audio feedback can greatly enhance learning through assessment, and can impact the tutorstudent relationship positively.
In future research it would be useful to consider whether audio feedback is scalable to larger cohorts. It would also be interesting to explore whether student self and peer assessment, combined with reflection would be a fruitful way to employee audio assessment in a studentled context. Finally it may be useful to focus upon an analysis of staff experience in providing feedback in terms of how modality of feedback affects quantity and type of feedback along with affective concerns for the tutors producing such feedback.