U-Behavior: Visual-Form Learning Analytics to Enhance Teaching and Learning

Learning analytics have great potential to support students’ learning process and instructors’ learning design, specially when presented as a visualization, visual-form LA, designed in conjunction with student reflections. This presentation represents a multi-year mixed-methods study that collected students learning analytics from participation in retrieval practice activities, low-stake quizzes, and presented this data as visual-from LA to help students to be cognizant of and reflect on their learning practices in order to improve retention and recall by implementing high impact learning practices. Quantitative and qualitative data was collected, analyzed, and integrated to generate insights regarding the impact of the design on students’ study behaviors and students’ self-awareness of these behaviors. Findings suggest that the integration is successful in developing autonomous learners that more often recognize and implement effective learning behaviors.


Introduction
There is considerable empirical evidence that provides students with guidance on the most productive way to spend study time. Zimmerman's self-regulated learning model is a threephase cyclical process that encourages students to regulate their study behavior and time by monitoring time including planning (forethought), practice (performance), and self-reflection (Zimmerman, 2000). Additionally, the testing effect, which is the power of retrieval as a learning tool (Brown, Roediger, & McDaniel, 2014), requires students to retrieve information through self-quizzing, planning their retrieval of content over time and mixing up their practice. This effect is strongest (i.e. has the biggest impact on student performance) when students practice retrieval with time at the center of their strategy (Brown, Roediger, & McDaniel, 2014). Although these well-established learning techniques provide students with guidance on how to spend their time, students often behave in less-productive ways (Karpicke, 2009).
This research paper describes a multi-year study that aims to use visual-form learning analytics (LA) to help students to be aware of and reflect on their study behaviors as well as to persuade learners to change their learning strategies. Visual-form LA consist of representing learner data as a visualization (Authors, 2019). To produce the visual-form LA, the researchers developed U-Behavior.
The study was conducted with a primary objective of teaching well-established learning strategies, such as retrieval, spacing, and interleaving, and apply those strategies in course assignments. U-Beehavior is a method that uses retrieval practice activities (RPAs), implemented in a learning management system (LMS) as low stakes self-regulated quizzes. The intention of the RPAs was to create an opportunity for students to implement the various learning strategies they were discussing in the course and to observe their own study behaviors after learning about retrieval, spacing, and interleaving. The aim of the study was to determine if learners can be persuaded to implement more productive and beneficial learning behaviors through reflection and self-regulation.

Literature Review
This section presents the literature used to support the study. A review of the current literature related to the use of learning strategies, particularly low-stakes quizzing, to improve retention and recall is highlighted. In addition, the concepts of visual-form LA to bolster learning is discussed.
The behavior of testing one's knowledge is a highly effective way of learning and retaining information (Bjork & Bjork, 2011;Larsen, 2018). Specific behaviors such as active retrieval of information through testing, spacing out one's testing practice, and interleaving (mixing up) subjects and content that is tested, are supported by decades of empirical research that reinforces the positive effects of these learning behaviors (Brown, Roediger, & McDaniel, 2014). Furthermore, empirical evidence suggests that these effects translate into the acquisition and retention of conceptual, procedural, and metacognitive skills (Roediger & Karpicke, 2006;Kornell, & Bjork, 2007;Bjork & Bjork, 2011).
To be considered a high impact learning practice, testing, or quizzing, must be understood as a learning strategy instead of an assessment tool (Brown, Roediger, & McDaniel, 2014). As a learning strategy testing boasts several benefits for learning such as: retrieval aids later retention; helps to identify gaps in knowledge; increased learning in subsequent study episodes; produces better organization of knowledge; improves transfer of knowledge to new contexts; can facilitate retrieval of material that was not tested; improves metacognition monitoring; prevents interference from prior material when learning new material; provides feedback to instructors; and encourages students to study (Roediger, Putman, & Smith, 2011). Studies have shown that low-stakes quizzes that allow multiple attempts, with or without feedback, removes the anxiety of high-stakes quizzes/exams, encourages the learner to focus on the material instead of on a grade, and helps students to perform better and achieve longterm learning (Brown, Roediger, & McDaniel, 2014;Trail-Constant, 2018).
Learning analytics are an effective tool for supporting students and informing instructional design (Gasevic, Jovanovic, Pardo, & Dawson, 2017). One way of helping students to make use of learning strategies that will contribute to their long-term learning is to show students their LA as a tool for supporting and improving their study behaviors. Presenting these LA as a visualization, visual-form LA, provide insight into how students are engaging with the activites instructors and designers are creating for them (Ellis and Goodyear, 2010) and is one way to provide students with clear feedback regarding their learning behaviors (Authors, 2019). Ware (2004) questioned how the visual representation of digital products could change users' behaviors by supporting their optimal decision-making process.

Methods
During Fall 2017, learners were given the opportunity to complete eight RPAs introduced as learning-based quizzes which they had the option to take up to ten times and retain their highest score. After the final retrieval practice activity (RPA) was offered, learners were presented with visual-form LA, in the form of a personalized RPA graph depicting their RPA attempt data, and prompted with reflection questions about the RPA graph and thus their learning behaviors. Based on the high score focus of the learners and their desire to compare their study behaviors with their classmates as well as the desire of the instructor to see learners engaging in the beneficial learning strategies presented in the course, an intervention reflective activity was incorporated in the Fall 2018 design. The intervention was introduced one month into the course, allowing students the opportunity to start building their knowledge regarding the content in the course. This intervention consisted of presenting RPA graphs from Fall 2017 and asking the students to reflect on them. At this time they were also told that they would be provided their own RPA graphs containinng personalized data of their RPA usage behavior near the end of the course and were prompted to forecast how they would like their RPA graph to look at the end of the course.
The researchers hypothesized that with this shift in expectation by the instructor, that students' RPA graphs would reflect effective learning strategies due to the presentation of exemplar RPA graphs and the understanding that the instructor and each individual student would actually see their personalized study behavior presented in visual-form as an RPA graph at the end of the semester.
The remainder of this paper will focus on describing the second phase (Fall 2018) of this research study, its findings, and how those compare with the findings from phase one conducted in Fall 2017.
This study used a mixed methods approach in which the investigators collected and analyzed both qualitative and quantitative data, integrated the findings, and then drew inferences using both qualitative and quantitative methods to provide a better understanding of the research problem (Tashakkori & Creswell, 2007;Creswell & Plano Clark, 2018). Prior to the first day of class, students were asked to complete the Study and Learning Behavior Pre-Survey. Once the course started, one RPA was offered each week in the form of a low-stakes quiz. After a month of classes, students were asked to complete the intervention reflection activity. After the eighth and final RPA was completed, researchers used the U-Behavior application to collect RPA attempt data and generated the current students' RPA graphs. Each student received their individual RPA graph and instructions to complete the reflection activity on their own RPA data. Lastly, at the end of the semester, students were asked to complete the Study and Learning Behavior Post-Survey.
This project was initialized in an online graduate level education course. Twenty-four adult learners were registered for the course with 19 consenting to be included in the research. The content of the course emphasizes research supported learning strategies, such as retrieval, spacing, and interleaving, and then creates opportunities for the students to apply the strategies when completing RPAs.
Overall this multiphase sequential study addressed the following mixed methods research questions: 1. What are students' perceptions of study and learning behaviors at the beginning of the course? 2. How do visual-form LA impact students' awareness of their study habits? 3. How do visual-form LA persuade students to change their study and learning behaviors?

Data Collection
The quantitative data was collected using web-based surveys and web-based retrieval practice activities. The pre and post surveys had seven multiple choice questions related to student's study and learning behaviors and were based on Kornell and Bjork (2007) studies. The U-Behavior application was developed by the researchers using Python and extracted the necessary data (quiz-log file) from the Canvas LMS. From this data, an RPA graph was generated for each student including the date and time of the attempt and the score received for each attempt. Each attempt is represented by a colored node and each of the RPAs offered during the course is identified by a particular color. Trend lines connect the RPAs taken in chronological succession. A single node not connected using trend lines represents either a single attempt of an RPA or interleaved attempts.
Qualitative data was collected through two web-based reflection activities. The first intervention reflection activity presented RPA graphs from the previous semester (fall 2017) and asked students to reflect on it considering the learning strategies and theories presented in the course. In the second personal reflection activity, students were prompted with a series of reflective questions based on their personal learning behaviors represented in their RPA graph. The reflection activity also included questions regarding the visual-form LA presented as an RPA graph so the researchers could collect feedback for future improvements to the RPA graph.

Data Analysis
Quantitative data analysis included descriptive statistics about the RPAs. RPA attempt data and learning strategy data (spacing, interleaving) identified in the RPA graph was also analyzed. Finally, comparison of data from pre and post web-based surveys to detect possible changes in students' study and learning behaviors after acquiring and practicing the various learning strategies studied throughout the semester was also quantitatively analyzed.
Qualitative data analysis of the students' RPA graph and reflections were analyzed and coded for the emergence of visual and written themes. The coding process was steered by one researcher and was conducted using Discourse Textual Analysis (DTA) (Moraes & Galiazzi, 2007). The DTA approach consists of building units of analysis from the corpus and identifying categories and themes from the previous units of analysis identified. It used the inductive method to look at categories that emerged from the units of analysis. The inductive method uses analytical induction (Lincoln & Guba, 1985) to identify categories, kinds of elements that have something in common. In order to ensure the trustworthiness of the findings, a second researcher reviewed the qualitative data and the categories. The two researchers conducted a social moderation process where they discussed the codes and categories until they came to agreement (Schaffer, 2017).

Findings
The primary difference between the two phases of research, Fall 2017 and Fall 2018, was the design in Fall 2018 which introduced the intervention reflection activity in an attempt to modify students' study behaviors. The researchers wanted to know if they were able to improve study habits by motivating students to incorporate various learning strategies based on the presentation and reflection of the exemplar RPA graphs from Fall 2017.

Retrieval Practice Activities
In order to determine any changes in study behaviors, the RPA graphs of each student were analyzed prior to the intervention reflection activity and post intervention. From that analysis two groups emerged: No Change Group -Students who did not change their behaviors after the intervention activity. Change Group -Students who did change their behaviors after the intervention activity.
Of the 19 students, 11, or 58% did not change their behaviors after the intervention activity. Of the 11 students, 36% (four students) practiced spacing and retaking prior to the intervention activity. They were categorized in the No Change Group, identified as not having changed their behavior, because they did not start to interleave after the intervention. So, even though they did not change their behavior, they already had used some of the learning strategies. In contrast, 64% (seven students) just retook the RPA , but did not space or interleave their practices. These students were focused on getting the highest score on each RPA, they are identified as high score oriented students.
Eight students, representing 42% of the total students, did change their behaviors after the intervention. Of those, one student started to space after the intervention, but did not interleave at all. Seven students, 37%, were already practicing spacing and retaking before the intervention and started to practice interleaving post intervention.
The data indicates that interleaving is the least practiced strategy utilized by students. Most of the students (63%, 11 from the No Change Group and one from the Change Group) did not interleave at all. The students who did interleave (37%, seven from the Change Group) only did so after the intervention.
The RPA graphs were analyzed to observe if students retook the RPAs after they had achieved the maximum score possible. Of the 19 students, only 26% (five students) retook an RPA after having achieved the maximum score. All five students were from the Change Group.

Personal Reflections
In coding the personal reflection activity, the themes identified from the Fall 2017 study (high impact learning practice, intentions versus engrained practices, high score orientation, and desire to compare) were used apriori themes. In addition, the researchers attempted to remain open to identifying emergent themes. The additon of a new theme regarding learning strategy application emerged in the Fall 2018 study.
From the No Change Group (students who did not change their behavior after the intervention activity), it was observed that the same themes from Fall 2017 were present and the presence of a new theme was identified, the application of learning strategies (AoLS) theme. The high impact learning practice (HILP) theme involves initial categories and units of analysis where students described the learning strategies they studied and how they identify them and understand them in relation to the RPA graphs. This differs from Fall 2017 where students mentioned a variety of learning strategies, students in the No Change Group from 2018 focused on spacing and retaking. The intentions versus engrained practices (IVEP) theme included commentss on study behaviors they applied during the semester, reflections about their learning behaviors, and reflections about the benefit of using RPAs. For example, Student N states that the RPA graph showed them falling into old study habits rather than applying the learning strategies presented in the course.
The High-score orientation (HSO) theme included students' description about how they focused on grades in spite of the low stakes nature of the RPAs and how they fixated on receiving the highest score possible. The desire to compare (DTC) theme involves students' desire to compare their performance with others. The application of learning strategies (AoLS) theme involves initial categories and units of analysis that demonstrate how students are applying the content they have learned from the course to other situations.
The analysis of Change Group students (students who did change their behavior after the intervention activity) included all of the same themes, however the high-score orientation theme was only present in one student's reflection. The HILP theme emerged in the change groups' reflections, and it extended beyond the No Change Group who primarily focused on spacing. In addition to discusing spacing the change group included pedagogical practices, such as interleaving, retrieval, and reflection. The IVEP theme for the Change Group was notable. Unlike studetns from the No Change Group, it was not surprising that this group discussed their shift away from engrained practices and how they actually changed their behaviors. The DTC theme was present for the Change Group, but fewer students mentioned their desire to compare their performance with their classmates as seen with the No Chanage Group students. The additional theme to emerge in 2018, AoLS theme, included the following reflections by students in the

Conclusion
The inclusion of persuasive design that incorporated an intervention illustrating the utilization or lack of utilization of various beneficial learning strategies along with the opportunity to reflect on this data created an opportunity to change student' behavior. This intervention had an apparent positive effect in student behaviors when comparing the Fall 2017 study with Fall 2018 study. The intervention resulted in more students integrating spacing, retaking, and interleaving. This study supports what is presented by Brown, Roediger, & McDaniel (2014), that testing, spacing, interleaving and varying practice, are perceived as less productive by students because they are harder to do than rereading and their effects are not immediately perceived. However, with this intervention the researchers were effective in changing the behaviors of some students. Quantitative and qualitative data provided evidence that after the intervention, students started to space and interleave their studies, using learning strategies that have been shown to contribute to long-term retention and performance. However, as identified by Karpicke (2009) some students persist in using less-productive strategies, thus the necessity of continuing to work on strategies that persuade students that testing, spacing, interleaving and varying practice are learning strategies that are valuable for long-term learning.