Design and Evaluation of Gamification Experiences in Computer Science Studies

This paper presents two gamification experiences developed in the Computer Science (CS) degree at the University of Barcelona. Usually, the CS degree suffers from low class attendance, which impacts the participation of students in in-class programming activities. Additionally, this degree includes a basic course related to human computer interaction that, despite of being valuable for the formation of students, they feel as boring and useless, far from their heavily computer focused interests. Then we decided to gamify a basic course of programming skills and the mentioned HCI course. We decided to use two different formats, online and physical. First, one experience was performed in “Human Factors in Computing” course of third year of studies, with 70 enrolled students. First, students attended to a 3D online theoretical class related to emotional design in “The education district” (TED) platform, which is a 3D Virtual World (VW) for educational purposes, developed by Virtway corporation. Afterwards, they participated in the ‘Game of thrones’ gamified activity where they evaluated the usability of TED. Second, we designed ‘physical’ gamified classes to increase attendance to Data Structures course, and in consequence, augment the number of programming exercises that students perform and the assistance they receive from the teacher. During the gamified sessions, the 120 enrolled students enhanced their skills in solving programming problems. The experience consisted of three kinds of challenges (with easy, medium, and hard difficulty level) solved in large, medium, and low-sized groups of students, respectively. The results of both experiences were satisfactory as evidenced by the percentage of students (>=66,7%) that rated the activities between 6 and 10 (in a 0 to 10 scale).


Introduction
Gamification is the use of game elements in non-game contexts (Werbach, 2012). Whatever the context, gamification design should consider two key elements: the goal and users' profile. Undoubtedly, any product design, either in architecture, fashion or gamification areas, works for the purpose and the users it was created for. What is right for a goal and profile of users may not be right for others. Additional factors that gamification design should contemplate are mechanics and dynamics that fit users' profile, assessment as well as needed resources and the format (online/offline, duration, activities) of the experience.
Gamification started in the business field but soon expanded to other contexts such as health, tourism, and education (Pereira et al., 2014) (Rodriguez et al., 2019 (Xu et al., 2013). Gamification design is a task that should be well thought so that it is important to follow a gamification design framework such as FRAGGEL (Mora et al., 2015) and LEGA (Baldeón et al., 2016a).
The literature of gamification in education shows, on the one hand, works that are entirely integrated in the curriculum of the course and thus the gamification extends to every teaching session (Iosup, 2014) (Mora et al., 2016). On the other hand, other gamification initiatives (Baldeón et al., 2016b) focused on concrete parts of the curriculum that, consequently, occupied part of the course planification, e.g. one or a few course sessions This research aligns with the latter works. This paper presents two gamification experiences for Computer Science undergraduates. Our students perceive human interaction course as boring and useless, far from their -heavily computer focused -interests. We also have students with low class attendance, which impacts their participation in in-class computer programming activities. Thus, we decided to gamify two different courses. The first experience was performed in "Human Factors in Computing" of third year of studies. The experience lasted a session of two hours and was performed in the 3D virtual world platform TED (https://www.theeducationdistrict.com/en/). The main goals were to introduce to the students concepts closely related to gamification, you will forgive the repetition, in a gamified environment, and to let them practice an usability evaluation method learned in the course while interacting within the 3D interface. Class attendance was voluntary, 40 out of 70 students were enrolled in the online class. The second experience was carried out in "Data structures" course of the first year of studies. The number of students was 120, although the gamified experience was designed and deployed in three groups of 40 students. This experience was developed in (physical) classes oriented to solve programming problems that make use of data structures.

Learning context
The course "Human Factors in Computing" focuses on human-centered design, User eXperience (UX) guidelines and user evaluation studies. The goals of the gamified 3D online class were twofold. First, make the students active participants of an emotional design, i.e the design of attractive and delightful user experiences. And second, give them the case study of TED to practice heuristic evaluation, a technique introduced in the course, which consists in finding good and bad usability aspects in the design of an interface (Nielsen and Molich, 1990). Collaterally, we also aimed to break the course routine with a disruptive activity, stimulate teamwork and students' creativity. The designed activity made use of gamification mechanics such as on-boarding, rewards, challenges and social engagement (we put them in italics when referred during the text). Regarding gamification assessment, we use a satisfaction questionnaire and analyze the participation of students and their grades in the heuristic evaluation.

Gamification design
The experience was performed in "The Education District" (TED), a 3D virtual environment (3DVE) to facilitate online learning. Although many of our students were usual players of MMOGs (Massively Multiplayer Online Games), which have features in common with 3DVE, this time they were to make "a serious use" of VW.
The session was divided into two parts (see Figure 1): a theoretical class in which the teacher presented the emotional design lesson in an outdoor space (garden) of the 3D virtual world; and a gamified activity in which the students collaborated in groups to construct a 3D building. Note that, while the students were building they had to elaborate a heuristic evaluation (usability) report of TED's user interface. The theoretical class lasted 30 minutes and the gamified session lasted one and a half hours. The former consisted on several stages: game rules explanation, houses' formation, castles building and conquering, and castles scoring. In the following, we detail them. Game rules. The teacher introduced the activity: "Welcome to game of thrones! (a happy murmur denoted that students were full delighted and excited). The houses of thrones, Stark, Targaryen, and Lannister, shall build their castles. The king leads the construction and hopefully make strategic decisions. The castle should have at least 2 towers, 2 access key doors, 2 windows, and the flag of the house. Afterwards, you must conquer your enemy's house. The teacher decides when castles construction begins and ends, and the house you have to conquer". Finally, students were informed about the information point situated in brick's landing point, with the game rules (on-boarding).
Groups formation. The teacher asked the students to sit in three groups (social engagement), and, to speed up the process (time was gold for us), grouped them by last name. To choose each king, the teacher repeated the following process three times. She asked a question related to the course, both via voice chat and textual chat. The student who wanted to answer by voice, raised her hand, but they also could send the answer by text chat. The latter was the preferred option (maybe it was the quicker way and was better for shy students). The teacher gave hints: "The answer contains the E", "it begins with "J", etc. The first student who answered the question correctly became king of one house of game of thrones. If nobody answered it or nobody got the right response, the teacher gave more clues. If they did not get it right either, she asked a new question. The king of each house led the distribution, monitoring and coordination of tasks to be performed by the members of his house during the castles building, and received a prize for developing this task (reward).
Castles building and conquering. Students had 30 minutes to build the castles (challenge, cooperate within the group, compete against other groups). The construction was facilitated by 3D bricks tools. Towers, windows, doors and walls of the castles were done by creating, moving, rotating different kinds of 3D shapes. Once the castles were finished, each house had to (challenge) conquer the enemy house, accessing the castle of another house and placing their flag. The 2 gates of the castle had words of passage, which were the answer to a question related to some lessons of the course. Students were informed about it so that they could review the material before the online class. When the students touched the gate of the enemy's castle, the question appeared and then, they should type the response. If the response was correct, then the door opened. The quicker house placing their flag on the enemy's castle received the reward of +1 point in the exam (group reward). The king of the winning house got a reward, the possibility of labeling a multiple-choice question of the exam as "King of game of thrones" (avoiding negative mark if the answer was incorrect) (individual reward). As a consolation prize, the other houses (i.e. players who did not win any reward) could win up to 0.5 points in the exam whether they sent the teacher, in the next few days, the heuristic evaluation report (group reward).
Castles scoring. Once the mission of building and conquering the castles was finished, participants should score competitors' castles. Unfortunately, we had no time to do it.

Gamification deployment and evaluation
We asked the participants to rate their satisfaction filling a questionnaire. Figure 2 show that 64,3% and 76% of the students scored greater than 6 (in a 0 to 10 scale) the garden and the bricks activities, respectively. Hence, we see that students enjoyed both experiences. Nevertheless, a few students complained about their Internet connection that impeded them to participate in the construction activity seamlessly, we guess they could be the ones that gave 0 score (right part of Figure 2). Some students found the construction activity quite stressful (In short time, they had to construct the castle, set the passage words, and set the flag), and they complained it was the first time they manipulated objects in this 3D virtual space. Regarding the heuristic evaluation report, 47% of the students did it, and the average grade was 9,3 (out of 10).

Learning context
The goal of "Data Structures" course is to provide students with the basic skills on data structures as well as to train them in the application of the different data structures in computer science problems. In addition, a secondary goal is to improve student's programming skills and their mastery of C++ language, since the resolution and implementation of complex projects that include data structures depend on this ability.

Gamification design
We decided to include gamification in this course in order to increase engagement of the students. Students are in their first course in CS degree and they still need a large amount of teachers' assistance for solving problems. Basically, our goals were: i) to augment student attendance to the classes devoted to solving problems; ii) to improve student understanding of course content; iii) to enlarge their skills for solving problems.
The course content is divided in different sessions: theoretical, problem solving, and laboratory sessions. We had a total of 120 students in the course. Students were separated in two theoretical groups (i.e., 60 students per group), three problem-solving groups with 40 students each and six laboratory sessions, each one with 20 students. Considering the main goals of our course, we decided to apply gamification to three of the problem-solving sessions out of 10 total sessions. Related to competences, the experience aimed to stimulate the work in group, creativity in problem-solving and improvement on their skills to move them towards self-sufficiency. The design made use of mechanics such as rewards, levels, social engagement, and medals that will give them a prize at the end of the course. We propose to assess goals achievement with an online questionnaire at the end of the course.
We divided the gamification of the course in three problem-solving Olympic sessions. Each one of them corresponds to a different level to acquire and it is focused on different topics of the course. First Olympic session was focused on the acquisition of a high level on C++ programming skills. Second one was centered in linear data structures, such as stacks, queues or lists. Finally, third Olympic session was devoted to the use and application to real problems of non-linear structures, such as binary trees or AVL trees.
It is important to mention that all the Olympic sessions had a duration of an hour and a half. First of all, before starting the problem-solving Olympic sessions, we detail the purpose of the gamified activity to the students and contextualized the activities that will appear in the session. Additionally, the teacher divided randomly students in large groups (10 students per group), by using a deck of cards. It was important to separate randomly the students to avoid alliances among them and to facilitate the learning progression of all of them. We spent 10 minutes to introduce the session. Afterwards, we started the Olympic session, which is divided in four activities of 20 minutes: an activity in large groups, another one in medium groups, an activity working in pairs, and an individual activity.
The Olympic session always starts with the activity in large groups. It was devoted to successfully solving a large problem-solving but easy in difficulty to favor social engagement and let students to understand the basic concepts of the subject. Next, students in large groups are reorganized in subgroups to form medium-size groups (4 or 5 students per group) and they start with the second activity, whose purpose is to introduce a problem with a medium difficulty that they must solve in 20 minutes. We maintain social engagement and we augment the difficulty with problems in which students need to apply advance concepts. In the third activity, groups are separated to form pairs of students. We focus the third activity on solving a medium-high level but short problem in pairs to consolidate the acquired skills. Finally, the last activity is focused on evaluating the final individual assessment of students. We prepared a Kahoot 1 for this evaluation. Every Olympic problem-solving session offers 100 reward points. Every activity had a reward for the participants that depends on the overall assessment and ranking, which are 15 for the winner in the activity in large groups, 25 for the winner in medium activity, 30 for the winner in activities in pairs, and 35 points to the winner of the Kahoot. It is important to highlight that all students receive a reward when they solve a problem (in groups, pairs or individually) although it is decreased according to the ranking.
With the three Olympic problem-solving sessions students were able to obtain a maximum of 300 points. Note that there are 100 points per session and after each one we publish a ranking of students to increase participation in the next one. Additionally, at the end of the course, we divide the ranking in quartiles. Those students that are in the first, second and third quartile receive a medal. First medal has a prize that consists of an increment in the final grade, second medal's prize is an increment in the exam, and third medal's prize is extra time to do the exam. Those students that fall in the last quartile do not receive a medal. The idea is to promote cooperation and social engagement with group work and a bit of competition among the students with the levels, medals and the prize that they may obtain.

Gamification deployment and evaluation
We asked the participants to give their opinions filling a questionnaire. First, we requested students to evaluate the Olympic sessions. Figure 3 (left) shows that 66.7% of the students scored greater than 6 points the sessions out of a total of 10 points. Note that 44.4% scored their satisfaction with 7 or more points. It is worth mentioning that a few students complained about the time for answering each one of the questions of the Kahoot. We will define much more time to answer this part next course. Finally, we also requested students how much useful they considered the Olympic problemsessions had been for understanding, learning and mastery in the topics of the subject. Figure  3 (right) shows that 66.7% of students considered that the Olympic sessions had been useful in improving their comprehension and skills in the Data Structure course, since they scored the questions with 5 or more points.

Conclusions
This paper presents two gamification experiences performed in CS studies. In the first experience students first attended an online 3D class of 'Human Factors in Computing" course and then competed in the 'Game of thrones' activity. The goals were to teach the emotional design lesson while making the students protagonists of a gamified (emotional design) activity, and to give them the case study of TED to practice an usability evaluation method introduced in the course. Most students had a satisfactory experience (60% scored it between 7 and 10), and liked the novelty and the dynamics of the activity. The second experience was performed in "Data Structure" course. The design was based on the Olympic games, where students worked in different group sizes to improve their knowledge and skills on problem-solving in the subject. Considering that 66,7% of students scored their satisfaction higher than 6 (in a scale 0 to 10), we will repeat the DS Olympic games next course. To conclude, the big effort of designing a gamified activity worthwhile. Thus, those teachers who plan to do it should have the resources (time, tools, assistants). In our case, TED platform proved to be secure and scalable. Although further research is needed, it seems that online 3D platforms help with students' engagement. Moreover, in the case of short-duration activities, we recommend, if possible, to perform a pilot activity to evaluate timing and anticipate problems (e.g. technical, coordination, misunderstandings) and their solutions.