Iterative Design of a Gamified Course in High Education : deployment and evaluation

During the last years, gamification has been used to engage students in more attractive educational activities in different Computer Science subjects at the university level, thereby improving their motivation and learning outcomes. This work continues a research that initially proposed a gamified design for the course “Distributed Artificial Intelligence”, an optional undergraduate course of the Computer Engineering degree. Specifically, we focused on reinforcing subjects related to Multi-Agent Systems (MAS) by means of fun hands-on activities to experiment theoretical concepts in practice. A first iteration of the design was deployed during two consecutive academic years, with good results in terms of students’ perceived learning, engagement and commitment during the class. Nevertheless, a posterior analysis of the design showed that the proposed mechanics did not consider some types of players such as disruptors -, and some of the learning profiles such as theoretical and reflexive -. Then, we proposed a card-based game to redesign the learning experience, using a LEarner-centered GAmification Design Framework (LEGA) that aligns both educational and gamification approaches. This paper focuses on this second iteration of the design, which has been deployed and evaluated during the last semester. The obtained results show that students liked the card game, were engaged and motivated during the gamified class, as well as they perceived an increased learning in the subject.


Introduction
Gamified classes are built upon fun and dynamic activities that make the learning experience lively and stimulating.Nevertheless, the design of such classes -that intend to boost students' involvement and go deep on conceptual subjects at the same time -is indeed a challenge for teachers.
Gamification should be a process carefully designed because of the inherent complexity of intervening elements (goals, users, context and resources, game elements, etc.).Much research work has studied the design of gamification, as a result several gamification design frameworks have been proposed such as the Gamification Model Canvas (GMC) (Escribano, Moretón, & Jiménez, 2016), the MDA (Mechanics, Dynamics, Aesthetics) (Hunicke, LeBlanc, & Zubek, 2004), DMC (Dynamics, Mechanics, Components) or others (Werbach, 2012;Marczewski, 2015).However, these frameworks are general guidelines for scaling up experiences in a wide range of fields such as business, events, or education.More concretely, in the education field, there are particular characteristics that should be taken into account: learning objectives, student profiles and the educational context (Kapp, 2013;Simões et al., 2013).Some more specific gamification frameworks have been proposed for the gamification of learning (Mora, Zaharias, González, & Arnedo-Moreno, 2015) (Sheldon, 2011) (Baldeón, Rodríguez, & Puig, 2016).Most of these frameworks conceive gamification as a spiral process consisting of several iterations that serve the purpose of design, validate and refine consecutive gamified solutions.This work continues a previous research focused on the gamified design of a Multi-Agent System course in Computer Science degree from the University of Barcelona.Until now, we have performed two iterations of the gamification process.In the first iteration, we proposed several activities that revolved around the learning task of defining a market model (Baldeón, Lopez-Sanchez, Rodríguez, & Puig, 2016).Afterwards, and based on obtained results, we provided a second design of the gamification which extended the previous one incorporating new mechanics based on strategy cards.This extension takes into account students' learning and playing styles (Baldeón, Rodríguez, Puig, & Lopez-Sanchez, 2017).This paper concretely focuses on the deployment and evaluation of the card-based redesign.

Previous work
There is limited literature in the gamification of teaching and learning MAS.Melo et al. (2006) used a first-person shooter in the course curriculum of Autonomous Agents and Multi-Agent Systems, achieving highly motivated students.Fasli and Michalakopoulos (2005) integrated a simulation game in their graduate course on Agent Technology for E-commerce to teach students the principles of electronic markets and strategic interaction.They achieved to engage and challenge students.Another work included role-playing games when teaching the content of a MAS, where players understood the basis, were motivated and had fun (Barreteau, 2001).Sakellariou, Kefalas, & Stamatopoulou (2008) utilized a multi-agent programmable modelling environment to increase active learning, where students enjoyed and their level of satisfaction increased.Soh (2004) introduced a game-based technique to his Multi-Agent Systems class in four game days, where students' teams competed against themselves in games related to MAS issues, achieving motivated and wise students.Aligned with previous experiences, our work aims to improve students' performance and compromise by means of the gamification of several theoretical and hands-on sessions.

Gamification design
We have followed the design process as stated in LEGA gamification framework.Briefly, LEGA defines five stages that guide teachers in the task of gamification for learning.Figure 1 gives relevant details of every stage in the second iteration of the design.This figure is self-contained regarding Stages 01, 02, and 03.However, stages 04 and 05 are more detailed in the next section.

Deployment
The gamification was structured in two parts.In the first part, the teacher introduced the game to the students and explained the first task to be performed before starting the game: each student had to propose 2 multi-answer questions related to MAS concepts.The teacher selected 10 questions to be included in the Kahoot!1 (Activity 1).The card game started in the second part, planned as a 4-hour long class with a coffee break.Activities were focused on the design of a MAS market collaboratively.Concretely, the design of their market models (Activity 2), and design of interaction protocol between agents (Activity 3), and specify a complete market (Activity 4).Cards were being used by students i) to form groups to work on a type of market product (products deck), ii) to select market strategies (strategies deck), and iii) to develop some Skills and perform some Challenges in activities 2, 3 and 4. For instance, the Spy skill card allows the student to observe the work of other teams.The Juglar challenge card invites the student to recite in verse an oral exposition, and then win points.Note that we proposed Skills and Challenges cards in the re-design to contemplate different types of players -such as disruptors2 -not considered in the previous design.
At the beginning, the teacher gave the students a personal scoreboard sheet to annotate rewards along the game.Students that proposed Kahoot!questions selected by the teacher received extra points.Then, students played the Kahoot!, updated their scoreboard with the obtained points, and the top 5 students in the Kahoot!ranking drew a card from the Skills deck.Additionally, a Challenge card was distributed to each participant.
After that, to favour the creation of (three) groups with students that usually don't work together, students should draw one card from the Product deck (Party, Dron and e-Book).The teaching staff formed pairs of students in each group.Once the couple was formed, they agreed who assumed the role of provider and who the role of buyer.Each role adopted a strategy, from among a set of three possible strategies using the Strategy deck (Imitation, Innovation, Reputation leadership for provider role, and Buy cheapest, Satisfy requests exactly, and Be loyal to the provider for buyer role).
The game continued through activities 2 and 3 by pairs.Before each activity, each player had to play all the cards that he/she wished to use during that activity.To make the design of the model and the protocol, players had a limited time to discuss and develop a proposal.When time ran out, each couple presented their design to their team.The rest of the team voted each couple's argument using virtual coins, and the winning pair received extra points and drew a card from the deck of Skills.
The last activity was to specify a complete market (Activity 4) model by each team.Each team adopted the market model of the winning pair of their team on activities 2 and 3, and dramatized a possible execution of the complete market model.Before dramatization, all players played the Challenge cards in their hand that they wished to use.For each completed challenge during activity 4, students obtained some extra points.At the end of the session, each student had the opportunity to elaborate -at home -an individual improvement of the specification done in activity 4, by means of a video or blog, that was voted by the rest of the class.The game ended and the game-winner was the student with the highest number of points.At the end of the game, the results were added to the assessment of the student's progress in the subject.The top three ranked students obtained an additional point on the subject grade, an additional point in laboratory practice or a voucher for a correct question on the theory test.This mechanism was an extrinsic motivation to enhance student participation in the game.

Analysis of results
We conducted a survey to obtain data about students' perception of both learning and gamified dimensions.Related to learning, the gamified activities helped the students to become familiar and better understand the concepts of the subject (See Figure 2a, 68% and 32% of students thought that learned a lot and most, respectively).When the analysis is done by activities (Figure 2b), the three activities where the students perceived they learned the most were those related to market model design (where 28% of responses signalled Activities 2, 3 and 4), which are the activities where players used the cards of Skills and Challenges.The Activity 1, were they played Kahoot!shows moderate results in terms of perceived learning (14%).Nevertheless, students' opinions about how Activity 0, where they proposed Kahoot!questions, has impacted their learning is very low (2%).
Figure 3 shows results on the gamified dimension of the activities.In Q1, the majority of students (79%) would recommend attending gamified sessions to their classmates.Moreover, the students felt high commitment and motivation during the activities (Q2 and Q3, with 100% of responses between scales 3 and 5).When asked about attendance (Q4), 64% of respondent had a highly positive response.The survey included several questions related to each activity.For the sake of limited space, we don't show results of Activities 0, 1, 2 and 3 in graphs, but briefly explain gathered opinions.In the Activity 0, the proposal of Kahoot! questions, students were little motivated.This may be because the activity was concretely designed for the learning styles reflector and theorist, and as there are other learning styles then the level of appreciation is scattered.However, almost all students liked the execution of Kahoot!In Activities 2 and 3, students worked on the market and the protocol design.They liked to work in groups and to share their proposals with other groups.However, a few students did not like to present their proposals to all the class, they did not feel good when presenting orally to the class.
Figure 4 shows survey detailed results related to Activity 4, the complete specification of the market model.Note that many students liked to compete in obtaining the best possible market model and presentation, but 21% of the class had a negative response because they probably do not meet with the profile of player that most like this kind of (competitive) mechanic.Moreover, 89% of students scored between 3 and 5 this question, then they enjoyed the gamification dynamics performed as defining a theme and a narrative for their proposal presentation (Q6).However, for a few students, the dynamics of defining a subject and a narrative, as well as the act of competing and presenting their full market models were not of their interest (Q7).Some students did not pretend recognition or did not like the competition.We think these are students with a predominant philanthropic player style.When students were asked about the gamification elements such as the use of points, prizes and a leaderboard, only a few liked a little the use of these elements.This may be due to the fact that there were students that do not fit the "player" player style (motivated by rewards).Regarding the use of Skill cards, students enjoyed them, they highlighted their usefulness.However, some students indicated that the Skill card that they played did not give them an advantage because they considered that their cards only blocked the effect of another card without the opportunity to initiate some influence in the presentation of another group, or because there was no need to use it, or because the skill offered was not useful at all.Thus, we should balance and redefine Skill cards with negative appreciations.In the case of the Challenge cards, and due to the design of the session, only some of the cards were distributed, of which most were very well valued, nevertheless, for future editions of the gamification we think we should distribute more cards to students.

Conclusions
This paper focuses on the iterative design of a gamified class in Multi-Agent Systems subject of Computer Science degree.Concretely, we deploy and evaluate a second iteration of the design.In this second iteration, we propose a redesign -including activities and mechanics -for player and learner styles not considered in the first one.Students perceived that they had learned in almost all activities.They also enjoyed and feel committed during the class.Some cards have to be revised because they complained about their usefulness.Nevertheless, they ask for more quantity of others.In future academic years, we plan

Iterative
Design of a Gamified Course in High Education: deployment and evaluation (a) (b) Figure 2. Students' perceived learning a) Overall perception b) By activity.Source: authors (2018).

Figure 3 .
Figure 3. General results of the survey about the gamified sessions.Source: authors (2018).

Figure 4 .
Figure 4. Survey about the full market model specification activity.Source: authors (2018).