Questionnaire design in gamification process for education: a case study at Universidad de Guadalajara – Mexico

Gamification techniques have demonstrated that students improve their learning process through mobile applications. However, every teacher creates his/her own questions for the game design, involving classroom response systems through a digital app such as Kahoot!, Socrative, Blicker, Clickers, Plickers, etc., without previous planning of the difficulty in every question formulated to the students. This work focuses on the evaluation design of a questionnaire for the final test of the Power Electronics subject, following Bloom’s Taxonomy methodology for every level of critical thinking within the cognitive domain of the learning process for students. Furthermore, an analysis of the V-Aiken for the reliability and validity of each question were taken into account. The final test implementation was carried out at Centro Universitario de Tonala, Jalisco, Mexico, involving nine students of the Energy Engineering grade of the fifth semester. The results show an analysis of the performance of the students when applying a designed test based on revised Bloom Taxonomy. It was better because the student demonstrates, in a more integral way, his/her mastery skill in different topics of the subject, while the results of the not-designed exam showed a poor student performance because all of their knowledge and skills are not evaluated when it is not applied a proper effective question.


Introduction
Modern education demands technology innovation (Westera W, 2004). Due to the growth of student population and technology innovation, the learning environment in schools has been changing the way a student can learn interacting with the new educational digital platforms. Nowadays, educational institutions requires the inclusion of new information technologies within a classroom environment where students must have access to online digital tools (Tak, Wong, Yuen, & Wong, 2018). Although technology is gaining greater importance in the modern learning environment and for educational purposes, it is essential to remember that technology serves as a tool for teachers to deliver knowledg to the students. Students perceive gamification activities as a strong tool for the scholar assessment and the student performance, it is an important key to harnessing the power of gameplay for educational purposes (Kingsley & Grabner-Hagen, 2018). The main problem of gamification activities, especially in Kahoot!, is that questions asked on the game are often very easy to answer by students or any other person who wants to play a game of a specific subject. Many times these questions do not follow a methodology to be better formulated or answer options are simple to discard because they are not related to the question (Wang & Hoang, 2017). In other words, the set of questions made by a game results inefficient for the students and have no value for the learning process during the course. Students perceive gamification activities as a strong tool for the scholar assessment and student performance. It is an important key to harnessing the power of gameplay for educational purposes (Kingsley & Grabner-Hagen, 2018). As a consequence, the main objective of this work is to evaluate the differences between a set of normal questions made on Kahoot! and a second set of questions based on the six level of learning skills according to the cognitive domain of Bloom Taxonomy, in order to stimulate critical thinking of students. To this end, Kahoot! was used in the subject of Power Electronics belonging to the Energy Engineering degree at the University of Guadalajara, Mexico, as a case of study.

Bloom Taxonomy
It is important to evaluate the learning process of students because it is an effective way to provide a valuable feedback on the design and the implementation of the study program in a continuous process and a periodic exercise. (Rajšp, Beranič, Heričko, & Horng-Jyh, 2017). One way to accomplish the above is through a hierarchical model used to classify educational learning objectives into a level of complexity known as Bloom Taxonomy. It divides the way people learn into three different domains. Cognitive is one of those domains, which emphasizes the intellectual performances of people, then is the affective domain, based on people emotions, and psychomotor which involves manual or physical skills (Bloom, 1956). The goals of the learning process are to acquire new skills, knowledge, and attitudes. Bloom taxonomy is commonly used in applications for exam papers assessment where the cognitive level is the prime objective of a school subject (Sivaraman & Krishna, 2015). Furthermore, it is also applied in scientific writing papers, such as literature reviews, where the author needs to develop and demonstrate the intellectual skills for every one of the six categories of learning in the cognitive domain (Granello, 2001). In the beginning of the 21 st century Lorin Anderson, a former student of Bloom, revised the cognitive domain in the learning taxonomy making two big upgrades: changing the names in the six categories from noun to verb forms and rearranging the evaluation and synthesis skills as seen in Table 1.1, this new structure of Bloom taxonomy reflects a more active and accurate form of the learning process (Anderson, et al. 2001).

Gamification in Education: case of Kahoot!
Popular online games have shown very interest in students, fomenting the competitivity and superiority of the smartest student of the group (Rajšp et al., 2017). Many works have demonstrated that gaming platforms result pleasant and fun to play, even though many of them do not deliver knowledge, only serve as a tool of reinforcing the knowledge of students (Sola Guirado, Castro García, & González Sánchez, 2018; Stott & Neustaedter, 2014). One of the most popular gamification tools being used in the academic field is Kahoot!, which is a free e-learning platform with more than 30 million users (Bicen & Kocakoyun, 2018). Using kahoot! Proffesor can set up conducted learning games, design own tests. Furthermore, questionnaires can be projected in the classroom and each student can answer it via web or smartphone application, also question punctuation, student accumulated score and winner can be displayed motivating a competition environment (Ares, Bernal, Nozal, Sánchez, & Bernal, 2018).

Content Validity
One aspect to consider while designing the questionnaire in Kahoot! is the validity of the questions on the game. Content validity is mainly used to test whether the items possess representativeness and comprehensiveness, and whether or not the items tested fully reflect the contests and theories seen in class. Hence, in order to obtain a valid and reliable measurement instrument of each item, it is a must to follow a systematic procedure to achieve that. One approach is the content validity coefficient and homogeneity reliability coefficient proposed by Aiken (1980), which can be used to quantify the validity rating of each item into one coefficient called "V value". The procedure for determining the V coefficient begins with the ratings of m items by a single rater. Validity ratings can be made on any convenient scale of c successive integers. In this particular case, the ratings will be 4, from 0 to 3. When ratings of m items are made by one rater, the V coefficient for that rater is computed in Eq. 1.

∑ [ ( )]
Where V is the item validity index; s is the score assigned by each item minus the lowest score in the used category , with r equals to rater category selection score and equals to the lowest scores in the scoring category); m is the number of items; and c is the number of categories that raters can choose. The V index value ranges from 0 to 1. The closer an item to 1, the better it is (Aiken, 1985).

Methodology
In this research two sets of 13 questions were conducted; the first one evaluating normal questions of the Power Electronics subject and the second one evaluating the questions based on the six levels of learning of Bloom Taxonomy within the cognitive domain. Each item was accompanied by a Likert scale from 1 to 3 to reflect the judge's opinion with each statement, as seen in Table 2.1.
The Kahoot! session was held in April 2019, using student's smartphones. The session was divided into two questionnaire application; the normal test was first and the design test (according to revised Bloom Taxonomy) was second. A total of 9 students belonging to the subject "Power electronics" of the Bachelor Degree in Energy Engineering of the Universidad de Guadalajara took both questionnaires. By gender, 11.11% were women and 88.89% were men. Considering age, 90% were students up to 21 years old while the remaining students were over 21 years.
The missing questions of the table were not relevant to write them because they are on the same cognitive level of knowledge, comprehension or application, making the first questionnaire on the Low Order Cognitive Skill (LOCS). The second questionnaire is located on the High Order Cognitive Skill (HOCS) due to the complexity of the formulated questions. Table 2.2 shows the categorization of questionnaire one and two according to the cognitive domain of Bloom Taxonomy. It is notable that many items belong to the Remembering and Understanding skill, while items three and eight only belong to the Applying skill. Furthermore, questionnaire number one consists only on LOCS, while questionnaire number two applies LOCS and HOCS for the item design. 3. Checking all the responses given on each item with a significant standard of content validity coefficient (V value). 4. Applying questionnaires #1 and #2 to the students. 5. Analyzing the game's outcomes for interpreting the results.

Results
In this section, the results obtained are presented in Table 3.1, where each question was evaluated according to Bloom Taxonomy and calculating the V-Aiken value for the validation of items by three expert judges (J1, J2, J3), with a confidence index (CI) for both questionnaires.  Student 6 20,085 10 3 Table 3.2 and Table 3.3 shows the outcomes of the games on Kahoot! after the revision of judges on item analysis was done to questionnaire 1 and 2. One aspect to observe was that students who obtained good results in the implementation of the first questionnaire do not achieve the same results on the second questionnaire application. Therefore, the number of correct answers or points obtained during the game do not necessarily make a student more intelligent, but it is subject to several interpretations to really know the level of knowledge of the participants in the game and their skills and mental abilities.

Discussion
The goal of this study has been to design a proper questionnaire for gamification in education, according to a methodology for a better performance of student's learning objectives by applying the Revised Bloom Taxonomy. It has proven to be a good mechanism to develop and create more effective questions than only "yes or no" answers.
Perhaps not many teachers would like the idea to redesign a quiz for the time that involves thinking, analyzing and classifying each question depending on the skill level of the cognitive domain. Although students prefer applying the test on digital platforms and taking quizzes over online games perhaps reality is that educational institutions do not like the idea of changing panorama even though digital tools go hand-to-hand with the new generation students. Overall, content validity is a powerful tool to measure the evaluate student's performance. Bloom taxonomy is ideal for better construction of learning objectives. Kahoot! is a fun, dynamic and interesting platform where competitiveness and learning among people come to challenge.
The first questionnaire showed that questions on the game that lacks complexity and difficulty can be fun to play but the second questionnaire was statistically validated with the use of V-Aiken, showing that questions are more challenging and therefore, requires more time to think and analyze the problem. It is good to enhance student's learning and stimulate their cognitive skills. All of them, LOCS and HOCS, result as important as formative evaluation. Some future research projects that can be done are the addition of a greater number of judges, validators or teacher who are Subject-Matter Experts for a better questionnaire validation of the V-Aiken. Also, the item discrimination index and item difficulty index can take place in order to find those items which are extremely hard to answer or too easy to solve. With that, a more comprehensive analysis can be made to discard questions on Kahoot! games.