A score methodology to assess concept maps in medical education in the context of pathophysiology teaching

Abstract The main objective of pathophysiology teaching is to facilitate the learning of mechanisms of diseases and the understanding of their expressions in patients (symptoms, signs and tests). This objective requires the application of basic biomedical science to explain the abnormalities expressed by the patients. The capacity to integrate this new organization of knowledge is essential to the understanding of pathophysiological mechanisms, which explain expressions of specific diseases. Our group has a longstanding experience in the teaching of pathophysiology to medical students using problem-based learning (PBL) and concept mapping (CM). This semestral discipline has a yearly intake of 190 students, divided into 18 tutorial groups and supported by 14 tutors. The students’ learning progress is evaluated by their performance during the tutorial sessions and the CM methodology has been introduced as an additional tool to visualize the integration of knowledge and how it is displayed in the different pathophysiological mechanisms. Until now, the evaluation of CM has been qualitative and used as an additional assessment tool by the tutors. The presentation will show how we are changing this approach by training the tutors and developing a scoring methodology, together with a preliminary application of the score to selected maps.


Introduction
Concept maps are graphical tools that depict relationships between concepts, representing the organization of knowledge structure. During construction of CM, students organize concepts related to a certain domain and link these concepts in a hierarchically organized knowledge framework (Novak & Gowin, 1984). CM have been increasingly used in medical education and in a recent review (Torre et al, 2013), it was clearly stated the importance of developing scores to assess the validity and reliability of CM to evaluate the students' ability to display acquired knowledge by connecting and integrating newly acquired information with previous knowledge and develop critical thinking (Torre, 2017). These skills are important in medical education where students have to integrate and apply basic sciences knowledge to the understanding of disease manifestations, diagnosis, and treatment. Pathophysiology is a discipline that helps to develop those skills.
Pathophysiology teaching was introduced at NOVA Medical School (NMS), in 1982 and pioneered the introduction of Problem-Based Learning (PBL) in undergraduate medical education in Portugal (Rendas et al., 1998;Neuparth & Rendas, 2016). The CM methodology was an additional educational innovation, starting in 2002(Rendas et al., 2006, to allow the visual display of pathophysiology learning during the analysis of the PBL cases. In the academic year of 2018-2019, it was decided to reorganize the tutorial sessions, reviewing the use of CMs without any change in the previous syllabus. The main reason for this review came from the feedback from the students of the previous years, namely 2017-2018, expressed in the anonymous responses to a standardized questionnaire, which is part of the institutional policy of NMS to evaluate the quality of teaching. Concept maps can be an important measure of knowledge structure in medical education, but there is limited information regarding the validity and reliability of CMs assessment scores (Schmidt, 2004). Since the introduction of CM in the pathophysiology course at NMS, no scoring methodology has been used and the assessment is still currently based on a qualitative analysis of the various components of each map. The current project aims to address this issue within the context of a whole course involving tutorial sessions to around 190 students, divided into 18 tutorial groups and accompanied by 14 tutors for a whole semester.
According to Torre et al. (2017), two main scoring systems are currently used: 1. Structural scoring based on the work of Novak & Gowin (1984) and adapted by West (2000) which considers five components: concepts, connecting links, hierarchies, cross-links, and examples. In this context, a concept is a label given to any information considered relevant and enclosed in a circle within the map; a proposition is a relationship between two concepts (two circles), with an arrow pointing to the direction of the relationship and a connecting word, or short sentence (connecting link), in between; in hierarchies, each subordinate level of concepts, usually more specific and detailed, is drawn below the more general, or global, propositions; cross-links, are connecting links between concepts or propositions displayed between hierarchical levels; examples, are specific objects or events that validate instances designated by the specific concepts or propositions.
2. Relational scoring developed by Yin (2004) aims to quantify the number of propositions in order to access the meaning of the relationships between the concepts, for example by constructing a CM with created connected words or with selected linking words.
In medical education, the situation is very complex because the clinical information about a patient with a specific disease usually encompasses various knowledge domains and new developments in science and technology are constantly applied to diagnosis and treatment. For all these reasons, we are reporting our work in progress to develop a scoring methodology that will consider these issues in the context of pathophysiological teaching and learning.

Background
In the academic years of 2017-2018 and 2018-2019, the pathophysiology unit was composed of lectures and tutorial sessions using CMs, based on an organ/body system approach, covering the following five blocks: digestion, respiration, circulation, renal and body fluids and the endocrine system. Each tutorial group participated in twenty sessions, four per block.
In the initial tutorial session of each block, the students received a clinical vignette (patient case), and discussed, with the assistance of the assigned tutor, the relevant information from the case, including the underlying pathophysiological mechanisms and the ways in which they can be visually displayed using CMs. The students installed and used the freely available software program Cmap tools (IHMC Cmap Tools, Florida, USA). In the following three sessions, the students, in smaller groups (3-4), improved the CMs, based on learning issues identified during self-learning which occurred between the tutorials. In the last session of each block, the whole group presented a final CM for discussion.
Concept maps were constructed to be read in a clockwise manner, beginning with the available clinical information of the patient in the center of the map (name, age), with the disease hypothesis and etiology placed on the right of the map. The pathophysiological mechanisms were placed in the center and below, hierarchically organized and related to the clinical information from the case. The final map also displayed multiple concepts placed at different levels of the hierarchy of the map and connected by cross-links (Fig.1). The qualitative assessment of the maps was based on the analysis of these different components. During the academic year 2017-2018 a coordinator group formed by the course director, the two most senior faculty members, a junior faculty member experienced on concept mapping and a medical student from the previous year performed an analysis of the CMs made by the students of that academic year and concluded that they were heterogeneous in many ways, namely regarding the structure (some of them were not readable in a clockwise manner, as described above, others would not have well-constructed propositions and the number of propositions used in each map was very variable). Regarding the content of the maps from different tutorial groups, the pathophysiological mechanisms were approached with different levels of detail and there was heterogeneity in the depth of representation of the main disease expressions within the same clinical vignette. The main hypothesis for this heterogeneity between maps produced by medical students was that there were lacking more strict guidelines and guidance for concept map construction. This analysis, together with the feedback given by the students to the questionnaire of NMS institutional policy for the quality evaluation of teaching were the starting points for the revision of pedagogical methodologies in the academic year of 2018-2019. The main steps introduced were a preparatory training period for the tutors and a revision of the CMs' methodology, including its construction, presentation, discussion and assessment criteria.
In the formative pedagogical training sessions, developed for the staff, besides the review of the pedagogical use of CMs, eight core concepts were identified for each clinical vignette, to provide a more homogeneous approach to the study of the mechanisms of disease displayed in the different tutorial groups. These core concepts were used to construct mini-maps, during the tutorial sessions in the following years, starting in 2018-2019. In each of the five blocks, the students constructed eight mini-maps, which were assembled and reorganized into a final CM, which was presented and discussed in the final session of each block.

Proposal of a CM scoring
The scoring methodology developed by the core group not only aimed to create a more homogeneous approach to evaluate knowledge acquisition, but also to provide guidelines and guidance for the students to construct the CMs, as it was stated by the teaching staff at the beginning of the academic year of 2018-2019. The proposed evaluation score is divided into two sections: qualitative (Table 1) and quantitative ( Table 2). The qualitative score intends to evaluate the overall structure of the map, including the clockwise reading methodology, the insertion of patient information from the clinical vignette, the inclusion of all eight nuclear concepts and their pathophysiological explanations and the usage of a legend. The quantitative assessment is based on the structural scoring from Novak and Gowin (1984) and adapted by West (2000), regarding usage of propositions, hierarchy levels, cross-links and examples. The number of propositions was limited to 25 in order to provide readable CMs in a printable A3-size and aiming to avoid the huge divergence of the number of propositions inserted between CMs that were assessed in the ones from the previous academic year. In this way, students were guided to choose only meaningful and pertinent concepts. Despite this overall educational planning the coordinator group decided, in the previous and current academic year, to apply only the qualitative score and to monitor the production of the maps in order to evaluate the effect of the training of the tutors and the increasing awareness of the students in the correct use of the CMs methodology.

The scoring methodology -work in progress
This proposal is currently being analyzed and tested by the coordinator group using maps produced in the last two academic years, by different tutorial groups from different tutors in order to assess the reliability and validity of the proposed score when applied to a large class of students, for a whole semester. Examples of the application of the score will be presented in the conference, using a limited number of selected maps.

General Organization
The map should have a general organization, using concepts and linking words, structured in hierarchies from top to bottom, from more general concepts to more particular ones. The map may also contain sequences of concepts, illustrating some pathophysiological mechanisms.
The map should be read in clockwise order, following the structure of disease definition, causes, classifications, pathophysiological mechanisms, clinical manifestations and diagnostic tests.
1 -badly structured map, limited number of connecting words, no hierarchical levels, not well-defined reading and structure.
2 -similar to 1 but with more connecting links.
3 -similar to 2 but limited hierarchical levels and disorganized structure.
4 -similar to 3 but with an organized overall structure. 5 -exceeding reference guidelines.

Maximum > 5 points
Information from the clinical case The map should contain the information relating to the clinical case, as described in the vignette, marked in a different color. The information should be appropriately inserted in the map and integrated into the pathophysiological mechanisms.

-no clinical information.
1 -some clinical information.

-some clinical information, well placed.
3 -all the clinical information displayed. 4 -similar to 3 but well placed.

-exceeding reference guidelines Maximum > points
Inclusion of core concepts Each clinical case has eight associated core concepts, which should be displayed in the map using different colors. The core concepts should be developed and integrated with the remaining information of the case, whenever applicable.
2 -all but not well placed in the map (lack of interrelations).
3 -similar to 2 and with an adequate display.
4 -similar to 3 and and well interrelated. Hierarchy levels Each hierarchical level corresponds to the new horizontal level of concepts originated by the linking word from the previous concept. Hierarchical levels are considered valid when the respective branch of propositions that originates has a pathophysiological meaning and corresponds to a progression from more general concepts to particular ones.

points per each valid hierarchy level
Maximum > 4x5 (20)

Cross-links
Cross-links are arrows with connecting words that relate concepts of different hierarchical levels and/or concepts of different map areas, in order to highlight a relevant pathophysiological relationship within a given context. Cross-links are marked on the map with a wider arrow (information to be included in the legend).

Examples
They allow exemplifying a certain concept, enriching the map. Considered valid whenever they are appropriate to the subject and have justification for their reference. The examples are marked in the map outside of circles or rectangles, in a simple text box, with an arrow whose linking word is "example" ("e.g.").