Engineering Graphics Concept Inventory : Instrument Development and Item Selection

More reliable instruments need to be developed to assess curricula and measure student learning. It is important to ensure that students properly understand fundamental concepts, as scaffolding learning on a poor foundation can have a negative cascading effect. A concept inventory is an example of an instrument that aims to assess student learning and identifying their misconceptions. Such an instrument is typically comprised of an assessment whose items are prudently chosen to test understanding of a single concept per item. The result of this careful selection of items for the Engineering Graphics Concept Inventory resulted in a 30-question multiple choice instrument that can be used to identify deep-seated misconceptions and to assess course outcomes. This paper will outline the development of the Engineering Graphics Concept Inventory, focusing specifically on developing distractors and the selection process for the items in the instrument. The instrument will provide a means to assess and streamline curricula for engineering graphics educators.


Background
At present, there is no nationally standardized means to assess misconceptions and competencies in engineering graphics.Such an instrument could be beneficial to graphics educators within the engineering community, as it will allow them to identify which misunderstandings prevail the most.The development of the Engineering Graphics Concept Inventory (EGCI) has been a result of the combination of efforts from researchers at Embry-Riddle Aeronautical University, Purdue University, Penn State Behrend, NC State University and The Ohio State University, made possible through the support of the National Science Foundation.The development of the assessment began with a Delphi panel of graphics experts who originally identified 10 fundamental concepts of engineering graphics, and is now in its final stages of completion.
The selection of items used in the EGCI has involved the consultation of graphics experts and several meetings where the research team gathered to identify important concepts, create items, and refine the collective instrument over several iterations to make certain the CI will assess the concepts it was designed to measure.Each item required thoughtful consideration to ensure a high quality instrument to accurately reveal any deep-seated misconceptions held by students regarding the identified concept

Introduction
A concept inventory is an assessment tool "designed to determine the degree to which students understand the concepts of a subject and to identify misconceptions and misunderstandings held by students" (Jacobi et al., 2003).Numerous concept inventories have been developed for a variety of subjects.The force concept inventory (Hestenes et al., 1992) was the seminal work that preceded concept inventories in thermodynamics (Midkiff et al., 2003), statics (Steif & Dantzler, 2005), heat transfer (Jacobi et al., 2003) and now engineering graphics (Nozaki, et al., 2016) each of which can be used to assess course outcomes and student misconceptions in their respective domain.Engineering graphics was identified as a suitable candidate for a concept inventory as it is typically found in the broad range of engineering curricula, and there is currently no instrument for assessment.The various educational and personal backgrounds of students contribute to a different understanding and perspectives of the concepts in engineering graphics; having an instrument that could assess such divergencies early on in the course of study is one way that the instrument may be helpful.Since 2013, a team of researchers from five different universities have collaborated on the development of the engineering graphics concept inventory (EGCI) (Nozaki, et al., 2016).The intended goal of the EGCI is to provide a means of identifying misconceptions with topics related to engineering graphics and to provide a mode of standardizing curricula for various instructors teaching the same course.
The framework adopted to create and validate the EGCI was the "assessment triangle" which has three interconnected key elements i.e. cognition, observation and interpretation.Cognition refers to the way in which students are believed to develop understanding, observation deals with the instrument utilized to assess the knowledge and interpretation involves the analysis of the measured results (Pellegrino, Chudowsky, & Glaser, 2001).
The first step for developing a concept inventory is the identification and classification of fundamental concepts in the subject area.A Delphi panel of graphics professional identified 120 unique topics that were considered important in the area of engineering graphics (Sadowski & Sorby, 2013;Sadowski & Sorby, 2014).These topics were further categorized into 10 main concepts.These concepts, which are shown in Figure 1, were used in the creation of a pilot study by the research team.

Visualizing in 2D and 3D
Projection Theory

Mapping between 2D and 3D Parallel Projection Methodologies
Planar Graphical Elements Drawing Conventions

Sectional Views Dimensioning
Methodologies for Object Representation Solid Modeling Constructs The pilot study comprised of 60+ open-ended questions that addressed each of the ten concepts.Student responses and interpretations were coded.Incorrect student responses formed distractor creation for the initial multiple choice version of the instrument.In order to identify misconceptions, the correct responses in the instrument are accompanied by multiple distractors, each deliberately selected to identify a single misunderstanding of a particular concept.In doing so, this ensures that when a student selects an incorrect answer it can be attributed to a particular misconception.Thus, each item in the assessment has one correct response and three incorrect responses.
Instrument development is best done in iterations to successfully refine instrument attributes.Once distractors were generated for each item, two additional iterations of the EGCI assessment tool, were used to develop the final version.The current instrument is comprised of 30 items and tests five different concepts -Sections, Projection Theory, Mapping Between 2D and 3D, Planar Geometry, and Dimensioning.

Method
The first step of item development in the project was a pilot study that was conducted with engineering students as participants.The participants were asked to respond to a series of open-ended questions.Incorrect responses from the initial assessment were then analyzed, and student narratives helped make it possible to categorize the misconceptions.The items and distractors formed from the pilot study responses were used to create the "alpha" version of the instrument.With a complete instrument, the research team used the psychometric quantities of difficulty and discrimination of each item, given by equation 1 and equation 2, respectively  = 100 −   * 100 (Eq. 1) Where D is the difficulty index, C is the number of correct responses for a particular item, N is the total number of responses, R is the discrimination index, H is the number of correct responses in the high performing group, and L is the number of correct responses in the low performing group.Based on Weirsma and Jurs (1990), the top 27% of participants were high performers while the bottom 27% were low performers.
Using the difficulty score and discrimination index, items from the alpha instrument that met predetermined criteria were selected to form the "beta" version of instrument.The items were again multiple choice and had a range of difficulty from very easy to very difficult (with a difficulty index between 20 and 80), and an adequate discrimination index.
Items with a discrimination of 0.30 and greater were determined to be good items and kept on the instrument, as this suggested that students who performed well on the instrument tended to score well on those items while low performers did not.
The overall objective of the assessment is to have several unique items per concept that measure student understanding of each concept.Distractors must only address a single unique fundamental concept.If any distractor addresses more than a single topic it is not possible to determine exactly where the students' misconceptions lie.Therefore, the distractors were intentionally chosen based on the responses to the open-ended questions in the alpha instrument such that only a single feature made the option incorrect.
In addition to the "traditional" engineering graphics concepts, the beta instrument also included Computer Aided Design (CAD) items which intended to measure the understanding of solid modeling.Later it was determined that the use of CAD items was not contributing to the instrument because for most items, there were multiple correct final solutions possible.Furthermore, many of the items had very low discrimination -meaning most participants were able to answer the questions correctly, regardless of overall performance.For more information on the selection of CAD items and the overall student performance on these particular questions refer to "Assessment of CAD Items for a Concept Inventory for Engineering Graphics" (Nabutola et al., 2017) The latest version of the assessment, the "gamma" instrument, consisted of 30 multiple choice questions which tested the understanding of traditional engineering graphics concepts without CAD.

Item Selection
Including items which test a single concept and clearly have one correct answer is very important in a concept inventory.An example of this can be found in the CAD items that were tested in the beta instrument, and were deemed unusable because they did not meet these criteria.Figure 2 shows an example of an open-ended CAD item tested.
Given the object below, describe in general terms how you would create it using a CAD system.You may use sketches to help your explanation.The question shown in Figure 2 can be answered correctly in multiple ways.For example, the "L" shaped profile seen in the right side view can be extruded, and a countersunk hole added directly to the base of the object as shown in Figure 3.This is an example of additive modeling where material is added to form the main features of the object.Alternatively, a rectangular profile can be extruded, and a smaller rectangular block can be removed from the resulting solid before introducing the countersunk hole as seen in Figure 4.This is an example of subtractive modeling where material is removed.These are just two examples of how the part can be modeled.The definition of a "correct" response in this case would depend on the modeling method, the CAD software utilized and the purpose/orientation of the part when in use -correctness can be quite subjective.Coding the responses would be extremely difficult, and subject to a high degree of confounding factors.As such, an openended question of this sort would not be very beneficial in developing a candidate item for a concept inventory.
The question shown in Figure 5 is the product of a well phrased open-ended question where there is one clear correct answer.The student is asked to select from one of four options the correct set of orthographic projections which correspond to the object shown.This item falls under the Projection Theory concept, and a student who answers this question correctly likely understands the layout, alignment and orientation of orthographic projections.Notice, there are four options, and each option tests the understanding of layout, alignment and orientation separately such that if a student is to select one of the incorrect responses it can directly be attributed to a misunderstanding of one of those three features.
Select the correct set of orthographic views for the object shown below.

Results
After asking suitable open-ended questions and appropriately categorizing the incorrect responses, the EGCI was created.From the 60+ questions originally used in the pilot study 30 multiple choice questions were chosen through an iterative process.These items, which all have a discrimination index greater than 0.30, cover the five concepts outlined in the introduction i.e. sections, projection theory, mapping between 2D and 3D, planar geometry and dimensioning.The gamma version of the instrument has been administered to over 800 engineering and technology undergraduate students at four different universities.The instrument is currently undergoing its final revision and should be available by Fall 2018.
The instrument can be used to compare the students' understanding of diverse concepts of engineering graphics and this information can then be sorted by instructor and used to relate the teaching style and time spent on each concept to the performance of students.This will ultimately provide a way to streamline instruction such that students are receiving the same information in pre-requisite courses as they advance into more challenging concepts.

Conclusion
A concept inventory can be a very useful tool to assess student learning if the items are judiciously chosen with deliberate distractors.The items presented need to have distractors that clearly show one particular misconception per option in order to accurately determine the source of misinterpretation.
With well posed open-ended questions and detailed responses from students, it is possible to develop an assessment that tests a number of concepts and indicate subjects where students' misconceptions affect them the most.This can potentially provide instructors an opportunity to revise their course content and utilize different resources and teaching techniques to target areas where students seem to be struggling.Another use can be to administer the instrument by multiple instructors at an institution and the results can be used to compare the performance of students for a number of faculty members all teaching the same course.

Figure 1 :
Figure 1: Engineering Graphics Concepts Identified by Graphics Experts