Differences in Research Literacy in Educational Science Depending on Study Program and University

The ability to purposefully access, reflect on, and use evidence from educational research (Educational Research Literacy, ERL) are key competencies of future professionals in educational practice. Based on the conceptual framework presented in this paper, a test instrument was developed to assess ERL, consisting of the competence facets Information Literacy, Statistical Literacy, and Evidence-based Reasoning. This contribution aims to delve deeper into the question of whether Educational Science students differ in their overall ERL proficiency depending on their study program and university. This comparison is based on a large-scale study of 1,213 Educational Science students (Teacher Training and Educational Studies) at six German universities in the winter semester of 2012/13 and in the summer semester of 2013. The results indicate that students seem to profit from their studies at the different universities. Moreover, the ERL competence facets differentiate to some extent between universities and degree programs, which can serve as the starting point for curricular quality development measures. Subsequently, the results are critically discussed, and the desiderata for future research are stated, e.g., the identification of predictors that cause the reported differences.


Introduction
Due to continuous scientific progress, the modern knowledge society is constantly changing (Grundmann & Stehr, 2012); hence, the ability to purposefully access, comprehend, and reflect on scientific information, as well as to apply resulting conclusions to problems, is important for social participation.When making educational decisions, this ability is referred to as Educational Research Literacy (ERL; cf., Shank & Brown, 2007).However, the ability to reflect on and use evidence is neither necessarily developed nor retrieved in an optimal manner in adulthood (Barchfeld & Sodian, 2009).As students, graduates, and professionals in Educational Science will soon be responsible for imparting relevant competencies to future generations, education and the systematic fostering of research competencies will play a central role.Future educators have to be trained to critically reflect on and use research knowledge in practice.Higher education institutions are particularly suitable for this because they provide research-based education.Therefore, research literacy is not only included in the general definition of the standards and objectives for higher education degrees (Standing Conference, 2005), but it is also formulated as a requirement for specific study programs in the field of Educational Science.In German higher education, Educational Science is used as an umbrella term for different study programs that address both the theory and practice of education and training.A typical study program in this field is Teacher Training, which aims to convey competencies in Educational Science, and it also features a strong practice orientation in subject-related studies and didactics.Since Teacher Training is federally organized, the nationwide Standards for Teacher Training in the Educational Sciences (Standing Conference, 2004) exist and include various requirements, amongst others, the demand that teachers should be able to engage themselves with, and reflect on and use, evidence from educational research in their work.Another frequently studied degree program is Educational Studies1 , in which education and training over the entire life span are covered both theoretically and empirically.In contrast to Teacher Training, the Educational Studies curricula vary considerably between higher education institutions; topics range from primary education, media education, and cultural education, to adult education and vocational training.
Traditionally, German higher education institutions offer one-tier study programs that lead to Diplomas or Magister Artium degrees, or they were completed, such as in the case of Teacher Training, by the so-called State Examination2 .However, following the Bologna Reform agreement of 1999, Germany has committed itself to switching over to the Bachelor and Master's degree system by 2020; this process was nearly complete as of 2011 Groß Ophoff, J..;Schladitz, S.;Wirtz, M.. (Federal Ministry of Education and Research, 2015).Conversely, in Teacher Training, only 11 of the 16 German federal states have implemented the two-tier degree system as of 2015 (Standing Conference, 2015).Blömeke and Zlatkin-Troitschanskaia (2013) emphasize that the ongoing reorganization and change processes that are currently underway in the heterogeneous tertiary sector in Germany require a theoretical and empirical foundation to both develop and implement sustainable measures of quality assurance and development.The investigation presented in this paper draws on this point by validating an instrument that measures ERL in higher education, and which is intended for assessment and evaluation purposes at the student, course, or institutional level.

Conceptual Framework
The core component of the conceptual framework of ERL is the so-called research cycle, which is described in theoretical (e.g., Pedaste et al., 2015) and curricular models (e.g.,

Methodology
The conceptual framework was used to develop a test instrument to assess ERL in higher education.To optimize the content validity of early drafts of this instrument, educational research experts (at the post-doctoral level or higher) reviewed the material.After the final revisions were made, more than 200 test items were available, each operationalizing one of the three competence facets (IL, SL, and ER).Due to assumptions of multi-dimensionality, and given the desired level of scale reliability, an incomplete block design was used in the standardization study to minimize participant strain (e.g., fatigue, motivation).To ensure standardized implementation, test administrators conducted the tests.During the test implementation phase, 40 minutes were allotted for test completion.

Results
Multidimensional latent regression models were analyzed using the R package Test Analysis Modules (TAM; Kiefer, Robitzsch, & Wu, 2016) to compare cross-sectional differences in the study programs at the different higher education institutions (see Table 1).Following the recommendations of Rose, von Davier, and Xu (2010), all omissions were treated as missing observations.Due to poor item fit (0.80 ≥ Infit/Outfit ≥ 1.20; cf.Adams, 2002), 33 items had to be excluded from the current analysis.Therefore, the results are based on 193 tasks (IL: 16%; SL: 37%; ER: 47%).The analysis draws on the comparison of the one-dimensional and multidimensional competence models reported by Groß Ophoff et al. (accepted).Depending on the focus of the analysis, either a unidimensional model or a three-dimensional model may be applied: Given that the general factor of ERL is dominant   The results for both models are reported in Table 2.The grouping variable (Study Program*University) was dummy coded, so that the regression coefficients represent the estimated difference between group means.The positive regression coefficients indicate that students in both study programs seem to profit from their studies at the different universities.The universities BW2 and LS4 seem especially strong in supporting ERL.However, while the Teacher Training students at LS4 were at the master's level, students at BW2 were at the bachelor level.As such, these results serve as an indicator of Teacher Groß Ophoff, J..; Schladitz, S.; Wirtz, M..

Differences in Educational Research
Training quality in ERL at the latter university.Students at RP3 showed the lowest (and for Educational Studies non-significant) regression coefficients.With respect to Teacher Training, this can be explained by the fact that ERL was not an explicitly stated study objective in the curriculum at that time.Somewhat surprising is the overall result whereby Teacher Training students show higher proficiency in ERL than Educational Studies students.One possible explanation for this is that the latter study program features a stronger qualitative research methodology orientation, which is somewhat contrary in nature to the quantitative focus of the test instrument described here.
With regard to the three-dimensional model, the various competence facets of ERL are somewhat able to differentiate between university and degree programs, despite the high intercorrelations of the subdimensions (r ≥ .46).For example, at universities BW2, RP3, and NW6, Teacher Training students showed lower ER proficiency when compared to their performance on the other two competence facets.Despite the overall competency differences between these universities, a similar trend was observed, insofar as performance on this facet of ERL was weakest.In another example, the rather small group of Educational Studies students at NW6 performed comparatively low in terms of IL; however, the negative determination coefficient indicates that the predictors did not contribute to the incremental validity of IL.

Conclusions
A reliable and economical test instrument was developed that is appropriate for verifying the significant and empirically plausible differences between study programs at different universities.The ERL model was proven to be valid, especially based on a one-dimensional model.However, the multidimensional model was also found to be substantial when assessing the competence facets SL and ER.Conversely, the differentiation between higher education institution and study program did not contribute to explaining variance of the competence facet IL.The strength of the presented research lies in the sample size and in the investigation of several universities from various German federal states.However, one must exercise caution when making inferences about institutional or curricular differences due to the non-representative sample included herein.Nonetheless, the results indicate that there are considerable differences in ERL between institutions and study programs.The results of another study, which compared the research competencies of German and Austrian Teacher Training students, further support this conclusion (Groß Ophoff, Haberfellner, Schladitz, & Wirtz, 2017).Thus, the assessment of ERL can serve as the starting point for curricular quality development measures, such as offering courses that address the entire competence spectrum (e.g., based on the Article Literacy Checklist; cf.Shank & Brown, 2007).This is further reinforced by the fact that less proficient students

Differences in Educational Research Literacy Depending on Study Program and University
are quite able to find and reproduce research information in tables, diagrams, and summaries, whereas only advanced students are proficient in evaluating scientific evidence and in critically appraising research-related conclusions (Groß Ophoff, Schladitz, Lohrmann, & Wirtz, 2014).To increase research competencies, it is also critical to foster self-efficacy in research, as well as to offer active participant learning opportunities throughout the course of study (e.g., Bell, 2016;Butcher & Maunder, 2014).
Given that the research presented in this paper follows the research cycle as well, the results necessarily raise new questions: Which predictors (e.g., institution-specific or coursespecific learning opportunities) produce the reported differences?Is it possible to identify critical proficiency levels?How can the development of different facets of ERL be reliably assessed and fostered, particularly at the curricular/institutional level?These (and other) questions need to be addressed in further research.
Literacy Depending on Study Program and University over the secondary factors, a one-dimensional model is feasible for assessing, and providing feedback on, learning gains at the student level.For the purposes of course or study program development, it is also acceptable to distinguish between the three ERL subdimensions, IL, SL, and ER.For the current sample of Teacher Training and Educational Studies students, the one-dimensional model (M1: AIC = 38,852; BIC = 39,892; CAIC = 40,096) appears to exhibit a slightly better fit when compared with the three-dimensional model (M3: AIC = 38,858; BIC = 40,025; CAIC = 40,254).

Table 1 . Number of enrolled students in the winter semester of 2012/13, as well as the sample sizes of the Teacher Training and Educational Studies students at participating German universities. University code German federal state Number of enrolled students: winter semester 2012/13 n (TT) n (ES)
standardization study was conducted in the winter semester of 2012/2013, as well as in the summer semester of 2013 at six German universities (see Table1).All of these higher education institutions offer study programs in Educational Science, and they particularly focus on Teacher Training and Educational Studies.The largest subsample originates from the smallest university (BW1).This institution is one of six Universities of Education (German: Pädagogische Hochschule) that emerged from teacher education academies approximately 50 years ago and developed into specialized Institutions for higher education in the federal state of Baden-Wuerttemberg.In other federal states, such as in the cases of RP3 and NW6, these teacher education institutions were integrated into the Educational Science departments of state universities by the 1970s at the latest.Similar to BW1, RP3 is a rather small university that places a strong focus on Educational Science and related disciplines, as well as on subject-related didactics.Together with BW2, LS4, and BE5, NW6 is one of the largest universities in Germany, and it offers a wide range of study programs characterized by a strong research orientation.In Teacher Training, these comprehensive universities typically tend to focus on subject-related studies.For this study, participants were recruited through participation requests that were made during lectures; participation was entirely voluntary and anonymous.The largest group of respondents is represented by 841 Teacher Training students, followed by 372 Educational Studies students.The samples of students from other study programs (e.g., Health Education and Educational Psychology) were considerably less frequently represented and were not considered further in this paper.A total of 74 percent of the total sample (N = 1,213) was comprised of female students with an average age of M = 23 years (SD = 3.40) and an average university entrance qualification grade (Abitur 3 ) of M = 2.4 (SD = 0.56).At the time of the study, universities RP3, LS4, BE5, and NS6 had already implemented the Bachelor and Master's degree system, even in Teacher Training.Only those universities in Note.N total = 1,213 students (Teacher Training and Educational Studies).Abbreviations: TT = Teacher Training; ES = Educational Studies.The Baden-Wuerttemberg (BW1, BW2) still carried out state examinations in Teacher Training programs at this time.The majority of the participating students were at the bachelor level (or at an early level) in their studies (number of semesters: M = 3.4; SD = 2.3).The only exception was LS4, where only one master's-level course in Teacher Training could be accessed (number of semesters: M = 3.0; SD = 3.1); these students were approximately six semesters ahead of the other study participants.