Assessing the Teaching and Achieving of Research Skills

We present a method to empirically analyse the relation between faculty wide intended learning and concomitant achieved learning outcomes. The method is applied to the Master of Science in Business Administration (MSc BA) program of the Faculty of Economics and Business (FEB) of the University of Groningen. Intended learning is measured from the students’ exposure to seven research aspects taught according to course specific learning goals. The achieved learning outcomes are derived from the grades that the FEB professors gave on these research aspects when grading the students’ final Master theses. We find that the intended learning of the research aspects in the MSc BA program does not benefit the students with a previous Bachelor of Science background at the FEB. However, external students with no previous FEB background do benefit from such teaching. Because of these plausible results we expect that our method may also be relevant for accreditation committees/managers of other Business Schools. Moreover, the application results may also be interesting for professors who (intend to) teach research skills.


Introduction
To be eligible for accreditation of the Association to Advance Collegiate Schools of Business (AACSB), a business school has to prove that it is able to achieve its targets 1 .In this context we focus on the program of Master of Science in Business Administration of the Faculty of Economics and Business (FEB) of the University of Groningen.This program aims at teaching content as well as research skills.We focus here on the intended teaching of the research skills and the concomitant learning outcomes.
At the FEB the basic research skills are taught in the Bachelor of Science program, while more content specific research skills are acquired by students in the Master program.In their final thesis the students have to show that they master the research skills.These research skills are measured at the end of the Master program by grades given on seven aspects of the Master thesis.These seven aspects are also explicit in several courses that precede the Master thesis.By studying whether the exposure to the courses in the program do indeed relate to the grades that students receive on their final Master thesis, we relate intended learning to the achieved learning outcomes at the curriculum level. 2 This paper contributes to the knowledge of teaching and learning in at least three ways.First, the AACSB accreditation assumes that a positive relation should exist between the intended learning outcomes and the achieved learning outcomes.Our method 3 may assist in giving evidence of such a relationship.Second, our results may be interesting for professors who teach -or intend to start to teach-research skills.
Third, we show what student groups (male/female, good or not, internal/external) benefit most of the teaching of the seven research skills.

Background, data and method
The dataset consists of students who finalized their Master program in the period 2008-2011.Students had to be exposed to courses with at least one research aspect before finalising their Master thesis and there should be an assessment form available with scores on all seven Master thesis aspects.4The passing grade for the thesis is a 6 and the maximum grade is a 10 (excellent), though the grade 9 (very good) is already rare.The frequency of the grades in our sample of 692 students is: 6 (sufficient, 18%), 7 (satisfactory, 49%), 8 (good, 28%) and 9 (very good, 5%).5 The FEB target is that at least 80% the students score standard (grade 7) or higher.
We measure the exposure of students to research skills teaching by analysing the actual courses that the student passed successfully.For each course with an intended teaching of research skills we know what research aspect(s) the course covers.Each of the covered aspects is then multiplied by the number of ECTS (European Credit Transfer and Accumulation System) multiplied by 28 hours for the specific course.Then all the hours for each aspect are summed over the courses that the student followed before finalising his or her Master thesis and that sum is divided by the total number of hours in the official program, namely 1680.This means that if a research aspect is taught in a 5 ECTS course, that this aspect gets a weight of (5*28)/1680=0.0833.A student who follows four 5 ECTS courses with this aspect, gets a weight on the relevant research aspect of 4*0.0833=0.333.The average on all seven research aspects for each student is dubbed "Overall Research Skills Exposure", or ORSE.The ORSE averaged over all students is 0.322.6The variety of the exposure to research skills teaching proves to be relatively large at the FEB.One reason for this is that the students could choose between nine programs within the MSc BA.Another reason is that quite some students started their Master program before the intended learning outcomes were instituted.A third reason is that many students followed more courses than the minimum number required.
We also know the gender of the student and whether the student was an external student, or had a Bachelor degree from the FEB.66.8% of the student population is male and 42.3% has an external background.Students are also dichotomized in the 4 The seven aspects that are used to evaluate the Master thesis are: 1) Introduction.This requires from the professor i.a. an evaluation of the initial motive and an assessment of the relevance of the problem from academic and professional perspectives.2) Research framework.This assesses whether the problem is adequately grounded in the literature, and whether the research objective, and research questions and/or hypotheses are clearly formulated.3) Research design.This aspect requires an evaluation of the research method and the data.4) Analysis, discussion and/or design.This evaluates whether the research data are adequately presented, analysed, interpreted and discussed.5) Conclusions and recommendations.This evaluates the relevance of the conclusions and whether the thesis conclusions are related to the thesis objective, and questions.6) Report.This aspect requires that the professor evaluates whether the report is well structured and that the references are complete, consistent and that the argumentation is clear and convincing.7) Process.Finally the professor assesses whether the student worked independently and made effective use of feedback. 5Detailed descriptive statistics of the data and additional information are available in a paper upon request. 6It is tempting to conclude that therefore on average 32.2% of the official Master time is research related.That is, however, not necessarily the case, because the relative time weights of the research aspects within a course are unknown.group of students with an average grade per previous attempt (AGPA) below the median (Low AGPA) and students with an average grade above the median (High AGPA).

Analysis
Ordered probit analysis (not reported) showed that the grades for all research aspects were significantly and positively related to the final grade on the Master thesis.Factor analysis (not reported) also showed that all seven research aspects scored high on the major factor.Therefore we measured whether the average exposure of the teaching of research skills aspects (ORSE) influenced the grades of the several student groups (not reported).This proved to be significant for the external students only.So the research skills teaching in the Master year had relevance for the students which did not get the previous FEB Bachelor's research skills training.Then we checked whether the grades given with the Master thesis on each of the seven research aspects depended on the exposure of the teaching of these aspects, on the interaction of the aspects' exposure and being an external student, and on the other two student characteristics (male/female) and (good previous performance or not).Because the grades on the research skills aspects follow an ordering from low (6) to high ( 9), we apply ordered probit analysis.
Table 1 shows that the exposure of internal students to teaching individual aspects is only significant for aspect 5 (conclusions and recommendations), but negatively so.External students, however, do benefit significantly from the teaching on several aspects as can be seen from the interaction of each aspect's teaching and the external students' dummy.The teaching improved external students' performance on aspect 1 (introduction), aspect 4 (analysis and discussion), aspect 5 (conclusion), aspect 7 (process) and (marginally significantly) on aspect 6 (reporting).Students with higher quality as indicated by higher average grades on previous attempts (AGPA) do perform significantly better on all aspects, while male students underperform significantly on aspects 1, 2, 3 and 6.Table 1.The impact of aspects teaching and the grades received on the seven aspects (Asp.) with a distinction between teaching internal and external students, MSc BA cohorts 2008-2010 (p-

Conclusions
We analyse the relation between intended learning outcomes and achieved learning outcomes of the MSc BA program of the Faculty of Economics and Business (FEB) of the University of Groningen.We focus on the intended teaching exposure that students received on seven research skills.We relate the teaching exposures to research skills received by individual students to the concomitant grades that they received on their final Master thesis.All research skills aspects are important in generating the grade for the final thesis.However the teaching of research skills does not influence the grade on the Master thesis positively, except for the students with no Bachelor background at the FEB.
These results would make it important for the FEB management to consider if the teaching of research skills should be improved for internal students, or whether it is satisfied with the low research skills benefits for internal students (who still have learned specialization specific content) and the positive benefits of teaching research skills for the external students.This choice was however not explicitly made, as there has been an overhaul of the program that focused even more explicitly on research driven education.We therefore suggest that the current research is repeated for the FEB in the future and at a less ad-hoc basis (Moskal, 2008).Such a research would, moreover, become more precise if explicit knowledge were available on the amount of time spend on teaching the several aspects in each course.
Also other Business Schools might benefit from our research, if they want to analyse the results of their teaching on a curriculum level, by applying a similar method.We show that this can be done without having recourse to assessments made by students on how much they learned from these aspects in separate courses.If there is a relatively large variety of courses available from which students can choose, our method gives quantitative results which may help the boards of directors to pinpoint teaching weaknesses and to improve teaching quality.Finally, we also think that professors who (intend to) teach research can benefit from this research.

1st
International Conference on Higher Education Advances, HEAd´15 This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License (CC BY-NC-ND 4.0).

1st
International Conference on Higher Education Advances, HEAd´15 This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License (CC BY-NC-ND 4.0).

1st
International Conference on Higher Education Advances, HEAd´15 This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License (CC BY-NC-ND 4.0).