Enhancing students’ preparation for the professional field: A quasi-experimental study on a new community service learning module for first year pedagogical sciences students

In this paper, we describe the implementation of a new teaching module in the first year of a Pedagogical Sciences programme based on Problem Based Learning, Community Service Learning and co-creation principles. In this module, first year students answered a real-life pedagogical question for a project partner from a professional organization. Students ‘co-created’ solutions for the pedagogical-themed question by working together with a university expert and a project partner from a professional organization. Results indicated that students involved in this new teaching module scored significantly higher on a range of self-reported outcomes: feelings of being challenged, being able to link science to practice, feeling prepared for the professional field in general, and intrinsic motivation. Significant positive results occurred specifically when levels of co-creation were relatively high. This study underscores the importance of involving societal partners and challenging students to work on real-life problems very early on in academic education, that is, already in their first year at university.


Background
The academic study programme Pedagogical Sciences aims to optimally prepare students for their role as scientist-practictioners specialised in the relation between developing individuals and those (professionals and caregivers) involved in their upbringing and education. Academically trained pedagogues are expected to be able to develop and implement theoretical and scientifically based solutions for (new) educational issues, in close consultation with a wide network of societal partners. However, a nation-wide evaluation amongst all students in The Netherlands (i.e., the National Student Evaluation, NSE), indicated that students in Pedagogical Sciences programmes felt poorly prepared for their prospective professional field (NSE, 2017). A broad evaluation of the bachelor's study Pedagogical Sciences at Utrecht University in 2016-2017 also indicated that students missed the connection between their studies and their prospective professional field.
A focus group of Pedagogical Sciences students at our University pointed out that particularly the first year of the programme was very theoretical and did not allow students to practice one of the key skills required of scientist-practitioners: bridging the gap between science and practice (OC kamer, 2017). Also, more incidentaly, students mentioned that the lack of contact with their prospective professional field in the first year of their studies was a factor which negatively affected their study motivation.
To address these issues, we implemented a new teaching module in which first year pedagogical science students answered a (real-life) pedagogical question for a project partner from a professional organization. The aim of the current quasi-experimental study was to compare student self-reported outcomes between three versions of the new module that differed with respect to inclusion of a real-life versus a hypothetical problem, and the level of contact with the project partner from a professional organization. Self-reported outcomes measures included students' feelings of being able to bridge the gap between science and practice, their sense of preparedness for working as a professional in the field, their feelings of being challenged, and their intrinsic study motivatoin.

Educational principles underlying the new module
In order to address the issues indicated by the students, we combined elements of various educational principles and approaches in the design of the new teaching module: Problem Based Learning (PBL), Community Service Learning (CSL) and co-creation. PBL is a student-centered teaching method which students collaboratively work through facilitated problemsolving on open questions (Schmidt & Moust, 2000). Crucially, teachers have no fixed answers to these questions in mind; the students thus have "ownership" of the issues they are working on, and this is an effective ingredient in problem-based learning (Savery & Duffy, 2001). CSL can be described as experiential education in which students, teachers and societal parties work together on social issues. It is a form of education that encourages students to apply and enrich academic knowledge and skills by working on real-life issues in a way that creates value for societal partners. Intended societal partners are organizations, social enterprises and other initiatives that work on specific public issues in local communities. Students are more satisfied with their university when they participate in CSL (Eyler et al., 2001). Co-creation, a term originally stemming from marketing theories, refers to processes in which value is co-created by consumers through playing a collaborative and active role (Prahalad & Ramaswamy, 2004). This concept is also applicable in higher eduction, in which students act as co-producers in the learning process (Mavondo et al., 2004). Students who consider themselves as co-producers 'take full responsibility for their learning and use teachers and other resources to support their effort and ensure more successful outcomes' (Mavondo et al.,2004, p. 46).
In the new module that we designed, principles of PBL, CSL, and co-creation were applied by letting students collaborate in groups of four to five on a (real-life) pedagogical question for a project partner from a professional organization. Students became members of socalled professional "focus networks", together with a project partner from a professional organization and multiple university teachers with relevant expertise on specific topics and skills (expertise in terms of content, expertise in terms of academic skill training), allowing for co-creation between all partners involved. The answer to the pedagocial question was unknown to both University teachers and project partners, and students were thus given full ownership. Students were enrolled in the module for a full academic year. They began with clarifying the problem after consulting the project partner from the professional organization, performed an analysis of the scientific literature, and finally wrote an advisory report for the project partner. The value for the project partner, then, was the advisory report in which their question was answered based on recent scientific literature.
In the current study, we aimed to test the impact of the CSL and co-creation elements of the module on student self-reported outcomes, by manipulating the type of problem (hypothesical vesus real life) and the level of co-creation (no versus low versus higher cocreation) to test the impact.

Hypothesis
We hypothesized that first year Pedagogical Science students that worked on a real-life problem would score higher on all outcome measures (i.e, enhanced motivation, feelings of being challenged and being able to link science to practice, and feeling more prepared for the professional field in general) (PBL + CSL group), than students working on a hypothetical problem (PBL only group).
In addition, we expected that more opportunity for co-creation would lead to higher scores on all outcome measures.

Sample and procedure
A total of 246 first-year-students from the Pedagogical Sciences track at Utrecht University took part in the study. Half of the sample enrolled in the academic year 2017-2018 (group 1, N = 121), and half the sample enrolled in the academic year 2018-2019 (group 2, N = 125). At the end of the academic year, in June 2018 and June 2019, respectively, students filled in a set of questionnaires. At that time, 10 (8%) and 12 (10%) students had quit the programme, respectively.

Design and module description
A quasi-experimental design with three groups was used. Students in group 1 either received the new module, but without any contact with professionals in the field (N = 70, 1a, PBL only, no co-creation), or were enrolled in a pilot group for the new module which did include limited contact with professionals in the field (N= 22, 1b, PBL + CSL, low level of co-creation). Students in group 2 were all enrolled in the new module with more extensive contact with professionals in the field (N = 91, 2, PBL + CSL, higher level of cocreation). The content of the module was as follows, for each of these three groups: Group 1a: students were introduced to a current pedagogical theme at the beginning of the first year, and given a question relevant to this theme. The question was made up by teaching staff and novel in the sense that there was no pre-defined answer. An example theme was "online integrity" and a question within that theme was "How can we support parents to help their teenagers become smart users of social media, in order to protect their online integrity?". In the first course of the first year (running from September to October), students were requested to analyse the theme and question from multiple perspectives, such as those brought forward in traditional and new media, and the scientific literature. In the second course of the first year (November -January), students were requested to more deeply analyse the scientific literature about their respective theme and question. In the third course (February -April), students were requested to write an advisory report in which they specifically answered the question, based on scientific literature. In the fourth course (May -June), students practiced their presentation skills and gave a presentation at an end-of-year-symposium, in which they showcased their results to their fellow students.
Group 1b: students in the pilot group followed the same courses and made the same assignments as the students in group 1a, except that the theme and question that they were working on were real and came from a professional organization in the field. An example theme from a youth care organization was "teen pregnancy", and the question within that theme was "can we use real-care babies as an intervention for vulnerable teenage girls, to help reduce their risk for teen pregnancies?". Students were introduced to the theme and question during a visit to the professional organization in September, and reported their results back to the organization during a second visit in May.
Group 2: students in group 2 followed the same courses and made the same assignments, and the theme and question that they were working on were real and came from a professional organization in the field, like in group 1b. However, students in group 2 had more intensive contact with the project partner from a professional organization in the field. Specifically, they had similar meetings with the professional organization in September and May, and an additional interim meeting in February. In the additional interim meeting, students were requested to pitch multiple answers and pieces of advice, based on the scientific literature. In the interim meeting, the project partner of the professional organization provided direct feedback to these ideas, and informed students of which pieces of advice might be feasible to implement in the organization, and which pieces of advice may not be feasible to implement and why. On this basis, students tailored their advisory reports to the specific needs of the organization. Members of the professional organization were thus actively involved in the process during the academic year, allowing for more cocreation than in the pilot group.

Measures
At the end of the academic year, students in all groups filled in a number of questionnaires.
General evaluation questionnaire: Students were given a number of statements and asked to rate to what extent they agreed with each statement. Statements concerned, amongst other issues, students' level of motivation to work on the module's assignments (1 item), the level of challenge they had experienced when working on the module's assignments (1 item), and the extent to which they had learned about bridging the gap between science and practice (1 item). Students were asked to answer on a five-point Likert scale, ranging from 1 (disagree) to 5 (agree). These items were filled out by 102 out of 111 students (92%) still enrolled in the programme at the end of academic year 2017-2018, and by 86 out of 113 students (76%) still enrolled in the programme at the end of academic year 2018-2019. National Student Evaluation: Three selected questions from the Dutch national student evaluation (NSE) were used and adapted for use in this study. These items concerned students' sense of feeling prepared for working as a professional in the field. Cronbach's alpha was .76. This scale was filled out by 96 out of 111 students (86%) still enrolled at the end of academic year 2017-2018, and by 86 out of 113 students (76%) still enrolled at the end of academic year 2018-2019. (Vallerand, Pelletier, Blais, Briere, Senecal, & Vallieres, 1992;translated to Dutch by Stevens, 2016). An adapted version of the Academic Motivation Scale was used. Students answered 11 questions on intrinsic study motivation. Cronbach's alpha was .84. This scale was filled out by 104 out of 111 students (94%) still enrolled at the end of academic year 2017-2018, and by 79 out of 113 students (70%) still enrolled at the end of academic year 2018-2019.

Results
First, we compared student self-report scores between students who had had no contact with a professional in the field (1a, PBL only, no co-creation) and those who had had only limited contact with a professional in the field (the pilot group, 1b, PBL + CSL, low cocreation). A MANOVA was run with group as fixed factor and the three questions from the general evaluation questionnaire (motivation to work on the module's assignments, level of experienced challenge, and learning about bridging the gap between science and practice), and the average NSE score (students' sense of feeling prepared to work as a professional in the field) as outcome. There was no significant effect of group (F(4, 80)= 1.475; p = .218).
Second, we merged group 1a and 1b and compared student self-report scores between students who had had no or only limited contact with a professional in the field (group 1, no + low co-creation) and students who had had more intensive contact with a professional in the field (group 2, PBL + CSL, higher level of co-creation). Again, a MANOVA was run with group as fixed factor. There was a significant effect of group (F(4, 171) = 12.582; p <.001). Tests of between-subjects effects showed that group 2 scored significantly higher on all outcomes (p's < = .001), although the effect on motivation to work on the module's assignments was only marginally significant (p = .055).
Third, in a separate analysis, intrinsic motivation was compared between group 1 and 2 (note that these scores had been collected on a separate form, which contained no identifying information, and could thus not be merged to the data that was used for the MANOVA reported above). There was a significant effect of group (F(1, 183) = 7.982; p = .005), and group 2 scored higher than group 1. Figure 1 shows that the effect sizes of the differences between group 1 and 2 ranged from small/moderate to large and were largest for students' experienced level of challenge and their sense of feeling prepared for working as a professional in the field. Cohen's d of .20 is typically considered a small effect size; .50 is considered a moderate effect size; .80 is considered a large effect size (Cohen, 1992).

Conclusion and discussion
This quasi-experimental study shows that enrolment in a new teaching module in which elements from PBL, CSL, and co-creation were adopted, significantly enhanced a range of self-reported outcomes in first-year Pedagocial Sciences students: students' feelings of being challenged, being able to link science to practice, feeling prepared for working in their prospective professional field, and intrinsic motivation.
Notabely, however, no significant positive effects of the new module were observed when we compared outcomes between subgroups of students who had received a hypothetical question (PBL only group) versus a real question from a societal partner (PBL+CSL group), when the level of co-creation was small in the latter group. Specifically, only the unique combination of a PBL+CSL approach with relatively high levels of co-creation led to enhanced student self-reported outcomes. As the positive effects of the module were observed in the second academic year that it was implemented (i.e., cohort 2018-2019 received PBL+CSL with relatively high co-creation), a further underlying cause of the observed positive effects could be that the team of teachers and coordinators had gained more experience than in the first year of implementation (i.e., cohort 2017-2018, no to low levels of co-creation). A limitation of the current study is there was no random attribution of the students to the independent variables, so student cohorts might be confounding. Another limitation was the number of students who filled in the set of questionnaires at the end of the academic year.
Although drop-out rates were very similar across the two cohorts (academic year 2017-2018 and 2018-2019), participation rate was much lower in the second cohort. This effect was most profound for the Academic Motivation Scale (intrinsic motivation scale), where 70% of students filled in the questionnaire in cohort 2 compared to 94% of students in cohort 1. Since no characteristics of the non-participating students are known, an attrition analyses could not be performed. Therefore, it is unclear whether attrition was selective and may have biased the results.
To conclude, the current study underscores the importance of involving elements of PBL, CSL and co-creation in higher ducation, and attests to the value of such an approach very early on in the academic curriculum, that is, in the first year. Moreover, the study shows that the specific combination of PBL and CSL with relatively high levels of co-creation led to positive student self-reported outcomes. Furthermore, findings highlight the importance of piloting educational developments and taking multiple years to fine-tune such developments in an increasingly experienced teaching team.