Adoption of evidenced-based teaching strategies in STEM and non-STEM courses after a common faculty development experience

North Carolina State University undertook a faculty development initiative, TH!NK, beginning summer 2014. TH!NK is a campus-wide initiative designed to develop faculty members’ abilities in cultivating students’ higher-order skills in critical and creative thinking and self-reflection. Faculty and courses in a wide variety of disciplines were involved in the initiative, with the ultimate goal being an institutional transformation in the way that teaching is approached across campus. This paper shares early outcomes of five years of the program, which engaged approximately 130 faculty members. We assess the adoption of teaching strategies and how adoption varied between STEM and non-STEM courses based on a 2019 survey of TH!NK-trained faculty (n=72). We observed that an intensive, multi-day, interdisciplinary faculty development institute, paired with longterm peer mentoring and accountability, led to a high rate of adoption of the strategies. While non-STEM faculty utilized a wider array of teaching strategies prior to training, both groups made gains post-training, with the greatest gains among STEM faculty. There were notable outcomes observed in faculty use of the strategies in other courses and sharing activities and assignments with colleagues inside and outside of their home departments.


Problem and Significance
In order to build the citizenry and workforce of tomorrow, learning environments must foster and support critical and creative thinking skill development. Higher education must shift the paradigm that often does not explicitly emphasize creative and critical thinking in lower level courses but then holds the expectation that students will be prepared for the higher-order thinking required in the capstone, project-based courses where students often struggle with the process (Atman et al., 2005).
Our campus has adopted a model for defining the critical and creative thinking process. The model incorporates raising questions and formulating problems; gathering and assessing relevant information; synthesizing and generating diverse ideas; considering alternatives; reaching reasoned conclusions; and effectively communicating. This process is iterative, with self-reflection on one's own thinking at the center. The process spans academic disciplines, though methodologies may vary. Explicitly articulating these skills helps develops student awareness, and also provides language to faculty for delivering specific feedback to students. While many publications have explored factors contributing to successful faculty development models (Czajka & McConnell, 2019;Lavis et al., 2016), we were unable to identify papers that quantified the frequency of adoption of evidence-based teaching strategies in classrooms after faculty training. Often, these data are not collected due to the lack of systematic evaluation of faculty development outcomes (Hines, 2009). It is routine practice for program assessment to be reduced to measures (e.g., participation, satisfaction) that neglect to tell the full story of how the program impacted a change in practice.

Program Goals
The TH!NK Program at NC State University seeks to bridge the gap between evidencebased research on teaching and teaching practices in the classroom through an intensive faculty development program. Through this work, we aimed to transform the culture of teaching on our campus away from teacher-centered instruction and toward studentcentered instruction that promotes higher-order thinking. Allen et al (2019), previously described implementation of various aspects of the TH!NK initiative.

Faculty Development Plan
The key individuals in the initiative are the TH!NK Director -responsible for creation and implementation of faculty development, recruitment of fellows and faculty, vision; the TH!NK Fellowsa team of five award-winning faculty members who created and implemented the program with the Director and provide mentoring to participants; TH!NK Faculty -the cohort of faculty who participated in the training, made enhancements to their courses, and assessed outcomes of their students; assessment professionals in both the Office of Assessment for student learning outcomes and the Office of Faculty Development for faculty learning outcomes; and of course, students. Faculty participation was voluntary, but a stipend was provided.
Approximately thirty faculty members participated in training each year (15 in the first year). Each year, a new faculty cohort participated in an intensive TH!NK Institute in May at the start of summer break. Faculty worked on course revisions over the summer with the support of an interdisciplinary peer cluster consisting of other faculty participants and a TH!NK Fellow, having small "cluster meetings" approximately three times over summer break. Each cohort reconvened in August prior to the start of the semester to share pedagogical innovations and receive structured peer feedback The primary goal of the Institute was to define the critical and creative thinking process across disciplines and provide faculty with the tools to create opportunities for students to practice the skills in the process, as well as tools to provide feedback on students' thinking process rather than just on the work product. Some of the themes integrated throughout are removing barriers to intellectual risk taking, prompting targeted self-reflection, and questioning assumptions. Evidence-based teaching strategies and concepts that faculty engage with during the institute included: defining and assessing the critical and creative thinking process (Ennis, 1985;Rhodes, 2010;Sternberg & Lubart, 1999); peer evaluation (Volz & Saterbak, 2009); divergent and convergent thinking (Schommer, 1990); critical thinking scenarios (Carson, 2015); data visualization and concept mapping (Novak, 1990); SEE-I (state, elaborate, exemplify, illustrate) (Nosich, 2011); metacognition and selfreflection (Bruning et al., 1995;Hofer & Pintrich, 1997).
TH!NK Faculty were held accountable by their Fellow and peer cluster to produce materials by August including: a revised course syllabus; a description of an activity where students evaluate the works of others; a description of at least one activity or assignment designed to encourage/integrate student self-reflection; and a capstone Critical and Creative Process activityan assignment that integrates the opportunity for students to practice and receive feedback on the critical and creative thinking process.

Research Questions
This paper addresses the following questions: 1. What are the trends in adoption of 13 evidence-based instructional strategies by faculty who teach STEM or non-STEM courses after a common faculty development experience?
175 Adoption of evidenced-based teaching strategies after a common faculty development experience 2. How commonly did faculty implement changes in additional courses, and share activities, assignments, or concepts gained during the training with colleagues?

Faculty Impact Survey
A survey was developed to gather matched pairs data from five cohorts of faculty who completed TH!NK training between 2014 and 2018. The survey asked faculty about their use of specific evidence-based teaching strategies in their TH!NK course before and after the training, as well as intended future use. In addition to the pre/post matched pairs data, course-level data was collected in order to examine strategy adoption in STEM and non-STEM courses. The survey had questions about whether faculty had implemented changes in their other courses, and whether they shared TH!NK activities and assignments, or mentored others inside and outside of their department. The survey was administered to 126 TH!NK faculty members and 72 responded (Response Rate=57%).

Data Analysis
First, descriptive statistics for the demographic data were summarized. Then, we used SPSS to analyze the matched pairs data, and calculated Cohen's d to measure the change in strategy adoption as an outcome of the faculty development training. Cohen's d measures 'effect size', or the magnitude of difference between the two means (pre/post), which is an indicator of the importance of the difference between them (Cohen, 1988).

Participant Demographics
Faculty of all ranks, and from every undergraduate-serving college on our campus, participated in the Institute. Of the 126 participants, approximately 60% were professionaltrack and 40% were tenured or tenure-track. Over 25% of participants in the training were tenured. In the training, 47% of faculty were transforming a STEM course and 53% a non-STEM course. The proportion of survey respondents was representative of the demographics of those who participated in the training.

Adoption of evidence-based teaching strategies in STEM and non-STEM courses
The survey asked participants to indicate, for each strategy, whether it was used in their course both prior to and after participating in training. For this analysis, responses were separated by STEM (e.g., science, technology, engineering, mathematics), or non-STEM (e.g., humanities, social sciences, education, business, design, arts) course types.
In Figure 1, for each strategy on the left, a stacked bar is shown for STEM (upper bar) and non-STEM courses (lower bar). The lighter shade represents the percent of survey respondents who used the strategy prior to training, and the darker shade indicates the increase in use, or total percent of respondents who used the strategy post-training. In the faculty population overall, not separated by STEM status, over half of the strategies evaluated showed over 0.5 effect size pre/post-training. The three strategies with the largest effect size for both STEM and non-STEM courses were SEE-I, self-reflection, and short written reflections with effect sizes of 1.0, 0.98, and 0.73, respectively.
It is important to note that among the low adoption strategies -journal writing, case studies, and critical reading -journal writing usage was low before and stayed similarly low, while case studies and critical reading were already highly utilized by approximately ⅔ of faculty prior to training. Thus, in some cases where there was a lack of new adoption it was due to high usage prior to training. It is therefore important to consider both gains in use and overall use in discussing which strategies were valued by faculty.
In almost all cases, with the exceptions of data visualization and data analysis, faculty teaching non-STEM courses were more likely to utilize the evidence-based teaching strategies post-training, compared to faculty teaching STEM courses. While both course types showed gains in adoption, the gains among STEM faculty were greater, thereby reducing the gap in usage. Notably, faculty teaching STEM courses reported the increased 177 Adoption of evidenced-based teaching strategies after a common faculty development experience use of short written reflections and other forms of self-reflection, as well as peer review, for which the analysis showed very large (over 1.0) effect sizes. In most cases, when we look at post-training usage, popular teaching strategies were embraced in both course types and unpopular strategies were mostly shunned by both types. Outliers are, again, data visualization and data analysis, which had greater use in STEM courses. Use of the synthesis matrix and journal writing was not extremely popular among either group, but both were utilized at a greater level in non-STEM courses. This may support the idea that while a small number of these strategies are discipline-specific, most are applicable to similar degrees across disciplines.
Across the board, faculty reported that prompting students to reflect on their own thinking was one of the most impactful enhancements to their courses. One faculty member wrote "I really feel like giving [specific self-reflective prompts] to all of my classes would benefit me in the future, as it has successfully given students a better platform to give more significant feedback to their learning gains rather than what they didn't like about the course or what they enjoyed about the course." We also asked faculty about their intention to use each strategy in the future. In all cases, faculty not only reported intending to continue usage, but many faculty indicated that they intended to try additional strategies in the future.

Informal spreading of ideas, assignments, or teaching strategies among faculty
Of respondents who teach more than one section of their TH!NK course, 92% reported implementing changes across sections. Of those who teach multiple courses, 100% report implementing changes to additional courses based on their learning in the program. Respondents also reported sharing TH!NK ideas, assignments, or activities. Approximately 85% of survey respondents reported sharing inside their own departments and 40% shared outside of their own departments.

Conclusions
We observed that the combination of an intensive, multi-day faculty development institute, paired with peer mentoring and accountability over a prolonged period of time, led to a high rate of adoption of a number of evidence-based teaching strategies among our faculty cohort. This aligns with research suggesting that faculty development programs include at least two of the following: a sustained effort, lasting a minimum of four weeks, feedback on instructional practice, and a deliberate focus on changing faculty conceptions about teaching and learning (Desimone & Garet, 2015;NRC, 2012).
Faculty teaching non-STEM courses utilized evidence-based teaching strategies focused on critical and creative thinking more than STEM faculty prior to TH!NK training for most of the strategies we introduced and evaluated. However, following training, while both groups made large gains, gains among STEM faculty were even greater, helping to close the gap between usage between STEM and non-STEM faculty. While there were some strategies that were utilized at greater levels by STEM faculty prior to training, the overall trend may indicate that STEM faculty were less likely to have been exposed to a wide variety of teaching strategy options prior to training. In most cases, strategies that were very popular were popular among both groups, while less popular strategies were not favored by either group post-training. This argues against a need to have separate training for faculty based on discipline when it comes to critical and creative thinking. And indeed, faculty overwhelmingly expressed that gaining perspective of peers from different disciplines helped them gain awareness in gaps in the way teaching is frequently done in their own discipline. One faculty member said "I highly value the relationship of [my faculty cohort] in the enhancement of my THINK course. Forming these relationships in the THINK training has enabled me to meet other members of the faculty from different colleges and disciplines, and I have learned from their different perspectives and approaches to critical and creative instruction." Across the board, faculty indicated making changes not only to their TH!NK course but to all of their courses. Many report sharing ideas, assignments, and teaching strategies with colleagues both inside and outside of their academic departments. This not only indicates a snowball effect of these ideas throughout campus, but also indicates that participation has encouraged interdisciplinary interactions in the realm of teaching and learning.
To date, approximately 11,000 students (student enrollments) have been impacted directly in TH!NK courses. We are currently analyzing and writing up the student learning outcomes from the first phase of our program focused on first-semester freshmen, and also beginning to gather longitudinal data on students in majors where multiple faculty intentionally incorporated critical and creative thinking skill development throughout all years of the curriculum. An initial bird's-eye view of the data points to gains in a greater number of unique critical thinking skill areas post-intervention compared to preintervention and faculty training. We also learned that our first-year students score very high in convergent thinking tasks, but have more room to grow in skills that require divergent thinking and where there is not necessarily an answer that is "best" from all perspectives. These student learning outcomes will guide us in planning the next phase of TH!NK faculty development.