Evaluating Learning for the Multiple Constituencies of Higher Education: A Call for Action, A Call for Research

This paper explores the rapidly changing world of higher education and the need for different ways to identify learner outcomes and evaluate student learning. In recent years, higher education has experienced significant demographic shifts in student populations. These shifts were the result of numerous variables including the increasing cost of higher education, the demand from business and industry to get people into high-demand occupations faster, and the decreasing number of individuals choosing postsecondary education immediately following high school. The year 2020 brought unprecedented challenges to the world with the pandemic caused by the coronavirus known as COVID-19. The pandemic accelerated the change that was already taking place in higher education. From how education was delivered to where it was delivered, higher education was forced to rapidly change a centuries-old model. This paper explores a tier one research university’s response to the changes in higher education by employing a proven process of mapping learning outcomes, assessing both new and prior learning using innovative technology, issuing microcredentials, and working with policymakers and employers to meet workforce demands.


Introduction
Ten years ago, the United States adopted policies that sought to increase college enrollment and graduation rates. Both federal and state governments established goals to increase the adult population's share with postsecondary credentials to 60 percent or higher by 2020 (Fry, 2017;Obama, 2009). This goal has yet to be met with about half of the adult population holding some type of postsecondary credential (including workforce certificates), although there are significant differences across the 50 states (Lumina, 2019). Postsecondary education plays a vital role in income and wealth mobility (Chetty et al., 2017) and adds to people's resiliency to the effects of economic downturns (Bureau of Labor Statistics, 2020). Resiliency and adaptability in postsecondary and higher education are not system challenges unique to the U.S. context alone. Globalization and rapid technologization have strained education and workforce readiness systems and structures worldwide. As a potential solution to this continued problem, we present within this paper an ambitious and broad set of solutions around developing learning maps, creating an authentic outcomes assessment system based upon the learning maps, using the learning maps to communicate competencies across courses and document the relationship of the mastered outcomes to business workforce needs, and issuing stackable micro learning credentials all while using and extending proven processes, technology, and lessons learned from the K-12 education sector.
In recent years, higher education has experienced significant demographic shifts in student populations. These shifts resulted from numerous variables including the increasing cost of higher education, the demand from business and industry to get people into high-demand occupations faster, and the decreasing number of individuals choosing postsecondary education immediately following high school. It is imperative that higher education look for diverse ways of filling the projected financial gap due to decreased enrollment. One possible solution is to engage learners across their lifespan and career span. For example, in the state of our tier one research university, it is projected that high school-age cohorts are getting smaller. Smaller cohorts limit the traditional pool of students and, therefore, tuition upon which universities and colleges rely.
The world of work, education, and industry have always been interconnected, complex, and dynamic. However, there have been consistent limits of infrastructure and challenges to the rate of change and responsiveness of learning systems. Simultaneously, in the U.S. higher education has seen decreasing public funding while student debt is mounting. Add to this the need for employers to find just-in-time training and development for employees. Therefore, it is the rate of change and, more importantly, the multiple convergences of change that we seek to discuss here. This work addresses the concomitant change across six different areas within the context of higher education: (1) the changing constituencies of postsecondary learners; (2) the need for and articulation of diverse competencies; (3) the need to track or at least recognize cross-cutting skills like critical thinking and teamwork which although critical often remain invisible in terms of outcomes to potential employers; (4) enhanced theories and methods in assessment research; (5) the evolution of learning progressions into more holistic learning maps; and (6) the maturation and interoperability of technology infrastructure.

The Five Components
We propose that the major constituencies of higher education -university administrators, faculty, students, potential employers, and policymakers -will all benefit from a wellarticulated system that has five components: 1. Learning maps that communicate the relationships among learning outcomes and competencies within a program of study. 2. Credit for life experiences. 3. Multiple assessment approaches for demonstrating mastery of learning outcomes. 4. Degrees based on stackable microcredentials. 5. A technology platform that articulates the components and facilitates interactions among universities, businesses, and students.

Learning maps
Learning maps are a model that articulates a network of precursor and successor skills, where mastery of a particular precursor skill or learning outcome increases the probability of mastering a successor skill. Learning maps make learning visible which supports the diversity in knowledge and experience that learners bring with them as well as showing the equally diverse pathways learners can take to gain mastery. Figure 1 presents a view of a portion of a learning map developed for a K-12 assessment program. The information within the specific nodes shown below are not relevant to this paper; it is presented to show that the structure can include one to one, one to many, and many to one relationship among nodes. Learning maps can be created based on existing empirical research or be based on expert opinion and validated based on the collection of assessment data (Adjei et al., 2014). Learning maps have served as the basis for formal assessment systems (Kingston et al., 2016) as well as an organizing structure for instructional practices (Kingston & Broaddus, 2017). Learning maps can be tagged to show which courses address which learning outcomes and which outcomes are deemed critical for which career paths. They can help administrators (and businesses) understand course overlap, outcomes not explicitly addressed by any courses, and outcomes intentionally addressed across the curriculum, such as ethical behavior and critical thinking. Using critical thinking as an example, we assume there are nodes related to critical thinking in our learning map and that our assessments gather information about student critical thinking that we can pull out and report associated with that node. In the past, all anyone ever saw was a grade for a course; i.e. Biology 101 or English 203. By using learning maps and appropriate assessments, students could demonstrate the mastery of learning outcomes attained over multiple courses, that were previously invisible within the grade for a single course.

Credit for life experience
While credit for life experience has long been considered by institutions of higher education, a variety of factors make it important to accelerate this trend. High performing businesses are increasingly self-identifying as learning-focused organizations and investing increasing amounts of resources into training. However, some common approaches to evaluating past learning experiences are expensive and do not provide evidence of validity and reliability or are not direct evidence of attainment of learning outcomes and workplace required competencies. Workplace competencies manifest in accomplishing tasks, not demonstrating answers to questions on related areas of knowledge. Given an increasing policy focus on adult education and training, stackable credentials provide a pathway for increasing educational attainment and career training among working adults who already hold a postsecondary credential (Meyer et al., 2020).

Multiple assessment approaches
At the heart of assessment in the service of learning is the use of principled approaches to assessment design, development, and implementation, along with the ongoing accumulation of evidence to support the claims of the assessment (Ferrara et al., 2017), which represent a validity argument (Kane, 2013;Sireci, 2013).
Criticisms of assessments have stimulated their evolution and transformation. Assessments have been criticized for not effectively capturing complex knowledge and skills as emphasized and needed in today's curriculum and workforce (Resnick & Resnick, 1992); measuring a relatively narrow range of knowledge and skills (Glaser et al., 1997); capturing information about performance set in one point in time and not the progression of knowledge, skills, and understanding over time (National Research Council, 2001); and not measuring what is taught (Black & Wiliam, 1998;Shepard, 2000). The evolution of assessments involves the reconceptualization of the nature of assessment to represent the documentation and inference about (a) developed competencies, (b) the process by which competencies are developed, and (c) the potential for the development of competencies.
To ensure the evolution of assessments is high quality representing competencies articulated in learning maps, the assessments should (a) assess the competencies, (b) be aligned to and inform the process of developing these competencies, and (c) be valid, reliable, and fair as demonstrated by evidence (Darling-Hammond et al., 2013;The Gordon Commission, 2013). The evolution of assessments represents an assessment enterprise collecting a variety of evidence of the competencies (e.g., portfolios, performance events, artifacts, paper-andpencil tests, computer-based tests, among others). This evolution of assessments to capture evidence of learner outcomes in higher education is growing internationally (Tremblay et al., 2012).
Several characteristics must exist to develop a sound assessment. These characteristics, as implied by using a validation framework approach, need to be based on evidence that support the claims made by the information that comes from these assessments. The competencies used as the basis for the assessment should be representative of the important elements of the domain they are supposed to represent by (a) competencies aligned to what they are supposed to represent; (b) the scoring of evidence produces accurate scores; (c) the scoring rubrics adequately reflect the differentiated performance indicated by the performance level descriptions; (d) the scored evidence empirically reflects the competencies; (e) the scores produced correspond to actual performance on competency; (f) the scores are fair; and (g) the scores are useful in representing the competencies.

Stackable microcredentials
Microcredentials are related to a formally approved or accepted set of standards or competencies. They can be "stacked" up to achieve a credential that is recognized both within institutions and business and industry. Therefore, they offer a more granular way to move through content/competencies than a traditional degree or certification program. They offer a timelier and more formative unit from which to be assessed resulting in both lower costs and stakes. However, we propose not merely microcredentials in and of themselves, but the analytical and cognitive process behind their development (i.e., learning maps and diverse assessments).

Technology platform to articulate components and facilitate communication
A technology-based infrastructure is necessary to allow the various components to work synergistically. It must include a bank of academic program-based learning outcomes from which faculty can choose their course outcomes, as well as the facility to add outcomes. This component must be able to communicate with existing learning management systems to reduce duplication of effort. Another component must support the development and visualization of the learning maps. A third component must support assessment via evaluation of work experiences, performance assessments, and technology-enhanced assessments that foster authentic assessment. Finally, the results of the assessment component must support stackable microcredentials, where credit is reflected as specific course attainment, rather than just a general awarding of credit. While existing commercial platforms are available, most carry out only one or two functions needed for full articulation of the components.

Necessary Next Steps
In order to respond to the changes in higher education discussed in this paper, we suggest the following next steps: 1. Research agenda examining the alignment of the content, the alignment of the assessment, the reliability of the decisions about competency, the generalizability of the scoring, the concurrent and predictive informational value of the scores and microcredentials, and the fairness of the results to examinees including evidence of the appropriate use of the scores and microcredentials (Darling-Hammond & Adamson, 2010;Shavelson et al. 2018). 2. Operational studies to examine the efficiency of obtaining, scoring, and certifying competencies (Wei et al., 2014;Cresswell et al., 2015). 3. User studies to examine and monitor the understanding and use of score reports, as well as the effectiveness in dissemination activities and informational campaigns of the competencies and microcredentials (Zenisky & Hambleton, 2012;Kirsch & Braun, 2020;Bennett & Folley, 2020). 4. Validation of microcredentials by (a) gathering corroborating evidence that achieving a microcredential corresponds to competency in the knowledge, skill, and/or behavior implied by the competency and (b) examining whether the use of microcredentials predicts subsequent performance in sequent educational experiences or job performance.