Assessing resilience at University

Promoting quality results also requires the ability to measure outcomes and to identify what context variables can affect the student’s success. The aim of this work is to analyze the distances in terms of the results obtained to a test on generalist skills among students with different family background (advantaged vs disadvantaged) and with brilliant performance, verifying whether these differences are significant. The analysis suggest that for resilient, the social background does not predict the student’s performance, thus we suppose that the skills achieved during the University studies could contribute to the reduction of the distance among students with different social background.


Introduction
The Prague Ministerial Conference (2001) showed the need to offer more details on the results of each University cycle.To this end, a group of specialists in higher education has defined the set of Dublin descriptors, defining which general knowledge and skills every graduate should possess at the end of each cycle of tertiary studies (Joint Quality Initiative, 2004).
In 2005, the Italian Ministry of Higher Education defined the Italian Qualifications Framework.According to that, upon completion of the first cycle graduated students should: • have demonstrated knowledge and understanding in a field of post-secondary education and is typically at a level that, whilst supported by advanced textbooks, includes knowledge of some latest developments of their field of study; • apply their knowledge and understanding in a manner that indicates a professional approach to their work, and have competences typically demonstrated through devising and sustaining arguments and solving problems within their field of study; • have the ability to gather and interpret relevant data (usually within their field of study) to make judgments that include reflection on relevant social, scientific or ethical issues; • communicate information, ideas, problems and solutions using both a specialist and non-specialist language; • have developed those learning skills that are necessary to undertake further study with autonomy.
Thus, promoting quality results also requires the ability to measure outcomes and to identify what context variables can affect the student's success.

Research objectives
During the last forty years, several studies have tried to identify which factors could influence student's performance: the shared finding of these studies lead to the family background that, in some way, can affect their success.
However, in the current scientific literature there are few studies that examine the performance of "resilient" students, that is those who have a disadvantaged socio-economic background, but achieve school success (Lombardi, Agasisti, 2012).Fewer are the studies that analyze the student's performance at the tertiary level.Notably, the PISA data (OECD, 2013) showed that, during 2003-2012, Italy recorded a significant increase in the number of resilient students.
The aims of this work is to analyze the distances in terms of the results obtained to a test on generalist skills among students with different family background (advantaged vs disadvantaged) and with brilliant performance, verifying whether these differences are significant.

Methods
The assessment of the learning outcomes of Italian graduates through a test, conducted by the Italian National Agency for the evaluation of Universities and Research Institutes (ANVUR), has set the ambitious goal of measuring the levels of critical thinking and written communication achieved by Italian undergraduates (Benjamin et al., 2013).CAE's Collegiate Learning Assessment (CLA+) is a performance-based assessment that measures higher-order thinking skills at the tertiary level within the United States and internationally.It consists of two sections, a Performance Task (PT), which requires students to generate a written response to a given scenario, and selected-response questions (SRQs).CAE collaborated with ANVUR to conduct two pilot studies from 2012 to 2015: in both cases, ANVUR selected one CLA+ International test to be translated, adapted to Italy and validated by a sample of Italian students.This process ensured that the test was as much as possible analogous and equivalent to the American version.ANVUR pretested the translated and adapted test and conducted a series of cognitive labs in April of 2013 and 2015.The final result was rebranded as TECO (TEst sulle COmpetenze).For its nature, the same test was administered to students regardless of the course of studies in which they were enrolled.
To ensure whether and how some background variables (e.g.demographic, territorial and related socio-economic and cultural background) can affect the TECO results, ANVUR proposed a questionnaire at the beginning of the test.
This study is based on the responses given to this questionnaire, taking into account the results obtained in the first section of TECO (Performance Task, PT).This test presents a problematic scenario, in which the student has to propose a possible solution through the use of a number of documents provided (Blog, Podcast, research reports, newspaper articles and interviews).The student is not required only to assess critically the situation and texts, but also to explain effectively its solution to the problem (Benjamin et al., 2013, p.5).
The PT section measures three specific abilities: Assessing resilience at University
Each student receives a score from one to six for each of these three abilities according to a defined scheme.A score of N/A was assigned to students who did not answer the prompt or whose responses were off topic.The student responses were randomly and anonymously assigned to two different scorers, identified within the teaching staff of each university (24 universities) involved in the TECO, review each written response in double-blind: they do not have information about the student nor the scores assigned by the other scorer.For any double-scored response that was inconsistent (i.e., the difference between the two total scores was greater than 3 points or the difference between two sets of subscores was greater than 2 points), the elaborate received a third correction by ANVUR supervisors.

Participating students
The eligible students for TECO 2013 (ANVUR, 2014) were those enrolled in the 3rd or 4th year of a three-year course or single-cycle master's course who had acquired all the necessary study credits (basic and characterizing).For the TECO 2015 (ANVUR, 2016), some changes were adopted: students had to be in their 3rd consecutive year enrolled at the university; students enrolled in a three-year first-cycle course, must have acquired 75% of the basic and characterizing study credits required by the course class; students enrolled in a single-cycle master course must have acquired at least 90 (from a total of 120) basic and characterizing study credits.Students who participated in the 2015 edition of TECO were 6586, but for this study we are taking into account only those 2085 students whose elaborate have received three separate corrections, considering the final score as the average of the three.

Results
The following analysis (table 1) show the correlation coefficients for the PT subscores (Analysis and Problem Solving, Writing Effectiveness, and Writing Mechanics) and the total score, showing that correlations are significant reliable (level>.01).We calculated the highest occupational status of parents (HISEI) and the highest educational level of parents converted into years of schooling (PARED) based on the Index of Economic, Social and Cultural Status (OECD, 2012).Table 2 shows the correlation with PT subscores.We then synthesized them through Principal Component Analysis into a single index, defined Background.The correlations with the test scores were then verified (Table 3).In order to measure the effect of the family background for the Italian undergraduates and estimate the differences in their performance using the Background index, the sample was divided in two groups of students, namely disadvantaged and advantaged students.This operation required a selection of the two extremes of the distribution of the Background index: students with high background values (over the 66th percentile) are defined Advantaged (Group 1); students with lower background values (below the 33th percentile) are identified as Disadvantaged (Group 2).
The intensity of the Background effect (independent variable) on students' abilities (dependent variable) is calculated by linear regression.The residues of the regression allow to identify three sub-groups, as reported in table 4.

Results and conclusions
Data show that in the Disadvantaged group, the distances between resilient (DTA) and low achievers (DLA) are large and significant for the total PT score, for the three subcategories (APS, WE, WM), as well as for the two extremes subgroups (ALA and ATA) of the Advantaged students.However, taking into account the distance between the resilient (DTA) and advantaged students with brilliant performances (ATA), we show that the gap is not existent.In addition, the two sub-groups achieve very similar scores, suggesting that, at least for the resilient, background does not affect their success.To graphically show this conclusion, all the upper and lower limits for the PT average score of each sub-group have been calculated.To obtain an error probability of 5%, parameters have been calculated to be included in the ±1,96*E.S. range.Results (see Figure 1) show that the differences between groups are significant.However, there are no significant differences between the Low Achievers (Disadvantaged and Advantages), Medium Achievers (Disadvantaged and Advantages) and Top Achievers (Resilient and Advantaged) sub-groups.
Assessing resilience at University The analysis suggest that for resilient, the background does not predict the test performance, thus we suppose that the skills achieved by students graduating from the Italian higher education system, could contribute to the reduction of the social distance among students with different social background.
We believe that the study of student's resilience could be an instrument to help the definition of new policies with the aim to increase disadvantaged student success, thus, increasing the overall equity of the tertiary education system.Universities, in this perspective, will have the role of "resilience catalysts".