Academic rankings as a source of metrics and benchmark tools for continuous improvement at Técnico Lisboa

Academic rankings ot only provide an opportunity to increase visibility and reputation of institutions but also can be used as benchmark tools for quality improvement, despite much criticism of biased coverage and flawed methodologies. As part of the University of Lisbon, Técnico Lisboa – a leading engineering and technology school in Portugal – has monitored and examined rankings through its Rankings Observatory in the context of strategic options, but institutional transformations have been an obstacle to exactly measure Técnico’s worth and weight in the context of the University of Lisbon. Those transformations have not contributed to leverage the positioning of the University of Lisbon in major rankings either. This paper discusses the impact of institutional transformations on the University of Lisbon and Técnico in rankings, describes Técnico’s attempts to overcome these constraints and gives examples of how it applies rankings as internal benchmark tools towards continuous improvement.


Introduction
The University of Lisbon (ULisboa) acquired its current status in 2013, after the merger of Universidade Técnica de Lisboa (UTL) and Universidade de Lisboa (UL). Among 18 institutions, Técnico Lisboa (Técnico), which was UTL's engineering and technology school, has maintained its name and role in the current ULisboa. This merger has allowed ULisboa to become a 'world-class university', in a trend of recent years in Europe (Docampo et al, 2015). However, it also brought major constraints. Técnico has undergone a decrease in autonomy and diminished visibility and ultimately it has affected its brand identity. It became difficult to exactly measure its performance or establish its 'worth' and reputation, given that the rankings list ULisboa and not Técnico.
In its recently adopted Strategic Plan 2020-2030 (2022), Técnico highlights its ambition 'to become a Europe's leading (top 20) Engineering, Science & Architecture school' by 2030, among other major targets in terms of governance and positioning in the national and European contexts. There has also been a wide internal debate on possibly changing its statutes and adopting a foundational framework, with resulting changes in governance, with more autonomy, more flexilibility financially, among other structural transformations.
Despite widespread critique, rankings have been increasingly used by institutions to be part of decision-making strategies or as a benchmarking of quality assurance (Hazelkorn, 2013). This criticism ranges from indicators adopted to the weight some have in scores (Fauzi et al., 2020). Through its Rankings Observatory (Observatory), Técnico has focused on specialised rankings in Engineering and Technology, in its fields and subjects of intervention. It operates as an intelligence unit on these matters, which has sought to overcome existing limitations by using ranking metrics to quantify its contribution to ULisboa, analyse its performance through indicator analysis, and compare Técnico's performance against its peer institutions.
Other limitations still endure. An example of this is the reputational component of rankings, given that some ranking sources have a strong focus on reputation. Reputation surveys (academic, employer) refer to ULisboa and not to Técnico, which makes it impossible to accurately determine Técnico's value and weight, as reputation has a substantial impact on ranking scores, as in QS and THE (Vidal & Filliatreau, 2014).
It is important to note that although the Observatory monitors different rankings, analyses and benchmark exercises focus fundamentally on QS and THE ranking sources.

The challenge to overcome the absence of Técnico in rankings
As previously mentioned, one of the major challenges faced by Técnico is the fact that it is not listed in any ranking, and its performance is diluted in the participation of ULisboa. As an added challenge, the merger of UL and UTL led to the situation in which some of the scientific fields of Técnico are not exclusive. This is particulary noticeable in areas such as chemistry, biology, physics, maths and even in some engineering fields. The attempt to overcome this barrier is to quantify, as precisely as possible, how much Técnico weighs in terms of percentage on ULisboa and consequently to understand how much it contributes to its performance in rankings by subject, in subject areas taught at Técnico.
For this purpose, we chose to look at two dimensions: scientific output (bibliometric data) and volume of students. The former is measured through the volume of publications indexed in SCOPUS whereas the volume of students corresponds to the current number of students enrolled. The subject areas where these dimensions come closest to 100% are those in which Técnico's weight is greater or almost exclusive in the context of ULisboa.
Based on the organization of broad fields in SCOPUS, table 1 provides key outcomes concerning the number of publications of Técnico and ULisboa, i.e. Técnico's percentage of contribution to ULisboa in each subject area. According to the above data, three broad fields taught at Técnico are the greatest contributors to the ULisboa's performance and outcomes: Mathematics (87.7%), Engineering (85.3%) and Materials Science (83.2%). These are followed by Energy (78.5%), Computer Science (71.5%) and Physics and Astronomy (71.1%). The remaining fields account for less than 70%, which suggests that it is reasonable to consider that other ULisboa's institutions, in particular the Faculty of Sciences, significantly contribute to its performance and results.
As regards the volume of students, table 2 provides the volume of students enrolled in Técnico vs ULisboa in the fields taught at Técnico and its weight in the framework of ULisboa, according to the data retrieved from the Portuguese Directorate-General of Statistics of Education and Science (DGEEC). The data shows that the number of students enrolled in Engineering, manufacturing and construction industries at Técnico accounts for 73.6% of the ULisboa student population, followed by Information and Communication Technologies (ICTs), with 39.9%.
As for Natural Sciences, Mathematics and Statistics, it is also reasonable to consider that other schools of ULisboa have a relevant contribution and therefore it is more difficult to exactly determine Técnico's weight and standing in ranking sources in these subject areas. This therefore justifies the option for the Observatory to focus on Engineering and Technology sector rankings.
In this regard, it is possible to understand how much Técnico weighs percentage-wise in the context of ULisboa, in subject areas and student population, but these outcomes only reflect an approximate idea of that influence. Finally, it can be said that this is an exercise of trying to measure the 'weight' of a merged institution in the context of its parent, which could be applied to other examples.

Applicability and instrumentalizaiton of rankings
This section disccuses and explores some examples of how Técnico's Observatory applies rankings, namely in the analysis of ranking indicators and scores, and benchmark excercises, which stimulates a quality culture within the institution in its areas of activity, as argued by Berbegal-Mirabent & Ribeiro-Soriano (2015).

Analysis of indicators and scores
Because the key challenge is to tackle the "non-presence" of Técnico in the rankings, the strategy involves improving the positioning of ULisboa, preferably in collaboration with other schools. In this regard, we would need to be able to accurately understand the impact of Técnico on ULisboa in all indicators, for which access to data (ie. student population, financial data, etc.) from other schools should be needed.
Ranking indicators provide a good comparison tool because they are a common benchmark applied equally to all institutions. Reputation indicators (academic, research and employers) depend on surveys that are conducted among stakeholders. At this level, Técnico's partners perceive their partnership with Técnico and not with ULisboa. Nevertheless, these surveys are conducted among parent institutions, in this case ULisboa, because Técnico is not eligible for that purpose and its reputation may not contribute significantly to the reputation of ULisboa. A number of indicators should also be defined to exactly measure the weight of Técnico in the positioning of Ulisboa and use them as benchmarks for improvement, given that the the analysis of indicator scores allows us to identify, for example, underperforming indicators. The examples that follow draw on the latest edition of the THE ranking by subject (2022) in Engineering and Technology. The figure below shows how each indicator evolved in the period 2020-2022. From the above, it can be said that these observations are critical to identify underperforming indicators or even those which can perform even better. The figure shows that the Citations indicator, which has been constantly decreasing across the 2020-2023 period, should object of concern as it weighs 30% according to the THE WUR by subject methodology. Table 3 reveals the outcome of an hypothetical exercise of applying a growth percentage to the score of an indicator and, thorugh that, calculate a precise rank. These results are merely hypothetical and rely on the assumption that the next edition of the ranking would remain unchanged. However, it gives an idea of how the global score and standing would be if we apply a growth percentage in some or all indicators. The inhouse score and rank were calculated by the Observatory, because the THE ranking classifies institutions in bands from the 201st place onwards (201-250…).

National benchmarking
Benchmark exercises include comparisons with peer universities, either in the national or international contexts. Figure 2 below allows us to observe how the ULisboa globally performed against its peers in Portugal, Universaity of Porto (UPorto) and Universidade da Beira Interior (UBI), in a defined time period: 2020-2023. It reflects the evolution of the global score in the ranking, and it is aimed chiefly at marketing purposes. The table below provides the annual mean variation between 2019 and 2023 in each indicator, for ULisboa and Univ. Porto. Based on the percentage rate of growth (or decrease) in that period, we can observe an estimated position for 2024. This can be a useful way of looking at the evolution of the institution during the period in question and take action in advance if a downward trend in the performance of ranking indicators is observed. The potential scenario for 2024 allows the governing bodies of the institution to anticipate a likely future estimate if performance keeps the pattern of recent years. It also also helps identify priority areas of intervention, according to ranking methodologies.

International benchmarking
In the pursuit of quality, international benchmarking has also been very important to observe ULisboa's performance in some international networks. how these targets may be re-defined. Considering the the scores in each dimension we may clearly underline that reaching the top 20 in 2030 seems to be unrealistic.

Conclusions
This paper briefly analyses what a subsidiary institution can do to measure its weight within the context of its parent for ranking purposes, as a result of a merger. This need comes from the fact that a potential Técnico position in university rankings cannot be done alone by looking at how the parent university is performing 'by subject' because we have to consider the potential input of other schools, specially those who share the same subjects with Técnico, such as as physical sciences, biology, math or chemistry.
Due to these constraints the analysis of rankings undertaken by Técnico, for purposes of its own autonomous strategies and policies, should mainly focus on Engineering and Technology field rankings to determine, as closely as possible, a potential Técnico position on university rankings. The analysis of ranking indicators and scores proves to be relevant to stimulate a quality culture through comparisons with peer institutions, nationally and internationally, in a number of activities, according to ranking methodologies.
As it is a recent activity at Técnico, there is still no visible impact so far in terms of objective results and improvements, however the rankings theme has gained significant awareness among the management board and it has made its way into strategic planning for the next 10 years and has a significant role in quality monitoring by providing key indicators for the yearly activity plan. Técnico is currently better aware of its performance and position among Portuguese universities.