Monitoring the Evolution and Benefits of Responsible Research and Innovation (MoRRI)

European efforts to 'mainstream' the notion of Responsible Research and Innovation (RRI) is challenged by a lack of organisational and individual incentives, but also by absense of relevant measures of RRI. The European Commission has launched a study with the aim of developing indicators capturing the evolution and benefits of RRI across Europe. This paper presents the preliminary set of indicators identified by the project consortium. In recent years, the notion of Responsible Research and Innovation (RRI) has emerged in European policy making. From the perspective of the ‘Science with and for Society’ (SwafS) scheme of the European Commission (EC), the purpose of promoting RRI is “to build effective cooperation between science and society, to recruit new talent for science and to pair scientific excellence with social awareness and responsibility” (EC, 2016). Building on work by Von Schomberg (2013), the EC defines RRI as a process which allows “all societal actors (researchers, citizens, policy makers, business, third sector organisations etc.) to work together during the whole research and innovation process in order to better align both the process and its outcomes with the values, needs and expectations of European society” (EC, 2016). Conceptually RRI reflects previous strands of activities such as anticipatory governance (Karinen and Guston, 2010) Constructive, Real-Time and other forms of technology assessment (Rip et al., 1995; Guston and Sarewitz, 2002; Grin and Grunwald, 2000), upstream engagement (Wilsdon and Willis, 2004), and value-sensitive design (Friedman, 1996). In operational terms, however, the EC brings dimensions of public engagement, gender equality, science literacy and science education, open access, ethics and governance under the RRI umbrella. These six RRI ‘keys’ have informed the composition of the work program for SwafS and the featuring of RRI as a cross-cutting issue of Horizon 2020, intended to be embedded across the priorities of the funding programme. So far, efforts to ‘mainstream’ RRI across the European research area have been modestly successful (Mejlgaard and Griessler, 2016). Studies indicate significant obstacles, pertaining not least to disincentivizing reward structures at both organisational and individual level (Smallman et al., 2015). While ‘pairing’ responsibility and scientific excellence is an explicit aim for the RRI agenda, they are in reality often perceived as contradictory demands by the individual scientists or viewed as unequally important concerns by the research performing and research funding organisations. While the production of high impact publications is, e.g., considered a core academic activity clearly carrying merit, engaging in public outreach or stakeholder dialogues might easily be considered peripheral activities without straightforward value for the individual scientists. Moreover, the lack of adequate measures of responsibility in research and innovation further hampers the mainstreaming of RRI. Inability to evaluate, compare, and benchmark ‘performance’ in terms of RRI at the national as well as disaggregated levels, constitutes a barrier to any revision of reward schemes and dilutes the potential vitality of the organisational or national ‘horse race’ for high performance in this area. Identification of useful indicators and metrics for RRI might then contribute to bringing issues of responsibility from a peripheral position and closer to the center of activity. Finally, evidence This work is licensed under a Creative Commons License: Attribution-NonCommercial-NoDerivatives 4.0 International.


Executive Summary
Since the turn of the century, the topic of gender equality in science and research has been intensively discussed. Accordingly, a broad range of literature, pilot projects and empirical evidence is available which deals with gender inequalities in this area. This provides the starting point for the discussion on gender within responsible research and innovation (RRI) and the development of indicators for the gender dimension in RRI. Following the recent political and scientific discourse, gender equality is defined as a three-dimensional construct aiming at:  integration of women in all fields and at all levels in research and innovation (reduction of horizontal and vertical segregation);  structural change in research institutions in order to abolish structural barriers for women (e.g. through the implementation of comprehensive equality plans, quotas for women, transparent decision-making);  integration of gender in the content of research and innovation to ensure that women's needs and interests are adequately addressed.
The goal of the dimension report on gender equality is, therefore, to outline a first set of indicators that represents gender equality in this comprehensive sense and allows comparative analyses for EU countries over time based on a literature and data review. The first set of indicators developed for the dimension report covers all three of the above aspects, although they do differ in terms of accuracy with regard to the underlying construct to be measured, comparability between EU Member States and availability in time series terms.
The first dimension of gender equality, female participation, includes eight indicators and is, therefore, well represented -especially for the university sector. The situation for female researchers in the non-university sector is covered less comprehensively. The proposed indicators include the share of female researchers by sector and an estimation of how long it will take to achieve equality in gender participation in science and research based on recent trends in female participation. The dissimilarity index represents horizontal gender segregation. Vertical segregation is indicated by the glass ceiling index as well as the number of graduates and female academic staff by grade indicators. Women's access to top management positions is covered by the share of female heads of research performing organisations (RPO). An additional indicator focuses on the project level and represents the gender composition of teams and management in EU funded projects.
Although there have been numerous pilot projects and case studies which focus on the implementation of equality policies in research funding organisations (RFO) and research performing organisations, there is less information available for a representative analysis covering several countries. Furthermore, only selective information is available on the integration of gender in research content. This is also due to the fact that administrative data do not yet by default consider gender aspects. However, recent initiatives by the Helsinki Group and the development of a monitoring system for European Research Area (ERA) goals have initiated progress in this respect.
The set of indicators covering the second dimension reveals the implementation status of equality policies in RPOs and RFOs and contains six indicators: The share of gender balanced recruitment committees in RPOs, the share of gender balanced research evaluation panels in RFOs, the share of RPOs with gender equality plans, the share of RPOs with female recruitment and promotion policies, the share of research projects with specific gender equality actions and the share of technical universities with organisational structures for gender equality. However these indicators do not fully depict the change in organisational structures that has been supported by these interventions: They only show the extent to which organisations have implemented policies -not any corresponding change. The literature review also reveals a lack of evidence on the direct effects of equality policies and the relevance of underlying assumptions. As a consequence, indicators addressing structural change in RPOs and RFOs are interpreted as input-oriented indicators.
The third dimension of gender equality is covered by three indicators representing the share of RFOs promoting gender content in research, the share of RPOs with policies to integrate gender in research content and the share of EU-funded research projects with a gender dimension in their content.
In summary, the gender dimension in science and research is covered well by the proposed set of indicators, although there are some shortcomings which need to be considered when interpreting indicators and developments. The link between the RRI dimensions with regard to gender and their coverage by indicators will be discussed in the next phase of the MoRRI project. Another focus of the subsequent phases of this project will lie on the development of indicators that address benefits as a supplement to input-related indicators.

Introduction -analytical and empirical aspects of Responsible Research and Innovation
This report is one of a series of six reports, each targeting a separate dimension of Responsible Research and Innovation (RRI). The six dimensions include 'citizen engagement and participation of societal actors in research and innovation', 'science literacy and scientific education', 'gender equality', 'open access to scientific knowledge, research results, and data', 'research and innovation governance' and 'research and innovation ethics'. The six reports collectively form the main output of Task 2 of the 'Monitoring the Evolution and Benefits of Responsible Research and Innovation' (MoRRI) project, and they are informed by the results of the literature review of RRI and its conceptual components, which was performed as Task 1 of the project.
The six reports emerging from Task 2 specifically address analytical and empirical issues relating to each of the RRI dimensions. Each report aims to:  provide an operational understanding of the RRI dimension it targets,  present existing empirical information about the RRI dimension, and  assess data availability and specify analytical levels and degrees of aggregation of available material The reports will provide a platform for the subsequent definition of metrics and indicators for the RRI dimensions in Task 3. The report at hand specifically focuses on the dimension of gender equality.
The report is structured in accordance with the main aims of Task 2 and also provides an outlet for the results of Task 1. In chapter 2, results from the literature review are presented. These provide a background for the following chapters. Chapter 3 is concerned with the development of an operational understanding of gender equality. The objective is to provide a functional vocabulary of gender equality by clarifying important analytical components and definitions of gender equality. This chapter includes a specification of the relationship and borderlines between the gender equality dimension and the other five dimensions of RRI. Chapter 4 considers selected existing empirical information on gender equality. It is based on a review of selected studies funded by the European Commission, along with a review of evidence from other empirically oriented studies which are considered particularly relevant for the gender equality dimension.
In chapter 5, the availability of existing data on gender equality is assessed. Following the scheme outlined in the MoRRI proposal, this chapter specifically considers the availability of data on gender equality relating first to its characteristics in terms of the intervention logic model, i.e. data describing the context, input, output and outcome of gender equality. More specifically, context relates to the environment and overall situation in a country; input to the activities carried out, measures taken, structures created or resources provided to address what is done in order to address issues of RRI and whether it is done in a systematic manner; outputs to the immediate or direct results of activities; and outcomes to the achievements (MoRRI Proposal 2014:64). Second, the availability of data is described according to the level of aggregation of these data, distinguishing data that describe the global level, the national level, the regional level, the institutional level, the programme/project level and the individual level.
Reflecting on the findings in Chapter 5, Chapter 6 considers issues relating to data gaps and assesses the overall need for primary data collection to fill such gaps.
Finally, Chapter 7 provides some initial thoughts on the development of indicators and metrics for gender equality, which will be the objective of Task 3.

Results of the literature review on gender equality
This chapter provides a list of the core literature on gender equality selected for review (10-15 papers were reviewed for each RRI dimension) as well as a synthesis of the literature review on this dimension. The literature review was performed in Task 1 of this project. The synthesis serves to summarize the main conceptual elements of the targeted dimension and forms the background for the subsequent chapter on the 'functional vocabulary' for the gender equality dimension.

Review of core literature relating to gender equality
The objectives of the literature review (Task 1) were to In order to meet these objectives and provide useful input for Task 2 and the other subsequent project tasks (which are strongly related in terms of both topic and methodology), the approach to the literature review was designed in close cooperation with the dimension and task leaders. In a first step, the five dimension leaders were asked -based on their long-standing experience in their respective fields -to select 10 to 15 key publications in each key RRI dimension for detailed review. A review template was then designed in order to a) ensure a systematic analysis of the selected literature and b) cover all relevant aspects and information required in Tasks 1 and 2.
Before it was rolled out to the individual reviewers, the template was subjected to a pre-test. The guidelines for the review process and the findings of the individual reviews are documented in the Appendix to this report.
The following key gender equality publications were selected and reviewed: calls for social justice and was embedded in the development of general antidiscrimination policies at both national and European level aimed at establishing equal rights for women in employment. Corresponding research focused on the career paths of men and women as well as on the complex interplay between the institutional arrangements and personal preferences that might serve to explain the underrepresentation of women, especially at the top levels (European Commission 2004;Caprile et al. 2012). Since the turn of the century, economic arguments have also been used increasingly to justify gender equality policies:  In the European Commission's (EC) view, realizing Europe's ambition to achieve a competitive knowledge-based society will require an increase in the number of researchers (European Commission: The Wake-Up Call for European Industry 2003). In 2012, the EC again maintained that boosting innovation in the EU would mean increasing the number of researchers in Europe by at least one million, given that the key role assigned to research and innovation in striving towards smart, sustainable and inclusive growth in Europe requires that the EU makes full use of its human capital -thereby involving both men and women, a particularly important aspect in light of demographic change (European Commission 2012). In order to achieve goals like 'competitiveness', 'innovation' and a 'knowledge-based society', it is evident that the talents and potential of women have to be developed, mobilized, leveraged and used more actively, deeply and completely.
 From the science and technology perspective, 'gendered innovations' enhance excellence in science, medicine and engineering both in terms of knowledge and personnel. They lead to gender-responsible science and technology, and seek to enhance the lives of women and men around the globe. 'Gendered innovation' is defined as the process that integrates sex and gender analysis into all phases of basic and applied research to assure excellence and quality in outcomes (Schiebinger & Schraudner 2011;European Commission 2013a).


In the business sector's view, the reasons why gender diversity 1 should be taken into serious consideration lie in women's talents, the economic power of women, the changing market structure and the positive impact of women on organizational excellence and financial performance (Catalyst 2004;McKinsey 2007).
The issue of the under-representation of women in top positions both in academia and in the business sector is widely discussed in the literature reviewed. This underrepresentation can be illustrated in two ways: professional/educational and organizational. While 45% of doctorates are awarded to female students, only 30% of active researchers and 18% of full professors are women (European Commission 2012). Berryman (1983) introduced the metaphor of the 'leaky pipeline' to describe the normative sequence of educational and employment stages that typically comprise a scientific career: at each moment of transition from one educational/professional stage to another, the pipeline loses more women than men 2 . Remarkably, the educational and professional under-representation of women has changed at a very low pace (European Commission 2006;Caprile et al. 2012), although this issue has been discussed intensively across the EU Member States since the Beijing Platform for Action in 1995 (EIGE 2014).
1 Catalyst (2004) defines gender diversity as "recruiting, retaining and advancing women". 2 This also conjures up the image of a pair of scissors. The scissors refer to the difference between the share of men and women in a typical academic career and represents the different career tracks of men and women: a larger percentage of men than women reach the upper levels of both academia and management (European Commission, 2009b).
The literature reviewed interprets and explains this persistent gender segregation at three levels: the individual level, the institutional level and the social/cultural level.
1. Individual level: Women's ambitions are often restrained by psychological obstacles like higher awareness of barriers or difficulties in identifying themselves with success (McKinsey 2007), for example through low assessment of self-efficacy and limiting expectations regarding their own career prospects. Furthermore, social and cultural values shape gender stereotypes, which influence degree course choice and the gender division in the labour force (Caprile et al. 2012).
2. Institutional level: In both academia and the business sector organisational and management models have historically been designed by men. The 'anytime, anywhere' performance model, a precondition for a top management job, is irreconcilable with the double burden facing women (work and domestic responsibilities). Mastering male 'codes' is viewed as the only way to rise through the ranks. This requires not only greater efforts to adapt on the part of women in making their way to the top but also the ability to promote oneself and be assertive about one's performance and ambitions. An added final handicap is that it appears harder for women to find mentors in science and research. This leads to gendered career paths, firstly because degree course choice remains largely gendered and, secondly, because the 'rush hour', i.e. the early stage of the scientific career in which family and career demands most often collide, clearly puts women at a disadvantage. Further career advancement after a career break is difficult (Caprile et al. 2012). Since work is organised in gendered ways, it is difficult for women to reconcile paid and unpaid work. Gender continues to be a structuring factor in the workplace in general and in research in particular. This results not only in the persistent gender pay gap but also in harassment, concentration of power and the guru/acolytes model of power relations (European Commission 2012).
3. It also leads to the persistence of male dominance in the scientific system. Gender discrimination in the scientific system is prohibited, but it still exists, albeit in more subtle forms than in the past. This can have an impact, for instance, on selection, hiring and promotion procedures, on the distribution of resources, or on the assessment of scientific excellence. So-called gatekeepers are established, i.e. (male) scientists or peers who control the definition of merit and the means of exercising academic power (Merton 1968). Consciously or unconsciously, similarto-me effects (sexism and nepotism) still influence assessment and selection procedures, e.g. in the peer review of research grant applications (European Commission 2004). The evaluation system that has already been established aims to be objective and meritocratic. But its approach is not only imperfect, it may even be hindering women in establishing scientific careers. Indeed, merit and talent do not suffice alone for a successful career in science: Resources, time, social networks and encouragement -unevenly distributed between the sexesare also prerequisites (European Commission 2004). 4. Social/cultural level: Conflicts between the self-image of women and the image of science and technology (S&T) are identified as key barriers to increasing female participation in male-dominated disciplines (European Commission 2006). One reason why many female students are not willing to enter the S&T disciplines is not that they have less talent than their male counterparts but that the image of S&T does not fit their (expected) self-perception. For example, the stereotyped construction of the self-image of women is that they are emotional and peopleoriented. In contrast, the image of S&T is logical, rational and machine-oriented. In addition, students (both male and female) fear isolation in their chosen degree environment, a situation that causes many female students to drop out of S&T courses 3 . This is supported by traditional gender roles (stereotyping of women and men). Women are defined as the people who take care and men as those who take charge. Evidence suggests that European women devote on average twice as much time as men to domestic tasks (McKinsey, 2007).
To better explain the complex processes of gender bias in science and research, Schiebinger identifies three interrelated political approaches, namely fixing the number of women in science, fixing the institutions and fixing the knowledge (Caprile et al. 2012). In 2007, the European Commission changed its policy approach from 'fixing the women' to 'fixing the institutions' in line with the process related approach of gender mainstreaming (Lipinsky 2014).
1. The individual level (fixing the number of women in science): This approach seeks to increase the participation of women by supporting them in education and in careers in science. The implicit assumption here is that science, medicine and technology institutions and research are gender neutral (Schiebinger & Schraudner 2011). However, this approach has proved insufficient in increasing the number of women in science, particularly in positions of responsibility, and has not helped to address the structural barriers that contribute to the well-known leaky pipeline phenomenon (European Commission 2012).

The institutional level (fixing the institutions):
This second policy approach focuses on institutional change/reform, which aims to amend gendered organisational structures and practices.
a. RFO: Gender mainstreaming in access to research funding, decision-making on funding, allowing for parental leave during a research project and/or evaluation procedure, etc. Examples of the tools applied include quota regulations and quantitative targets (management by objectives) (Caprile et al. 2012).
b. RPO (e.g. higher education restructuring/reform): The recent introduction of New Public Management (NPM) strategies into higher education and research means that important changes have been made not so much with regard to the goals (e.g. raising the proportion of women in higher career positions), but in terms of the steering mechanisms used to achieve them. Several policy instruments, e.g. corresponding legislation or positive action (such as quota regulations), co-exist alongside more recent "mainstreaming" mechanisms 4 and new steering instruments like target/incentive-linked resource allocation. This change is also reflected in the shifting responsibilities of women's representatives and/or equal opportunities officers (Müller et al. 2011).
According to the experts, there are three essential elements which should be considered as prerequisites by all organizations undertaking structural change.
The first of these is knowing the institution, which can be achieved by developing statistics and indicators which ensure that the situation in each institution is widely known and acknowledged. The second is obtaining top level support, i.e. the support of the people in positions of power. The third element is generating effective management practices, e.g. by ensuring the availability of gender 3 It is not only female students but also female faculty who feel isolated in S&T area. Gilmer et al. investigated the experiences of female academics in STEM disciplines with isolation and related factors such as department fit and communication. They also examined the strategies that women use to overcome this isolation, primarily networking and mentoring (Gilmer et al. 2014).
4 Gender mainstreaming became an important issue in the debate on higher education sector reform at the end of the last century (Rothe et al. 2008). expertise and raising awareness (European Commission 2012) and by introducing greater transparency both in screening procedures and in the monitoring and evaluation system (European Commission 2004, 2009).
3. Gender analysis/gendered innovation in scientific knowledge and technology design (fixing the knowledge): This third approach integrates a gender dimension into research and innovation content and academic curricula (Schiebinger & Schraudner 2011;Lipinsky 2014). It focuses on overcoming gender bias in science and technology by incorporating gender analysis into all phases of basic and applied research, from setting priorities to funding decisions, establishing project objectives and methodologies, data gathering, evaluating results and transferring ideas to markets. This mainstreaming of gender analysis into research ultimately creates "gendered innovations" (Schiebinger & Schraudner 2011), which could also serve to build inclusive scientific communities in which men and women assume an equal role at all levels, i.e. in decision-making, in policy setting and in defining and carrying out research. This would, in turn, lead to a change in the scientific 'subculture' with its male hegemony.
Lipinsky (2014) stresses the importance of the (socio-)political level and points out that governments can initiate institutional change in RPOs and RFOs by creating a conducive legal and political environment and by providing incentives for change. This can include incentives for research establishments to recruit female academics, funding programmes for gender equality in research or support for innovations which combine output related strategies aimed at advancing women in middle and senior academic positions (output in numbers) with input related incentives for institutional change. Similarly, and according to EIGE (2014), governments could support gender equality by providing institutional mechanisms for gender mainstreaming 5 . As components of gender mainstreaming, this report lists 'commitment', 'structure', 'involvement of civil society', 'gender awareness training and advisory services for government bodies', 'focus on legal reform in the following areas: family, employment, social security, income tax, education, positive measures to advance women, perceptions and attitudes and creation of a culture that supports gender equality', 'sufficient budget resources and professional capacity' and 'tools'. Gender budgeting, as defined by the Council of Europe, is, in turn, the application of gender mainstreaming in the budgetary process (Rothe et al. 2008). In line with EIGE (2014), Caprile et al. (2012) note that "gender policy is not only made by regulation and legal changes but mostly by leadership and a commitment to changing structures and cultures".
As a consequence of all the above, a broad policy mix has been developed to support women and overcome gendered structures. This includes both the provision of career support for women (mentoring, coaching, training, networking, leadership programmes) as well as institutional measures (provisions to facilitate a work-life balance, e.g. flexible working hours, childcare facilities). Empirical evidence (European Commission 2006) shows, however, that simply offering work-life balance options is not enough: The organisational culture (as evidenced in communication regarding such practices and, in particular, in the reactions of supervisors and peers) must also be truly supportive of such policies. This includes the need for a reflection on organisational norms, including the (often implicit, mostly incompatible) traditional notions of the 'ideal worker' and the 'ideal mother'. These normative beliefs are heavily influenced by gender stereotypes which are similar across cultures and which 5 The BPfA (Beijing Platform for Action) points out that the main task is to support government-wide mainstreaming of a gender equality perspective in all policy areas. The essential conditions for effectiveness are location at the highest level in government, sufficient resources and process management inter alia (EIGE 2014). relate to the 'separate spheres' of home (i.e. care, children) and work (i.e. career). While such norms appear extremely resistant to change, the good news is that in reality these spheres increasingly overlap and are no longer defined by one gender (European Commission 2009b). Furthermore, experts contend that insufficient process management is an important factor that affects the slow progress towards gender equality. While many reports and conferences express commitment to gender equality, this does not always lead to corresponding efforts and/or the implementation of policies in this direction. There is evidently a lack of management commitment to sustainable and significant change in which planning, decision-making, implementation and evaluation follow the customary process cycle (European Commission 2006).
Although numerous initiatives have been introduced in this field, there is a lack of evidence regarding the direct effects of the corresponding interventions. The literature on this topic identifies several shortcomings in gender equality policy design (Müller et al. 2011;Caprile et al. 2012): 1. The lack of a more thorough theoretical foundation of projects and research which implement and evaluate gender equality policies (i.e. an unclear link between statistics and gender inequality). Most of the evaluation studies that have been conducted are descriptive and lack explicit theoretical references. This lack of an explicit theory is a further handicap to tackling structural and cultural change.
2. The lack of an unambiguous definition of gender equality, i.e. an open discourse on the dimensions gender equality entails, how progress towards gender equality can be measured and what constitute the indicators of success.
3. The lack of common quality standards for evaluation. Indeed, a common evaluation framework could also prove useful for addressing the related problem of detecting structural change. This also points to the need to make the normative component of many evaluation studies explicit.
4. The need for research into the long-term effects.
5. The gaps between policy implementation and changes in societal values.
The literature also reveals several gaps in research and data. Some of these gaps are the result of insufficient statistics or measurements, e.g. lack of genderdisaggregated data (EIGE 2014), lack of knowledge on specific disciplinary career paths, advancement and obstacles (Müller et al. 2011) and lack of information on nonnormative scientific careers (Caprile et al. 2012). Others relate to the discussion of 'scientific excellence' and the need for new, unbiased measurements (European Commission 2004;Husu 2004). The problems of defining and measuring excellence have not been solved. Experts agree that scientific excellence is not 'a universal fact' but rather a social construction and, as such, it is open to many kinds of biases. Following this logic, the spectrum of activities and achievements included in the definition of scientific excellence must be broadened to include, for example, other dimensions of scholarly practice, such as education, participation in committees, administrative tasks, external consultancy and contribution to public debates (European Commission 2004).

Functional vocabulary of gender equalitydefinitions and terminology
Building on the results of the literature review, this chapter seeks to develop a functional vocabulary of gender equality. This vocabulary establishes the relevant definitions and terminology that are (to be) used to address gender equality consistently within the MoRRI project. A further goal of this chapter is to outline any potential links between gender equality and the other RRI dimensions.
MoRRI follows a social constructivist understanding of gender. Thus, gender is understood as a social construct which results from performative practices (Butler 1990;West & Zimmermann 1987). It encompasses not only the differences between men and women, but also the distinctions which are made in order to replicate social gender roles as well as the mechanisms of inclusion and exclusion associated with these roles. Gender does not denote a simple dichotomy of two supposedly homogeneous gender groups but must instead always be conceived as intricately linked with other structural categories such as age, socio-economic status, race/ethnical background, disability, sexual preference, etc. The aim is not only to differentiate between men and women but to follow an intersectional approach (Hancock 2007) and inherently consider other relevant criteria for differentiation in any kind of gender analysis.
Consequently, gender equality is understood as a three-dimensional construct whereby gender equality is reached when (1) women and men are equally represented in all disciplines and at all hierarchical levels, (2) gendered barriers are abolished so that women and men can develop their potential equally, and (3) when the gender dimension is considered in all research and innovation activities.
If we look more closely at this three-dimensional definition of gender equality, it becomes evident that gender equality policies in science and research demand more than just the promotion of women in male-dominated fields or to male-dominated positions. Indeed, there should be three pillars to equality policies.
The first pillar comprises measures to promote women in fields where they are under-represented as well as to increase female participation in management and decision-making positions. The goal here is to reduce gender segregation. In employment terms, gender segregation refers to the tendency of women and men to work in different occupations and sectors. We distinguish here between two types of segregation. Horizontal segregation is understood as the under-representation or over-representation of a specific group of workers in occupations or sectors not ordered by any criteria, whilst vertical segregation refers to the under-representation or over-representation of a group of workers in occupations or sectors at the top of a ranking based on 'desirable' attributes -income, prestige, job stability, etc. In the literature, vertical segregation is sometimes referred to as the 'glass ceiling', which points to the existence of visible or invisible obstacles that lead to the underrepresentation of women in positions of power and decision making functions. This is completed by the concept of the 'sticky floor', which describes the forces that tend to maintain women at the lowest levels in an organisation (Caprile et al. 2012).
The second pillar comprises structural measures aimed at a changing existing organisational arrangements in order to progressively eliminate barriers for women on their path to top positions or factors which induce women to drop out of science. This is in line with the gender mainstreaming 6 approach, which aims at "the (re)organisation, improvement, development and evaluation of policy processes, so that a gender equality perspective is incorporated in all policies at all levels and at all stages, by the actors normally involved in policy-making. In concrete terms, this implies that the needs, interests, competences and skills of both women and men are taken into account" (Council of Europe 1998). In its recent recommendations to Member States on gender equality standards and mechanisms, the Committee of Ministers of the Council of Europe recalls the importance of adopting methodologies for implementing the gender mainstreaming strategy, including gender budgeting, gender-based analysis and gender impact assessment (Council of Europe 2011). A gender-based analysis is designed to establish a description of the realities facing men and women in a specific field. The goal is to provide empirical evidence for policy development that adequately considers the socio-economic realities of both genders. Gender budgeting refers to the application of gender mainstreaming to the budgetary process. This includes a gender-based assessment of budgets, bringing a gender perspective into all levels of the budget process and restructuring revenue and expenditure to promote gender equality. Gender impact assessment has its roots in the environmental sector and is a typical example of an existing policy tool that has been adapted for use in gender mainstreaming. Gender impact assessment allows for the screening of a given policy proposal in order to detect and assess its differential impact or effects on women and men, so that these imbalances can be redressed before the proposal is endorsed. Gender impact assessment can be applied to legislation, policy plans, policy programmes, budgets, concrete actions or government bills as well as to reports or calls for research.
The third pillar of gender equality -the integration of a gender dimension in research and innovation content -is legitimised by the gender mainstreaming strategy on the one hand and by quality standards in science and research on the other (Caprile et al. 2012). Gender studies are now either well-established or at least partly in place in almost all fields of research. Indeed, it is argued that research results are not valid or reliable if they only consider male research subjects. This point has been discussed at length, for example in medicine with regard to false diagnosis or medication (e.g. the false diagnosis of heart attacks among women or the different effects of the same dosage of medication on men and women). Mainstreaming gender analysis into research creates gendered innovations, while a gender bias (as described above) limits the potential benefit of science and innovation to society. Hence, it is important not only to identify gender bias in science and innovation but also to understand how it operates. On this basis, gender analysis is seen as a resource that stimulates gender-responsible science and innovation.
The successful implementation of equality policies which include all three pillars should lead to a cultural change both in science and research as well as in RFOs and RPOs. In academia, the concept of excellence plays a crucial role in this regard. Excellence as a (national) higher education and research strategy defines the institutional framework for a career in science, academia and research. This ostensibly focuses on the implementation of meritocratic principles in science, academia and research, which are shown in feminist research to be gender biased. To achieve cultural change, new standards for excellence are needed in the academic and research system. This requires a reflection on the inherent gender bias in the definition of principles to support excellence such as output orientation, the breaking down of research findings into least publishable units, strategic publication planning ('A' journals), research niche 6 Gender mainstreaming was established as a major global strategy for the promotion of gender equality in the Beijing Platform for Action produced at the Fourth United Nations World Conference on Women in Beijing in 1995. building and the assessment of ability based on the amount of third party funding generated (Matthies & Zimmermann 2009;Beaufays & Krais 2005).
To sum up, in the context of the MoRRI project, we understand gender equality as a three-dimensional construct. To achieve gender equality, progress is required on all three dimensions. Accordingly, the three equality goals are:  integration of women in all fields and at all levels in research and innovation (reduction of horizontal and vertical segregation) 7 ,  structural change in research institutions in order to abolish structural barriers for women (e.g. through implementation of comprehensive equality plans, quotas for women, transparent decision-making), and  integration of gender in research and innovation content to ensure that the needs and interests of women are adequately addressed.
This definition of gender equality in science and research is in line with the equality goals formulated for the European Research Area (ERA) strategy and, thus, in Horizon 2020. The strategy on gender equality in Horizon 2020 aims at:  fostering gender balance in research teams, in order to close the gaps in the participation of women,  ensuring gender balance in decision-making, in order to reach the 40% target for the under-represented gender in panels and groups and 50% target in advisory groups, and  integrating the gender dimension in research and innovation (R&I) content, in order to improve the scientific quality and societal relevance of the produced knowledge, technology and/or innovation.
Considering the three-dimensional construct of gender equality in science and research consequently, gender is also relevant in other RRI dimensions. In the course of the interim evaluation of the Science-in-Society programme, in particular the section on "Assessment of Future Options" (Bührer et al. 2012a), the connection between RRI and the (former) Science-in-Society topics was also a subject of investigation. In concrete terms, an Impact Assessment workshop was used to ask 26 expert participants from different thematic fields to express their views on the interconnectedness of the different topics and their potential integration into the RRI concept. The results of this workshop are depicted in Figure 1.

7
The focus on gender also means that men have to be addressed in order to reduce horizontal segregation -e.g. in fields where they are underrepresented (e.g. humanities, pedagogy and veterinarian medicine).  Figure 1 shows the strong connections between the different RRI dimensions. It is notable that, in almost all cases, the experts are in agreement on the strength of the bonds between the dimensions and the fact that these connections are reciprocal. With regard to the gender dimension, the participating experts perceived particularly strong connections with Ethics and Governance (reciprocal and strong), medium connections with Science Education (reciprocal but medium), non-reciprocal connections with Public Engagement and no connection with Open Access.
This does, however, raise another question, namely how these interrelations can be defined with regard to content. The questions/aspects formulated below serve here only as a first step towards the more systematic analysis of linkages between the gender dimension and other dimensions, which will be carried out in the course of the MoRRI project. Science education: PISA results indicate a significant gender difference in interest and self-confidence in science among 15-year-olds. In order to tackle this gender difference, teaching and learning strategies will need to be reformed. As a consequence, teacher education will also need to be changed to support the implementation of such reforms. In addition to measures addressing the pre-school and school sectors, measures addressing adults should also be designed in a gender sensitive manner (e.g. science labs, science museums).
Ethics: it should be standard practice in good (reliable, valid, transparent) science to consider gender as a central topic in all research. From an ethics point of view, it is important to avoid the (re-)production of gender stereotypes in all stages of research (formulation of research questions, research process, analysis, reporting) as well as in the application of research findings. Gendered norms and values as well as unintended consequences and impacts, particularly for women etc., could play a role here.
Governance: All steering mechanisms in science, research and innovation should consider the three-dimensional construct of gender equality. This is especially important in the budget allocation context as well as in the development, implementation and evaluation of science, research and innovation policies. The different instruments intended to influence behaviour and/or processes, e.g. regulatory acts like quota regulations, also play an important role here.

Review of existing empirical knowledge of gender equality
In this chapter, which constitutes the bulk of the report, the focus turns to empirical studies in the gender equality field. It presents the results of Sub-task 2.2 and Subtask 2.3, which review the state of knowledge regarding the RRI dimensions, including the empirical knowledge emerging from EC-funded studies on the RRI dimensions. In doing so, it focuses on those results which apply specifically to the gender equality dimension.
The chapter is divided into two parts. It begins with a review of selected EC studies which contain particularly rich empirical information on gender equality. This is followed by a summary of a selection of other studies which offer equally rich information on gender equality. The aim of the review of the EC studies is: 1. To specify the questions concerning gender equality that are (partially) answered in these studies, 2. To tentatively identify the indicators that can be harvested from these studies, 3. To assess whether the information contained in these studies relates to the context, input, output, or outcome of gender equality following the intervention logic model, 4. To specify the analytical level of the information and distinguish between global, national, and sub-national (regional, institutional, programme/project, and individual) levels, and 5. To specify whether the studies provide quantitative or qualitative data.
Our aim with the extensive list of other relevant empirical studies is to summarize the information sources, the analytical level at which the information is presented and the key focus of the studies, in order to pave the way for a subsequent qualified selection of existing gender equality indicators in Task 3 of the MoRRI project.
This review of studies containing empirical information on gender equality will then serve as the background for assessing the overall availability of empirical information on gender equality (see Chapter 5).

EC studies and projects in the gender equality field
A number of EC projects have explored the gender equality dimension. For the purposes of this report, eight projects which are considered particularly relevant for the gender equality dimension in terms of identifying empirical data for further analysis have been reviewed. These projects are listed in Table 1 below.
Seven of these projects aim at supporting gender equality in RPOs through the implementation of gender action plans which address several areas simultaneously. These interventions are targeted at: 1. Increasing female participation in STEM disciplines and in management positions, 2. Building up gender competence among all stakeholders with a special focus on management, 3. Tackling structures that hinder women on their path to top positions (with a focus on recruitment practices), and 4. Supporting the integration of a gender dimension in research and innovation content.
They also aim at initiating a broader discourse on gender equality in science and research by addressing not only RPOs but also RFOs, politicians and experts (in particular the GenSET, STAGES and GENOVATE projects). The eighth project, MORE2, does not explicitly address gender but does focus on two aspects that are of high relevance to gender equality barriers, namely mobility patterns and career paths (including remuneration aspects).
A key instrument for progress towards gender equality in RPOs is the development and implementation of targeted gender equality plans. This requires the development of a comprehensive policy mix for research performing institutions, which addresses any problematic aspects (e.g. gender gaps and their origin) revealed in a gender analysis. It is ultimately to be assumed that the measures in a consistent and coherent policy mix will support and supplement each other. Hence, it is not possible to determine direct relationships between a single measure and an achieved outcome (e.g. change in female participation in management). On the contrary, change is understood as a consequence of the policy mix as a whole.
All these EC projects include case studies where targeted gender equality plans and other specific tools have been implemented. One of the explicit goals of the projects is/was to develop general guidelines or supporting tools based on the analysis of experiences made within the case studies. The target group for these guidelines, tools, etc. are RPOs interested in the implementation of gender equality policies.
The analyses of the case studies provide starting points for the development of context, input and output indicators with regard to gender equality at institutional level. In most, cases input indicators cover a broader range of dimensions of gender equality than output indicators. Conversely, output indicators focus in most cases on one dimension of gender equality, i.e. horizontal and vertical segregation. It would appear to be difficult to use quantitative indicators to measure output with regard to either structural or cultural change or the integration of the gender dimension in research and innovation content.

PRAGES -Practising Gender Equality in Science
The PRAGES project, which was carried out in 2008 and 2009, aimed to analyse "existing practices to support universities and research institutes, both in European and extra-European (Australia, Canada, USA) countries, willing to implement genderequality oriented measures in their research management". 8 As main outputs of this extensive analysis and stocktaking exercise, a database containing an assessment of 109 promising gender quality programmes in S&T and a set of guidelines were produced. These guidelines include 31 recommendations, 61 lines of action and 219 specific examples relating to how gender equality can be promoted in S&T (Cacace 2009). The vast amount of stocktaking data collected, the recommendations produced and the cross-cutting analysis performed provide a very rich foundation for further analysis in terms of indicator development within the gender equality dimension (see also Table 2).

GENSET -Increasing Capacity for Implementing Gender Action Plans in Science
The GenSET project, which was carried out in the period from 2009 to 2012, aimed "to improve the excellence of European science through inclusion of the gender dimension in research and science knowledge making." It is a "forum for sustainable dialogue between European science leaders, science stakeholder institutions, gender experts, and science strategy decision-makers, to help implement effective overall gender strategies". 9 To facilitate the sustainable dialogue, a range of participatory mechanisms -such as consensus seminars, mentoring workshops and gender summits -were implemented with the purpose of producing "practical guidelines for implementing gender action plans within existing institutional mechanisms". One related goal was to explore how existing gender knowledge and expertise could feature effectively in European science institutions in order to "increase women's participation in science" with regard to (1) science knowledge-making, (2) the research process, (3) recruitment and retention, (4) the assessment of women's work, and (5) the science excellence value system (cf. Recommendations for action on the gender dimension in science 2010:6).
The set of recommendations produced with regard to constructing an "overall gender strategy in scientific institutions" include actions in four different areas (see Figure 2). The actions recommended could be a useful resource in the development of indicators which characterise gender equality. The range of additional experiences, outputs and outcomes produced throughout and beyond the project phase are also considered relevant for further analysis. For instance, the European Gender Summits explored "how gendered methodologies can stimulate innovation and advance scientific excellence" with regard to main policy initiatives. At the first Gender Summit, a manifesto for integrated action on the gender dimension in research and innovation was signed by more than 2,300 researchers. This manifesto stresses the important role of research funding institutions and calls for the consideration of gender in all stages of all research projects. The recommendations of the subsequent Gender Summits forge a collective commitment to strengthening human capital development, scientific research and innovations and the transformation of higher education and research institutions via an integrative focus on gender equality. Hence, gender has to be integrated in research and innovation content, methods, analysis policies and practices. The conclusions of the fourth Gender Summit once again stress the relevance of gender criteria for all Horizon 2020 calls as well as for their monitoring and evaluation.

GENIS LAB -The Gender in Science and Technology LAB
The GENIS-LAB project (2011-2014) "aims to implement structural changes in a group of selected scientific organisations in order to overcome the factors that limit the participation of women in research". Furthermore, the project aims to construct and implement effective gender equality policies which have the capacities to bring about tangible changes in scientific organisations. The project focuses primarily on the nanotechnologies field as well as two other STEM areas, namely Physics and ITC. In contrast to earlier measures which focused primarily on producing cultural change, the GENIS-LAB project promotes actions centred around the structural conditions/factors that impede effective gender equality in scientific organisations. Such factors are seen to be related both to organisational systems as well as to the relationship between the organisation and the individual. In order for systematic actions to be effective, "cooperative support" is deemed necessary at local, national and European levels. With this established premise, GENIS LAB implemented an "integrated and systemic approach, focusing on three levels": 10  The organisational level (scientific organisation partners). Development of specific management tools and formulation of self-tailored action plans aimed at promoting internal structural changes.


The social/environmental level. Training for HR managers aimed at fighting against stereotypes (de-constructing the stereotyped relationship between women and science). Training will support cultural changes within the organisation through the re-definition of excellence evaluation criteria.


The transnational European level. Promotion of networking/mutual learning among involved scientific organisations to support the exchange of experiences, practices, and efficient management tools. The driving idea is to promote and support structural changes on the basis of self-tailored action plans in order to establish a more equalitarian approach to (female) talents that is based on the recognition of skills and competencies and is suitable to overcoming gender discriminations.
Some of the key tools for obtaining the project's objectives include the participatory gender audit methodology (PGA), gender budgeting, evaluation of excellence and a training programme for HR managers. The project is currently in its closing phase, and final results have not yet been published. It is to be assumed that the range of results, experiences and recommendations produced will serve as an inspirational source for indicator development. For instance, the specific PGAs produced at the six partner institutions already provide relevant data for further analysis. The PGAs analyse specific factors which impede women's participation in scientific research decisionmaking and develop "additional tools to assess gender equality issues within organisations (e.g. redefinition of criteria for organisational assessment related to human resource and gender; gender stereotypes; gender responsive budgeting)" (Periodic Report Summary 2013:2). The adaptation of the International Labour Organization's PGA and the performance indicators used therein (see www.ilo.org) could also provide relevant data for performance indicator developments at the organisational level.

INTEGER -Institutional Transformation for Effecting Gender Equality in Research
The INTEGER project (2011INTEGER project ( -2015 aims to "engage problems of gender equality in academia and research institutions in the STEM sector involving every part of themmanagers, researchers, administrative staff etc." 11 . Its main objective "is to escalate career progress of women in research and academia." Accordingly, it will strive to improve organisational matters, raise awareness of gender correspondence and initiate a more comprehensive working environment, thus increasing the number of women researchers. 12 Four key areas of intervention and analysis are to be implemented over the course of the project, namely: INTEGER further aims to create so-called transformational action plans as a key output. These are defined as "a user-friendly and effective implementation model, written utilising an 'approved code of practice' approach" 14 . Evaluation methods (GESIS) will be used to measure the effects and impacts of these plans in order to improve them and make them available for use beyond the duration of the project and in institutions outside the consortium. The project chose to approach gender equality at the institutional culture and organisational structure levels rather than focusing on individual factors, an approach which has had a tendency to zoom in on how the individual can change (Convention report 2013:2) (this concept is also described by Londa Schiebinger as "fixing the women"). GENOVATE also aims to implement GEAPs in six European universities. As a part of this objective, a "social model of gender equality implementation" (SMoGEI), underpinned by "the gender Change Academy Model" (CAM) will be constructed. Key elements in this process are: 1. the establishment of a consultation model, using suitable mechanisms such as online surveys, online forum, etc. (in discussion with UNINA), or E-portfolios; 2. the documentation of each partner institution's perceptions and best practices in implementing the GEAPs, at micro and macro institutional level (personal, team, institutional levels); 3. discussion with all partners on the use of the CAM and the social model of equality for creating the GENOVATE SMoGEI (rationale, applicability, projected outcomes). (Convention report 2013:8).
The project is currently still in the implementation phase. Nonetheless, its "social model of gender equality implementation" is expected to be able to provide data on relevant mechanisms regarding barriers to gender quality, best practice, effects of implemented actions plans, etc.

STAGES -Structural Transformation to Achieve Gender Equality in Science
The STAGES project (2012)(2013)(2014)(2015) broadly aims to support "the career advancement of women researchers" through the implementation of institutional action plans and "structural change strategies" and by working in close cooperation with HR management at the participating research institutions.
The three main objectives of STAGES are: 1. Applying different self-tailored action plans aimed at introducing gender-aware management at all levels in each of the participating organisations. Each action plan includes activities in one or more of the 3 strategic areas identified in PRAGES: Building a women-friendly environment, promoting gender-aware science and supporting women's leadership in science; 2. Producing a deeper understanding of the dynamics surrounding structural change efforts by constantly analysing, monitoring and assessing the process activated in each institution, in order to initiate mutual learning practices among partners; 3. Spreading information among European universities and research institutes on successful negotiation strategies implemented to build consensus on and commitment to structural level gender equality initiatives which address different leadership levels and the many stakeholders directly or indirectly involved in change (Periodic Report Summary 1 2014).
Due to the ongoing status of the project, no final results have yet been produced. However, the results that are already available (see Table 7) and the final set of guidelines and recommendations to be generated are expected to provide relevant material for further data review in indicator development terms. Also of relevance are the set of five main criteria adopted for the regular evaluations performed in the project (effectiveness, efficiency, impact, sustainability, relevance), which cover more than 70 indicators (Progress evaluation report no. 3 2014). WHIST -Women's careers hitting the target: gender management in scientific and technological research WHIST (2009WHIST ( -2011 aimed to "improve the situation of gender diversity in science, by inter alia improving transparency in recruitment, promotion, and nomination" and to "increase the capacity of S&T institutions in monitoring, managing and feeding gender diversity in their own organisation, at all levels" 16 . The project followed the strategic directions for conducting successful interventions that were identified in the FP7 project "Practising gender equality in science" (PRAGES) to provide new knowledge on gender dynamics in scientific and technological research institutions. Experimental initiatives in building a women-friendly environment, promoting gender-aware science and supporting women's leadership in science were carried out in three organisations (Fraunhofer IAO, Stuttgart, Germany, the University of Aarhus, Denmark, and the European Space Agency, France) with the aim of identifying effective solutions.
The main objective of the WHIST project was to "provide for a review on the main areas of risk for gender diversity in research settings as well as on the correspondent regimes to cope with them that will allow the drafting of the provisional version of guidelines to be used for the implementation of experimental activities" 17 . To provide a solid base for the experimental initiatives, the guidelines were discussed in interactive workshops. The initiatives included "both the direct promotion of new programmes and the support to programmes promoted by the organization" to help revise the guidelines on their potential to support gender diversity in science.
In its "Guidelines on gender diversity in S&T organisations", the lessons learned were formalised in obstacles to the gender equality activity and recommendations to achieve the established objectives as well as information on the key role of negotiation activity and the size of interventions. The direct impact of the project was primarily the increased knowledge obtained on discriminatory dynamics, but it did also produce an increase in institutional capacity building and better awareness of the topic, as was pointed out in the internal evaluation of the project. 18

MORE2 -Support for continued data collection and analysis concerning mobility patterns and career paths of researchers
The aim of the FP7-funded MORE2 project is to "investigate remuneration and working conditions of researchers across 40 European and 10 non-European countries" because "these factors strongly impact the decision of researchers whether to become mobile or not during their career or whether to work in the academic or non-academic sector." 19 The main objective of MORE2 is to "provide internationally comparable data, indicators and analysis in order to support further evidence-based policy development on the research profession at European and national level." MORE2 is the follow-up to the MORE project and builds on its predecessor's "results and methodologies, which will be improved, fine-tuned and expanded, where needed, both methodologically and conceptually." The "Researcher Indicators Report" focuses on the selection and update of "indicators on the state and development of the European research system (EU and Member State level), particularly the stock, mobility and overall career path of researchers" 20 . Besides indicators that are related to various dimensions of mobility (geographical mobility; intra-EU 27 mobility and mobility in or out of EU 27; sectorial mobility; virtual mobility), it also offers information on the remuneration of researchers in over 45 countries. The data collected allow the comparison of remuneration aspects across countries and includes not only salaries, stipends and benefits by job position and employment contract but also social security systems, labour legislation in the Higher Education Institution (HEI) sector, the tax system, etc. Unfortunately, it only provides a very limited number of gender-segregated indicators.

Other recent empirical studies on gender equality
In addition to the EC-funded studies identified and reviewed above, a number of other studies also offer relevant empirical information on issues related to gender equality in the research and innovation contexts. Table 10 lists 31 such studies. For each entry, the analytical level in terms of aggregation is specified along with a brief indication of the key focus of the study. The publications included in this list provide helpful input for the development and discussion of indicators because they all explicitly or implicitly address formulated assumptions about direct relations between measures/policies and changes/effects. In the following, we will now discuss selected examples of such claims for the three dimensions of gender equality.

Participation of women
There is a lot of evidence pointing to the (under-)representation of women in science and research both at national level as well as in international comparison (mainly EU countries). The description of female participation in science and research refers in most cases to the image of the leaky pipeline. This image is characterised in most European countries by a female dominance among students and graduates and a decreasing share of women in higher levels of the hierarchy. 21 Women account for only 20% of professors (Grade A, EU 27) and 15% of heads of higher education institutions. However, although women are still underrepresented both in science and research as well as in top positions, the share of women increases and the leaky pipeline constricts slightly in the university and government sector at least. In the business sector, 19% of researchers are women (EU 27). However, there is less evidence available on vertical segregation in the business sector and how it differs from the university and government sectors.
Women are also still under-represented in STEM disciplines: one in four PhD graduates in engineering, manufacturing and construction, and 40% of PhD graduates in science, mathematics and computing is female. In contrast, almost two thirds of PhD graduates in education science are female.
A well-known problem with regard to horizontal segregation is that higher education course choice is strongly determined by experiences in school. PISA surveysespecially those focusing on maths and science -show that boys not only perform better than girls in these subjects, they also show a higher level of self-esteem and interest here (OECD 2013a+b, 2012. This is already the starting point for numerous programmes to increase the interest of girls and women in science (e.g. various STEM initiatives, specific degree course advice, job coaching, etc.).
One specific problem with regard to female participation in STEM disciplines is the drop out phenomenon. Several recent studies and articles discuss the reasons for female drop out in specific STEM disciplines (e.g. Cundiff et al. 2014;Hatmaker 2013;MacPhee et al. 2013;Robnett 2013;Britton et al. 2012;Good et al. 2012;Judson & Kulinna 2012;Stout et al. 2011). These publications focus on the male-dominated culture in STEM disciplines, which is the result of traditional and gendered practices, persistent stereotypes and the lack of role models. These (and other) factors create a working culture which lacks a sense of belonging for women. This also leads to a reduced self-perception of their academic skills among women, despite the fact that no gender gap in performance is evident (e.g. MacPhee et al. 2013). Conversely, research also indicates that if women stay in the STEM field, they enjoy equal career opportunities to their male counterparts (Best & Schraudner 2014).
In most countries -and at European level -several policies have been implemented to increase female participation in professor level and management positions (e.g. anti-discrimination legislation, positive action, quota regulations, etc.). These policies have indeed contributed to an increasing participation of women, especially in the public sector (including universities). However, has this also led to a change in culture in academia? Kanter (1977) formulated the hypothesis that increasing female participation would lead to a change in organisational culture. Indeed, it is assumed that the interests of an under-represented group cannot be ignored by an organisation if the share of that group reaches a critical mass (between 25% and 33%), since other topics will then feature on the agenda, decision-making criteria will change and -last but not least -better and more innovative decisions will be made. This assumption is also corroborated by research for the corporate sector: McKinsey and Company (2007) and Catalyst (2004) both demonstrate a striking and positive correlation between female participation in management and economic performance indicators. However, there is also research which shows that women in decision-making positions in universities are as gender biased as men (e.g. Moss-Racusin et al. 2012). Accordingly, in order to reduce gender bias in decision-making, female participation is a necessary, but not sufficient, condition for change. Instead, this requires the building up of gender competence among both female and male managers.
In recent years, quota regulations have been discussed at length as a measure to increase female participation in decision making. In academia, the quota discourse is characterised by a clash of two different logics: the goal of increasing female participation contains a social justice argument that is seen as a threat to excellence criteria, and is therefore denied by many academics. Balafoutas and Sutter (2012) analyse acceptance of several measures to promote women (including quota regulations) and the effect of such measures on performance and willingness to cooperate with "quota women". They found a significant positive impact of quota regulations on women's willingness to participate in a competitive situation and no negative effects on performance or cooperation in teams.

Structural change
A common goal of the EC studies presented in Chapter 4.1 is the elimination of structural barriers to women's careers in science and research institutions. It is assumed that structures and processes in RPOs and RFOs contain an inherent gender bias, which is reproduced in everyday practices. In most cases, this gender bias is not intentional but just "somehow happens" (Martin 2003(Martin , 2006. The implementation of NPM in academia supports or even strengthens traditional, gender-biased practices. NPM brings about new managerial control mechanisms based on quantitative performance indicators intended to raise efficiency and intensify competition in science (e.g. Jansen 2007).
There is a vast amount of research dealing with questions of gender bias in the context of appointment procedures or promotion, access to funding or within peer review. Van den Brink et al. (2012) analyse appointment procedures for professorships in the Netherlands and show that the success of women in appointment procedures is highly influenced by the design of the procedure (public advertisement or not). Wroblewski (forthcoming) shows for the Austrian context the scope of action universities have in designing non-discriminatory appointment procedures ( The study showed that peer review is not as "neutral" or "objective" as it claims to be. Despite the quality of the proposal, the applicant's gender or affiliation significantly influenced its chance of being funded. Gender bias in the research funding context is disturbing as it contradicts one of the core myths in science: decision-making should only be based on meritocracy (excellence). Several national funding organisations as well as the European Research Council (ERC) have analysed gender bias in success rates, or are currently doing so. 23 Most studies show an under-representation of women in funding (basic research) and offer different explanations for this phenomenon. Fischer and Reckling (2010) argue for the Austrian context that women are under-represented because of the gender segregation in disciplines. Jänchen and Schulz (2005) also use segregation to explain the low participation of women in funding, but focus on vertical segregation. The main reason why only a few women receive funding from the Swiss National Fund lies is seen to lie in the low number of formally qualified women.
Critiques from gender scientists on the current ways of measuring excellence focus on bibliometric indicators, i.e. advanced analytical tools used to assess scientific productivity, visibility and impact -factors which are interpreted as proxies for quality and excellence. The increasing use of quantitative measures for the assessment of research performance has enhanced the real consequences of bibliometric indicators for the allocation of positions and resources (Van den Brink et al. 2013;Husu & Cheveigné 2010;Weingart 2005). Previous studies have revealed significant gender differences in scientific productivity. On average, female scientists tend to publish fewer peer-reviewed papers than their male colleagues, although this difference has been decreasing since the 1970s (Xie & Shauman 2003). However, there are notable indications of a gender bias in bibliometrics in favour of male researchers (e.g. Abramo et al. 2013). For example, the classification of "excellent" and "normal" publication channels is influenced by mainstream approaches and topics, to the disadvantage of (female) researchers, who deviate from the prevailing norm in notions of research. Research which focuses on gender relevant questions is often not in line with mainstream research in a specific field. Instead, it is frequently interdisciplinary or transdisciplinary and is therefore difficult to assess using disciplinary excellence criteria. The increasing literature on gender and research funding highlights the importance of gender equitable measures of research performance.
Such findings confirm the need for structural change and support the calls for specific measures to initiate structural change in research performing as well as research funding institutions. Important instruments in this context include gender action plans or equality plans, which often form the focus of the case studies described in Chapter 4.1. It is assumed that a comprehensive bundle of measures (a policy mix) which addresses all three pillars of gender equality will bring about cultural change in research institutions. This is based on the assumption that the successful implementation of equality plans requires a reflection on existing practices with regard to an inherent -and probably unintended -gender bias. The success of such an approach depends on gender competent management. Consequently, measures also aim at increasing gender awareness and gender competence among decision-makers and management. To date, no comprehensive impact analysis of equality plans is available.

Gender in research and innovation content
The discussion of gender aspects in research and innovation content emerged several years ago and is part of a shift towards a "benefit-orientation" in the debate on gender equality.In the meantime, the need to integrate gender aspects into the research and innovation process has become largely accepted and, to some extent, even institutionalised, e.g. through specialised institutes like the Institute of Gender in Medicine (GiM) at the Charité University Clinic in Berlin.
The European Commission supported the improved consideration of gender aspects in research through different approaches such as (1) the mandatory integration of gender impact assessment in research proposals during FP6, (2) the existence of explicit programme lines within the Science-in-Society programme in FP7, and (3) the continuous monitoring of research project achievements with regard to gender aspects through the inclusion of corresponding items in the reporting questionnaires. 24 During an expert workshop organised in the course of the interim evaluation of the Sciencein-Society program (see above), the following aspects were identified as necessary steps for an improvement in gendered research and innovation in the future: gender bias (dominance of male as the norm) in science knowledge making, gender balance 24 Template Project Final Report, part 4.3: Report on societal implications; Science and society reporting questionnaire. Most recent recent developments are described in the document "Vademecum on Gender Equality in Horizon 2020 http://ec.europa.eu/research/ swafs/pdf/pub_gender_equality / vademecum_gender_h2020.pdf) in teams as enhancement of collective intelligence and new ways to promote interdisciplinary advancements (e.g. linking areas where women are in a majoritysuch as life sciences -with areas where they are in a minority -such as photonics).
In a large explorative study, the Fraunhofer society developed a conceptual framework and subsequent guidelines aimed at supporting researchers in the identification of gender aspects in their research processes (Bührer & Schraudner 2006). They also developed a checklist which helps guarantee proper consideration of gender aspects in the application of empirical social research methods like surveys, interviews, focus groups, etc. (Bührer 2006;Kane & Macaulay 1993). The conceptual framework of the Fraunhofer project elaborates four different aspects of the gender construct (Bessing 2006): (1)  The Gendered Innovations Website (http://genderedinnovations.stanford.edu/) offers a series of tools and case studies aimed at helping researchers and innovators to identify gender aspects in research. Its conceptual framework covers the complete research and development process, from the identification and determination of topics to the utilisation of results. It is pointed out, for example, that the definition of research content priorities is largely shaped by the availability of (public) funding, the dominant reward systems for the respective careers as well as existing norms and stereotypes (Schiebinger & Klinge 2010).
There are numerous examples of how the neglecting of gender aspects leads to suboptimal or even harmful results, e.g. with regard to the lack of appropriate diagnostic and therapeutic instruments for heart disease in women or the under-diagnosis of osteoporosis among men (for further examples in the health sector see IOM 2010; Wajcman 2010). Harmful product examples include car seatbelts and airbags which do not take into account the safety of pregnant women. Research and development processes can also reinforce gender stereotypes, e.g. by designing "male" and "female" computer games (Kafai et al. 2008). Likewise, the use of particular standards and reference models can lead to a certain bias if, for example primarily young white men are defined as norm (a famous example here are crash test dummies). There also several examples of dysfunctional product developments, e.g. assistance systems for the elderly (household robots) which neglect the fact that the main target groupelderly women -is not tall and strong enough to manoeuvre such robots or voice recognition systems that cannot recognise female voices, etc. In contrast, the potential for improved user-orientation through participatory research and design that involves both female and male target groups is also described in several studies (Schraudner 2006;Leung et al. 2004;Oudshoorn & Trevor 2003;Greenwood et al. 1993).

Assessment of the availability of data on gender
Based on our review and presentation of empirical studies on gender equality above, this chapter provides an overall assessment of the availability of data on the gender dimension for indicator development purposes. It discusses the issue of data availability in terms of 1) the extent to which the empirical studies provide relevant information across the three dimensions of gender equality identified in the functional vocabulary, 2) the balance and availability of both quantitative and qualitative data, 3) the extent to which the available data address the four analytical levels specified in the intervention logic model, and 4) the availability of data at different levels of aggregation.

Data availability across gender equality categories
The studies presented in the previous chapters offer rich empirical information on gender in science, research and innovation. Due to the substantial support provided by the European Commission through its systematic embedding of gender (and ethics) across all projects and programs since FP6 and the establishment of different working groups and networks (e.g. Helsinki Group, ETAN Group, etc.), many of the studies cited above include comparative elements, at least with regard to the EU Member States.
Following the three-dimensional definition of gender equality applied in MoRRI, there are great differences in the availability of data on 'horizontal and vertical participation of women in research' on the one hand and 'structural change in institutions' and 'gender in research and innovation content' on the other. The continued dominance of data on individual participation has been tackled recently by the enhancement of existing data sources like the "She Figures" (European Commission 2013).
Developed by the European Commission in co-operation with the Statistical Correspondents of the Helsinki Group on Women and Science, the "She Figures" contain statistics and indicators on the critical mass or scope of women in science, on female participation in different scientific fields, on seniority and careers as well as on the setting of the scientific agenda. They allow us to measure and monitor the extent of gender imbalances in science from a comparative perspective and provide empirical evidence to support the design of policies to reduce such imbalances. The "She The monitoring of ERA gender goals refers to "She Figures" with regard to female participation, but also focuses on cultural and institutional changes in organisations and gender quotas in committees. 25 The empirical basis is provided here by the ERA surveys of RPOs and RFOs.
The 2014 ERA survey is a simplified version of the 2012 questionnaire and was designed to reduce the response load and collect adequate data for the indicators agreed with Member States. ERA surveys gather information from 1,265 public or publicly funded research organisations (universities, institutes, hospitals, research agencies, etc.). With regard to RPOs, they cover about 20% of the total research population in the EU. In financial terms, the RFO responses represent around 34% of total government budget appropriations or outlays for R&D (GBAORD) in the EU. There is a great difference in representativeness between countries in both categories. The data are used to classify Member States 1) according to the existence of specific measures in support of the ERA, and 2) by the level of implementation of RFOs and/or RPOs in relation to the EU average.
In addition to these primary sources of gender data, there are also a number of projects that collect information on the implementation of gender equality plans and related measures on a case study basis (see Chapter 4.1 for details of corresponding EC projects). The CESAER Gender Equality Survey 2012/2013 merits specific mention here because it provides data on the state of play of gender equality and gender equality management for 48 European universities of science and technology based on a standardized survey (Horvat et al. 2014). It covers the organisational structure for gender equality, gender equality plans, initiatives and measures supporting gender equality, barriers and statistics on the gender structure of staff (on different hierarchical levels) and also provides examples of best practices.
With regard to institutional strategies, the CESAER survey covers (1) existing gender equality plans and whether they are integrated into overall strategy, (2) monitoring, evaluation and/or benchmarking activities, and (3) implemented measures like gender budgeting or internal communication and supporting measures like training, manuals, etc. In terms of implementing strategies and plans, the survey distinguishes between nine different activities, including work-life-balance support, programs for attracting female students, networking opportunities, quotas, etc. With regard to organisational structures and approaches to promote gender equality, the survey differentiates between a special unit for gender equality, gender equality as part of the other responsibilities of a unit, one person dealing full-time with gender equality, one person dealing part-time with gender equality, no unit or person dealing with gender equality and other ways of supporting gender equality. It also looks at requirements in place for gender diversity in appointment committees.
The CESAER report depicts barriers to gender equality in the participating institutions. Typical barriers mentioned here are lack of tailored regulations or policies, lack of resources for the implementation of gender equality and internal resistance. Change resulting from gender equality activities are described in terms such as the top institutional level takes responsibility for gender equality, women get more visibility, dedicated institutional structures, increased gender awareness, changed institutional cultures, improved working environment, changes in quantitative terms, positive impacts of specific programs, targeted measures to support institutional change, planning, monitoring, evaluation, benchmarking and gender equality performance as criterion in university ranking (Horvath et al. 2014:39).
Other interesting projects which collect data on science and research are also in place, although they are not all adequate for use in gender analysis. For instance, MORE2, a study to support the continued collection and analysis of data on the mobility patterns and career paths of researchers, covers gender relevant aspects in science (e.g. career development and remuneration of researchers). However, gender issues are not an explicit focus, and gender segregated data is only presented for a few aspects (such as the stock of researchers, satisfaction with scientific career or the gender wage gap to researchers in the private sector).

Availability of quantitative and qualitative data
Due to improvements in the provision of sex-segregated data, progress in gender equality can be identified in many aspects on the basis of quantitative data. This applies especially for the participation of women in different fields and positions in research. The challenges for a monitoring of the gender dimension in RRI are 1) to go beyond sex segregated data and assess gender differences, 2) to measure gender inequalities beyond female participation, 3) to quantify progress in structural changes in institutions, and 4) to indicate the integration of gender in research and innovation content.
Most of the available quantitative data address the horizontal and vertical segregation of women and men in science and innovation, an indicator which is thoroughly documented by the "She Figures". The share of female researchers serves as a rough indicator for gender inequalities in careers in science, research and innovation. Differences relating to fields and levels of research positions indicate hierarchical differences between women and men. Complex indicators like the Dissimilarity Index for horizontal segregation or the Glass Ceiling Index for career opportunities of women in academia allow a comparative analysis between countries in their respective contexts. Similarly, participation indicators cover individual researchers by gender for EU Member States, candidate countries and EFTA countries based on R&D statistics.
However, there are also some shortcomings in the comparability between sectors (higher education, government, business), especially with regard to scientific fields or career positions. In the higher education sector, sub-categories of the major science and technology fields can be applied because corresponding detailed administrative information is available. This is not really possible, in contrast, for the private and non-profit sectors. Similar difficulties arise with regard to hierarchical differentiation in non-university institutions, where there are no equivalents to the rigid hierarchy of formal positions in higher education. Likewise, it is not easy to differentiate between activities in the field of research and innovation and related technological, industrial, administrative or other supporting activities in a non-university setting.
Researchers can be differentiated according to their gendered social roles, for instance by differences in parental roles. The 'maternal wall' refers to the multiple barriers faced by women scientists with family responsibilities. Because of the lack of quantitative data on the family situation in research data, "She Figures" uses EU-SILC 26 data to report on the parenthood status of researchers, with the deficiency that researchers cannot be identified precisely in this data.
Similar challenges occur in approaches to measuring female disadvantages beyond participation: There is no quantitative gender-segregated data on career paths or the working conditions of researchers that can be used for a monitoring. The gender wage gap can be interpreted as the central indicator of gender-structured labour markets. As a synthetic indicator of multiple inequalities between men and women, this gap is determined by differences in educational attainments, labour market experience and tenure, sectoral affiliation and occupations as well as by wage discrimination, etc. Because of the lack of harmonized and comparable gender-segregated data on the remuneration of researchers, the Structural Earnings Survey is used, with the focus on men and women with ISCED 27 5 and 6 level qualifications in ISCO 28 2 and 3 occupations as a proxy for researchers, with the major disadvantage that this does not cover the government sector.
Women in decision-making positions can be measured by their share in top positions in research and management as well as by female participation in recruitment committees and evaluation processes. The ERA surveys provide a basis for such indicators, but with the drawback that the coverage of institutions is limited.
A number of projects deal with the implementation of policies and programmes, including guidelines and monitoring criteria for the promotion of structural change towards gender equality in science (see Chapter 4). However, the availability of data on structural change is much less satisfactory than for female participation. Experiences in statistical surveys show that dealing with diverse gender equality measures is difficult. Accordingly, the data collected data on implemented measures for gender equality are hardly comparable between countries according to intensity or scope. Information on the existence of gender equality plans, recruitment and promotion policies only indicate if and how gender equality is on the agenda of RPOs and RFOs.
Possible indicators for assessing the integration of gender in science and research content include whether gender criteria are used in research project or programme evaluation processes or whether other activities to promote gender content in research exist. The challenge here lies in transferring these qualitative data into quantitative indicators. However, it is rather challenging to operationalise change of structures and processes as a consequence of the introduction of systematic screening of gender aspects in research proposals or the institutionalisation of governance structures which shall enable their detection. Process-oriented and structural criteria of this kind are typically dichotomous, indicating merely the existence or lack of the respective structures. This is the case for the ERA surveys mentioned above, and is mirrored by the criteria used for the final reports of FP projects, which respond to questions like "Was there a gender dimension associated with the research content?" [Yes, please specify / No] 29 , or "Type of actions mentioned within the GAPs" 30 (Design and implementation of an equal opportunity policy, Clear defined targets to achieve a gender balance in the workforce, Organisation of conferences and workshops on gender, Actions to improve work-life balance, Other [text box]). 31 The illustrated examples of indicators provide valuable information on the implementation of policies but not on the quality of the implementation processes or the changes the bring about. Accordingly, the implementation as such remains a black box.

Availability of data across the analytical levels included in the intervention logic model
Following the MoRRI proposal, indicators will be considered for different levels or phases of the 'logic model' of gender equality interventions. These levels include the 'context', i.e. the overall environment for gender equality, the 'input', i.e. the activities carried out, measures taken, structures created or resources provided to improve gender equality, the 'outputs', i.e. the immediate or direct results of such activities, and the 'Outcomes' i.e. the medium-and long term achievements and consequences of a better integration of women in research and innovation (from both a position and a content perspective).
The empirical information that emerges from the studies presented in chapter 4.1 mainly address the input and the outcome levels of the 'logic model' of gender interventions. The relevant context, i.e. the overall environment for gender issues, relates to the gender division of paid and unpaid work. Output indicators, i.e. the immediate or direct results of gender equality policies, are scarcely available because there is little evidence on causal effects. Accordingly, we will interpret most of the indicators as outcome.
A large share of the available data addresses indicators for the representation of women in different fields and decision-making positions. Depending on the underlying goal, specific indicators can be interpreted as context, input, output or outcome indicators. For instance, the number of formally qualified women in a specific field establishes the basis for female professors in that field. The share of women in decision-making positions can also be interpreted as an input indicator for cultural change in universities. As explicit goals of specific measures (e.g. recruitment processes for professorships), they can also be interpreted as the direct output of activities or as the long-term outcome. Following the logic of the gender dimension in MoRRI, the integration of women in all fields and at all levels in research and innovation is one of the main goals. Therefore, we will consider participation of women mainly as an outcome.
Input indicators relate to a range of different activities covering institutional measures like regulations, institutional settings, etc. as well as to substantial measures like individual career promotion, work-life-balance activities, etc. This includes context factors that directly influence the integration of women into research (like working hours arrangements, care infrastructure, working conditions in the business sector, equal pay legislation, etc.).
A large proportion of the literature reviewed depicts the influence of context factors on gender equality in science and research (e.g. working hours arrangements, care infrastructure, working conditions in the business sector, equal pay legislation). The analysis of outputs and outcomes is less common: If these are the subject of a study, they mainly occur in terms of improved performance (e.g. more frequently cited publications, improved products and services or integration of new aspects in research content etc.).
However, there is also a growing body of literature on how to evaluate gender equality measures and activities. How this can be used in the definition of typical outcomes, impacts and benefits should therefore be explored further. Indeed, one of the main challenges of the continued work within the MoRRI project will lie in developing indicators for the impact/benefit or output of gender equality activities.

Availability of data at different levels of aggregation
A significant number of the empirical studies presented above provide empirical information on gender equality at the European level (EU Member States, candidate countries and EFTA countries). Likewise, several studies also target the national level. But there is also rich empirical evidence referring to the sub-national level, and in particular the institutional level, since single (R&I) institutions are also implementing structural change. The individual level is also featured, for example, in discussions of the individual success factors for career advancement.

Data selection for RRI monitoring -reflections on current data gaps and required data collection
The purpose of this chapter is to assess data gaps and reflect on the need for primary data collection in order to mitigate these data gaps based on the content and results of the previous chapter as well as the list of promising indicators constructed in Chapter 7.
With regard to the data available for monitoring gender equality, we can conclude that while harmonized quantitative data is available for the first dimension (participation), the data available for the structural change and gendered innovations dimensions can be described as a patchwork of mainly qualitative data. Although there are several projects implementing and monitoring equality policies in research organizations (including Helsinki Group initiatives and the development of monitoring for ERA goals), there is less representative data available for cross-country analysis. Even less satisfying is the supply of data on the gender dimension in research and innovation content. With the exception of EU projects that are subject to specific evaluation criteria, hardly any data exists on the integration of gender in research. This dimension can only be covered by input indicators limited to specific fields. However, the data on the integration of women in all fields and at all levels in research and innovation has several shortcomings. Vertical segregation can be presented only for the higher education sector by grade, and no data currently exists that would allow an analysis of the hierarchical position of female scientists in sectors other than higher education. Other indicators illustrating structural gender inequalities for researchers (e.g. income distribution) are faced with the challenge of defining what constitutes a research occupation. Last but not least, gender discrimination in excellence evaluation criteria can scarcely be illustrated using quantitative indicators.

Early thoughts on gender equality indicators
This chapter provides a space for compiling promising indicators based on the existing empirical information identified throughout the report. The intention here is to prepare the ground for Task 3, in which the existing indicators will be selected and new indicators will be developed.

Name of indicator
Women's participation in paid work

Brief description
Quantitative indicator on women's participation in paid work to illustrate the context of female employment in science and research

Source of data Labour Force Survey
Date 2013

Measurement level
Metric -share of women in total working population

Unit of analysis Country
Coverage 33 EEA countries

Name of indicator
Share of female researchers by sector

Brief description
The percentage of female researchers depicts the (under-)representation of women in research. Its differentiation by sectors indicates different opportunities and barriers.

Analytical level (logic model)
Context or outcome-related

Source of data Eurostat: Statistics on research and development
Date 2011

Time series
Most countries biennial -but data availability differs according to countries

Measurement level
Metric -share of female researchers

Unit of analysis Countries
Coverage R&D statistics are currently available for EU Member States and Candidate Countries, EFTA Countries, the Russian Federation, China, Japan, the United States and South Korea. Regional R&D statistics are available for EU Member States, Candidate and EFTA countries. Besides national and regional statistics Eurostat calculates and disseminates aggregates at the EU-and Euro-area-levels (EU-28, EU-15 and EA-18).but data availability differs over the years.

Information Item G3
Name of indicator Years to achieve gender equality in research participation

Brief description
Estimation of the years required to reach equal participation (50%) of women and men in research, based on the average growth rate of female participation in research between 2003-2011 and the share of female in researchers in 2011. This indicator is very responsive to progress and refers to the status quo in female participation.

Input-related
Analytical level (aggregation)

Source of data
Eurostat: Statistics on research and development

Time series
Most countries biennial -but data availability differs according to countries

Measurement level
Metric -estimated time to reach equal participation of women and men in research in years

Unit of analysis Countries
Coverage R&D statistics are currently available for EU Member States and Candidate Countries, EFTA Countries, the Russian Federation, China, Japan, the United States and South Korea. Regional R&D statistics are available for EU Member States, Candidate and EFTA countries. Besides national and regional statistics Eurostat calculates and disseminates aggregates at the EU-and Euro-area-levels (EU-28, EU-15 and EA-18). -but data availability differs over the years.

Attributes
Annual growth rate of female participation (in head counts) between 2003 and 2011 (geometric mean) Years to achieve 50% women in research Years to achieve 50% women in research (Y) are given the difference between 50% and the current share of female researchers of I in year a (2011) in percent (R fIb ) in relation to the average annual rate of growth g ab .

Brief description
The Dissimilarity Index provides a theoretical measurement of the percentage of women and men who would have to move to another field of science to ensure a gender balanced distribution across fields. It measures the distance from balanced gender distribution across fields for horizontal segregation in research.

Analytical level (logic model)
Outcome

Source of data She Figures 2012
Date 2011

Time series
All 3 years (at least up to now)

Measurement level
Metric -share of men and women for the distance of balanced gender distribution across fields

Unit of analysis Countries
Coverage 2011: 29 countries; EU 27

Attributes
Higher education sector and government sector

Brief description
The Glass Ceiling Index measures the relative chance for women, as compared with men, of reaching a top position for vertical segregation. It compares the proportion of women in grade A positions to the proportion of women in academia (grades A, B and C).

Analytical level (logic model)
Outcome

Source of data She Figures 2012
Date 2010

Time series
All three years (at least up to now)

Measurement level
Metric -share of women in grade A in relation to share of women in academia

Unit of analysis Countries
Coverage 2010: 29 countries; EU 27 Attributes A Glass Ceiling Index of 1 indicates equality between women and men being promoted, a score below 1 means an over-representation of women in grade A level and a score above 1 an under-representation of women in grade A.

Time series
All three years (most countries for graduates biennial)

Measurement level
Metric -share of women in different grades and ISCED levels

Grade C Grade B Grade A
Information Item G7

Brief description
The Gender Wage Gap illustrates the observed unadjusted difference in average gross annual earnings of male and female paid employees as a percentage of the average gross annual earnings of male paid employees. Persons with tertiary education corresponding to the ISCED codes 5 and 6 who are employed in occupations in the major groups 2 ("Professionals") and 3 ("Technicians and Associate Professionals") of the ISCO classification are used as a proxy for defining researchers in the non-academic sector.
The Gender Wage Gap can be interpreted as a synthetic indicator of multiple inequalities between men and women. It is determined by differences in educational attainments, labour market experience and tenure, sectoral affiliation and occupations, etc., as well as wage discrimination etc.

Analytical level (logic model)
Outcome-related Source: MORE2 Report on case study of researchers' remuneration Information obtained from responses to ERA RPOs survey question 35: Please specify the gender of the person who was head of your organisation at the end of the calendar year in 2013 (Head of organisation: highest decision-making official in the organisation (e.g. rector or equivalent in the academy, president or equivalent in non-academic research organisations)

Analytical level (aggregation)
National on the basis of information about organisations  Information Item G9

Name of indicator
Share of gender-balanced recruitment committees at RPOs

Brief description
This indicator depicts the share of recruitment committees for internationally recognised researchers (e.g. team leaders, management positions, full professors, etc.) which are gender balanced (i.e. reach the threshold of 40% of the underrepresented gender). It can be interpreted as an indicator for women in decisionmaking process.
The data is obtained from responses to ERA RPOs survey question 39: How many recruitment committees for leading researcher positions did your organisation set up in 2013 for the recruitment of researchers? and question 40: Amongst them, how many recruitment committees for leading researcher positions reached the threshold of 40% of the under-represented sex?
Analytical level (logic model)

Analytical level (aggregation)
National on the basis of information about organisations

Source of data ERA facts and figures 2014 on the basis of data from ERA RPOs Survey
Date 2013

Time series
Not yet

Measurement level
Metric -share of committees

Coverage
Research performing organisations in 28 EU countries, covering about 31.6% of staff (headcount) in research organisations in the EU

Name of indicator
Share of gender-balanced research evaluation panels in RFOs

Brief description
The indicator measures the share of evaluation panels which reach the threshold of 40% of the under-represented gender in RFOs. It relates to panels which are responsible for the evaluation of research projects and programmes as well as performance at the institutional or individual level. The outcome of the evaluation may be linked to the allocation of research funding and/or other resources.
The data is obtained from responses to ERA RFOs survey question 27: How many research evaluation panels did your organisation set up in 2013? and 28: Amongst those, how many panels reached the threshold of 40% of the under-represented sex?
Analytical level (logic model)

Analytical level (aggregation)
National

Brief description
The existence of a gender equality plan indicates institutionalised activities for gender equality. A gender equality plan is a consistent set of provisions and actions aimed at ensuring gender equality.
The information is obtained from responses to ERA RFOs survey question 36: In 2013, has your organisation implemented a gender equality plan or equivalent?

Analytical level (aggregation)
National on the basis of information about organisations

Name of indicator
Share of RPOs with female recruitment and promotion policies

Brief description
The indicator depicts the share of research organisations that have implemented recruitment and promotion policies for female researchers. This is an indicator of special actions to increase the participation of women in research.
The information is obtained from responses to ERA RPOs survey question 37: As part of the gender equality plan or equivalent, which of the following measures or actions have been implemented by your organisation in 2013?
Recruitment and promotion measures / Targets to ensure gender balance in recruitment committees / Flexible career trajectory (e.g. provisions for interruptions of career, returning schemes after career breaks, gender aware conditions, provisions on dual careers) / Work-life balance measures (e.g. parental leave, flexible working arrangements) / Support for leadership development (e.g. mentoring or networking opportunities for female researchers) / Other

Analytical level (logic model)
Input-and outcome-related

Time series
Not yet

Measurement level
Metric -share of organisations with female recruitment and promotion policies

Unit of analysis Countries
Coverage 28 EU Member States, the respondents in the ERA RFOs survey 2014 account for about 34% of total GBAORD in the EU.

Share of RFOs promoting gender content in research
Brief description This indicator illustrates the integration of gender as part of the research design and process. It entails sex and gender analysis being integrated into basic and applied research.
The information is obtained from responses to ERA RFOs survey question 26. When allocating research and development funding in 2013, did your organisation include the gender dimension in research content? (Yes, in half or more of the projects/ programmes / Yes, in less than half of the projects/ programmes / No / Not known / Not applicable)

Analytical level (aggregation)
National on the basis of information about organisations

Source of data ERA facts and figures 2014 on the basis of data from ERA RFOs Survey
Date 2013

Time series
Not yet

Measurement level
Metric -share of organisations

Unit of analysis Countries
Coverage 28 EU Member States, the respondents in the ERA RFOs survey 2014 account for about 34% of total GBAORD in the EU.

Attributes
Existence of gender content in research Yes / No / Not known / Not applicable

Brief description
This indicator summarizes activities to integrate the gender dimension in research content that can address research design and process gender analysis.
The information is obtained from responses to ERA RPOs survey question 38: Does your organisation include a gender dimension in research and innovation content of programmes, projects and studies? (Yes / No / Not known / Not applicable)

Analytical level (aggregation)
National

Name of indicator
Share of research projects with specific gender equality actions

Brief description
This indicator asks for the existence of specific gender equality actions and whether these actions are perceived as effective or non-effective.
Three types of gender action types are differentiated: design and implementation of an equal opportunity policy; set targets to achieve a gender balance in the workforce; actions to improve work-life balance

Analytical level (aggregation)
Project level of cooperation projects within completed FP7 projects (by June 2013) that reported specific gender equality actions and gender action types

Date
Published 2013 for the time period 2007-2012

Measurement level
Metric -share and number of projects

FP7 Projects
Attributes Number and share of projects according to priority areas with specific gender equality actions; assessment of the gender action types as effective / non effective

Analytical level (aggregation)
Project level of cooperation projects within completed FP7 projects (by June 2013) that reported gender aspects and with specific gender equality actions and gender action types.

Date
Published 2013 for the time period 2007-2012

Measurement level
Metric -share and number of projects

Coverage
Final Reports of FP7 projects mentioning gender aspects (N=737)

Attributes
Number and share of projects according to priority areas which report gender aspects; number of projects where gender dimension was associated with the research content, per priority area and total

Measurement level
Metric -share of women and men in grants

Coverage
Final Reports of FP7 projects mentioning gender aspects (N=737)

Attributes
The roles of the contact persons differentiate between coordinators and participants on the one hand and seven other individual contact roles on the other (contact person, contact person for legal aspects, contact person for scientific aspects, Marie Curie individual fellows, first administrative officer, principal investigator, secondary administrative officer)   (2004)

Background and objectives
The purpose of this template is to provide each member of the review team with a common framework and reference point to conduct the literature review and, one the reviews are conducted, to facilitate a systematic and structured analysis of the literature.
According to the TOR, the main objective of this first task in the MoRRI project is to  Important definitions or other central statements may be copied into the template; please always make reference to the page number of the review document  Given the diversity of literature covered in this review, it is difficult to provide guidance on how extensive each review should be. For a "normal" journal article we expect the filled-in template to count roughly about 8-10 pages.
If you have any questions, please get in touch: Ralf Lindner, ph.: +49 (0)  The purpose of the study Meta-analysis of gender and science research was to collect and analyse research on horizontal and vertical gender segregation in research careers, as well as the underlying causes and effects of these two processes.
The objectives of the study were to: • Provide an exhaustive overview and analysis of research on gender and science carried out at the European, national, and regional levels.
• Make the study results accessible to researchers and policy-makers via an informed bibliography (online database) and a set of reports.
• Steer policy-making on gender and science and define future research priorities within the Framework Programme, in particular through good practice examples and gap analysis in the various research topics. (P. 15) It shows that women's advancement in science is too slow. It unravels and exposes the subtle mechanisms that maintain gender inequalities in research institutions, and demonstrates that the traditional view of science as genderneutral is flawed. On the other hand, and this should come as no surprise, there is also enough evidence that science benefits from the greater involvement of women. (P. National experts in the 33 countries covered by the study were in charge of selecting the most important contributions to the national literature from 1980 onwards and preparing an informed bibliography, which included the bibliographical reference, English title and abstract, as well as additional codified information addressing thematic and methodological issues. National experts codified the publications according to the following conceptual dimensions and topics: According to the author(s), which type of evidence/data is missing to better support the claim? (e.g. data gaps, limitations with regard to analytical levels, lack of indicator specifications etc.)

Research gaps:
Overall, the meta-analysis of the literature provides a clear overview of the most under-researched themes as regards gender and science: • Non-normative scientific careers is a largely neglected topic. In general, studies concentrate on academia and focus on scientists that pursue the most standard path. Little is known about those scientists who leave the academic pipeline or fail to adjust to the rigidity of academic 'tempos'. Industrial research and other science and technology-related professions remain under-researched.
• There is a lack of theoretical and empirical research on the criteria and procedures for assessing scientific excellence. Particularly, studies about research funding are noticeably absent, specifically analysis of the recruitment practices for gate-keeping positions, as well as of the practices of the different bodies and scientific committees that award research grants and funds and assess scientific excellence. Overall, the lack of transparency in awarding procedures hinders empirical research.
• Research on pay in scientific professions is scarce. It is a rather new topic of study, for three reasons: First, there is a lack of available official data on income and gender income differences. Second, in an important number of research institutions wages are entirely determined by rank and seniority. Third, in some countries and in some cultures, discussions of earnings are taboo.
• Research addressing the evaluation of gender equality policies in science and research is scarce. There is a relative abundance of position statements, conceptual clarifications and recommendations dealing with gender issues in science across most countries.
There is also a relatively large body of research documenting horizontal and vertical segregation in science. However, there are comparatively fewer systematic evaluations of policy measures. (p.20) Comments on 11.
Main findings: 4. Overview of the most important trends as gender segregation in science and related research: current research focuses on four sets of factors in order to explain gender segregation: gender stereotypes, choice of study field, gender division of labour and time constraints, and covert barriers and biases in organizational practices.

Gender segregation in scientific careers:
A large strand of the literature refers to gender differences in scientific careers, with a focus on three critical moments: choice of studies, which remains largely gendered; the 'rush hour', i.e. the early stage of the scientific career, in which family and career demands most often collide, a fact that disproportionately disadvantages women; and career advancement, which shows persistent gender inequality.
6. Gender, institutions and knowledge: with a focus on three different themes: the gender dimension of current institutional changes, gender analysis in research content and policies towards gender equality in science.

Key dimensions of RRI
(For literature dealing with one or more of the 5 key dimensions.) 12.1 How is the key dimension defined?
(terminology applied, central features/characteristics) Gender segregation in employment refers to the tendency of women and men to work in different occupations and sectors. The literature usually distinguishes between different types of segregation: Horizontal segregation is understood as the under-(over-) representation of a certain group of workers in occupations or sectors not ordered by any criterion, whilst vertical segregation refers to the under-(over) representation of a group of workers in occupations or sectors at the top of a ranking based on 'desirable' attributes -income, prestige, job stability, etc. In the literature, vertical segregation is sometimes referred to as the 'glass ceiling', which points to the existence of visible or invisible obstacles that lead to the scarcity of women in power and decision-making positions. This is completed by the concept of the 'sticky floor', which describes the forces that tend to maintain women at the lowest levels in the organisation. (p. 26) Leaky pipeline: Berryman (1983) introduced the metaphor of the 'leaky pipeline'. The process of becoming a scientist can be conceptualized as a 'pipeline'. The science pipeline refers to the normative sequence of educational and employment stages that typically comprise a scientific career. From this point of view, the decreasing proportion of women moving up the educational/professional hierarchy is attributable to women's higher rates of attrition from the science pipeline: at each moment of transition from one educational/ professional stage to another, the pipe line loses more women than men. (p.26.) Scientific excellence: The definition of scientific excellence is elusive. The scientific community acts as if excellence were an obvious quality, and seldom feels the need to define it clearly. According to the documents written by professionals and agencies whose mission is to foster scientific excellence, it can be defined as follows: Scientific excellence is the ability of a scientist or an institution to impact on a field of study producing a major change, leading other scientists towards asking new questions and producing new, important and useful contributions to knowledge, using new methodologies. The quality of excellence must be proven by a number of means, (such as publications, citations, funding, and students) and recognized by the peers by the bestowing of various honours, prizes and other awards.
The scientific community seems to act as if the meaning of scientific excellence were obvious and agreed on by all participants of the scientific enterprise. It behaves as if scientific excellence were an uncontested terrain and as if the procedures and criteria that lead to the selection of the top layer of scientists who are considered excellent were given, known, and unproblematic. However, contributions in the literature (Addis & Brouns, 2004) underline the need to engage in a critical reflection on the concept of excellence as well as on the processes and procedures that lead to the creation and recognition of excellence. Excellence is the final result of procedures that place scientists and scientific institutions in different positions within the network and the hierarchy of their fields. The fact that women scientists do not achieve excellence at the same rate as would be predicted by their outputs in the earlier stages of their scientific career is the product of a number of social processes within and outside the scientific community. (p. 27.) Subtle gender discrimination: Gender discrimination in the scientific system is prohibited, but still exists, though it adopts more subtle forms than in the past. It may impact on the selection, hiring and promotion procedures, on the distribution of resources or on the assessment of scientific excellence. (p.

27.)
Cumulative advantages and disadvantages: Merton (1968Merton ( , 1988 coined the term 'Matthew effect' to describe the pattern of cumulative advantages in science ('For to all those who have, more will be given, and they will have abundance; but from those who have nothing, even what they have will be taken away'. Gospel of Matthew 25:29). The Matthew effect refers to the social processes through which initial advantages in terms of capacity, structural location and available resources make for successive increments in advantage such that the opportunities for undertaking scientific research and receiving symbolic and material rewards for its results tend to accumulate for some scientists and scientific organisations.
(p. 28.) Gate-keeping: Gate-keepers are established scientists or peers that control the definition of merit and the means of exercising academic power (Merton, 1973

General comments and remarks
With the overall purpose of promoting gender equality in science by facilitating non-linear careers and degendering, the main priority of research should be to build more consistent links between analysis and policy making.
Recommendations can be grouped into four main issues: 7. Better statistics: information on qualitative aspects of their employment is very limited.


The European Labour Force Survey is a valuable source of data for the analysis of scientific and technological employment. It offers rich information on personal and family variables. However, it does not make it possible to distinguish clearly between professional and research activities.


European "She Figures" on a tri-annual basis since 2003 constitutes a unique attempt and opportunity to build a comparable European database in order to monitor the relative position of women in science. Collecting more systematic sex-disaggregated data on pay and research funding should be a priority. In particular, research funding requires proper monitoring whilst the lack of transparency in the allocation of research grants and awards is a major obstacle.


Major hindrances for research are the lack of sexdisaggregated data on personal and career developments (including demographic variables such as the number of children, marital status, etc.) and the lack of longitudinal data.


Research suffers from a lack of panel data, which hinders the development of longitudinal research, which is the best way of analyzing patterns of cumulative advantage and disadvantage that shape gender differences in scientific careers.
8. Broader scope of research: Overall, research on gender and science should be less descriptive and more theoretically embedded within the strand of literature that analyses divergent patterns of feminization and change in highly qualified professions. More research is needed to fully understand the complex mix of structural barriers, discrimination and cumulative disadvantages that account for women's underrepresentation in the highest scientific positions. Research should also address the development of science-related professions in nonacademic settings and its gender dimension, including technicians working as research staff and technology transfer professions.
9. Mainstreaming sex and gender analysis  Developing internationally agreed upon methods of sex and gender analysis.


Training current researchers and evaluators in gender methodology.


Holding senior management accountable for developing evaluation standards that take into account the proper implementation of gender analysis in research.


Training the next generation in methods of sex and gender analysis.
10. Focus on institutional change and evaluation of gender equality policies  The need for common quality standards for evaluation: a common evaluation framework might be useful for addressing the related problem of detecting structural change. This also points to the need to make the normative component of many evaluation studies explicit.


The need for theory and interdisciplinarity: Most studies are descriptive and lack explicit theoretical references. This reinforces the isolated nature and lack of comparison between case studies across Europe.


There is a need for research on long-term effects. (pp. 20-24) Concluding remarks: 11. A cultural change in terms of challenging traditional gender roles, specifically in terms of more genderbalanced decision making in research, will be required.
The key challenge is not to change women but, on the contrary, to change the culture of science and research. This change would concern not only the definition and assessment of excellence but also issues relating to career and family balance.
12. At present, the main challenge is not to define new policies but to reinforce their effects through an in-depth evaluation of measures and transferability of good practices. It implies developing sound theoretical frameworks, appropriate methodological tools and shared evaluation standards.
13. In the end, the new European perspective on gender and science comprises the idea that gender policy is not only made by regulation and legal changes but mostly by leadership and a commitment to changing structures and cultures. The connection between gender diversity and corporate financial performance was examined in this study.
The key findings are 14. The group of companies with the highest representation of women on their top management teams experienced better financial performance than the group of companies with the lowest women's representation. This finding holds for both financial measures analyzed: Return on equity (ROE), which is 35.1 % higher, and Total Return to Shareholders (TRS), which is 34.0 % higher.
15. Financial performance was also analyzed by industry, and in each of the five industries analyzed, the group of companies with the highest women's representation on their top management teams experienced a higher ROE than the group of companies with the lowest women's representation.
16. In four out of the five industries analyzed, the group of companies with the highest women's representation on their top management teams experienced a higher TRS than the group of companies with the lowest women's representation This publication compares and presents the progress of Member States in the area of institutional mechanisms and gender mainstreaming since 2006 when the first report on institutional mechanisms was developed by the Finish Presidency of the Council of the EU.
The main findings show that by 2012 all Member States had established governmental bodies for gender equality and bodies for the promotion of equal treatment on various grounds. Notwithstanding positive trends in institutional settings over the last decade, the bodies responsible for gender equality are often marginalised in national governmental structures; split into different policy areas; hampered by complex and expanding mandates; lack adequate staff, training, data and sufficient resources; and experience insufficient support from political leadership.
The report emphasises several important trends. Firstly, although recognised as a fundamental value of the European Union, the status and profile of gender equality currently shows signs of decreased importance. Independent bodies for the protection against discrimination on the ground of sex are increasingly replaced by bodies for the protection against discrimination on various grounds. Whereas the importance of acknowledging the heterogeneity of women and men in terms of age, class, disability, ethnicity/race and sexual orientation is crucial to the recognition of diverse experiences among women and men, the consequences of downplaying gender as a structural dimension and underlying element of all inequalities should not be overlooked. The political, social and administrative remit of gender equality has started shifting towards legal and procedural mechanisms addressing discrimination at the individual level. Gender equality is more seldom addressed and promoted through policies and institutions that tackle gender gaps and the disadvantages of certain groups of women and is more often viewed as a human right requiring legal measures to protect individual citizens against discrimination. This approach marginalises gender equality as a political goal and undermines gender equality as an important policy area in itself. (P.

Institutional mechanisms for gender equality:
 The BPfA defines these as the national machinery for the advancement of women regarded as the central policy coordinating unit inside government. Its main task is to support government-wide mainstreaming of a gender equality perspective in all policy areas.
The following necessary conditions for an effective functioning of institutional mechanisms for gender equality:  location at the highest possible level in the government, falling under the responsibility of a cabinet minister;  institutional mechanisms or processes that facilitate, as appropriate, decentralised planning, implementation and monitoring with a view to involving non-governmental organisations and community organisations from the grassroots upwards;  sufficient resources in terms of budget and professional capacity;  the opportunity to influence the development of all government policies (P. 11).
19. Gender mainstreaming  A Council of Europe study described gender mainstreaming as 'the (re)organisation, improvement, development and evaluation of policy processes so that a gender equality perspective is incorporated in all policies at all levels and at all stages by the actors normally involved in policymaking'.  What currently is known as the dual approach to gender equality, 'meaning the integration of the gender dimension into all policy areas and specific measures' (European Commission, 2010b). The Council of Europe developed the conceptual framework for gender mainstreaming and identified the following tools and methods for gender mainstreaming:  analytical techniques and tools -statistics; surveys and forecasts; cost-benefit analysis; research; checklists, guidelines and terms of reference; gender impact assessment methods;  educational techniques and tools -awareness raising and training courses; follow-up action; 'mobile or flying expertise'; manuals and handbooks; booklets and leaflets; educational materials for use in schools;  consultative and participatory techniques and toolsworking or steering groups and think tanks; directories, databases and organisational charts; participation of both sexes in decision-making; conferences and seminars; hearings.  20. statistics disaggregated by sex To produce sex-disaggregated data and information, where appropriate, for planning and evaluation purposes, for gender impact assessment and, in general, for an effective gender mainstreaming. (P. This report is the result of a joint initiative between the three following entities: the Gender Studies and the Governance, Science and Technology Programmes of the Robert Schuman Centre for Advanced Studies, EUI, the Women and Science Unit, Research DG, European Commission, and the Women and Science Network, Joint Research Centre DG, European Commission. It seeks to stimulate debate on the construction of notions of 'excellence' in the shaping of scientific careers. In particular, it considers in what ways gendered assumptions underpin constructions of excellence, and what these imply for both women and men. (The aim of the workshop was thus to explore the mechanisms and present modes of evaluation of scientific excellence, and the operation of gender within these.)

Components of gender mainstreaming
The report comprises a synthesis of the papers at the workshop and the discussions that took place around them. It offers the reader diverse resources for thinking about the problems of defining and measuring excellence and will lead to some new research initiatives and improved practices which will have benefits for all.
Whilst the setting considered here is predominantly academia, the discussion and research findings are also relevant for other institutional settings -public and privately funded research institutes, and industry. (pp. 7-8) 22. In the first session, mechanisms of evaluation were discussed in relation to careers.
23. The interaction of gender in the definition of excellence and characteristics of evaluation systems was the scope of the second session which sought to address more specific practices of evaluation, for instance through peer review, in gate-keeping mechanisms, and in the decision-making processes of promotion committees.
24. The final session explored strategies to combat gender bias. The workshop participants were scholars from a range of disciplinesincluding sociology, gender studies, history, economics -either specialists in women and science and in the measurement and evaluation of scientific achievement, or in a related field that could offer insight into constructions of excellence. The exchanges between them made for an extremely lively interdisciplinary event. Very different points of view and paradigms were heard, and a wide range of data sources and modes of analysis and interpretations were brought into productive dialogue with one another. 1. Sexism and Nepotism in the peer review of research grant applications: in 1997,Wennerås and Wold published their groundbreaking Nature article on sexism and nepotism in the peer review of research grant applications to Sweden's Medical Research Council (MRC). The article showed that the peer review system is not as 'neutral' as it claims to be. Male applicants and researchers with an affiliation with one of the evaluators were more successful in their applications to the MRC for postdoctoral research grants. The article concluded that whilst the quality of the proposal was an important factor in assessing the scientific competence of research grant applicants, so was the gender of the applicant, as well as his or her affiliation to one of the members of the evaluation committee.
This evidence of gender bias was particularly disturbing because it contrasts with one of the scientific community's core beliefs about its own internal governance. Decision-making should be based on meritocracy, hierarchy on individual performance in furthering scientific inquiry. This belief is rooted in the heart of the scientific ethos, connected with the struggle of science to liberate itself from theology and other societal powers (Merton, 1942).
A science that is oriented by non-scientific judgments on the performance of some scientists lacks the fundamental quality of objectivity. In this context, the supposition of attributing 'excellence' mainly and mostly to male scientists becomes problematic for all scientists. 2. Evaluation system is hindering women in establishing scientific careers: According to Cozzens' input into the discussion, the longlasting debate on 'objectivity' and 'fairness' of assessment systems ended more or less in the statement that although the system is not perfect, it is the best available (Cole and Cole, 1985). However, with the growing interest in gender issues in the field of research policy, the picture seems to be changing: the system is not only imperfect, it may even be hindering women in establishing scientific careers. Merit and talent are not sufficient conditions to become a successful scientist. Resources, time, social networks, encouragement -unevenly distributed between the sexes -are necessary prerequisites.
3. Gender bias: An important issue is that of the hegemonic position of the 'hard sciences' vis-à-vis the humanities and social sciences, in combination with the relative absence of women in the hard sciences.
4. Problems related to measuring scientific excellence: Simplism is surely one of them. The dangers that derive from using an oversimplistic model of excellence are illustrated by the following set of graphs. (pp. 13-16) If we consider all that, the model of the first box deconstructs in a rather more complex scheme. Excellence as we see it today is just one of many possible consensuses about what excellence is. This consensus is shaped by gender relations in the scientific community and in society at large. But the standards used may be different, and the distributions may be received differently.
Excellence can change. Thus excellence becomes a contested terrain. The effort to measure excellence is also a battleground. Existing measures, like bibliometrics, are not gender-biased, but this is not the same as saying that there is not structural gender bias in the larger environment (Feller).The effort to establish criteria and build indicators that take into account the difference in men's and women's lives and abilities is an effort to redefine excellence so that excellent people of both genders may contribute to science. (pp.11-12.) 11.4 According to the author(s), which type of evidence/data is missing to better support the claim? (e.g. data gaps, limitations with regard to analytical levels, lack of indicator specifications etc.) 25. A lack of reliable data comparing men and women working under similar conditions (age, experience, etc.) makes it difficult to draw conclusions on gender differences in publication rates across disciplines and age. (p. 17) 26. More research is needed for a better understanding of where the bias comes from. (p. 29) Comments on 11. Conclusion: The fact that the workshop did not produce consensus on numerous points -including on appropriate methodologies and interpretations of research -is hardly surprising, and this multiplicity of perspectives remains a feature of this report and one which we consider to be amongst its positive contributions.


Based on evidence presented from their own research, participants agreed that scientific excellence is not "a universal fact" but rather a social construction and, as such, it is open to many kinds of biases.
 Several aspects of possible gender bias in the production and evaluation of scientific excellence were discussed in the workshop. Gender bias can occur (1) in the characterisation of scientific excellence, (2) in the criteria used to assess it, (3) in the choice of the explicit and implicit indicators for scientific excellence, (4) in the way the criteria are applied to men and women, (5) in the failure to integrate women in scientific networks, and (6) in the procedures through which criteria are applied to people.
Gender dynamics as regards scientific excellence are multifaceted, are often invisible, and apparently related to gender differences in social capital and in the attribution of competence, and to a scientific culture in which the 'similar-to-me' process unwittingly seems to favour men scientists details: 1. DEFINITION AND MEASUREMENT OF EXCELLENCE: THE MAINSTREAM: Scientific excellence is essentially difficult to grasp. How to achieve the disinterestedness and the ability to be objective is the cornerstones of the scientific ethos.
However, different problems could be observed in the mainstream measurements a. Bibliometrics i. Tension between reliability (quantity) and validity (quality). Indicators that are measured easily and unequivocally -and provide a reliable way of counting -are not necessarily the most valid. In other words, bibliometrics are not necessarily the best indicator of scientific quality, as argued by Feller in this volume. The quantitative is a reduction of the qualitative, which is not easily measured in an objective manner. ii. the connection between short-term publication and long-term scientific impact is rather weak. Early measures may not be an accurate predictor of the longterm impact of a scientific discovery: Wittgenstein would not have survived such a system (Dummet 1991 cited in Feller in this volume). iii. The validity of the Science Citation Index regarding scientific excellence is limited as it rarely includes sources in languages other than English, and covers only a minority of the scientific journals in humanities and the social sciences (see chapter below). The system of judgment employed in bibliometrics privileges well-established fields with long-standing publication traditions and clear boundaries. iv. The way individual scientists react by producing more and more publications: The way scientific excellence is measured creates a specific atmosphere in which competition leads to high numbers of publications but not necessarily to good science. "Publication numbers themselves can be an outcome of a certain form of masculinity", as Hearn stated during the workshop. v. the result of the same computer technology that made bibliometrics possible: scientific production has increased massively. There has been a veritable inflation of literature, while the information processing capabilities of humans has stayed the same. vi. Issue of gender and publications-productivity puzzle -: On average, women tend to publish fewer articles than men. Recent publications clearly show that productivity appears to be related to academic rank. The lower productivity of women can be explained by the fact that they are working at lower professional ranks than men. Within the same category, it seems that there is no significant difference by gender (Bordons et al., 2003). In addition, there are important differences between the scientific fields in terms of women's participation and of publication rates and citations. Discipline-specific publication traditions can explain the existing gender differences in productivity. An alternative explanation for gender differences in publication rates emphasises family responsibilities. During the workshop, Palomba presented research which showed that there is a family effect on productivity: the publication peak for men is earlier in their careers than for women. b. Peer review: There was general consensus among the participants in the workshop that excellence is not an 'universal fact' or a 'natural given', or a 'supra-disciplinary' fact. It is a social construction and, as such, it is open to many kinds of bias.
i. a composite of many skills -carefulness, originality, clarity, complexity, etc. -and is achieved through a process of training, networking, accumulation, and resources. The judgment of excellence depends on the importance attributed to each of these characteristics. ii. The most obvious difficulty is the evaluation of original, innovative research. Innovation is not always recognized immediately, and may sometimes even be rejected as 'bad science'. Other problems arise because of the idiosyncratic character of the judgments. According to Feller (this volume), there seems to be a lack of attention to or endeavour in the construction of alternative measures. The challenge for future work assessing scientific performance and excellence is to develop metrics that better capture the dynamics of scientific discovery, as well as encompass the array of societal objectives that led to the initial public policy decision to fund the research. iii. In addition, the presupposed disinterestedness of the peers and the objectivity of the system are the subject of debate. A standard called 'objective' can be interpreted differently and there is no golden standard -all those participating in the decision-making process must agree on a standard and/or rules to decide whose standard will prevail in order for a decision to be made. Thus, choosing the standards, proper indicators, and devices capable of measuring the standards is a point of contest between different viewpoints. The 'objectivity' of the final decision is the result of negotiation. iv. 'Similarity' seems to be a major aspect steering the evaluation process. Although existing research is ambiguous, there is some evidence that peer reviewers prefer proposals that are similar to their own work (Guetzkow, Lamont & Mallard [in press] cited, with permission, in Griffin in this volume). Knorr-Cetina (1999) has called practices based on similarity 'epistemic cultures', a primary orientation and research styles characterising research groups and research fields. This implicit cloning mechanism limits the chances of research proposals and publications that do not fit in with the traditions. v. Matthew Effect (Merton, 1968) -achievements: evaluators tend to overestimate the accomplishments of scientists with an established reputation, whereas unknown researchers meet more reserve. Matilda Effect, has also been documented: achievements of female researchers are frequently attributed to their male colleagues or otherwise minimised and underestimated (Rossiter, 1993;Stamhuis, 1995 the recruitment of new managers is closely related to processes of cultural cloning, pointing at an often unintended preference by men for men or, as Kanter stated some decades ago, homosocial reproduction (Kanter, 1977). c. Male bonus: Women scientists seem to encounter trouble in becoming part of loose networks, subtly excluded ven by colleagues who are not explicitly sexist in any way. One possible explanation of this fact is that men competing against each other can expect large honour gains when they win and only small ones if they lose. In competition with a woman, the picture changes: men do not want to compete with women because the gains from winningthe competition are relatively small, and the risk related to losing the competition is high, because this implies large honour losses (Addis, this volume). As a result, men treat women differently from the way they treat men, and women remain 'the others'. Under these circumstances, it is far easier for men to gain scientific credibility from an overwhelmingly male scientific forum than it is for women. CROSSING BORDERS Four dimensions of 'situated decision-making' were discussed during the workshop: disciplinary differences, mono-and interdisciplinarity, different modes of science, and geographical location at the centre and periphery. Sometimes the relationship with gender is quite clear -e.g. the disciplinary differences -but in some cases the gendered character remains indefinite. There is -for instance -some evidence that women scientists have a stronger tendency towards interdisciplinary research and towards research aiming at social issues, but the results are not unequivocal. (pp. 11-27.) 12. Key dimensions of RRI (For literature dealing with one or more of the 5 key dimensions.) 12.1 How is the key dimension defined?

SOCIAL DYNAMICS
(terminology applied, central features/characteristics) Gender bias: Gender bias is the often unintentional and implicit differentiation between men and women situating one gender in a hierarchical position to the other, as a result of stereotypical images of masculinity and femininity steering the assessment and selection process or the gendered structure of the scientific system. Explicit gender bias is prohibited, but still existsdiscriminatory practices considering recourses seemed partly to explain the under-representation of female scholars at the highest positions (MIT 1999 In this report, the different modes of science are discussed. The Agora Model refers to RRI exactly.
The text: One of the central lines of debate during the workshop referred to the changing position of the sciences as a social institution in the emerging knowledge-based society. Different concepts have been developed to cover these transformations. Gibbons et al. (1994) describe them as Mode 1 versus Mode 2 models of knowledge production and transmission; Laredo (this volume) addressed it as a 'third mission' of the universities -a responsibility to ensure more direct links with societal and economic needs. Brouns (this volume) refers to it using the metaphor of Mount Olympus versus the Agora. The classic but still powerful metaphor for science is the Olympus model which situates scientists, in their unselfish and disinterested quest for truth, at the top of the pyramid, far removed from the concerns of everyday life. In the Agora model, science is analysed as a societal practice, tightly bound to other such practices. In the context of the knowledge-based society, the sciences are moving into the Agora, but this is hardly recognised in the evaluation systems. Scientific knowledge refers to creators, transmitters and users (Blagojevic, this volume), but only the first ones are acknowledged in the dominant system of measuring scientific quality.
The Centre de Sociologie de l'Innovation has designed a 'researchcompass card model' to map the complex world in which researchers have to operate (Laredo, this volume). It would be counter-productive for research institutions to consider only academic excellence when, at the same time, policies demand other missions to be developed. What is needed is for indicators to be developed for performance in other research activities. Laredo argues for a transition from scientific excellence to scientific performance, which clearly engages inscientific activities in a broader scope.
Concept "Science-in-context" is discussed and refers to PE and RRI Instead of adapting traditions and behaviour to the standards, it is argued that it is necessary to broaden the spectrum of activities and achievements to be included in the definition of scientific excellence. It is important to include other dimensions of scholarly practice, such as education, participation in committees, administrative tasks, external consultancy, and contribution to public debates. In other words, it is important to emphasise not only production, but also relevance and the different users of scientific knowledge. Measures are meaningful if they are based on the context of their production and in the uses of the knowledge. But the question of how exactly this science-

Evidence
More research is needed for a better understanding of where the bias comes from. Gender is a deep cultural construct that operates at symbolic and institutional levels, and gender bias may assume different forms in different cultures, which may go unnoticed unless systematically explored. There are fields where women have fared better than in others; this needs systematic investigation.
General recommendation: Funding of research in some neglected areas, such as differences between disciplines, epistemic cultures, and national and regional contexts. These differences should be compared and investigated in order to improve our understanding of the gender dimension of science and scientific organisations.

Awareness
An important first step is to make all scientists, male and female, aware of the extent and the consequences of the problem of gender bias in measuring excellence. In particular, those in charge of screening procedures should be trained to understand gender bias and its consequences, so as to minimise it.
General recommendation: Special training programmes on gender awareness, designed by gender specialists. Development of reading material on gender bias in evaluating research.

Field boundaries
Particular attention should be paid to the fact that the definition of the boundaries of a field of inquiry and the specific activity profiles of scientific institutions matter when deciding who is excellent.
General recommendation: Much greater recognition of the positive contribution of interdisciplinarity, new research fields and gender studies to scientific excellence. Discuss also the relevance of productivity measures as a primary way of assessing researchers' performances.

Networks
The issue of networks is strongly linked to the issue of awareness and power. Scientists should be more aware of gendered difficulties in engaging in dialogue and networking with scientists of the other sex. More formal processes and criteria need to be set up to allow more women to pass the 'fuzzy screening' of networks.
General recommendation: gender balance in officially funded networks to be achieved by the imposition of recommended quorums formulated with reference to women's and men's respective presence in the field.

Procedures
Procedures for assessing excellence are not obvious or natural; there needs to be a critical examination of their interaction with gender.
General recommendation: in order to minimise gender bias, it is of particular importance that screening procedures be made more transparent and evaluations more public. (p. 29-31.) 16. Relevant  In 2003 the European Commission published The Wake-Up Call for European Industry. In order to realize Europe's ambitions in achieving a competitive knowledge-based society, the number of researchers must be increased. Business is a crucial partner for mobilising talent and women are obviously the source of untapped potential. Increasing the participation of women is fundamental to achieving the European innovation goals.
Although the proportion of women reaching top positions in government and business has increased, Europe-based companies still have a long way to go in attracting and retaining female talent.
Early in 2005 a group composed of 20 company representatives and 5 experts in economics engineering and social sciences (WiST) worked to make the business case for gender diversity in Science and Technology. This is a set-up either for a rich learning experience or for a "dialogue de sourds" (dialogue between deaf people). In any case it has helped form the judgment on many issues, enlarge the understanding of the issues and better perceive all of its facets, including non-business aspects like socio-economical undercurrents.
The aim of the expert group was to analyse the possibilities for the promotion of women in Science and Technology from a business perspective and to develop an integrated approach to the cultural change involved. And most importantly, the group wanted to give new impulses to these ongoing changes.  This report presents the findings of the dialogues between the members of the Women in Science and Technology (WiST) group. The WIST group (Women in Science and Technology) was set up as a strong collaboration between almost twenty leading multinational companies and five experts from several disciplines (engineering, cultural change, econometrics, economy and policy) to discuss and study these issues (i.e., a better understanding of the present situation, the reasons why change happens so slowly, and what can be done to speed up the processes of change) and to improve our understanding. This set-up allowed researchers to be confronted with companies'experiences and analysis of good practices; companies to be confronted with scientific analysis of recent developments at the micro and macro levels; the results to be communicated to the public and an open European forum involving top managers of technology-driven companies to be organised. The expert group has met five times to discuss presentations from companies and experts. Pierre Bismuth, HR for Schlumberger, was chair of the group -he invited the companies, encouraged the experts and structured the discussions. According to the author(s), which type of evidence/data is missing to better support the claim? (e.g. data gaps, limitations with regard to analytical levels, lack of indicator specifications etc.) For human resource management: Kochan et al. (2002)20, who investigated the business case for gender diversity, make a very clear statement: companies need to adopt a more analytical approach and produce better HR data to be linked to business performance. This will improve the learning capacity of HR practitioners, so that they themselves can answer the question: under what conditions do gender diverse teams outperform or under-perform other teams? Laure Turner, in her paper, expressed a similar need for aggregated data as a business tool, in order to improve the explanatory power of the econometric modelling techniques. Collecting, sharing and using the data will not only improve the general knowledge of what's going on, but also provide an understanding of what is not going on, as regards gender diversity.

Comments on 11.
Conclusion: This progress is slow because of a subtle and strong resistance. In many instances, academia in this domain seems far too satisfied with the status quo. Many HR managers would readily admit that they are not really prepared for the surge of dual careers. Top managers are still hesitant to take risk on talented women by accelerating their career or to openly raise the issue as a strategic move for the company.
The participating companies and experts share the conviction that attracting, developing and employing men and women equally in Science and Technology requires a significant cultural change, which is essential for innovation, growth and competitiveness.
This synthesis reflects the most important insights, presenting pieces of a complicated puzzle. It paints a picture of what has been established, while at the same time mapping the road still ahead of us. Although the issue has been on the agenda for more than a decade, sometimes even causing a kind of fatigue, the problem has not been solved. Across all diversity, the companies share a conviction that something needs to be done in order to speed up change -they take responsibility for future balance in the workforce. (pp. 4 and 8) Findings regarding the 5 issues mentioned in 11.1 1. Regarding mechanisms of the resulting in leaky pipeline:


The positive statement is that S&T talent is not an innate quality of men and women -it's mostly culture (e.g. selfimage: women=emotional, people-oriented vs. S&T=logical, rational machine-oriented) and therefore open to change. The negative statement, however, is that culture hasn't changed much over the years.  On the basis of these empirical investigations, it might conclude that the gender difference is partly reality and partly perception but always contextual. Emphasizing differences isolates women and freezes their identity.  According to Graham the 'women-only' events are very important for many girls and women in order to feel comfortable in a primarily technology-driven situation, mostly dominated by men.  it is important to understand not only why women leave S&T, but also why women find other functions or sectors more attractive: both financially -the gender wage gap is largest in male dominated occupations -and in terms of decision-making.


This means that not always the best will survive the pipeline to senior positions in the business sector. Apart from all the effects this has on individual lives, the social, cultural and financial costs are easily underestimated. The group saw this as a collective failure. Acknowledging this discrepancy between lived experience and organisational discourse on gender equality is a first step in framing the real issue -which is about changing organisational practice and its core value system. (pp. 9-13) 2. Regarding work-life balance issues:


The import issues  Companies have to perform better in supporting dual careers.
 Companies need to take some calculated risks with the best performing women to give them the chance to catch up with the gap.
 Companies need to pay particular attention to dual career men and women in the early 30s as they then experience the maximum pressure of work/life balance.


Companies are -apart from being producers of commodities and services-also social institutions producing time structures (reason: Time sovereignty and flexibility seem to be crucial. When workers feel supported and able to control the amount and conditions of their work, their perception of conflict between work and family diminishes (Gerson and Jacobs), social networks and meaningful relationships. Companies do not sufficiently acknowledge their societal roles.

Regarding organisational change:
 The expert group identified process management as an important factor affecting the slow progress towards gender diversity. Many reports and conferences express commitment, but this does not always imply major efforts and implemented policies. As Michel Domsch states in his paper, there is a lack of management commitment to sustainable and significant change, in which planning, decision-making, implementation and evaluation follow the ordinary process cycle.  Domsch introduced a social audit instrument -the Gender Dax. The Gender Dax is one example that measures not only numbers, but also processes, planning and available tools. Audits and self-assessment are regularly used tools, but organising feedback from gender action plans is rather seldom within these companies. Setting more transparent goals and using instruments and concrete measures would contribute to the quality of process management. Accountability is a driving force behind many business processes, as is managing diversity.


One of the powerful strategies is to connect gender diversity closely to the strategy of the company. Only a few of the participating companies explicitly link participation of women to their core business strategy and the content of the business. One of them is a strong advocate of gender mainstreaming. This means that gender diversity is no longer an isolated target, but an integrated aspect at all levels of the company (an integrated approach).
4. Regarding the complexity of managing diversity:  Firstly, there is the question of whether firms managed by a gender-balanced top team perform better than firms led by men only (what is the specific contribution of female leadership styles?)


The study found that if there is a positive relationship, this is due to board diversity affecting firm performance and not the opposite. From this point of view, we can conclude that there is a business case on gender composition of the top levels of the companies, especially related to welleducated women. More turns out to be better.


Secondly, there is a general question of whether heterogeneous teams outperform homogeneous teams.
The empirical literature does not support the simple notion that diverse groups perform better; on the contrary, if not managed, diversity is likely to have a negative effect, leading to conflict and turnover (see also Martha Maznevski's report in this document). The positive effect is that gender-balanced groups have more constructive interaction than either predominantly male or female groups (p.29).


Thirdly, is there a positive performance effect of engaging more women in traditionally male dominated forms, teams and units, such as S&T?  This last conclusion was supported by the results from Laure Turner's investigation of team performance in four participating companies (Turner, this volume). She found that the individual performance was highest in teams that were gender balanced (those which were 33-66% women) followed by predominantly male teams and finally the predominantly female teams. However, only the increase in women's individual performance proved to be significant. Hard evidence of the positive impact on collective performance was difficult to obtain due to the low number of investigated cases (69 teams), but the data indicate a positive effect of gender diversity.


The challenge of managing diversity: In order to make diversity productive, major investments in quality of management have to be realised by focusing on similarities and differences. Maznevski has introduced the MBI approach: Map, Bridge and Integrate differences.


Diversity management is about valuing difference and trust. The key is inclusiveness. The study showed that diversity tends to have a negative effect on performance if there is a low level of inclusiveness -the extent to which the members of a group feel connected to each other in one team -and a positive effect if there is a high level of inclusiveness. In other words, inclusiveness is the determining factor for the effect of diversity on process indicators -such as learning, participation, and communication -and performance indicators. But this is never easy: Diversity doesn't happen by accident!

General comments and remarks
Recommendations: What is lacking in many respects is a sense of urgency.  To repair the "leaking pipe line" which loses sometimes as many as 2/3 of female students with a good track record and strong interest in math and physics between high school and college.  To provide solutions for dual career couples and single parents and respect their basic need for work-life balance by combining the efforts of the business community and society at large.


To manage this cultural change with the same holistic approach as the one followed for example in the search for quality.


To develop in managers the ability to optimize the efficiency of a diverse group.  Finally to measure this efficiency and objectively observe the business impact of gender diversity as it is being deployed.
Ensuring that company culture is in tune with the evolving needs of our times is vital, and our delay in addressing this issue is not a good sign. This issue may not be an obvious one but it has deep economic and social implications. The companies in this group which are ahead and enjoy the most balanced gender diversity are struggling to sustain their level of diversity because of the limited reservoir of women graduating in technical disciplines from universities. The companies at intermediate level see the risk of a plateau. Finally, the ones that are merely beginning are meeting the strongest internal resistance.
By working together as a group, we are showing young graduates our commitment to making the technical and scientific business world a much less discriminatory work environment, as quickly as possible. In all European countries and beyond, women are having difficulties getting ahead in research careers. Women are heavily underrepresented in research decision-making in Europe, and thus have fewer opportunities to influence the research agenda. Since access to resources is a major key to success, this report focuses on research funding across Europe, mainly but not exclusively from a gender perspective. It is the result of the work of a EU expert group set up by the European Commission to provide recommendations 'on the improvement of transparency and accountability of procedures used in selection committees for grants and fellowship awards and of access to research funding in general'. The report analyses the gender dynamics among applicants, recipients and gatekeepers of research funding, in funding processes, instruments and criteria, and the role of key funding organizations in promoting gender equality in research.
The focus of the expert group included national grant awarding procedures and accessibility of gendered data on success rates, amounts awarded and peers taking part in the decision-making and evaluation processes, distinguishing according to disciplinary fields. It centred on the funding of academic and basic research, on key public funding organisations in each country, and on competitive project funding and individual grants. Private funding organisations and charities, and bulk funding for institutions were not included.
This report should be seen as a systematic effort to map the European research funding landscape from a gender perspective and highlight key issues and needs for future action and research. (P.5) Publicly available data were collected from websites, publications of the funding organisations and other stakeholder organisations, and from relevant research. When data were not publicly available, they were requested from the funding organisations. Other national experts were consulted, in order to complement and assure quality of the data obtained.

Main focus
An overview of the national situations in terms of research landscape and gender settings is annexed to the report.
The full national reports have been posted on the Science in Society web portal so that the work put into this analysis is made available to all interested parties for both policy-making and further study.

Political measures:
A number of innovative national policies which affect research funding were noted, such as gender balance targets (for example, in Slovenia or Switzerland) and legislation on gender quota of up to 40 % of the minority gender in committees (in Finland, Norway and Iceland). In a number of countries, integrated policies increase university funding based on their performance in terms of gender equity (for example, Germany, Netherlands, Ireland). Some have also set up specific national gender quality structures with strong prerogatives, which actively support their policies. infrastructures to monitor and promote gender equality in research, launched ambitious GEAPs, set up specific measures to promote women in research and conducted or are planning in-depth studies and monitoring activities on gender and research funding. Policy improvement can also be boosted by active engagement of the scientific community. An example of a bottom-up action is the Czech Republic National Contact Centre on Women and Science, which has succeeded in having funding mechanisms improved. A number of actions specifically targeted at women, to promote gender equality, are implemented by many funding organizations. They range from actively encouraging women to apply, or setting targets for proportions of women funded, to specific programmes for women, supporting them at the start of their career, aiding them to return to research after a career break or providing additional assistance for mobility. Various measures facilitating work-life balance in research for both women and men have been built into some funding schemes.
Gate-keeps: Detailed gendered data have been provided on gatekeepers in many of the countries under consideration.
Success rate: Success rates by gender and discipline, concerning This group of 16 experts has provided contributions to this report by gathering the necessary national data to the 7th Framework Programme for Research and Technological Development. The expert core group consisted of twelve experts who provided data and analysis of the national contexts. Each expert examined several countries, to ensure that all were covered. In addition, four experts were invited on a shorter basis. The experts came from the European Union or European Economic Area and brought a wide and high-level expertise from various disciplines and countries, as members of national funding committees, administrators of funding organisations, or academics with research experience on the area. The division between the proactive and the relatively inactive countries appears to follow rather well the global gender gap rankings of the World Economic Forum, with most proactive countries having relatively small societal gender gaps, and most relatively inactive ones larger societal gender gaps. (For literature dealing explicitly with responsible (research) and innovation. If the publication deals with one of the 5 key dimensions, please proceed to 11.) 9.1 Which definition of RRI is being used? (author's definition or reference to other source) 9.2 Which aspects of RRI receive special emphasis? (e.g., certain normative goals, procedural approaches, reference to one or more of the 5 key dimensions, …) 11. Claims regarding the effects of RRI and / or the key dimension (benefits, costs, disadvantages, trade-offs) 11.1 What claims are being made?
The balanced representation of women and men in science has been part of a strategic approach to bring forward equal opportunities in the field of scientific research, enhance European competitiveness, and to realise fully the European innovation potential. Clear progress has been made in the last 10 years with the European Commission playing a key role by providing much needed impetus.
Equally crucial is the objective of mainstreaming gender in scientific research. The scientific job market should include more women at all levels of seniority. Female researchers, scientists and professors should be able to participate fully in the production of knowledge and research. (p. 4) 11.2 Which arguments are used to support the claim(s)?

11.3
What evidence is presented to support the claims?
(e.g., data, indicators, research results, case studies, anecdotal evidence) 11.4 According to the This European level synthesis highlights the existence of very author(s), which type of evidence/data is missing to better support the claim? (e.g. data gaps, limitations with regard to analytical levels, lack of indicator specifications etc.) good practices in the field of transparency and accountability of research funding systems which could be applied in other settings. However, data monitoring is not systematic, and publication of research funding results per gender per discipline is far from perfect.  Data are missing from French-speaking Belgium, Croatia, Cyprus, the Czech Republic, Greece, and Hungary.  From Israel only data from 2000 were available, and data were only obtained from one UK Research Council.  From Austria and Luxembourg, data by discipline were not obtained.


No very systematic patterns appear in the data obtained.  Another dimension of success in funding is the amount of funding obtained, for which success rates were obtained for only a few countries. Better monitoring is clearly needed here.  Some very partial data were obtained on post-doctoral fellowships. Although no particular problems were noted, previous research has flagged up strong gender differences at this stage. This question needs clarification. (pp. 5-7) Comments on 11. Findings: 1. Success rates: the expert group has not found a large and systematic gender imbalance in terms of success rates in research funding in the funding systems studied, although a few exceptions exist.
b. No clear relation could be observed between the proportion of women in a field and their chances of success in obtaining funding. For instance, in some funding chemes and organisations women had higher success rates than men in engineering and technology or in natural sciences, the most maledominated fields across Europe, and in others lower. Nor was any large and universal imbalance observed in favour of men.
c. Some cases of imbalance can be observed, with various degrees of statistical significance. In a number of cases, on the contrary, women have significantly higher success rates than men. An example is the Dutch NWO, where, because of low representation of women in research, particular attention is paid to the quality of evaluation, and where promotion of women in research is an important policy goal.
d. Various 'excellence initiatives' aimed at rewarding the very best researchers and including substantial amounts of research funding were also examined. These instruments generally showed particularly strong gender imbalance. This was also the case with the European Research Council Starting Grants.
2. However, there is a clear difference in application behavior: women are less likely to apply for funding than men and they request smaller amounts of money. Again, further research is needed to explore this phenomenon, to understand the dynamics and reasons behind it, and to elaborate counter-strategies. 3. Decision-makers and Gate-keepers are men: In most of them, decision-making and other gatekeeping activities in research funding, including peer review, continue to be dominated by men, in some cases overwhelmingly so. Allmale committees and evaluation panels still exist in many countries, even in those where the proportion of women in research is relatively high. The recruitment procedures, in particular for peer reviewers, whose choice may be crucial, are often not clear. 4. Eligibility rules for applying for funding concern age or academic age, degrees completed, place of residence or citizenship, and present position. Age limits are in many cases increased -by up to three years -if the applicants have children. Rules requiring that applicants have a permanent position and forbidding them to fund themselves within their project are particularly penalizing for women. 5. The existence of an efficient system for monitoring the outcomes of research funding is an essential element of transparency. (pp. 5-7) 12. Key dimensions of RRI (For literature dealing with one or more of the 5 key dimensions.) 12.1 How is the key dimension defined?
(terminology applied, central features/characteristics) Gatekeepers: members of national science and technology councils, funding organisation directors, managers, board members and staff members, members of evaluation committees and panels, and external reviewers.

General comments and remarks
Recommendations: The Gender and Excellence expert group was set up to provide recommendations on the improvement of transparency in the procedures used in selection committees for the award of grants and fellowships and in access to research funding in general.
32. Taking the gender challenge seriously, backing specific actions, supporting structures to monitor gender equality, and encouraging research on this area, all with strong political will. The denial of or lack of interest in gender equality appeared to be one of the main sources of imbalance in a large number of European countries.
33. Increasing applications from women researchers. This implies encouraging and training women to apply and to request more funding. Measures for better work-life balance are essential.
34. Improving gender balance among the gatekeepers of research funding, including committee or panel members and reviewers, and organising gender training, for all involved in the funding process. Allowing women more equal access to the inner mechanisms of research funding could also have major impact on improving their application rates. (It provides women researchers more opportunities to learn how the funding and evaluation system works and to become integrated into important networks, and allows them a valuable overview of current frontline research.) 35. Gender monitoring and publishing of funding statistics on a regular basis, differentiated by discipline and research instrument.
In-depth monitoring exercises, both quantitative and qualitative, should be carried out and should include an analysis of the pool of potential applicants, the study of team composition in proposals and generally of the gender impact of funding actions. The European Commission decided to continue the WiST initiative (2006) for two more years due to the interest expressed by the participating companies. The WiST2 working group was thus established, giving more companies the opportunity to join the group, and at the same time expanding its scope to universities (4). Seven international experts were invited to address one or more of the issues above, by inviting the participation of companies and universities from the WiST2 group for data collection.

Generally improving accountability and transparency
Many of the organisations participating in WiST2, slowly but surely, provided access to employee samples for data collection, as well as other sources including performance information and communication materials.
The 6 experts' reports represent an intriguing, multi-method, multidisciplinary, cross-culturally comparative collection of insights into the working lives of women (and men) in S&T.
The topics of 6 experts' reports are:

Driving Attraction and Commitment with a Work-Life Proposition: Special
Focus on Science and Technology Employees (Corporate Leadership Council, CLC): The Council surveyed more than 50 000 employees from 35 different organizations across 20 industries. The majority of data presented in this study was collected using an existing employee survey instrument, The Corporate Leadership Council's Employment Value Proposition (EVP) Survey, which was first used in 2006.
38. Work-life balance and performance (Laure Turner): This study draws on three sets of data: data on employees' answers to the CLC Survey, data on individual performance, and data on industrial project performance and work-life balance in teams.
39. Flexible working policies, gender and culture change (Suzan Lewis): The study adopted a research strategy based on the initial phases of Collaborative Interactive Action Research (CIAR) (Rapoport et al, 2002). CIAR is a process of mutual enquiry that yields new ways of thinking about issues within organisations. This involves a number of stages: establishing case study organisations; establishing collaboration within the companies and a work unit to participate in the research; data gathering and analysis; reflection on the analysis.  The working group focused on the following issues: 11.2 Which arguments are used to support the claim(s)?

11.3
What evidence is presented to support the claims?
(e.g., data, indicators, research results, case studies, anecdotal evidence) 11.4 According to the author(s), which type of evidence/data is missing to better support the claim? (e.g. data gaps, limitations with regard to analytical levels, lack of indicator specifications etc.) Comments on 11. Conclusion: 1. Contemporary and future employees value work-life balance, and are expected to continue to do so in the future. Especially for dual career couples with young children, flexibility (in terms of timing and location of work) and an appropriate workload are in high demand. 2. Offering work-life balance practices is not enough -the organisational culture (as evidenced in the communication about these practices, but especially in terms of the behavior of supervisors and peers) must be truly supportive of the utilization of these policies. If the message is negative ("you will have to work extremely long hours and put in face-time in order to get promoted"), or mixed ("of course you can work from home, as long as I can expect you to come in at short notice"), many will not utilize what is on offer, and those who do, are likely to fear the consequences. 3. Central to organisational cultures in relation to gender diversity and WLB practices are our (often implicit, mostly incompatible) notions of the "ideal worker" and the "ideal mother". These normative beliefs are heavily influenced by cross-culturally similar gender stereotypes and relate to the "separate spheres" of home (i.e. care, children) and work (i.e. career). While many of us consider such norms extremely resistant to change, the good news is these spheres in reality increasingly overlap and are no longer defined by one gender.
Finally, Research in fact shows that ideology will follow policy (Sjöberg, 2004): in countries that implemented family policies towards the support of a dual-earner family, normative beliefs progressively shift away from traditional roles. (pp. 9 and 10) Findings regarding the 6 exports' reports mentioned in 8.1 1. Report of CLC: Flexible work schedules together with an appropriate workload tend to play a key role in determining employees' attraction and commitment. 2. Report of Turner: HR and line managers should try to prevent "frustrated" high potentials from dropping out, and "unbalanced" high performers from burning out. Furthermore, HR may need to screen the performance review system, as it may (implicitly) penalize those who use work-life practices to achieve balance. 3. Report of Lewis: for implementing change and improving efficiency, such as reducing the length of meetings and moving meetings to an earlier hour, from which everyone, not only parents, will benefit. 4. Report of Herman: Work life balance policies (including parental leave, flexible working and reduced working hours) can have the unintended consequence of reinforcing gender stereotyping within the workplace if it is only mothers/female careers who make use of these and not fathers or male careers. 5. Report of Connolly & Fuchs: possible routes for universities in reducing the leaky pipeline for women in academia in general and S&T in particular: by offering tenure tracks with the possibility to "stop the clock" during maternity or parental leave, by enhancing flexibility and emphasizing alternatives to the "long hours culture", and by making performance appraisal and promotion systems more transparent and standardized. 6. Report of Wächter: Many of these images essentially reflect and thus reproduce asymmetry, exclusion, numerical underrepresentation, and gender stereotypes, by showing women as support staff and men as engineers in the field, and by relating work-family issues only to women. (pp. In order to make sure that this communication between S&T companies, universities, experts, and the EU DG Research will continue beyond WiST2, we will look for innovative ways such as an on-line community or network of practitioners in order to provide a platform for and support communication between parties involved in WiST2. By these means, we can help create sustainability in combining career and care, which is of critical importance to HR and diversity practice in S&T companies.

General comments and remarks
Recommendations: 43. S&T companies need to keep on creating, promoting, and supporting custom-made WLB practices that fit the individual's needs and preferences, that match the strategic HR agenda of the organisation, and that are aligned with the national context in terms of legislation.
44. Employers can project their vision of the "ideal" diverse and inclusive organisation by paying extra attention to the images and language on their website and in corporate brochures.
45. Employers can protect their high performers and high potentials from burning out and/or eventually opting out by re-examining the nature of the performance appraisal process and making sure the utilization of WLB practices is not penalized unnecessarily or disproportionally.
46. Employers can better manage transitions ("off-and onramps") for those who take career breaks, and make sure that career trajectories take such career breaks or reduced hours into account. Boosting innovation in the EU means increasing the number of researchers in Europe by at least one million if the EU is to remain competitive and build on its strengths. To make sure that people starting research careers find it attractive to stay in science is necessary. This is especially true for women: while 45% of doctorates are awarded to female students, only 30% of active researchers and 18% of full professors are women.
A group of high level experts has been brought together in order to investigate the reasons behind existing trends. This is their report. The experts have reviewed a large body of evidence, have identified where the problems lie, and have clearly formulated the conditions needed to remedy a waste of talent which has already lasted too long.
The report argues that gender-aware management of universities and research organisations would have a positive impact on policies and practices in the recruitment, promotion and retention of both women and men, thus ultimately benefiting the very quality of research. There is no trade-off to look for between promoting gender equality and excellence in research. Instead there is a win-win situation for all researchers, their institutions, and for Europe.
The report rightly stresses that progress in integrating gender in research and innovation relies on firm and sustained top-level commitment. Based on recent scientific findings and research practices, this report analyses the progress made so far in legislation, participation and policy, describes the problems remaining for research institutions in Europe and stresses the role that EU policy-makers, science institutions and gatekeepers of excellence must play in order to advance gender equality in research and innovation 11. Claims regarding the effects of RRI and / or the key dimension (benefits, costs, disadvantages, trade-offs) 11.1 What claims are being made?
The key role given to research and innovation in striving towards a smart, sustainable and inclusive growth in Europe means that the EU should make full use of its human capital -thereby involving both men and women. (p. 6) Cost of no action: There are four consequences that are of concern: 48. According to the author(s), which type of evidence/data is missing to better support the claim? (e.g. data gaps, limitations with regard to analytical levels, lack of indicator specifications etc.)

Comments on 11.
Problems in the process of improving "gender equality" were identified by the study: 52. Insufficiency if the focus was merely at program level: at the beginning, the focus was initially on specific programmes to help women pursue scientific careers. However, those programmes have proved to be insufficient to increase the number of women in science, particularly in positions of responsibility, and have not helped to address the structural barriers contributing to the well known leaky pipeline phenomenon. 53. A shift in focus towards addressing the structural transformation of institutions: a systemic, comprehensive and sustainable approach was applied. The US has led the way with the ADVANCE programme, funded by the National Science Foundation. Some initiatives have also been taken in Europe, but the scale of these needs to be increased.
54. In the EU, the progress made so far in legislation, participation and policy, describes the problems remaining for research institutions in Europe and stresses the role that EU policy-makers, science institutions and gatekeepers of excellence must play in order to advance gender equality in research and innovation. i. statistics show that EU Member States still have a gender pay gap, and gender continues to be a structuring factor in the workplace, also in research: Work is organized in gendered ways, which makes it difficult for talented women to reconcile work and family; harassment, concentration of power, and the guru/acolytes model of power relations are also factors affecting women negatively.

Key dimensions of RRI
(For literature dealing with one or more of the 5 key dimensions.) 12.1 How is the key dimension defined?
(terminology applied, central features/characteristics) Equal opportunity indicates the absence of barriers to economic, political and social participation on the grounds of sex. Such barriers are often indirect, difficult to discern and caused by structural phenomena and social representations that have proved particularly resistant to change. Equal opportunities, which is founded on the rationale that a whole range of actions are necessary to redress deep-seated sex and gender-based inequities, should be distinguished from equal treatment, which merely implies avoiding direct discrimination.
In gender-sensitive research, gender is consistently taken into account throughout the research cycle Gender-specific research focuses on gender itself as a subject matter Gender-blind research does not take gender into account, being based on the often incorrect assumption that possible differences between men and women are not relevant for the research at hand Gender bias is the often unintentional and implicit differentiation between men and women by placing one gender in a hierarchical position relative to the other in a certain context, as a result of stereotypical images of masculinity and femininity. It influences both the participation of men and women in research (hence the underrepresentation of women) and the validity of research. An example of gender bias in research is research that focuses on the experience and point of view of either men or women, while presenting the results as universally valid.
Gender audits are evaluations that monitor and evaluate the implementation of gender issues into procedures. Unlike regular audits, they are based on self-assessments of how gender issues are addressed in internal organizational processes, and not on external evaluation.
Gender impact assessments provide help for policymakers in incorporating a gender perspective into policies that take account of the different needs, characteristics and behaviours of the users at whom they are aimed.
Gender proofing is a check carried out on a policy proposal to ensure that any potential gender discriminatory effects arising from that policy have been avoided and that gender equality is promoted. (p.

General comments and remarks
Suggestions of the study: structural change in science institutions as the means to address each of these five sets of problems, so that decision making is more transparent, unconscious bias is removed from institutional practices, human resources management is modernized, excellence is promoted through diversity, and research and innovation are improved by the integration of a gender perspective.
In addition, it signals three essential elements which should be considered as a prerequisite by all organizations undertaking structural change: knowing the institution, by developing statistics and indicators, so that the situation of each institution becomes widely known and acknowledged; getting top level support from persons in positions of responsibility; generating effective management practices, by ensuring gender expertise and by raising awareness.
While a lead is required from the EU and its Member States, a wider range of actors also need to play an active role in modernizing the way in which R&I is conducted in Europe. Some of the most successful innovators are paving the way but others are still lagging behind. Universities and research institutions, funding bodies and some learned societies still operate with the stereotypical gender regime of a full time breadwinning man and a female second earner. This report also proposes key recommendations to help different types of actors to improve their performance.
In Annex is gender equality strategy attached: The goal of the Expert Group was twofold: to provide scientists and engineers with practical methods for sex and gender analysis, and to develop case studies as concrete illustrations of how sex and gender analysis leads to new ideas and excellence in research. To match the global reach of science and technology, the case studies and methods of sex and gender analysis were developed through European and international collaborations. These fields reflect priorities set in the new European Framework Programme Horizon 2020 that will cover the period 2014-2020.

What is Gendered Innovations?
Thirty years of research have revealed that sex and gender bias is socially harmful and expensive. Gender bias also leads to missed market opportunities. In engineering, for example, assuming a male default can produce errors in machine translation. In basic research, failing to use appropriate samples of male and female cells, tissues, and animals yields faulty results. In medicine, not recognizing osteoporosis as a male disease delays diagnosis and treatment in men. In city planning, not collecting data on caregiving work leads to inefficient transportation systems.
It is crucially important to identify gender bias and understand how it operates in science and technology. But analysis cannot stop there. Gendered Innovations offer sophisticated methods of sex and gender analysis to scientists and engineers. Integrating these methods into basic and applied research produces excellence in science, health & medicine, and engineering research, policy, and practice.  [males] in terms of reproductive organs and functions based on chromosomal complement and physiology.sex is globally understood as the classification of living things as male and female, and intersexed.
Gender-a socio-cultural process-refers to cultural and social attitudes that together shape and sanction "feminine" and "masculine" behaviours, products, technologies, environments, and knowledge.
Gender analysis is presented in twelve methods in this project. Researchers may analyze sex or analyze gender. Gender analysis is the umbrella term for the entire process. Researchers will consider each of the twelve methods and choose the interacting methods that apply to their particular project.
Innovation in this project refers to new ideas, new knowledge, and new technologies and design.
Gendered Innovations are defined as processes that integrate sex and gender analysis into all phases of basic and applied research to assure excellence and quality in outcomes. When analyzing human standards and reference models, researchers/engineers will want to consider the following questions: j. Does the existing model differentiate between women and men?
k. Are existing standards up-to-date, or based on old data that might be invalidated by trends? For example, the incidence of obesity has increased significantly in highly developed countries over time (WHO, 2011). Japan, Brazil, the U.K., and the US have all seen rates of obesity roughly triple in less than 30 years (Jeffrey et al., 2008).
l. If a model does not consider sex, is it based on research in both sexes, or is it in fact a male reference model (or, in some cases, a female reference model) that is being improperly used as a generic "human" model?
m. If standards do consider sex, how important is sex to the reference model? Have researchers adequately investigated non-biological influences due to gender and other social or biological factors?
n. Beyond considering sex differences, does the model address sex-specific factors among women (such as pregnancy) and men (such as susceptibility to prostate cancer)?
o. Does the existing model take into account differences between women's and men's attitudes, needs, and interests?
When analyzing experimental reference models, researchers will want to consider the following questions: p. A re reference models by default based on one sex but taken to be valid for the species overall?
q. Do data for one sex lag behind data for another sex, so that sex-specific reference models may not be equally developed or validated?
r. What criteria are used in selecting species, strain, and sex of model organisms used in research that will be translated to humans? Abstract (copy and paste) The AAFAWCE mission was to increase the representation and promote the advancement of academic women in chemistry and engineering, thereby developing a more diverse science and engineering workforce. The AAFAWCE's goals were to (a) increase recruitment of women faculty in chemistry and engineering, (b) enhance retention of academic women by mentoring and networking, and (c) promote leadership of women within the universities and in their scientific and engineering fields. (P. 4) In order to develop and implement adequate activities to reach the goals mentioned above, AAFAWCE conducted an online-survey to collect data regarding the faculty climate. Based on literature review and the Faculty Climate Survey, AAFAWCE conducted a qualitative study (19 semi-structured interviews) to examine women STEM faculty's experiences with isolation and related factors such as department fit and communication. They also investigated strategies that women use to overcome their isolation, primarily networking and mentoring.

AAFAWCE ADVANCE-PAID Objectives:
Recruitment of Women Faculty: To assure the recruitment of women faculty to the sciences and engineering by providing opportunities, best practices and strategies for hiring women faculty in these areas.
Retention of Women Faculty through Mentoring and Advising: To assure the retention of women faculty in the sciences and engineering the Alliance will provide opportunities, infrastructure, and resources for mentoring and advising assistant and associate professors.
Promotion of Leadership Among Women Faculty: To increase the number of women in chemistry and engineering capitalizing on their leadership skills for career advancement and the attainment of leadership positions. Analysis of initial regression models of the AAFAWCE Faculty Climate Survey indicates significant differences in ways that women and men responded to the items related to their sense of isolation, fit, and communication. (P. 98) The items were scaled from 1 (strongly disagree) to 4 (strongly agree). "The mission of the AAFAWCE is to increase the representation and promote the advancement of academic women in chemistry and engineering, thereby developing a more diverse science and engineering workforce." (P. 34) "Our central theme across the AAFAWCE is to improve the climate for women faculty in chemistry, physics, and engineering by providing structures, strategies, and best practices for hiring, retaining, and providing leadership opportunities for women faculty in our five state institutions in Florida." (P. 122) To change the faculty climate (especially the isolation of women), AAFAWCE promotes: Mentoring/Networking/Collaborating "Our findings suggest that the sense of isolation that women STEM faculty experience is due to a lack of social capital, driven by insufficient institutionalized mentoring and networking, a deficiency that can be addressed through explicit institutional efforts. These findings indicate that we should not focus on "fixing" women STEM faculty, but rather that institutions should undertake to implement support mechanisms to connect women more directly to their colleagues, departments and institutions." (P. 110) Recruitment "The Faculty Climate Survey from the start of the grant informed us of the issues women faculty faced at our institutions: women's sense of isolation, decreased sense of department fit and communication, and their view of the climate (including recruitment efforts) for women faculty, in comparison to responses from the men faculty." (P. 126) Leadership "We decided to include leadership as one of our three foci for the AAFAWCE grant because women academics that excel and become leaders will have more of a chance of changing the culture within academia and in their profession." (P. 165) "Literature on women as leaders in the academe suggest that efforts to increase the number of women leaders work best if they both enhance women's skills and knowledge and directly confront extant organizational culture that can get in the way of women's advancement." (P.

General comments and remarks
Challenges in gathering survey data: small cell problemconcerns regarding confidentiality and anonymity -> low response rates; solution could be a bigger survey which includes more institutions (to reduce anonymity concerns with cross-aggregation); to market the survey as a general survey of faculty (instead of targeting only departments with very few women or minority groups); or do the faculty climate study in a more qualitative way. The study suggests that the companies where women are most strongly represented at board or top-management level are also the companies that perform best. Confirming the existence of the gender gap -most notably in the composition of corporate management bodies -the McKinsey study offers fact-based insights into the importance for companies of fostering the development of women in the business arena, so that a greater number attain positions of high responsibility.
Finally, building on these insights and observations, and highlighting the main barriers to female representation on management bodies, this study seeks to bring the practical debate of how to make the transition from awareness of the situation to the implementation of change.

2.
Abstract (copy and paste) The following article summarizes the meta-analysis of policies towards gender equality in science and research across Europe spanning the years 1980 to 2008. Observed overarching trends in the research literature are summarized, including the impact of higher education restructuring on gender equality in science and research and measures for advancing women's science careers. The article closes by stressing three key challenges: first, the integration of gender policy assessment with theories of social change; second, the gendering of innovation policy; and third, readdressing the question of power and political struggle in relation to policy. There are two difficulties in this report: 1. Establishing clear-cut relations between certain policy measures and the overall representation of women in science is problematic; 2. Some of the policies or measures examined showed no statistically significant correlation with the proportion of women in science. As the authors argue, however, this should lead to a more thorough examination of measures and initiatives at sub-national levels (ibid., p. 38). Local and small-scale initiatives could have a more decisive impact on women's participation in science than large-scale programmes. The report on policies towards gender equality in science and research aimed to close this gap. ( u. Regarding monitoring programmes and networking activities: these findings are based largely on qualitative interviews, reflecting the first hand experiences of participants. However, what is striking is the absence of any negative statements from the evaluation reports -although this "underestimation" of the real complexities of mentoring relations is a fairly familiar pattern in evaluation studies (Eby & Allen, 2002;Tenner, 2004). (pp.307-308) 74. A crucial gap in the available literature concerns the lack of knowledge on specific disciplinary career paths, advancement and obstacles.Research on appointment procedures and scientific excellence from the Netherlands showed that "implementation of very general policy measures targeted at academia as a whole is not the best way to obtain a gender-balanced workforce in the upper echelons in universities" (van den Brink et al. 2006, p. 39). In contrast, measures that take into account disciplinary differences seem to be a more promising alternative in the long run. Policy measures will need to take into account these specific disciplinary aspects in order to be successful. (p. 308) Comments on 11. Aim of the report is 75. To summarize and evaluate the main findings of the policy report 76. To identifying the major shortcomings of policy evaluations for gender equality in science and research and suggests ways to move forward. The main challenge from our point of view consists of overcoming the almost exclusive focus on a human resources approach to gender equality policy and to achieve a tighter theoretical integration of what are often isolated evaluation studies in order to tackle the difficult issues of promoting and fostering cultural change.
12. Key dimensions of RRI (For literature dealing with one or more of the 5 key dimensions.) 12.1 How is the key dimension defined?
(terminology applied, central features/characteristics) HE restructuring/reform: the latest introduction of new management strategies into HE and research means that important changes have been made not so much with regard to the goals (e.g. raising the proportion of women in higher career positions), but rather in terms of the steering mechanisms used to achieve them. Several policy instruments such as legal/rights measures, positive actions (such as quotas), co-exist alongside more recent "mainstreaming" mechanisms and new steering instruments such as target/incentive-bound resource allocation. Women representatives or equality officers reflect this change in their shifting responsibilities and tasks.

Advancing Science Careers:
The whole field of career development is focused on the core issue of promoting women in science. The well-known, albeit misleading, metaphor of the "leaky pipeline" bears witness to the fact that women are more severely under-represented the higher they climb up the career ladder. Policy transfer and innovation studies: Whereas in the past, excessive focus was given to the role of individual actors (politicians, bureaucrats, etc.), currently a more ecologicallyoriented perspective is being put forward, where individual agents operate under the constraints of past policies, existing socio-economic conditions, ideological climate or the efficiency of the available bureaucratic and administrative infrastructure (see Dolowitz & Marsh, 1996, p. 353ff.) (p. 310).
NPM strategies for recent higher education reform. On a very basic level, NPM offers a solution for streamlining an apparently inefficient and oversized bureaucratic state apparatus by introducing market logic into the non-market public sector (Hood 1991;Bouckaert & Pollitt 2005)  Theory of Change i.e. gaps between policy implementation and changes in societal values.

General comments and remarks
The final section of this article aims to provide a further reflection on the relationship between policy towards gender equality in science and its evaluation. What is lacking in a certain way is a more thorough theoretical engagement of projects and research which implement and evaluate gender equality policies. As Verloo stresses "...gender impact assessments merely make gender visible, by producing statistics for instance, but they fail to provide an analysis of such statistics in terms of their link to producing gender inequality, and therefore are not really gender-sensitive, let alone transformative." (Verloo 2005, p.357). The lack of explicit theory is a further handicap to tackling these aspects of structural and cultural change.
 A strong theoretical model of how gender inequality intersects with other social inequalities and is continuously reproduced in society will be a vital element. Mary Daly (2005) concludes that while gender mainstreaming is "trumpeted as fundamentally transformative, it lacks, as yet anyway, a full articulation of a theory of change" (p. 447). As she furthermore contends, this shortcoming is due to a missing sociological core that would enable reflection on the relationship and gaps between policy implementation and changes in societal values.  A further crucial resource should be to explicitly build on innovation policy studies. The private R&D sector is the most important factor in determining the proportion of women in research. Innovation policy is primarily directed towards the private business sector. Women entrepreneurs encounter significantly more difficulties attracting investment for their firms than men ( The question this project focused on is: how can Gender Budgeting be applied as a powerful instrument in the budgeting policy of an organisation? The intention is to show which dimensions and which phases of the budgeting process have to be considered. Furthermore, some basic steps for a systematic integration of gender issues into the budgeting process were developed.
The main findings are: 77. A concentration of power within university managements could be observed in all universities; 78. The decision-making concerning budgeting is still male dominated in all universities; 79. If gender equality topics expand into the budgeting planning process, there are always quite hard conflicts about them, despite the quite small sums actually concerned; 80. The budget processing gets less transparent: informal networks gain more importance. This leads to disadvantages for women; 81. General lack of gender awareness, of a critical reflection of the relationship between women and men as well as a basic change in the male dominated organizational culture of most universities.

Recommendations:
82. On the level of the universities: this refers e.g. to the need for more sensitisation and awareness raising for gender equality in science, for the implementation of sufficient institutions for the promoting of gender equality, for the operationalisation and implementation of gender equality objectives on grounds of sex-disaggregated data, for an institutionalisation of a Gender Impact Assessment, for the equal participation of women and men and the inclusion of gender equality institutions in all phases of the budgeting process. Important elements: transparency of the budgeting process, integration of gender objectives and gender analyses into all parts of the accounting system, the distribution of financial resources by indicators and the integration of gender issues in all agreements on objectives, the implementation of gender sensitive measures for a modified personnel recruiting, introduction of a gender controlling system to ensure effectiveness and sustainability of the whole process.
83. At national and state level: a strong legal framework for gender equality, to gender equality policies at ministerial level, once more to the collection of sex-disaggregated data and the implementing of gender objectives into performance agreements. Further recommendations concern the distribution of funding by indicators and the introduction of Gender Budgeting for third party funding. Additional further affirmative actions for the advancement of womenin science are necessary.

At the European level:
we recommend for example the implementation of Gender Budgeting into all research activities of the EU and more funding for projects on the implementing of Gender Mainstreaming and Gender Budgeting into science. We suggest to establish a European gender accreditation system and to develop a set of common gender equality objectives throughout Europe. An improvement of the European database on sex-disaggregated statistics. And finally we strongly recommend the integration of the subject of gender equality as a top level issue into the European agenda. (pp. 7-12)

Gender sensitive indicators
Gender-sensitive Impact of instruments and strategies can be assessed on five different levels: Examples for relevant aspects on the five levels mentioned above could be:

Input
As human resources are the most important input of scientific organisations, the analysis of possible effects on personnel is fundamental for the GIA as well as the analysis of the initial situation at universities. For the gender analysis of the users of activities and services for students, the beneficiaries of research as well as the general public, the main questions for analysing gender sensitive effects of measures and instruments concerning the users or recipients are:  How would the measure/instrument influence the usage by women and men of the universities' activities and services?  Does the measure/instrument have an influence in case the activities meet the different needs and requirements of women and men?  Important features are therefore: women and men in students' representative bodies (=participation); number of people who use certain services.

Outcome level
For the gender analysis of the outcome of a measure, of an instrument or a strategy which concerns indirect effects for users or individual and general external effects, two different kinds of effects need to be examined: the direct effects for users on one hand and general or individual external effects, either positive or negative ones, on the other hand. The main questions therefore are: The report is based on the following methods of data collection and analysis : 85. An analysis of the national framework for women and men in science and the financing of the university sector in Austria, Germany and Poland.
86. An analysis of the specific situation of women and men and of the process of budgeting at the three cooperating universities, the University of Gdansk, the University of Augsburg and the Vienna University of Economics and Business Administration.
87. On the basis of these findings, instruments and measures for the implementation of gender budgeting in scientific organizations were developed.
88. As part of the support action a selection of instruments and measures to the specific situation of each cooperating university was adapted. (For literature dealing explicitly with responsible (research) and innovation. If the publication deals with one of the 5 key dimensions, please proceed to 11.) 9.1 Which definition of RRI is being used?
(author's definition or reference to other source) 9.2 Which aspects of RRI receive special emphasis? (e.g., certain normative goals, procedural approaches, reference to one or more of the 5 key dimensions, …)

9.2
Which arguments are presented in support or rejection/criticism of RRI? 9.3 To which concepts, theories, approaches, schools of thought, communities (scientific or practice) in the area of research and innovation does the literature relate or make reference to? (e.g., STS, constructive TA, anticipatory governance, foresight, deliberative democracy, …) Comments on 9.
10. Policy context of RRI (For literature dealing explicitly with responsible (research) and innovation. If the publication deals with one of the 5 key dimensions, please proceed to 11.) 10. 11.4 According to the author(s), which type of evidence/data is missing to better support the claim? (e.g. data gaps, limitations with regard to analytical levels, lack of indicator specifications etc.)

Comments on 11.
Structural disadvantages and the waste of female talent in science have been diagnosed for all states in the European Community by several studies of the European Commission.
Parallel to this annoying situation Gender Mainstreaming has become an important issue in the debate on reforms for the higher education sector during the last years of the former millennium. Thus the European Commission called upon the member states to implement and intensify gender equality measures and the Commission insisted on a gender action plan as an obligatory request for an application in the 6th Framework Programme on Research and Technological Development. One request was the development of Gender Mainstreaming instruments for the scientific field and another was the implementation of a gender watch system. The definition of Gender Budgeting by the Council of Europe which is also used by the European Union: Gender budgeting is an application of gender mainstreaming in the budgetary process. It means a gender-based assessment of budgets, incorporating a gender perspective at all levels of the budgetary process and restructuring revenues and expenditures in order to promote gender equality. (p. 8) The following steering cycle of budgeting was applied in the study: scientific knowledge and technology design. Realizing the full potential of gendered innovations in the next decade will require deep interdisciplinary collaborations between gender experts, natural scientists, and engineers. Realizing the full potential of gendered innovations will also require international coordination, as recommended in the 2010 European Commission genSET Consensus Report and the 2011 United Nations resolutions on Gender, Science and Technology. ( In 2009, the Clayman Institute for Gender Research at Stanford University initiated the Gendered Innovations in Science, Medicine, and Engineering Project (Gendered Innovations). This project has been expanded internationally through a collaboration with the European Commission in 2011 entitled Innovation through Gender. Systematic methods of sex and gender analysis are being produced in a series of expert meetings in 2011 and 2012. These meetings bring together gender experts, basic scientists, engineers, public health and medical experts, policy makers, and technology designers. The purpose is to develop practical methods of sex and gender analysis for researchers. The Gendered Innovation project demonstrates methods through case studies. Each section below presents a case study highlighting a problem, a method of sex or gender analysis important to overcoming the problem, and a solution, or gendered innovation. (pp. 158-159)  90. It is important to point out, that increasing women's participation in science and engineering will not be successful without restructuring institutions and incorporating gender analysis into research.
91. The ultimate goal of gendered innovations is to enhance scientific and technological excellence. Research must control for sex and gender. Sex and gender analysis act as yet further controls one set among many standard methodologies that serve to provide critical rigour in science.
92. Gendered innovations also seek to create gender excellence; that is to say, to build inclusive scientific communities where men and women share equally at all levels in decision making, policy, and defining and carrying out research.
93. Gendered innovations seek: 1) to create gender equality; 2) to enhance creativity; 3) to stimulate economic and technological development (or business innovation); 4) to make research more responsive to society. This first approach seeks to increase women's participation by supporting Women's education and careers. While critically important, this approach has also been criticized for "fixing the women". The implicit assumption is that science, medicine, and technology institutions and research are gender neutral. Consequently, this approach fails to look beyond women's careers to the need to reform scientific institutions and research methods. (p. 156) This second policy approach focuses on institutional reform while often assuming that what goes on inside institutions-basic and applied research-is gender neutral. Restructuring institutions is important, but must be supplemented by efforts to eliminate gender bias from research and design.
Change needs to come also at a third level: gendered innovations in scientific knowledge and technology design. (pp.156-157) With respect to gender, ethnicity, and muchelse, science is not value-neutral. Gender mainstreaming, adopted by the United Nations Fourth World Conference on Women in Beijing (1995), entails the systematic integration of gender equality into all systems and structures, policies, programmes, processes and projects, into ways of seeing and doing (Rees 2002). Gender mainstreaming now needs to be expanded to include gender analysis in basic and applied research. Mainstreaming gender analysis into research creates "Gendered Innovations". (pp. 157-158) 11.3 What evidence is presented to support the claims?
(e.g., data, indicators, research results, case studies, anecdotal evidence) 11.4 According to the author(s), which type of evidence/data is missing to better support the claim? (e.g. data gaps, limitations with regard to analytical levels, lack of indicator specifications etc.) Comments on 11.
12. Key dimensions of RRI (For literature dealing with one or more of the 5 key dimensions.) 12.1 How is the key dimension defined?
(terminology applied, central features/characteristics) To better understand gendered innovations, we distinguish three approaches taken by policy makers, institutional administrators, and scientists and engineers over the past three decades (Schiebinger 1999;2008). The first focuses on programmes designed to increase women's participation. The second approach seeks to increase women's participation by transforming research institutions. The third focuses on overcoming gender bias in science and technology by designing gender analysis into all phases of basic and applied research from setting priorities, to funding decisions, to establishing project objectives and methodologies, to data gathering, to evaluating results, and transferring ideas to markets. ( v. Granting agencies can require that all applicants specify whether, and in what sense, sex and gender are relevant in the objectives and the methodology of their project. Research projects that fulfil this criterion might achieve a higher score for funding. Researchers might also achieve this score by demonstrating that sex or gender is not relevant to a particular project. It is important, however, that the issue be addressed.
w. Hiring and promotion committees can evaluate researchers and educators on their success in implementing gender analysis. Knowledge and use of methods of sex and gender analysis can be one factor taken into consideration in hiring and promotion decisions.
x. Editors of peer-reviewed journals can require sophisticated use of sex and gender methodology when selecting papers for publication. Do not delete anything after this (non printing) line