The 2009 Association of Research Managers and Administrators conference took place in Southampton from 1st to 3rd June. I attended the full three day event which was a great opportunity to find out about good practice in research administration at other institutions, as well as the latest news in HE funding. Since returning I’ve had some time to reflect on lessons learned and the big themes and issues which arose during the conference.

Here are some roughly coherent thoughts based on notes I made during the event (for the bitesize version, just search #arma09 on Twitter):

Impact

I attended the pre-conference specialist session on the impact of research. This was timely because you can’t go anywhere without hearing about impact in HE these days. As most in the sector are aware, Research Councils introduced a requirement for Impact Summaries and Plans into all grant proposals earlier this year. HEFCE have also revealed that assessment of impact will play a key role in the Research Excellence Framework.

Paul Hubbard and Graeme Rosenberg of HEFCE both presented at the conference on the ways in which they might go about measuring impact. The short version is this:

How far and in what ways a body of research activity makes a demonstrable difference within and beyond the academic sphere.

There are many ways in which a piece of research contributes to the “health, wealth and culture” of the nation (as the Research Councils put it), but few concrete examples were offered.

HEFCE and RCUK argued that measuring impact is not new, it’s just been made more explicit. The policy environment has changed and research must be seen to do some good to users outside academia.

BBSRC sees itself as “a customer seeking to procure research from the science base”. Therefore, impacts and outcomes of research are crucial, particularly when reporting to the government on how they’ve spent their money on funding research. Impact statements are seen as  a way of promoting cultural change in the mindsets of researchers.

The BBSRC also stated that they intend to ask an applicant to rewrite an Impact Plan/Summary if the proposal as a whole is recommended for funding. It’s clear that all Research Councils are still committed to funding based on the excellence of the research as the highest priority, not the Impact Plan.

Bill Wakeham, V-C of Southampton, thinks the sector should communicate impact through “stories not metrics”. The challenge is in keeping track of stories, particularly since there may be a significant time-lag between basic research and associated impact.

Metrics

Measuring impact – or indeed other factors involved in research excellence – is not simply down to metrics. HEFCE have been making significant moves away from bibliometrics in particular in recent months. While the REF was initially envisaged as heavily metrics-based, the focus has changed. The headline is now expert review informed by metrics.

A pertinent quote from the session from HEFCE’s Paul Hubbard:

Bibliometrics have a great deal to offer, but there is no discipline in which they are a complete answer; there are some disciplines where they are not part of the answer at all.

Wellcome Trust quoted Einstein on this point: “Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted”.

Where bibiometrics or citation data does feed into the assessment, this will be produced centrally by HEFCE on the basis of “standard algorithms” (yet to be determined), so that the burden does not fall on institutions.

The Research Excellence Framework

The REF will consider outputs, impact and research environment – the difference from RAE2008 is that esteem is no longer included. It was felt that esteem was too subjective and difficult to measure. You can’t measure research impact either, according to HEFCE, but indicators will be used to aid expert assessment and a mark will be awarded.

The primary focus in the REF will be excellence, as in the RAE. HEFCE are not particularly interested in lower quality research, so selectivity of staff will still be a key part of the exercise. They do not want to capture all research outputs – just the best ones. Other similarities to RAE2008 will be quality profiles.

Impacts within the assessment period will be considered, but HEFCE also suggested that impacts occuring within assessment period could be based on research carried out earlier. How much earlier wasn’t specified, but an average time lag of research -> impact of 17 years was mentioned. This could be significant, especially for younger institutions such as Lincoln.

Impact will be judged by an “evidence-based narrative statement”, similar to the RAE submission. This, like the other elements in the REF will be subject to review by panels of experts. Note that it’s expert review rather than peer review because there will be non-academic members on the panels.

The timetable at present looks like this:

Autumn 2009: HEFCE produced consultation on structure of REF

2008 – 2012: Census period

Late 2012: Submission of return

2013: Assessment period

2014: Funding allocation determined

Questions still to be answered include:

1. Do researchers take impact with them when they move institutions?
2. Will panels of research users contribute to the design of the assessment, or actually assess the work?
3. Will it be necessary to submit 3 or 4 outputs per selected researcher?
4. How many UoAs will be required? Fewer than 63 (as in RAE2008), but more than 15 was the only indicator.