What should a Teaching Excellence Framework look like?

Jo Johnson, the new Minister for Higher Education, surprised nobody last week by asserting his determination to deliver the Conservative Party’s manifesto commitment to “introduce a framework to recognise universities offering the highest teaching quality”. This will now be known as the Teaching Excellence Framework.

He did not go into a great deal of detail in his speech about what the framework would look like or what precisely the recognition would be for high quality teaching, although he did indicate during Q&A that he was open to considering financial incentives and rewards. In fact the level of openness and willingness to listen and respond to what the sector thinks was quite striking and positive. I think it is imperative that the sector engages positively with BIS over this, because I’m not sure it is an initiative where a strategy of kicking it into the long grass or hoping for death by a thousand cuts will work.

I think a key consideration in developing the TEF ought to be that it is light touch. I’m sure all universities that participated in the REF would want to avoid replicating the bureaucratic burden involved in that. The TEF should be based on metrics, ones that already exist or could be gathered with relative ease. Jo Johnson was also clear about this in his speech when he said “I have no intention of replicating the individual and institutional burdens of the REF. I am clear that any external review must be proportionate and light touch, not big, bossy and bureaucratic”.

Based on that principle, I think the data should fall into three themes:

  1. Input
  2. Output
  3. Peer judgement

1. Input

By input, I mean the quality of the content and delivery of teaching and wider academic experience of students.

There are a number of data that could be used, or newly collected to inform this but I think we have one already in the bag and a couple of others that could be gathered quite easily if we try hard enough.

  • Qualified Teacher Status: HESA has already been gathering data on this, I know it’s not a measure the whole sector has united behind yet, but with one more push and some finessing of the HEA’s Professional Standards Framework, we could get some sector-wide, comparable data on this. There will be plenty of wailing and gnashing teeth about this I am sure, but it is notable that this came top of student expectations when HEPI surveyed them recently, we should meet that expectation, it won’t do anyone any harm!
  • Research Impact: I was trying to think of a way of quantifying the fact that students should have the opportunity at university to learn and understand the very boundaries of knowledge and understanding in their chosen subject and learn from those involved in defining and discovering the new boundaries of knowledge and understanding. The REF scores would tell some of this story, but it occurred to me that the impact scores are probably a more effective and targeted way of measuring the ability to communicate and engage students with world-leading research.
  • Enrichment: it is widely accepted that a students’ academic experience is defined not only in the classroom, but also the co-curricular activities that are available. We are already engaged in an initiative to gather all this data to inform the Higher Education Achievement Record (HEAR). Be it work placements, study abroad, community engagement, leadership of sports clubs or academic representation, these activities improve student academic development and make a significant contribution to an excellent learning and teaching experience. It would take some work to finesse our systems to gather this data, but it can be achieved and should make a useful especially feature of a TEF.

2. Output

By outputs, I mean demonstrable measures of students gaining knowledge and skills during their their studies. I think there could be three data sets that could be used here, two that we gather already and one that could be derived from existing data.

  • Learning Gain: this term means many different things to different people. As a school governor, I have become familiar with the concept of ‘value added’ where a contextual measure of a students knowledge, skills and ability on entry and then looking at progression from that at various points up to a pupil leaving the school. I am not at all convinced that a measure that looks at UCAS tariff through to degree classification is the most effective measure of learning gain, mostly because of all the problems we know exist with the comparability of degree classifications. I think a measure should be developed that looks at the entry qualifications that students arrive with and then how far along they get with their higher education. That way, universities who recruit students with no or few qualifications could be rewarded and see the recognition increase as students achieve higher level HE qualifications. For example a student with no qualifications attending a university and achieving a Level 5 HND would have travelled the same distance as a student with the traditional A levels  and then achieving a Level 6 BA (Hons).
  • Employment data: we currently have the Destination of Leavers from Higher Education Survey that captures reliable data about what students are doing 6 months after graduation. There is talk of improving this to use HMRC data to show the real earnings of student at 1, 3, 5 and 10 years post graduation which would be a positive development I think. Either way, this is a key piece of data to show what students are capable of doing and achieving after graduation and should be included in any measure of teaching quality.
  • Grades: so long as we have the old honours classification system the data is not comparable. But, if a concerted effort is made to move to the GPA system, we might achieve more comparability and be able to develop data that reflects and measures that.

3. Peer review

A purely data driven approach would not give sufficient space for input from peers. I include students as part of the peer community here, because I feel strongly that we should consider their feedback and views of the quality of teaching to be of equal, if not greater, value than that of peers because, above all else students are the only people who can tell us what it is like to be a student.

  • The National Student Survey (extended to PGT): I’m pleased that the recent review of the NSS appears to have concluded that the survey is immensely valuable and not in need of a radical overhaul, but needs refreshing to sort out some of the dated and more nebulous questions (personal development, anyone?!). So this survey and its outcomes should undoubtedly form an important part of the TEF, I would argue that it should be heavily weighted in comparison to other metrics too.
  • External Review: I am not advocating a return to subject review or inspection regimes of old, they ran out of steam over 15 years ago and everything we learnt about them then still stands (burden, bureaucracy, gaming the system, diminishing quality of the ‘inspectors/reviewers’). But I think a positive judgement from the university’s most recent QAA institutional review should be a pre-requisite for inclusion in the TEF, to demonstrate that core academic standards are being maintained and the teaching is built on solid foundations.

This is an attempt by me, from my perspective, to list what should be included in a Teaching Excellence Framework. Each of the individual sets of data listed above can (and I’m sure will) be picked off on its own and rubbished as an insufficient measure of teaching quality. This may be true, for example DLHE only tells you if students get jobs, it doesn’t tell you if they would have got them anyway or whether their degree actually helped them. BUT nested in with a number of other data sets (some that we gather already, some that we need to start gathering) it can start to piece together a rich picture of the quality of teaching and learning.

This approach of picking apart the minutiae has been very successful for the sector in the past, it may succeed again, but I am doubtful. We have an opening to engage positively and constructively with this initiative and to shape it; if we try to nit-pick and shower the whole thing in treacle, I suspect we’ll get the sort of TEF we deserve!

Advertisements

15 responses »

  1. […] “Jo Johnson, the new Minister for Higher Education, surprised nobody last week by asserting his determination to deliver the Conservative Party’s manifesto commitment to ‘introduce a framework to recognise universities offering the highest teaching quality’. This will now be known as the Teaching Excellence Framework …” (more) […]

  2. dkernohan says:

    Aren’t many of these metrics subject to far more potential for “gaming” than the external inspection route ever was?

    Would also note that the Small Business Act powers allowing for read-across between Student Finance data and HMRC records would be a more reliable (& less gameable) way of identifying graduate salaries?

    • derfelowen says:

      I agree that the changes to employability data will be much more reliable and useful than DLHE, but it’s not clear how/when that is going to be used. Perhaps in time for TEF?

      I’m not convinced NSS, DLHE, Entry/progression/awarding data can be gamed. Might be some slight massaging on the periphery but negligible. Also, when all these data are taken in aggregate, they will tell a richer story than any of them do on their own.

  3. rmforsyth says:

    It still bothers me that almost all of these ‘measures’ are collective – what’s the incentive for an individual to excel, or even more complicated, for an institution to recognise the excellence of an individual teacher?

    Would you see ‘Qualifed Teacher Status’ as including a ‘remaining in good standing’ requirement?

    • derfelowen says:

      Fair point Rachel. I think the score should be collective for universities, once you try to break it down to department, course or individual, a national scheme will become a nightmare of bureaucracy and poor comparisons.

      I think if there is collective score for a university, then it is imperative that universities themselves build effective structures to reward, promote and recognise those that are demonstrating leadership and excellence in teaching. There are some good examples out there, but more needs to be done to make this feel real to staff.

      Yes, I think if QTS is included it should be a snapshot of how many staff have the status either newly gained or ‘remain in good standing’

      • rmforsyth says:

        I would be pleased to see a more systematic approach to recognising and rewarding good teaching, at both institutional and individual levels.

        We will definitely need to be very clear about how it works, though. the REF has had plenty of difficult consquences for individuals and, perhaps, disciplinary and institutional values. In teaching, it is already possible to see measures which are only suitable for institutional use being used for traffic light-style ratings for individual modules (and then, in a short leap, to identify module leaders who are at the red light).

        What are the chances of (keeping?) an academic culture which repudiates gaming and encourages institutions to work together to enhance teaching…?

      • derfelowen says:

        I think any public system of accountability will be subject accusation of gaming, I’m not sure we’ll ever avoid that.

        I think maintaining the focus of TEF at institutional rather than department/course will protect against a too instrumental approach. I don’t think we should fear a system that puts a ‘red light’ against struggling modules/programmes. We need to make sure that the first reaction is to support and improve teaching capacity rather than punish poor scores.

        I think this is the sort of approach that enlightened universities are taking to the REF.

  4. newtonwonder says:

    Thanks Derfel for thinking about this.

    I’d like to suggest that you play the following mind game. Will your system work if we compare two very different students?

    1. Young, white, female, middle class student, privately educated with a family tradition of higher education, 3 A*’s at A level

    2. Mature, male, BME, widening participation student, studied previously at a ‘bog standard’ college, first in family entering HE, qualified with a BTEC or Access Course

    Do your criteria still pick out teaching quality, or would it confirm that the students who entered with advantage leave university with their advantage intact?

    I know the question sounds a bit aggressive and uses two extremes but I think it’s important to push at the boundaries of your model.

    Cheers

    Ed

    • derfelowen says:

      Hi Ed,

      Thanks for your comment.

      I think I’m still important to note that what is under discussion here is a teaching excellence framework. Widening participation is immensely important to the sector, and as someone from a what would be classified as a WP background myself, it is also very important to me. I have posted about it before. I think it’s important to separate the two.

      Having said that, I have suggested that the ‘learning gain’ measure should be focussed on rewarding universities that take non-traditional students. Those universities that recruit mostly students that arrive with conventional qualifications (typically A levels) and then go on to do well, gaining conventional HE qualifications (typically BA, BSc) would be recognised but not especially rewarded. Those universities that go out of their way to recruit students without conventional qualification (e.g. with a strong focus on recognition of prior experiential learning, or without A level/BTEC equivalent) and then support them to achieve HE qualifications, would be recognised more generously.

      I think that’s a far better approach than focussing on UCAS tariff on entry.

      • davidbaume says:

        Learning gain or value added is a brilliant measure for a TEF.
        If QAA ensures that all degrees are of the same standard / level / quality (assume a liberal sprinkling of quotation marks throughout this reply), then measuring learning gain, as Ed’s examples above suggest, at once shows that most Russell Group Universities add far less value than do most pst-92 Universities. Logically unarguable. But – politically acceptable? I doubt it. But I’d love to be wrong!

      • derfelowen says:

        Hello David,

        My suggestion is that the ‘learning gain’ element of the TEF starts with the entry qualifications of students and then offers the most recognition for universities that take students on the longest journey to the qualification they leave with. Russell Group universities might not score very highly on this measure because they tend to recruit students with more conventional qualifications (although it is important to stress the the RG is by no means alone here!).

        That said, it is important that widening access and participation is not the only measure of teaching excellence, hence including other measures.

  5. newtonwonder says:

    Hi Derfel

    I’m not for one moment suggesting that WP ought to be the only measure. I think that the other measures you describe are sensible, pragmatic ways to measure teaching. My point was that you could in theory have two identical teachers/ departments/ institutions at different institutions but because one takes a more educated, more middle class intake that one is very likely to appear to better. Learning gain may deal with that problem, but as David says it all depends whether or not it’s acceptable or how important it is when compared to graduate employability or the number of ‘good degrees’

    I think for me it comes back to that messy problem that the students themselves are one of the inputs. If the TEF doesn’t take that into account, it feels like it will be extremely myopic. Hopefully the whole point of this discussion is that in some way it helps warm up whoever gets to talk to the ministers about what the TEF should include.

    Cheers

    Ed

  6. […] was a wide range of views and opinions. I’ve already laid my cards on the table in an earlier blog post about what should be measured and I’m not sure I’ve heard anything to convince me […]

  7. Ian Scott says:

    GPA is no more comparable than the Hons classification system (pure fallacy) A students employability is far more closely correlated with their social class than the University they attend – thus the HMRC data tells us nothing about what a University adds in terms of a student’s employment prospects. Any chance of a more informed evidenced based debate?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s