Asking all the right questions: the BIS Select Committee announces details of QA/TEF inquiry

CommitteeFollowing on from yesterday’s announcement that there will be an inquiry into the assessment of quality in Higher Education the BIS Select Committee have today announced a series of questions and topics that they will seek to explore.

  1. What issues with quality assessment in Higher Education was HEFCE’s Quality Assurance review seeking to address?
  2. Will the proposed changes to the quality assurance process in universities, as outlined by HEFCE in its consultation, improve quality in Higher Education?
  3. What should be the objectives of a Teaching Excellence Framework (‘TEF’)?
    1. How should a TEF benefit students? Academics? Universities?
    2. What are the institutional behaviours a TEF should drive? How can a system be designed to avoid unintended consequences?
    3. How should the effectiveness of the TEF be judged?
  4. How should the proposed Teaching Excellence Framework and new quality assurance regime fit together?
  5. What do you think will be the main challenges in implementing a Teaching Excellence Framework?
  6. How should the proposed connection between fee level and teaching quality be managed?
    1. What should be the relationship between the Teaching Excellence Framework and fee level?
    2. What are the benefits or risks of this approach to setting fees?

This is a very good set of questions, exactly what should be asked about current developments.

Written submissions are due by 30 October.


Select Committee to investigate HE quality

So, this is going to fun!

Mark Leach‘s eagle eye has spotted a statement from the Chair of the House of Commons Business, Innovation and Skills Select Committee setting out his strategy for the next five years.

I have vivid memories of the last select committee inquiry and report into HE – Students and UniversitiesI think it’s fair to say that it wasn’t an awful lot of fun for the sector.

I’ve always been a consumerist sell-out!

CNbvXhgWIAESZIl It’s always nice to get a blast from the past.

I was tweeted this photograph a couple of days ago by Christina Leston-Bandeira a Professor in the Politics Department at the University of Hull. It’s of an article that was written about Hull University Union when I was student President in 2004/5.

A few weeks ago, I wrote about a polemic, dressed up as research, that had been published arguing that the partnership model of student engagement was fatally undermined by the fact that students’ unions have commercial activity and provide services to their student members (and for that matter, other members of the university community).

If you look at the caption to one side of my photograph, you can see that I gave the game away over 10 years ago:

Independent thinker: Derfel Owen, the union president says that generating its own income has been the answer

I’m clearly a neo-liberal headbanger after all, hell bent on destroying the very fabric of higher education!

I think the point I was trying to make during the interview is that the successful commercial activities of the students’ union had made us a more engaged and visible contributor university life and to the local community, also that those commercial activities had grown our income streams and helped us to invest in representation, volunteering and social activities for students.

Top tips for surviving the TEF

I just read this great, succinct piece by Philip Plowden on the Guardian HE pages – How to approach the Teaching Excellence Framework with confidence.

It’s very balanced and sensible. I particularly liked recommendations 1, 4 and 5:

1) Stay Passionate about your subject area

Too often a commitment to learning and teaching is presented in opposition to engagement with research and scholarship, but the two should be inextricably linked. Universities are above all places where knowledge and practice are questioned, and our understandings extended. Without that continuing commitment to the discipline, any teaching can become rote, and the value to students is lost.

The emphasis on the link between research and scholarship and teaching is spot on, universities are meant to be places where students can and should learn about the boundaries of knowledge and understanding in their chosen subject; sharing that knowledge and working together to push the boundaries of understanding is what is should all be about.. Most important is the need for students to feel a sense of passion, commitment and excitement from their lecturers, this can come from linking teaching to their own research but not necessarily, it’s about having a sense that the person in front of you teaching you cares just as much (possibly even more) than you about what you are studying.

4) Talk with the students

“Co-production” has become something of a buzzword, and there is increasing challenge, including from students, to the idea that it is in part their responsibility to shape the learning experience that they are now funding. But if nothing else, it reminds us that students are best placed to comment on how a programme or a module feels to them. And that gives us our first intelligence as to whether we are achieving what we intended.

I’ve written a book about student engagement (get yourself a copy on Amazon!) so get very excited by concepts such as co-production, change agents, partnership etc. However, I have always been of the view that when boiled down to what can and should every member of staff at a university be doing, it’s this, talking to your students. They are the only people who know what it is like to study on your module or programme, taking time to talk to them and listen to them will be invaluable. This can be through formal structures such as student-staff committees or feedback forms, or less formal opportunities at the end of a lecture or around assessment hand-in times.

5) Get assessment right 

When we get assessment right, it becomes a seamless part of the learning process. There can be an unspoken anxiety that unless assessment is burdensome – for student and academic alike – it cannot have been rigorous. Perhaps this is why the assessment process often remains the most traditional element in any programme and module. If you can get the assessment right, the rest of the module will follow.

This is what it all comes down to, this is the biggest burden and stress factor for every student and academic. There are so many shibboleths and myths about assessment that lead to over-assessment but also over-burdensome methods of assessment for students and staff.

The other recommendations are good too.

The NSS: A case for the defence

nss218I was really downhearted when I read this article published by the Guardian on its Higher Education pages.

The author argues that the National Student Survey is a waste of time and is detrimental to learning and teaching.

Personally, I think it is a cause for celebration that students report such high levels of satisfaction and positive experiences. In particular this year when the skies were meant to fall down upon us because the first generation of £9k fee payers were filling in the survey.

The NSS is a survey that draws in hundreds of thousands of respondents each year; just in order for the results to be published a response rate of over 50% must be achieved (and this is massively exceeded every year), this provides reassurance that the data is robust and is genuinely representative of student views about their experience.

To complain that the scores are clustering misses the point entirely and ignores the facts. Would we prefer it if more students were having an utterly miserable experience for the sake of having a wider range of scores? I doubt it.

Then there is the argument that a single survey is not appropriate because universities and the courses they provide are different. Of course they are! Universities are different and the courses that are delivered will vary in content, teaching and assessment methods; the survey does not ask students to compare their experience with students at other universities and it would be absurd to do so because students (by and large) only have one experience to reflect on. The NSS, rightly, asks students a series of pointed questions about their experience of teaching, assessment, support, resources etc. at their institution, the questions are identical for all students in their final year so the focus is on comparability of outcome not input. It is perfectly possible for two courses in the same subject to be taught and assessed in completely different ways but for students to come out at the other end highly satisfied with their experience. It would be disastrous to try to bring about uniformity of input.

Another point to consider here is the ‘masking effect’ of the mean scores for institutions. These scores that aggregate the level of satisfaction of all students covers up the fact that there will be significant variation between departments and even within departments between courses. This in itself is not a problem, but it does counter the suggestion that all this is pointless because there isn’t a wider gulf between the highest and lowest scoring universities. This also gives an opportunity for departments to benchmark themselves against colleagues in university and also peer departments in other universities.

The suggestions in the article of changes to the curriculum that have been made because of the NSS seem very odd to me. If methods of assessment are being changed to move away from over-reliance on essays and exams, then I can only cheer it along as a good thing in itself. These traditional forms of assessment are tried and tested and have very important part to play in universities. But why not add other methods of assessment into the mix? Ones that challenge students to develop their thinking and negotiate a final piece of work with a group of other students, ones that encourage  and support them to present their research and ideas in new and innovative ways. I’ve just spent the Summer at UCL reading through hundreds of external examiner reports. They are all experienced, senior academics from other universities, and their feedback is routinely positive about new and innovative methods of assessment as a way of maintaining high standards and challenging students in new and interesting ways; far from discouraging new methods of assessment, they actively encourage, support and promote it and these are the opinions of other academics, not students.

Another criticism that is often made (not though in this Guardian article) is that the questions only ask about satisfaction and do not ask student to reflect on their teaching, learning and assessment experiences. I disagree. To take four questions as examples:

  • Staff have made the subject interesting (Q2 in the NSS)
  • The course is intellectually stimulating (Q4)
  • The criteria used in marking have been clear in advance (Q5)
  • Feedback on my work has helped me clarify things I did not understand (Q9)

Who would not want to know the answer to these questions? If, for example, only 30% of students agree that the course is intellectually stimulating I think it’s very important to know that and to do something about it! Equally if, 70% of student agree with the statement “Feedback on my work has helped me clarify things I did not understand”, that’s a good score, but still one third of students are saying that feedback is not helping them and that’s a worry.

No mass survey, containing only 23 questions is going to be perfect. But I for one welcome this annual opportunity for students, en-masse, to take a moment to reflect on their entire experience and tell us what it’s been like, it is an important feature of UK higher education and a critical part of the relationship we have with our students.

UPDATE: Andrew McRae has ‘fisked’ the Guardian piece with magisterial style on his blog .

The TEF beat goes on

I attended an excellent workshop run by the University Alliance last week, officials from BIS and HEFCE addressed an audience about progress developing the Teaching Excellence Framework and the ‘Quality Assessment’ regime. My impression that BIS are genuinely determined to build a framework that the sector can live with (I’m not sure we’ll ever reach a situation where we’re happy with it!) was reinforced.

The event organisers challenged us to think hard about the purpose of the TEF, what we want to measure and what metrics we might want to use. Unsurprisingly, at this early stage, there was a wide range of views and opinions. I’ve already laid my cards on the table in an earlier blog post about what should be measured and I’m not sure I’ve heard anything to convince me otherwise yet.

A few reflections:

  • I was surprised by the level of support for including research excellence in the TEF (not surprised because I think it’s not a good measure, I very much think it is and should be included in the TEF) because I thought some universities would think this would favour one type of university over another. On the whole I think there was recognition that a balanced set of metrics will include datasets that are good for one type of university and other data that are good for an alternate set. This is a good sign.
  • BIS have not made their minds up yet. This is not an exercise in battering the sector into thinking it supports something that is not in its interest (imagine a sector body trying to do that!). The TEF is going to happen, but what it looks like, how it operates and what incentives are attached are very much open to discussion.
  • What do we mean by ’employer input’? A number of contributors expressed a desire to have employers directly involved in defining and measuring teaching excellence. I am not convinced by this. Those subjects where graduates require professional accreditation (e.g engineering, architecture, pharmacy), already have employers heavily involved in shaping the curriculum, other subjects (mostly but not exclusively social sciences and humanities) it is very unclear what an employer could effectively contribute to a judgement on excellence. In terms of graduate skills etc. employers already make their contribution by giving (or not giving as the case may be) students a graduate job as measured by DLHE.
  • All will have prizes. The TEF lite that will be introduced next year will be light touch, probably making use of a small number of metrics and may have a low threshold because the ‘reward’ is needed by most universities and will not cost a phenomenal amount. We might find that the TEF lite rewards the vast majority of universities. This is sensible I think when trying to introduce something in such a short space of time; to introduce something that rewarded only a small number of universities before it had properly matured would kill it at inception.
  • Fee increases may not be the only reward. Of course students will say that fee increases are not a reward at all, but I think it was also clear that as the TEF matures and the economic climate improves there may be additional funding released for higher education that could by funnelled towards excellence identified by the TEF.

My former colleague and Comrade Professor Andrew McRae, Head of the Department of English at the University of Exeter, also posted his thoughts about the TEF on his blog on Monday. Characteristically sensible and challenging it is great to read an academic arguing that this is an opportunity not to be missed.

The TEF: What should it measure?

What should a Teaching Excellence Framework look like?

Jo Johnson, the new Minister for Higher Education, surprised nobody last week by asserting his determination to deliver the Conservative Party’s manifesto commitment to “introduce a framework to recognise universities offering the highest teaching quality”. This will now be known as the Teaching Excellence Framework.

He did not go into a great deal of detail in his speech about what the framework would look like or what precisely the recognition would be for high quality teaching, although he did indicate during Q&A that he was open to considering financial incentives and rewards. In fact the level of openness and willingness to listen and respond to what the sector thinks was quite striking and positive. I think it is imperative that the sector engages positively with BIS over this, because I’m not sure it is an initiative where a strategy of kicking it into the long grass or hoping for death by a thousand cuts will work.

I think a key consideration in developing the TEF ought to be that it is light touch. I’m sure all universities that participated in the REF would want to avoid replicating the bureaucratic burden involved in that. The TEF should be based on metrics, ones that already exist or could be gathered with relative ease. Jo Johnson was also clear about this in his speech when he said “I have no intention of replicating the individual and institutional burdens of the REF. I am clear that any external review must be proportionate and light touch, not big, bossy and bureaucratic”.

Based on that principle, I think the data should fall into three themes:

  1. Input
  2. Output
  3. Peer judgement

1. Input

By input, I mean the quality of the content and delivery of teaching and wider academic experience of students.

There are a number of data that could be used, or newly collected to inform this but I think we have one already in the bag and a couple of others that could be gathered quite easily if we try hard enough.

  • Qualified Teacher Status: HESA has already been gathering data on this, I know it’s not a measure the whole sector has united behind yet, but with one more push and some finessing of the HEA’s Professional Standards Framework, we could get some sector-wide, comparable data on this. There will be plenty of wailing and gnashing teeth about this I am sure, but it is notable that this came top of student expectations when HEPI surveyed them recently, we should meet that expectation, it won’t do anyone any harm!
  • Research Impact: I was trying to think of a way of quantifying the fact that students should have the opportunity at university to learn and understand the very boundaries of knowledge and understanding in their chosen subject and learn from those involved in defining and discovering the new boundaries of knowledge and understanding. The REF scores would tell some of this story, but it occurred to me that the impact scores are probably a more effective and targeted way of measuring the ability to communicate and engage students with world-leading research.
  • Enrichment: it is widely accepted that a students’ academic experience is defined not only in the classroom, but also the co-curricular activities that are available. We are already engaged in an initiative to gather all this data to inform the Higher Education Achievement Record (HEAR). Be it work placements, study abroad, community engagement, leadership of sports clubs or academic representation, these activities improve student academic development and make a significant contribution to an excellent learning and teaching experience. It would take some work to finesse our systems to gather this data, but it can be achieved and should make a useful especially feature of a TEF.

2. Output

By outputs, I mean demonstrable measures of students gaining knowledge and skills during their their studies. I think there could be three data sets that could be used here, two that we gather already and one that could be derived from existing data.

  • Learning Gain: this term means many different things to different people. As a school governor, I have become familiar with the concept of ‘value added’ where a contextual measure of a students knowledge, skills and ability on entry and then looking at progression from that at various points up to a pupil leaving the school. I am not at all convinced that a measure that looks at UCAS tariff through to degree classification is the most effective measure of learning gain, mostly because of all the problems we know exist with the comparability of degree classifications. I think a measure should be developed that looks at the entry qualifications that students arrive with and then how far along they get with their higher education. That way, universities who recruit students with no or few qualifications could be rewarded and see the recognition increase as students achieve higher level HE qualifications. For example a student with no qualifications attending a university and achieving a Level 5 HND would have travelled the same distance as a student with the traditional A levels  and then achieving a Level 6 BA (Hons).
  • Employment data: we currently have the Destination of Leavers from Higher Education Survey that captures reliable data about what students are doing 6 months after graduation. There is talk of improving this to use HMRC data to show the real earnings of student at 1, 3, 5 and 10 years post graduation which would be a positive development I think. Either way, this is a key piece of data to show what students are capable of doing and achieving after graduation and should be included in any measure of teaching quality.
  • Grades: so long as we have the old honours classification system the data is not comparable. But, if a concerted effort is made to move to the GPA system, we might achieve more comparability and be able to develop data that reflects and measures that.

3. Peer review

A purely data driven approach would not give sufficient space for input from peers. I include students as part of the peer community here, because I feel strongly that we should consider their feedback and views of the quality of teaching to be of equal, if not greater, value than that of peers because, above all else students are the only people who can tell us what it is like to be a student.

  • The National Student Survey (extended to PGT): I’m pleased that the recent review of the NSS appears to have concluded that the survey is immensely valuable and not in need of a radical overhaul, but needs refreshing to sort out some of the dated and more nebulous questions (personal development, anyone?!). So this survey and its outcomes should undoubtedly form an important part of the TEF, I would argue that it should be heavily weighted in comparison to other metrics too.
  • External Review: I am not advocating a return to subject review or inspection regimes of old, they ran out of steam over 15 years ago and everything we learnt about them then still stands (burden, bureaucracy, gaming the system, diminishing quality of the ‘inspectors/reviewers’). But I think a positive judgement from the university’s most recent QAA institutional review should be a pre-requisite for inclusion in the TEF, to demonstrate that core academic standards are being maintained and the teaching is built on solid foundations.

This is an attempt by me, from my perspective, to list what should be included in a Teaching Excellence Framework. Each of the individual sets of data listed above can (and I’m sure will) be picked off on its own and rubbished as an insufficient measure of teaching quality. This may be true, for example DLHE only tells you if students get jobs, it doesn’t tell you if they would have got them anyway or whether their degree actually helped them. BUT nested in with a number of other data sets (some that we gather already, some that we need to start gathering) it can start to piece together a rich picture of the quality of teaching and learning.

This approach of picking apart the minutiae has been very successful for the sector in the past, it may succeed again, but I am doubtful. We have an opening to engage positively and constructively with this initiative and to shape it; if we try to nit-pick and shower the whole thing in treacle, I suspect we’ll get the sort of TEF we deserve!

Teaching Excellence Framewpork – Jo Johnson’s speech to UUK is a game changer

Jo Johnson has made his second major speech since being appointed Minister for Higher Education today. Contents can be found here.

It is a substantial speech, I wasn’t there but judging by the tweets about the Q&A afterwards, he has really got to know his brief and means business.

The key part of the speech starts like this:

It is striking that while we have a set of measures to reward high quality research, backed by substantial funding (the Research Excellence Framework), there is nothing equivalent to drive up standards in teaching.

That is why my priority as Universities Minister will be to make sure students get the teaching they deserve and employers get graduates with the skills they need by introducing the Teaching Excellence Framework we promised in our manifesto.

My aims for the TEF are:

  • To ensure all students receive an excellent teaching experience that encourages original thinking, drives up engagement and prepares them for the world of work.
  • To build a culture where teaching has equal status with research, with great teachers enjoying the same professional recognition and opportunities for career and pay progression as great researchers.
  • To stimulate a diverse HE market and provide students with the information they need to judge teaching quality – in the same way they can already compare a faculty’s research rating.
  • To recognise those institutions that do the most to welcome students from a range of backgrounds and support their retention and progression to further study or a graduate job.

He does make it clear that the TEF will not replicate the bureaucratic burden of the REF, which is welcome.

In response to questions, he did make it clear that there would be incentives for teaching excellence. These will be consulted upon, but it is as clear an indication as any that this will somehow be linked to funding and/or student fee levels. I look forward to the consultation.

There’s a lot more in the speech too about access, standards, graduate prospects. All very interesting and balanced, I’ll try to post something later about these.

As a sideline, I mentioned in a post I wrote on WonkHE earlier in the week that HEFCE’s timing of its consultation on the future of QA was curious given that BIS was going to be consulting on the TEF, which is a much more substantial and significant change for the sector. This just reinforced that.

HEFCE hitting the reset button, don’t forget student engagement.

I’ve posted a piece on WonkHE this morning about HEFCE’s proposals for the future of quality assessment in England. I think it’s a mixed bag of good ideas, well meaning but flawed proposals, some contradictions and some very bad ideas. Overall, I think it has failed to produce proposals that would reduce the burden or cost of QA to universities, it merely shuffles the burden around.

One thing that I didn’t have space to comment on in that piece was the importance of student engagement. A great deal has been achieved on this over the past 10 years, not only in sharpening our focus on the student experience and student expectations but also involving students as active participants in quality assurance and enhancement. This is not very prominent in HEFCE’s proposals and could be one of those valuable features that gets thrown out with the bathwater.

Official: The partnership model is dead, because students eat Mars bars!

An interesting piece in today’s THE “Students’ unions face uphill battle in anti-consumerism fight”. It is covering an academic article that appears in this month’s edition of the British Journal of Sociology of Education, “Students’ unions, consumerism and the neo-liberal university”. It’s an interesting contribution, but one that I think it seriously flawed.

In the THE, one of the article’s authors is quoted saying “The irony for students’ unions is that one of the main means of retaining independence and being able to resist consumerist agendas, is by embracing commercialism and providing services to paying students”. What utter nonsense!

The first assumption that is made in the article is that consumerism or market-like behaviour by students is something new, a phenomenon that started in 2012, or perhaps the early 1990s. In fact many of the behaviours that are citied as all-of-a-sudden prevalent among students “see a degree as a private investment (rather than a public good); be prepared to accumulate significant debt in order to acquire it; and actively ‘shop around’, comparing institutions and courses to secure the ‘best’ possible education” can be identified in students as far back as 600BC in Ancient Greece (I talk about this and many other historical links through the centuries in one of the chapters in the book I co-edited that’s still available in all good book shops i.e. Amazon!). So this is nothing new, and needs to be accepted by the “we’re all going to hell in a handcart, HE doom” brigade if we are ever to build a meaningful and rich understanding of the very complicated relationship students have with their places of learning.

I feel the authors of this piece have made the error of confusing the multi-faceted and high complex relationship universities and unions have with an idealised, single communitarian model. The existing system is critiqued at length and often badged “neo-liberal”, but no alternative is articulated. The truth is that Universities, and for that matter Students’ Unions are very complex organisations and the relationship with students is equally complex.

At its core, a Students’ Union is established to advocate the views and rights of students, this is certainly best achieved through a spirit of partnership with the University. But unions have also developed sophisticated and valuable structures in addition to this, such as organising volunteering opportunities, job shops, sports and societies and commercial services that reflect the wider needs of students and the university community in which they exist. This article seems to assume that this activity is an unnecessary distraction from partnership. I don’t think it is! Students, while at university want a rich and varied set of experiences and it is perfectly acceptable and reasonable for a students’ union to play a role in providing them. If students want to eat Mars bars, then why on earth should a students union not help them to get one?!

I think also, the mistake is made of assuming that consumerism and partnership are mutually exclusive and totally incompatible. I don’t agree with this. If you look at the student journey through education, it is reasonable to conclude that they might make some decision in very similar ways, irrespective of whether it is a consumer or partnership model on offer. I’ve often shown the following table as an illustration of this when giving presentations on the matter:

Consumer Partner
Choice of a range of providers X X
Choice between a range of courses X X
Make informed choices X X
Free to chose course that suits you X X
Pay fee X X
You can provide feedback X X
You can suggest improvements X X
Provider is the expert X
You are responsible for your education X
If it breaks, you can take it back X

The key difference, as is highlighted at the bottom of the table is the level of responsibility students have for their own education. If students do not actively engage with the learning opportunities available to them at University, then they will gain nothing and will not be to go back and request a replacement of themselves after three years of study!