The TEF: What should it measure?

What should a Teaching Excellence Framework look like?

Jo Johnson, the new Minister for Higher Education, surprised nobody last week by asserting his determination to deliver the Conservative Party’s manifesto commitment to “introduce a framework to recognise universities offering the highest teaching quality”. This will now be known as the Teaching Excellence Framework.

He did not go into a great deal of detail in his speech about what the framework would look like or what precisely the recognition would be for high quality teaching, although he did indicate during Q&A that he was open to considering financial incentives and rewards. In fact the level of openness and willingness to listen and respond to what the sector thinks was quite striking and positive. I think it is imperative that the sector engages positively with BIS over this, because I’m not sure it is an initiative where a strategy of kicking it into the long grass or hoping for death by a thousand cuts will work.

I think a key consideration in developing the TEF ought to be that it is light touch. I’m sure all universities that participated in the REF would want to avoid replicating the bureaucratic burden involved in that. The TEF should be based on metrics, ones that already exist or could be gathered with relative ease. Jo Johnson was also clear about this in his speech when he said “I have no intention of replicating the individual and institutional burdens of the REF. I am clear that any external review must be proportionate and light touch, not big, bossy and bureaucratic”.

Based on that principle, I think the data should fall into three themes:

  1. Input
  2. Output
  3. Peer judgement

1. Input

By input, I mean the quality of the content and delivery of teaching and wider academic experience of students.

There are a number of data that could be used, or newly collected to inform this but I think we have one already in the bag and a couple of others that could be gathered quite easily if we try hard enough.

  • Qualified Teacher Status: HESA has already been gathering data on this, I know it’s not a measure the whole sector has united behind yet, but with one more push and some finessing of the HEA’s Professional Standards Framework, we could get some sector-wide, comparable data on this. There will be plenty of wailing and gnashing teeth about this I am sure, but it is notable that this came top of student expectations when HEPI surveyed them recently, we should meet that expectation, it won’t do anyone any harm!
  • Research Impact: I was trying to think of a way of quantifying the fact that students should have the opportunity at university to learn and understand the very boundaries of knowledge and understanding in their chosen subject and learn from those involved in defining and discovering the new boundaries of knowledge and understanding. The REF scores would tell some of this story, but it occurred to me that the impact scores are probably a more effective and targeted way of measuring the ability to communicate and engage students with world-leading research.
  • Enrichment: it is widely accepted that a students’ academic experience is defined not only in the classroom, but also the co-curricular activities that are available. We are already engaged in an initiative to gather all this data to inform the Higher Education Achievement Record (HEAR). Be it work placements, study abroad, community engagement, leadership of sports clubs or academic representation, these activities improve student academic development and make a significant contribution to an excellent learning and teaching experience. It would take some work to finesse our systems to gather this data, but it can be achieved and should make a useful especially feature of a TEF.

2. Output

By outputs, I mean demonstrable measures of students gaining knowledge and skills during their their studies. I think there could be three data sets that could be used here, two that we gather already and one that could be derived from existing data.

  • Learning Gain: this term means many different things to different people. As a school governor, I have become familiar with the concept of ‘value added’ where a contextual measure of a students knowledge, skills and ability on entry and then looking at progression from that at various points up to a pupil leaving the school. I am not at all convinced that a measure that looks at UCAS tariff through to degree classification is the most effective measure of learning gain, mostly because of all the problems we know exist with the comparability of degree classifications. I think a measure should be developed that looks at the entry qualifications that students arrive with and then how far along they get with their higher education. That way, universities who recruit students with no or few qualifications could be rewarded and see the recognition increase as students achieve higher level HE qualifications. For example a student with no qualifications attending a university and achieving a Level 5 HND would have travelled the same distance as a student with the traditional A levels  and then achieving a Level 6 BA (Hons).
  • Employment data: we currently have the Destination of Leavers from Higher Education Survey that captures reliable data about what students are doing 6 months after graduation. There is talk of improving this to use HMRC data to show the real earnings of student at 1, 3, 5 and 10 years post graduation which would be a positive development I think. Either way, this is a key piece of data to show what students are capable of doing and achieving after graduation and should be included in any measure of teaching quality.
  • Grades: so long as we have the old honours classification system the data is not comparable. But, if a concerted effort is made to move to the GPA system, we might achieve more comparability and be able to develop data that reflects and measures that.

3. Peer review

A purely data driven approach would not give sufficient space for input from peers. I include students as part of the peer community here, because I feel strongly that we should consider their feedback and views of the quality of teaching to be of equal, if not greater, value than that of peers because, above all else students are the only people who can tell us what it is like to be a student.

  • The National Student Survey (extended to PGT): I’m pleased that the recent review of the NSS appears to have concluded that the survey is immensely valuable and not in need of a radical overhaul, but needs refreshing to sort out some of the dated and more nebulous questions (personal development, anyone?!). So this survey and its outcomes should undoubtedly form an important part of the TEF, I would argue that it should be heavily weighted in comparison to other metrics too.
  • External Review: I am not advocating a return to subject review or inspection regimes of old, they ran out of steam over 15 years ago and everything we learnt about them then still stands (burden, bureaucracy, gaming the system, diminishing quality of the ‘inspectors/reviewers’). But I think a positive judgement from the university’s most recent QAA institutional review should be a pre-requisite for inclusion in the TEF, to demonstrate that core academic standards are being maintained and the teaching is built on solid foundations.

This is an attempt by me, from my perspective, to list what should be included in a Teaching Excellence Framework. Each of the individual sets of data listed above can (and I’m sure will) be picked off on its own and rubbished as an insufficient measure of teaching quality. This may be true, for example DLHE only tells you if students get jobs, it doesn’t tell you if they would have got them anyway or whether their degree actually helped them. BUT nested in with a number of other data sets (some that we gather already, some that we need to start gathering) it can start to piece together a rich picture of the quality of teaching and learning.

This approach of picking apart the minutiae has been very successful for the sector in the past, it may succeed again, but I am doubtful. We have an opening to engage positively and constructively with this initiative and to shape it; if we try to nit-pick and shower the whole thing in treacle, I suspect we’ll get the sort of TEF we deserve!

Teaching Excellence Framewpork – Jo Johnson’s speech to UUK is a game changer

Jo Johnson has made his second major speech since being appointed Minister for Higher Education today. Contents can be found here.

It is a substantial speech, I wasn’t there but judging by the tweets about the Q&A afterwards, he has really got to know his brief and means business.

The key part of the speech starts like this:

It is striking that while we have a set of measures to reward high quality research, backed by substantial funding (the Research Excellence Framework), there is nothing equivalent to drive up standards in teaching.

That is why my priority as Universities Minister will be to make sure students get the teaching they deserve and employers get graduates with the skills they need by introducing the Teaching Excellence Framework we promised in our manifesto.

My aims for the TEF are:

  • To ensure all students receive an excellent teaching experience that encourages original thinking, drives up engagement and prepares them for the world of work.
  • To build a culture where teaching has equal status with research, with great teachers enjoying the same professional recognition and opportunities for career and pay progression as great researchers.
  • To stimulate a diverse HE market and provide students with the information they need to judge teaching quality – in the same way they can already compare a faculty’s research rating.
  • To recognise those institutions that do the most to welcome students from a range of backgrounds and support their retention and progression to further study or a graduate job.

He does make it clear that the TEF will not replicate the bureaucratic burden of the REF, which is welcome.

In response to questions, he did make it clear that there would be incentives for teaching excellence. These will be consulted upon, but it is as clear an indication as any that this will somehow be linked to funding and/or student fee levels. I look forward to the consultation.

There’s a lot more in the speech too about access, standards, graduate prospects. All very interesting and balanced, I’ll try to post something later about these.

As a sideline, I mentioned in a post I wrote on WonkHE earlier in the week that HEFCE’s timing of its consultation on the future of QA was curious given that BIS was going to be consulting on the TEF, which is a much more substantial and significant change for the sector. This just reinforced that.

HEFCE hitting the reset button, don’t forget student engagement.

I’ve posted a piece on WonkHE this morning about HEFCE’s proposals for the future of quality assessment in England. I think it’s a mixed bag of good ideas, well meaning but flawed proposals, some contradictions and some very bad ideas. Overall, I think it has failed to produce proposals that would reduce the burden or cost of QA to universities, it merely shuffles the burden around.

One thing that I didn’t have space to comment on in that piece was the importance of student engagement. A great deal has been achieved on this over the past 10 years, not only in sharpening our focus on the student experience and student expectations but also involving students as active participants in quality assurance and enhancement. This is not very prominent in HEFCE’s proposals and could be one of those valuable features that gets thrown out with the bathwater.

Official: The partnership model is dead, because students eat Mars bars!

An interesting piece in today’s THE “Students’ unions face uphill battle in anti-consumerism fight”. It is covering an academic article that appears in this month’s edition of the British Journal of Sociology of Education, “Students’ unions, consumerism and the neo-liberal university”. It’s an interesting contribution, but one that I think it seriously flawed.

In the THE, one of the article’s authors is quoted saying “The irony for students’ unions is that one of the main means of retaining independence and being able to resist consumerist agendas, is by embracing commercialism and providing services to paying students”. What utter nonsense!

The first assumption that is made in the article is that consumerism or market-like behaviour by students is something new, a phenomenon that started in 2012, or perhaps the early 1990s. In fact many of the behaviours that are citied as all-of-a-sudden prevalent among students “see a degree as a private investment (rather than a public good); be prepared to accumulate significant debt in order to acquire it; and actively ‘shop around’, comparing institutions and courses to secure the ‘best’ possible education” can be identified in students as far back as 600BC in Ancient Greece (I talk about this and many other historical links through the centuries in one of the chapters in the book I co-edited that’s still available in all good book shops i.e. Amazon!). So this is nothing new, and needs to be accepted by the “we’re all going to hell in a handcart, HE doom” brigade if we are ever to build a meaningful and rich understanding of the very complicated relationship students have with their places of learning.

I feel the authors of this piece have made the error of confusing the multi-faceted and high complex relationship universities and unions have with an idealised, single communitarian model. The existing system is critiqued at length and often badged “neo-liberal”, but no alternative is articulated. The truth is that Universities, and for that matter Students’ Unions are very complex organisations and the relationship with students is equally complex.

At its core, a Students’ Union is established to advocate the views and rights of students, this is certainly best achieved through a spirit of partnership with the University. But unions have also developed sophisticated and valuable structures in addition to this, such as organising volunteering opportunities, job shops, sports and societies and commercial services that reflect the wider needs of students and the university community in which they exist. This article seems to assume that this activity is an unnecessary distraction from partnership. I don’t think it is! Students, while at university want a rich and varied set of experiences and it is perfectly acceptable and reasonable for a students’ union to play a role in providing them. If students want to eat Mars bars, then why on earth should a students union not help them to get one?!

I think also, the mistake is made of assuming that consumerism and partnership are mutually exclusive and totally incompatible. I don’t agree with this. If you look at the student journey through education, it is reasonable to conclude that they might make some decision in very similar ways, irrespective of whether it is a consumer or partnership model on offer. I’ve often shown the following table as an illustration of this when giving presentations on the matter:

Consumer Partner
Choice of a range of providers X X
Choice between a range of courses X X
Make informed choices X X
Free to chose course that suits you X X
Pay fee X X
You can provide feedback X X
You can suggest improvements X X
Provider is the expert X
You are responsible for your education X
If it breaks, you can take it back X

The key difference, as is highlighted at the bottom of the table is the level of responsibility students have for their own education. If students do not actively engage with the learning opportunities available to them at University, then they will gain nothing and will not be to go back and request a replacement of themselves after three years of study!

The future of quality assessment; a bigger role for the state? Dismantling brand UK?

An interesting post on WonkHE yesterday about the future of Quality Assessment.

There are clearly some interesting proposals brewing, and it is quite refreshing to know that sector bodies are willing to be creative and go back to core principles. The most important principle when it comes to the regulation of UK Higher Education, in my opinion, is that of institutional autonomy. The reason UK and U.S. Universities top every global league table going is because our universities, unlike the vast majority of other sectors, have had the freedom to develop and grow free from direct government intervention.

HEFCE is a government body, it’s is established by statute and reports directly to BIS. I was a little concerned therefore when it unilaterally announced, some months ago, that it was going to review ‘Quality Assessment’ in England. I posted about it on my blog.

The statute in the 1992 Further and Higher Education Act under which HEFCE gains authority for quality assessment is as follows:

(a)secure that provision is made for assessing the quality of education provided in institutions for whose activities they provide, or are considering providing, financial support F1. . .,

It is a little ambiguous, I think, but as I wrote back in October, it limits HEFCE to making sure that quality is assessed, it does not imply that HEFCE should be the assessor! A subtle but essential difference. The ambiguity does place a heavy responsibility on HEFCE and a big risk on the sector though; arguably HEFCE could extend it’s reach and interpret this statute in away that allows it to regulate and intervene directly in the management of quality in Universities.

I sincerely hope it doesn’t do that.

Some of the proposals that have been floated could see HEFCE, perhaps unintentionally, cross that line. Annual reviews of data will undermine existing internal processes and give HEFCE the levers to intervene in institutional prioritisation and management decisions. I’m not sure how dismantling external review by QAA and replacing it with a (presumably) ‘non-peer’ review as part of HEFCE’s five-yearly review of governance is going to lighten (or for that matter enlighten!) the system.

One final concern is the unity of the UK system. The existing QA system, just about, holds the UK together with English, Welsh, Scottish and Northern Ireland universities buying in to a broadly common system. Scotland walked away from the review at the outset, I’m not sure whether Wales and Northern Ireland have stayed the course. It would be bad news indeed if the brand of UK Higher Education were to be undermined.

Lots of detail and matters of significant principle to get stuck into.

UPDATED: A subtle shift in student visa policy?

Jo Johnson gave what I think is his first high profile speech a Minister for Higher Education yesterday at the British Council’s Going Global conference.

Full text of the speech can be found here.

Most it is blandly welcome, much as you’d expect from a new minister. Important to note the very warm words about “rolling out the red carpet for the brightest and best” and emphasising the Prime Minister’s personal commitment to this. But again we’ve heard all that before.

Reading the text closely though, I wonder if there is a subtle and important shift noted in this paragraph:

It is also right that we have reformed the student visa system to ensure that students who are not genuine cannot abuse the system. And we will take proportionate action to deal with overstaying wherever our new system of exit checks shows it to be an issue.

Maybe I’m reading too much into it, but this Is softer language than appeared in the manifesto and sort of implies that the focus will be on making the recently introduced system of exit checks work and checking whether/where there are problems before imposing a new draconian set of regulations on to Universities.

Ever the optimist . . .

UPDATE: David Cameron announced yesterday that he will be setting up a series of “Implementation Task Forces” to oversee delivery of key Government policies. There’s been a lot of media attention paid to the fact that he himself will chair the Immigration Task Force rather than Theresa May. What’s more interesting is the Terms of Reference:

Delivering annual net migration in the tens of thousands; controlling migration from the European Union, including through reforming welfare rules and reducing reliance on migrant labour; ensuring that there is a hostile environment for illegal immigrants at work, in housing, and in accessing services; clamping down on the exploitative employers and landlords who make it possible for migrants to undercut local workers; enhancing UK border security, and ensuring that visitors and immigrants who come to the UK contribute to growth.

No mention of student controls. Again, maybe I’m being overoptimistic . . .

Why there won’t be an HE Bill!

Throughout the last Parliament it seemed as though many in the HE sector were waiting with baited breath for the coalition government to publish an HE Bill. I always thought this was unlikely, and I still think it is now that a majority Conservative Government has been elected. There are a number of reasons for this.

1) The barnacle to beat all barnacles!

One of Lynton Crosby’s most notorious actions on taking over the Consevative Party’s campaigning operations in the UK was to clear the ‘barnacles off the boat’. By this, he meant removing all the clusters of policy and legislation that was getting the Government stuck in the mud but was not making a big contribution to the big agenda and messages that the Conservatives wanted to communicate, i.e. getting the economy on track. Who now can remember what all the fuss was about with forest sell-offs?!

I think this is one of the very best lessons anybody has taught a party of government, focus on your big ticket items because there’s only so much you can do in a few years. I don’t see why this Government would want to waste any political capital on legislating HE which never, ever gets positive coverage!

2) It is not required to deliver manifesto pledges

Read the Conservative’s manifesto. There is no mention of anything that would require legislation. The expansion of the sector to include private providers has probably not moved ahead as quickly as David Willetts would have expected in 2010, but it is progressing slowly. There have been numerous calls for an HE Bill to tidy up some of the regulatory anomalies that are left over now that the Governement barely funds teaching, most notably from the HE Commission and backed up recently in article by Professor Roger King.

The Conservatives have pledged to do not very much (directly) with HE and this is exactly what I expect to happen! There was some vague comment about a framework to recognise excellence in teaching. That won’t require legislation. The, regrettable, clampdown on student visas might need legislation but it will be steered through by the Home Office and not as part of an HE Bill.

3) HE has stayed with BIS

BIS, and its predecessor departments, has not been a legislative powerhouse. It is an influencing department, working with the sectors it ‘represents’ to bring about change. Sajid Javid as Secretary of State has given a raft of interviews since the election and every single one has focussed on his determination to cut red tape and reduce burden on business. Why, in that case, would the department trigger a piece of legislation that is all about creating and regulating a sector more closely?

4) It is not required!

The argument that is made by some in the sector, and by the HE Commission report, is that the landscape for HE has changed so much in the past 10-15 years, that a new legal settlement is required. I am doubtful. What has changed?

i) Fees and funding – yes students now pay almost all of the cost of their education, but that hasn’t changed anything that needs new legislation. The money now follows the student rather than coming in a block grant; there would need to be a vote in Parliament if the fee cap were to be lifted or removed, but that could be taken as a single issue. I suspect the only changes that will be made are to the thresholds for repayment of loans and to link the £9k to inflation; neither require a full piece of legislation and could be mopped up in one of George Osborne’s budget Bills.

ii) Legitimacy of Funding Councils? – now that HEFCE barely funds teaching, its authority to regulate the sector is gone, so the argument goes! The relevant law states that each funding council shall “secure that provision is made for assessing the quality of education provided in institutions for whose activities they provide, or are considering providing, financial support”. So HEFCE isn’t quite dead yet and we can see that in the way it has asserted its role in reviewing the work of QAA.

iii) Private Providers – yes they have come to the fore and a few private/for profit organisations have gained degree awarding powers. They are not funded by HEFCE but they are bound into the system because they need to be reviewed by QAA in order to gain Tier 4 sponsor status and to access student loans. A case is made that legislation is required to set the bar high for private providers to enter the market. It is already high and could be made higher without legislation! Degree Awarding Powers are actually very (and rightly) difficult to achieve. These are granted by the Privy Council, on the advice of QAA. The Privy Council could toughen its expectations and the level of scrutiny required without legislation.

5) We shouldn’t want this!

There is only one direction in which regulation of UK Higher Education could go. That is the same direction as pretty much every other HE sector in the world, where Governments and state departments regulate and interfere far more than we have grown used to in the UK. That would be extremely unwelcome. Any cosy ideas we might have that new regulations would ‘go easy’ on established providers and focus on new/private/for profit providers is nonsense and we should not delude ourselves.

So the current legislative and regulatory landscape is messy and unkempt, but it just about covers all bases and keeps Secretaries of State at bay. We should celebrate that.

The impact of immigration policy on HE

Screen Shot 2015-04-27 at 22.24.49

So, I got my moment in the General election limelight this week. Commenting to BBC Wales on the impact of the General Election on HE policy, in particular immigration policy.

The piece can be found here. Section on HE starts at 20m 30s.

What would UKIP do for Higher Education?

  

Who cares!! 0 out of 30

What would the Conservatives do for Higher Education?

 At the weekend, I posted some thoughts about what the General Election means for HE and promised to scour (aka quick search for key terms) through the party manifestos to see how they scored against the 4 critical policy areas (as I saw them). I’m giving double weighting to Tuition Fees and Immigration because I think they are the most acute areas of impact for the HE sector. On Monday, I scored the Lib Dem Manifesto, yesterday the Labour Party, today the Conservatives.

1) Tuition Fees: The Conservatives supported the establishment of the cross-party ‘Browne Review’ before the 2010 election, when the report was published the were in coalition government and had to implement the necessary reforms. The current regime falls short of some of Browne’s wilder recommendations, but signalled a major shift in policy as the variable fees cap was lifted to £9k meaning that almost all funding for tuition came from students with negligible state funding for some of the science subjects.

This policy has shifted the burden of University funding on to students and I must admit to sharing the concerns at the time that this would act as a deterrent to many able students, especially those from disadvantaged backgrounds. Fact is, the exact opposite has happened; student numbers have continued to grow, with rate of growth being particularly strong among students from non-traditional backgrounds. I think credit for this must go to David Willetts as the Conservative Minister for HE.

An important point to note is that because of this policy, in a period where public sector spending has been reduced dramatically, funding for teaching in HE has grown above the rate of GDP growth over last parliament. The graph below makes the point very clearly.

 

The manifesto effectively offers a continuation of this policy (although there is mention of some sort of teaching excellence framework which is worrying!) so this would offer the stability I think Universities desperately need in this area as well as the continuation of a successful regime. 5 out of 5

2) Research Funding: There is no commitment to safeguarding research funding in the manifesto. This is disappointing because the current government has been supportive of research and ringfenced the science budget. A number of specific initiatives such as the Royal Institute for Advanced Materials are reannounced but the heavy lifting looks as though it will come after the election when the Nurse Review of the research councils will require a response.

It looks like we will miss David Willetts’ influence and support here so a disappointing but balanced 3 out of 5.

3) Immigration: There was some vague hope that the Conservatives might see the light and be pursuaded by the Lib Dems to remove international students from the net immigration figures. No such luck! The manifesto, rightly emphasises the need to tackle abuse and I think there is some (limited) room for the government and the HE sector to be pleased with the work done to clamp down on ‘bogus colleges’ which were not helping to enhance the reputation of the sector and were undermining the case for a more liberal approach to student fees. 

Regrettably, that’s all that can be said that’s positive. There’s a random proposal to clamp down on London campuses, with no explanation as to why this might be necessary. A committment to continue to review Tier 4 status and introduce exit checks for international students will add further bureaucratic burden to universities and send the wrong signal to potential international students.

Hopeless; but a single point in recognition of success tackling genuine abuse of the system, 1 out of 5.

4) Membership of the EU: Universities need stability here and the Conservatives offer very little. I think the rhetoric has shifted a little over the past 12 months, Cameron is clearly signalling that he would campaign to stay in a moderately reformed EU and none of his negotiating points would have a major impact on HE. However, the very fact of a prolonged renegotiation and a divisive referendum would limit the UK’s access to EU research funding (which other European University would want to bid with a UK partner of our presence in the EU might be shortlisted?!). A slight lift because ultimately I think the Conservatives in government would negotiate and then campaign to stay ‘in’. 2 out of 5.

Overall 17 out of 30.