It is standard to hear a manager in the English Premier League say “the table doesn’t lie” as they bemoan their lowly position or celebrate their success. By contrast it has been equally standard to hear university recruiters put the case that various league tables are wanting in terms of nuance, specificity or even veracity. But it may become even more complicated if university league table compilers have a direct, commercial interest in the outcome of the table and its impact on students.
In a recent article in The PIE, the Chief Development Officer of Times Higher Education (THE) outlined plans for millions of international students who consult its rankings website each year. He said, “We want to stay top of the funnel and maximise the number of students coming to the site. What we will then do is identify a network of complementary, trusted partners that we will send those students to.” The potential for universities to find themselves excluded or obliged to pay large sums for access seem obvious.
Regulators, governments and the sector’s networking bodies would do well to consider whether this manipulation of the recruitment process through commercially driven league tables is in the interests of the institutions and the students. Back in October 2018 the Office for Students Director of External Relations wrote of the “challenge for policymakers….providing information responsibly and well as accessibly” but it is difficult to see any action to head off the private sector. Allowing brands that have been built with substantial public funding to be used as click-bait providing a return to private money certainly does not seem the best way forward.
Selective, Subjective and Subject to Manipulation
It is equally troubling to think that students may find themselves railroaded into choices by an organization that decides how the league table is compiled and has commercial partners who may have more than a passing interest in the result. Elsevier have quoted Lydia Snover, director of institutional research at the Massachusetts Institute of Technology, as saying, “every ranking is based on the available, comparable data, and is built on the subjective judgement (over indicators and weightings) of its compilers.”. Even when league tables are independently audited, consulted upon and done with good intentions they are about choices.
UC Berkeley’s Center for Studies in Higher Education has suggested that “universities with frequent QS-related contracts experienced much greater upward mobility in both overall rankings and in faculty-student ratio scores over five years in the QS World Rankings”. HEPI’s president, Bahram Bekhradnia, did not find this a surprise and noted, “QS is a commercial organisation. They’re there to make money and their rankings are not objective.” The higher education sector, while complicit in working with rankings media, is aware that this is a double-edged sword, and it may be that commercial imperative is sharpening one side.
Those factors are made worse by the documented cases of universities deliberately manipulating the data they submit in order to secure a place higher up the ranking. A University World News article in 2019 highlighted how the University of Oklahoma had been supplying US News and World Report rankings with incorrect information for nearly two decades. Occasional errors seem forgivable but the more complex and wide ranging the tables the more scope there would seem to be for manipulation.
Legitimization and Lost Perspective
It seems a long time ago that in the late 1990s a few national UK newspapers would produce university league tables once a year as part of their wider agenda of news coverage. But since the early 2000s league table compilation and publication has become increasingly central to the activity and business model of some HE sector-oriented media organisations. Universities have played their own part in legitimizing the ranking races that may undermine their reputation and their ability to compete for students.
Many university planning offices have also spent hundreds of hours analysing league table performance and working to advise their senior colleagues on the levers that can elevate the institution’s position. It would be difficult to believe any Vice Chancellor who says their university’s league table performance is not considered in strategic discussions. League tables have become silent and increasingly oppressive enforcers influencing decision making, reputations and student experience.
It is certainly plausible that one of the factors influencing grade inflation at UK universities has been the weighting of a ‘good degree’ in the league tables. When one university sees a perceived competitor getting league table marks for awarding a higher proportion of ‘good degrees’ the argument to amend marking criteria can be positioned as not disadvantaging students. Almost without realising it institutions and academics may find their autonomy compromised by external factors.
Methodology, Misalignment and Misunderstanding
Over and above that, the dizzying array of league tables has become a way for compilers to open new routes for advertising income and securing influence. Universities under 50 years old may welcome the chance to trumpet their performance against similar institutions and it allows the sector to applaud its own achievements. But when high placings are used as advertising and marketing fodder to attract students the institutions are validating a process which is almost entirely out of their control and where interests may not be aligned.
In 2004 the Times Higher Education (THE) began its University World Rankings but that has now been joined by 18 other main categories including World Reputation Rankings, Young University Rankings, Emerging Economy Rankings, Subject and Teaching Rankings. The latest addition of Impact Ranking assesses universities against the UN Sustainable Development Goals (SDGs). The accompanying launch events, announcements and conferences drive substantial content, which may be the purpose of media organizations but that is not the same as the purpose of universities.
The QS Rankings also began in 2004 and now covers 11 main categories, with several similar to THE but some noticeable differences such as Employability and System Strength. They have built a student-facing event series – the QS World Tours – to bring students together with admissions directors at events. Conferences and consultancy services also build out of the rankings as a source of revenue.
The variability of methodology that universities are trying to deal with shows in the league table results. The THE and QS most recent “top 10 global universities” and “top 10 under 50 years old” show seven as being the same in each category but three different. It’s a discrepancy which seems unhelpful if you are a student really wanting to know which were the best of breed in either category.
So, even when compilers are gauging similar categories they are making subjective choices about what to include, how to weight it and whether it will be important to their readers. But in what is largely a game of statistical musical chairs there is some evidence that there are also fundamental misunderstandings about what is driving the performance of institutions. Research by QS has suggested, for example, that students believe that a university’s ranking is substantially linked to employability of graduates when this only makes up a small element of the overall score.
It seems indisputable that league tables have become very big business for organizations that compile them and are influential enough to be a source of power over university decision making. The prospect of them being leveraged to influence student choice and the recruitment potential of institutions has been made clear. An informed, open discussion leading to collective action by the sector would be a step towards restoring balance.