I was invited to participate in the academic reputation survey for the new Times Higher Education World University Rankings. (The invitation email said that I was “statistically selected” but gave no details on the sampling procedure.) So how does the survey look like?
It’s pretty basic. First it asks some background information of the respondent such as research area (I could choose Social Science -> Political Science), year of PhD, geographic region, affiliation, etc. Then it simply asks to name the ten “best” (no criteria given) universities for research and, separately, teaching in my region, and the procedure was repeated for the whole world. The end. (There is a last step in which respondents can weigh the importance of several factors for the comparison of universities, such as publication output, reputation, number of classes, faculty/student ratios, etc. However, this is not taken into account in the reputation scores as far as I can tell.)
I am disappointed. I believe that systematic comparisons of departments/universities can in principle be useful (I have done my small part in this regard), especially as the (international) mobility of students and researchers increases, but this specific survey seems too vague to be really informative. The reputation data are complemented with objective indicators such as publication output, but still I think that they could design a better survey. Given the importance that these rankings have in the eyes of administrators, students, and policy makers, they certainly should.