The lead article in the 25th anniversary issue of the peer-reviewed Journal of Public Affairs Education , by Heather Campbell, is a deep-dive review of research that throws significant shade on student evaluations of teaching (SETs). SETs do not measure student learning, and may actually have the wrong sign (not to mention that their inherent gender, age, and racial/ethnic bias means their use for personnel decisions is probably illegal). Universities like my own, in which SET scores are the main, or often the only, teaching evidence used in promotion and tenure decisions are systematically damaging student learning.
From the abstract:
In many if not most colleges and universities in the United States,
raw scores from Student Evaluations of Teaching (SETs) are the
primary tool of teaching assessment, and teaching evaluations
often have real consequences for promotion and tenure. In 2005,
JPAE published an article on teaching evaluations, and this article
added to what was at that time a somewhat thin literature indicating
that SETs are systematically biased against female faculty,
and probably against older and minority faculty. Since that time,
this literature has swelled and grown and now the evidence that
SETs are invalid and systematically biased is too strong to ignore.
Why do we keep doing this? I have four hypotheses:
(1) A body of research of which I am unaware and can’t find refutes the findings Campbell summarizes. This is certainly the most generous conjecture, but JPAE is peer-reviewed and reviewers would have rejected this paper if they knew about such material. Not to mention reviewers for all the publications Campbell cites.
(2) Confirmation bias. We used to think SETs were valid indicators, and we subconsciously reject evidence that would change our mind. This mode of inference has recently been validated by our US president, who “just knows things” that are contradicted by every kind of expertise and evidence.
(3) Fecklessness and laziness. We assert our commitment to good teaching if asked, but actually we just don’t care enough to do anything that would actually advance it. The joke is, “Teaching is the tax you pay to do your research; tax evasion is a crime, but tax avoidance is the duty of a citizen” and the corollary is, “why are you talking about whether students are learning, when I have a research article to finish writing for my academic peers to admire?” SETs take a distasteful task (collegial responsibility for better teaching) off my desk and load it onto an unpaid, docile labor pool (students); what’s not to like?
(4) Fear. I have never received evidence I can respect as a scholar, or any other way, that I am any good at teaching or could become so (I do have evidence of this kind that I can do research OK and that I’ve gotten better at it over the years). Teaching is affectively fraught, and like everyone I know, I’m sure my emotional intelligence is not what it should be. The ego hit of talking while a roomful of people write down everything I say (lecturing, just as an example of a dubious pedagogical habit) is a lot to risk by trying to learn a new skill. Anyway, improving my teaching will take a lot of time and my job depends on publishing.
We would be a lot better off if we could shift our attention more generally from summative evaluation (at promotion time) to formative methods (coaching and experimentation between these high-anxiety moments). I suppose one could believe that college teachers only respond to money and status incentives, so if we reward good ones and ding or fire bad ones, we will eventually have only good teaching, but one would be wrong (ask any successful manager whether you can fire (or bribe) your way to success). One would be especially wrong if your filter for “good teachers” doesn’t measure student learning.
What we need is not a cheap, lazy way to pretend we are improving our teaching, but a real quality assurance program that a Google or Toyota manager, for example, would recognize as such. Got kids choosing a college? on your junior year visit, ask what their QA program is, be sure it doesn’t depend on SETs, and don’t be distracted by the fancy athletic facilities. Are you a student, paying through the nose with your time and money for the best possible education? Do the same, and if you don’t get good answers, recruit your classmates to go in the quad with pitchforks and torches.
Interesting that there is a claimed bias against female teachers when the overwhelming majority of students are female.
“If you don’t get good answers, recruit your classmates to go in the quad with pitchforks and torches.”
Addressing an angry crowd is perhaps the ultimate test of public speaking. The record is held by the Emperor Claudius’ Greek freedman Narcissus who broke a mutiny of four legions at Boulogne in AD 43 just by addressing them. (One despised foreigner and ex-slave against 20,000 angry and heavily armed professional soldiers in one of the best armies in history).
Wikipedia says it was done by humour:
You have to suspect that Narcissus staged the intervention to break the ice and gain a hearing. He still had to win the legionaries over and get them on to the invasion ships with some mixture of flattery, bribes and threats. He couldn’t use threats much, as the legions were themselves the Empire’s ultimate threat.
The analogy with teaching is imperfect. The object of rhetoric is persuasion, in a good or bad case. The teacher’s goal is to elicit thinking. There are common elements: gaining attention and a hearing is one.
I will say that, to the extent you’re right, it applies mostly to research-oriented institutions. My career was spent at teaching-oriented institutions, and my experience was that not only were factors other than CTEs important in the assessment of teaching, the importance of non-CTE factors has increased over time.
(Just a reminder, if it’s really needed) than not all of us-indeed, likely not even most of us-word in research-first institutions.