Category: Education

Do professors care whether college students are actually learning?

The lead article in the 25th anniversary issue of the peer-reviewed Journal of Public Affairs Education , by Heather Campbell, is a deep-dive review of research that throws significant shade on student evaluations of teaching (SETs). SETs do not measure student learning, and may actually have the wrong sign (not to mention that their inherent gender, age, and racial/ethnic bias means their use for personnel decisions is probably illegal). Universities like my own, in which SET scores are the main, or often the only, teaching evidence used in promotion and tenure decisions are systematically damaging student learning.

From the abstract:

In many if not most colleges and universities in the United States,
raw scores from Student Evaluations of Teaching (SETs) are the
primary tool of teaching assessment, and teaching evaluations
often have real consequences for promotion and tenure. In 2005,
JPAE published an article on teaching evaluations, and this article
added to what was at that time a somewhat thin literature indicating
that SETs are systematically biased against female faculty,
and probably against older and minority faculty. Since that time,
this literature has swelled and grown and now the evidence that
SETs are invalid and systematically biased is too strong to ignore.

Why do we keep doing this? I have four hypotheses:

(1) A body of research of which I am unaware and can’t find refutes the findings Campbell summarizes. This is certainly the most generous conjecture, but JPAE is peer-reviewed and reviewers would have rejected this paper if they knew about such material. Not to mention reviewers for all the publications Campbell cites.

(2) Confirmation bias. We used to think SETs were valid indicators, and we subconsciously reject evidence that would change our mind. This mode of inference has recently been validated by our US president, who “just knows things” that are contradicted by every kind of expertise and evidence.

(3) Fecklessness and laziness. We assert our commitment to good teaching if asked, but actually we just don’t care enough to do anything that would actually advance it. The joke is, “Teaching is the tax you pay to do your research; tax evasion is a crime, but tax avoidance is the duty of a citizen” and the corollary is, “why are you talking about whether students are learning, when I have a research article to finish writing for my academic peers to admire?” SETs take a distasteful task (collegial responsibility for better teaching) off my desk and load it onto an unpaid, docile labor pool (students); what’s not to like?

(4) Fear. I have never received evidence I can respect as a scholar, or any other way, that I am any good at teaching or could become so (I do have evidence of this kind that I can do research OK and that I’ve gotten better at it over the years). Teaching is affectively fraught, and like everyone I know, I’m sure my emotional intelligence is not what it should be. The ego hit of talking while a roomful of people write down everything I say (lecturing, just as an example of a dubious pedagogical habit) is a lot to risk by trying to learn a new skill. Anyway, improving my teaching will take a lot of time and my job depends on publishing.

We would be a lot better off if we could shift our attention more generally from summative evaluation (at promotion time) to formative methods (coaching and experimentation between these high-anxiety moments). I suppose one could believe that college teachers only respond to money and status incentives, so if we reward good ones and ding or fire bad ones, we will eventually have only good teaching, but one would be wrong (ask any successful manager whether you can fire (or bribe) your way to success). One would be especially wrong if your filter for “good teachers” doesn’t measure student learning.

What we need is not a cheap, lazy way to pretend we are improving our teaching, but a real quality assurance program that a Google or Toyota manager, for example, would recognize as such. Got kids choosing a college? on your junior year visit, ask what their QA program is, be sure it doesn’t depend on SETs, and don’t be distracted by the fancy athletic facilities. Are you a student, paying through the nose with your time and money for the best possible education? Do the same, and if you don’t get good answers, recruit your classmates to go in the quad with pitchforks and torches.

An aspirational goal for teaching (more music)

Andy Narell frequently plays and teaches with young people. Here he is visiting with the UNT steel band, playing with their admirable jazz ensemble. Everyone here is making music at a high level, but compare the affect-expressions, body language, everything-of the kids on the left side of the screen (who are actually wearing uniforms, symbols of identity suppression and servility) with those on the right (who are not). Which would you like your students to display?

How do we make this happen in, say, a statistics class?

Museums behaving badly

I love museums. Science museums, history museums, art museums; there’s nothing like looking at real stuff in person. Whether it’s an antique automobile, a big old beetle in a case, or the Ardabil carpet in the V&A, being able to walk around it, get close, and engage on my own time is one of my top-level pleasures.  I’m sure I learned as much natural science in the American Museum of Natural History as a child as I did in school; whenever I’m traveling, I make a beeline for local museums.

The affection is not entirely requited in art museums, mainly  because so many of them transparently disrespect me (and all the other visitors) by pointless, insouciant, arrogant stinginess with the information that makes the art accessible.  This weekend I was at the Huntington, the Getty Villa, and LACMA in LA. The Huntington and the Getty do a pretty good job with long, informative labels that provide context, history, and some guidance about what to attend to in the works on display, but LACMA left me really steamed.

A featured exhibition was several galleries full of contemporary political art by Iranians that reached back to the Shahnameh for analogies and references, a show with appropriate local interest (there are lots of Persians in LA, including refugees from before and after the shah’s overthrow) and in any case an interesting and fruitful concept.   You should go and see it, but unfortunately you will miss a lot unless you’re already hip to recent (and ancient) Iranian history, and can read Farsi. The labels were tiny and short and one after another very political work full of incriptions, signs, and text in Farsi was untranslated. One faceplant in particular seemed to sum up art museums’ worst instincts to make not only the typical visitor, but almost any visitor, feel unqualified and inadequate.

The work, by Koushna Navabi, is a couple of dozen rings in different metallic finishes with the same portrait, a little over an inch each way:

This is the entire label we were offered:

Know whose portrait this is? Only because I’m old enough to almost remember the period, and spent some time in Iran after the coup that overthrew him, I recognized Mohammed Mosaddegh, about whom Wikipedia says “Many Iranians regard Mosaddegh as the leading champion of secular democracy and resistance to foreign domination in Iran’s modern history. Mosaddegh was removed from power in a coup on 19 August 1953, organised and carried out by the CIA at the request of MI6.” In  the oppressive regime of the shah and his SAVAK secret police that followed, Iran’s oil remained in the hands of western oil companies and their US and British protectors. Of course by 1979 this arrangement went off the rails because the Iranians had had enough of it.

Any of that useful in engaging with this work? Or is the (I presume) affectionate but rather obscure pun in the title all you needed? I hung around and asked at least a half-dozen visitors if they knew whose portrait was on the rings; none had any idea. Here’s what the curator thought she was doing with this show; I’m sure her middle-east specialist colleagues were impressed, but an exhibition like this is a lot of work: I guess she just didn’t have a minute to actually think about the visitors who would walk in the door.

Actually, Don’t Follow Your Dreams

I often listen to books on tape. The ones you pay for generally have excellent readers, like the magnificent George Guidall. But the free ones (e.g., through LibriVox) are of uneven quality. I am in the midst of listening to a famous Victorian Era novel through LibriVox, and some of the chapters are read by a woman with such a thick accent and poor command of English words and cadence that I keep laughing because her rendition reminds me of this Monty Python sketch.

But it’s no big deal. People can certainly read the book on their own if they don’t like someone else’s reading, and she’s not harming herself. She’s a volunteer doing something she finds intrinsically enjoyable in what I suspect is her retirement. Indeed I get the sense she really admires the author of the words she is mangling.

But it makes me think of a situation I and many other professors encounter where the stakes are much higher, namely when someone is at the beginning of their adulthood, and just not that good at what they want to do in their career. What is my or any other faculty member’s responsibility to the pre-med student who dreams of being a surgeon but is better suited to being a barber? What onus is on the high school basketball coach to tell his star player that he’s never going to be an NBA player, so he should focus instead on doing well in math, science and language class?

This is a very un-American topic to raise. “Gotta have a dream” American movies are a dime a dozen, and virtually all of them feature a scowling older person who tells our young hero/ine that s/he’s never gonna be a movie star/top athlete/brilliant scientist/successful musician etc. Most professors don’t want to be that scowling figure; it’s easier to tell everyone to go confidently in the direction of their dreams, even when we suspect it’s going to be a train wreck (Food for thought here: I never should have followed my dream).

A couple years ago I was talking to a high-ranking professional staffer at one of the country’s leading universities. I asked him how he chose his career, and he said he owed it all to the mentor who told him in graduate school that he was never going to achieve his ambition of becoming a chemistry professor at a great university. His mentor told him, compassionately, that he didn’t have the scientific chops or drive it took to make independent research breakthroughs, and that he would probably end up at a community college or low-tier 4 year college where he would be required to teach all the time (and he didn’t like teaching). He dropped out and is now in a very different but well paid and meaningful line of work. His tribute to his beloved mentor stayed with me: “Thank God he didn’t believe in me”.

I don’t claim certainty about how faculty can know in advance for which mentees unwavering support is the right medicine and for which it is cruel. But I do feel pretty confident that American academic institutions tend to worry less than they should about leading on optimistic, unrealistic, dream-chasing students.

Berkeley athletics task force ducks engagement, steams onto rocks

Berkeley’s Task Force on Intercollegiate Athletics (TFIA)  has submitted its report  on what to do about an enterprise that soaks up tens of millions of dollars of subsidy each year while the university as a whole is being asked to eat almost $200m of budget cuts (see the TFIA assignment below). It’s appropriate to thank the members for the time they spent on this project. I wish it were possible to recognize their efforts as consequential or useful, but no such luck.  My expectations were modest, but this result dashes even my cautious hopes, and as I know several of the members to be smart people with the best intentions, it saddens me to say so. Eleven members, working for almost a year, have come up with five single-spaced pages of content (no, this is not the executive summary, and it is not a memo boiled down for Donald Trump), and one big table of financials uncritically assembled from IA’s annual NCAA P/L reports.

A bitter irony is the appendix listing (without links to access any of them) five previous TFIAs, from 1991,’92,’99,2000 and 2010. Isn’t a standard definition of insanity “doing the same thing again and again expecting a different result”?

Continue reading “Berkeley athletics task force ducks engagement, steams onto rocks”

Languages

This is a very tough post to write, because I have to confront a collision of legitimate values I hold strongly.  I just received an invitation to sign the petition here. I am not going to sign, even though I am the world’s biggest fan of learning languages. I regret that among the six with which I have some competence, none is really foreign (to me, that is, non-Indo-European). I think real command of at least one foreign language, meaning conversational comfort, writing a business letter, and reading a novel, not passing a written exam, should be a graduation requirement at any college that respects the idea of a liberal education. Requirement, period. I deplore the feeble command my students have of languages they have studied for two and three semesters in courses.  And by the way, every new language is easier than the one before. Continue reading “Languages”

Why colleges don’t “host” speakers—and why it matters.

Those of us whose ceilings are not silicates mostly know that raucous protestors this past Thursday—some, though probably not all, students—prevented Charles Murray from speaking at Middlebury College, and confronted him in ways that amounted to assault (injuring a professor who was with him) when he tried to leave.

There’s no lack of commentary on this; I have little to add to its substance. Like many, I think that judging Murray a poor scholar and a vicious racist (not far from my own opinion) does not constitute even a weak case for shouting him down or trying to beat him up. But less attention has been paid to how we talk about invited speakers in the first place.

Consider how Time described a letter from alumni, put out before the event, deploring the Murray invitation

In a letter published Wednesday [the link to the letter has since been moved to here—AS], more than 450 Middlebury graduates called the college’s decision to host Murray ‘unacceptable and unethical.’

The college’s decision to host Murray. I understand how a tweeter or journalist trying to save words would say for short that Middlebury “hosted” Murray. (Later in the piece, Time similarly paraphrases Middlebury’s spokesperson as defending “the decision to host Murray”). But that locution is completely misleading and extremely pernicious.

Colleges and universities officially host commencement speakers and a handful of other speakers (e.g. when a speaker series is called a “university lecture” or is part of a “Presidential lecture series”). But otherwise, universities don’t host speakers at all. The hosts are, on the contrary, the independent entities that essentially do the actual educating at universities and to which universities supply, in effect, a common bureaucracy, some imperfect quality control, and a crucial brand. Those entities include academic departments; university- or grant-supported centers and academic programs; and, as in this case, student groups. Only such units, and not “the university,” should be held responsible for their diverse and independent judgments regarding whom they decide to invite. Continue reading “Why colleges don’t “host” speakers—and why it matters.”

Why “Sweet Spots” Are Neglected in Political Debates

Over at The Incidental Economist, I have a post on “sweet spots” in public policy. When you understand sweet spots, you will know that many absolute policy positions many people adopt (e.g., More spending on schools will help children learn more, cutting healthcare spending doesn’t harm health, more cops reduce crime, increased incarceration doesn’t reduce crime etc.) are all, for lack of a better word, wrong. The same policy can be effective, ineffective or counter-productive depending on whether the current intensity at which it is applied is already in the “sweet spot” or not.

If you want to review examples of this phenomenon from education, crime, health and labor policy, please see the TIE piece. What I am writing about here is why the sweet spot reality is so underappreciated when people argue sincerely about public policy (I emphasize sincerely because of course some people know about sweet spots and ignore them anyway to further their own agenda, but here I am talking about people who truly believe what they are saying).

Our cognitive system has frailties. Kahnemann and Tversky showed how vivid, easily-recalled examples seem more representative than they are. Someone may “know” that more education spending always helps kids because they had a highly memorable experience of seeing a crumbling, underfunded school being turned around by additional investment. In contrast, a different person may be aware of a vivid example of where massive spending (e.g., Mark Zuckerburg’s $100 million dollar gift to Newark’s school system) didn’t help kids learn at all. By definition, single examples are going to be sampled from a range of possibilities that may or may not be in the sweet spot for a given policy, and because we are prone to misjudge such easily recalled instances as representative, we are often insensitive to counter examples that fall outside the range whence our example comes.

Continue reading “Why “Sweet Spots” Are Neglected in Political Debates”

Another Cal athletics moment

With our intercollegiate athletics department’s typical management finesse, we extended our football coach’s contract about a year ago…and fired him this week. He walks away with about $6m in severance, but that’s OK because we’re just finishing up the zillion-dollar severance payments for the previous coach, and the AD who is now at Penn State, so there’s lots of money just lying around that would otherwise be wasted on fixing classrooms, or scholarships for non-athlete students who just play sports for fun and don’t put any eyeballs on TV commercials. The intercollegiate athletics program at Cal costs about $30m a year (net), a sixth of a $180m campus deficit; a task force of alums, faculty, and staff is working on proposals to fix this.

The athletic director shares a set of insights that deserve attention, and translation:

We are continuously evaluating our program and looking for ways to make it better - whether that’s through additional academic support, recruiting, facilities, staffing, culture, leadership or anything else that can help our football program succeed. [1] Primarily, we want what’s best for our student-athletes [2] and have a head coach in place who is fully committed to our program  and our university [3].

….Our objective is long-term financial sustainability for our department. In order to do this, we understand that investing in football is critical [1]. We believe that this change will reinvigorate the program, stimulate lagging ticket sales and renewals, and energize our donor base. [4]

….We want to win championships. The success of our football program is vital to both our department and our university community [5], and its influence can be felt well beyond Berkeley.

1: Almost everything in this list costs money, and we intend to keep spending it no matter what that task force says, or what weird mission the outgoing or incoming chancellor thinks a university has.  It’s our tradition of a decade here at Cal to keep throwing money at a mediocre football program, and we take our traditions seriously. Sooner or later, maybe as little as $6m later, inshallah, the larger forces of big-time college sports will abate, the bleeding will stop, and we will reach some sort of equilibrium, right?

2: To be clear; we retain the services of the conditioning coach who killed one football player, sent another to the hospital, and cost us $5m in a liability settlement. We certainly aren’t going to rein in practice times so they can sit around in classrooms or labs, or do a bunch of wussy problem sets. “What’s best for our student-athletes” is not what people outside the cult might think the phrase means.

3: This is just sports PR blather, of course, the language of press conferences and after-game interviews; $3m college coaches are fully committed to their careers and if anything, expect their employers to be committed to them with limitless staff, facilities, money, and perks. Pete Carroll’s effortless leap to the Seahawks from the shambles he left at USC is instructive.

4: Chronicle reporting is occasionally sloppy, and in this case we are not informed whether Williams clicked his heels together three times and closed his eyes as he said this last.  Nor whether the “we” actually includes any living person on earth with a three-digit IQ.

5: This combines a statement of fact with a religious utterance based on faith. No, it’s not vital to the community, not even close, though its ruinous cost certainly inflicts a lot pain on the rest of us. My department just completed a faculty search and not one of the candidates asked about the prospects of the football team or even knew our record. We lost a prof to Stanford a few years ago, and in all our discussions of his move he never once brought up Stanford/Cal football. I have talked to dozens of undergraduates and grad students over the years and not found one who came to Cal because our football (or men’s basketball) teams were better than those at other schools to which they were applying. I have been here semester after semester, talking to colleagues in social science, humanities, and hard science across that university community, and the salience of football in our socialization, community spirit, and plain water-cooler schmoose is similar to the salience of pro wrestling. Big-time sports may be ‘vital’ for Clemson or Florida State, but not for us.

What can we expect in the near future?  The new coach will need to do some really desperate recruiting of defense players at the least, and develop a quarterback (unless he decides to bring in a graduate transfer ringer as Dykes did this year).  The program will be even less attractive to high school stars, however, so I have probability of about 0.3 that we will be reading about recruiting violations of the type fictionalized in the memorable movie Blue Chips, or recruiting and oversight failures of the type that recently humiliated Baylor. Meanwhile, ticket sales will keep going down, the athletics deficit will grow, and the new chancellor will find his feet stuck in the big muddy from his first day.

An Anecdote About Campus Microgressions and Intolerance

On a brisk autumn evening, I was about to cross Campus Drive when I noticed an acquaintance waiting for the shuttle. He was a retired Eastern European diplomat and consequently had something of an old world feel about him. After we shook hands and started chatting about politics, another acquaintance happened by. She was a graduate student versed in women’s studies.

After I had introduced them, the diplomat whipped off his glove and extended a hand for her to shake. He had kept his glove on while shaking my hand, and the female graduate student noticed the difference.

If you have been reading Conor Friedersdorf’s or Greg Lukianoff and Jonathan Haidt’s writings about campus culture in The Atlantic, you can imagine what happened next.

The women recoiled from the gendered micro-aggression and lambasted the diplomat: “Do you think women are too frail to touch a gloved hand or is this some kind of creepy come on?!!”.

The diplomat’s face registered shock and he began stammering “I didn’t…I don’t understand..”

The student continued “Well you should understand and it’s not my job to educate you. I don’t have to put up with your patriarchal bullshit!”.

Red-faced and near tears, this gentle, cultured man apologized repeatedly to both of us and retreated down the sidewalk, so upset that he forgot his briefcase on the shuttle stop bench as he fled.

Continue reading “An Anecdote About Campus Microgressions and Intolerance”