Bryan Caplan  

How to Fix Seminars

PRINT
Russ Roberts on Adam Smith's R... The Miracle of Compound Intere...
I'm sick of academic seminars.  Even if the presenter has a good paper and speaks well, the audience ruins things.  How?  Two minutes into any talk, the fruitless interruptions begin.  Half the time, they're premature quality control: "Are you going to deal with ability bias?"  "Uh, yes.  On the next slide."  The other half the time, they're bizarre pet peeves.  "How does this relate to sequential equilibrium?"  "Uh, it doesn't.  It's an empirical paper."  "Yes, but that reminds me of something Rudi Dornbusch once said.  Now I'm going to talk for four minutes."

Once fruitless interruptions begin, there's no such thing as winning.  There are only different degrees of losing.  Even if you limit each question to a single minute of time, your ideas lose their flow.  A well-planned talk handles complexities in logical order, allowing relative novices to follow your reasoning.  Premature quality control smashes that order to pieces, leaving novices confused and experts bored.  And after three pet peeve questions, the audience struggles to remember the topic of the talk - never mind its thesis.

Can't speakers deflect the pointless interruptions to make the talk train run on time?  It's tough.  You can't just say, "Please, only good questions."  And once you take one silly question, every other squeaky wheel feels entitled to put in his two cents.  "Hey, my question's clearly better than the last guy's!"  As a speaker, then, you have to choose between offending a fifth of the audience or boring the entirety.  Professionally speaking, the former is the greater danger.

Still, I come to fix seminars, not to abolish them.  My colleague Dan Klein inspired the following alternate rules:

1. Split the talk into two parts.  Part 1 is the first two-thirds of the allotted time.  Part 2 is the last third of the allotted time.

2. During Part 1, the audience may not ask any questions.  No exceptions.

3. However, the speaker retains the option to ask the audience questions during Part 1.  If the speaker sees a lot of confused faces, he can query, "Are you familiar with the efficiency case for Pigovian taxation?" and adjust his presentation accordingly.

4. The speaker scrupulously ends Part 1 on time, then turns the rest of the talk over for questions. 

Step 4 has two main advantages over the standard method. 

First, it's easier for the speaker to filter out bad questions.  Since there is a dedicated question period, multiple people will normally raise their hands.  If an audience member asks bad questions, the speaker can call on other people first.  If an audience member asks too many questions, the speaker can gently say, "Someone who hasn't already asked a question..."

Second, the question pool will generally be better.  Once the audience understands your point, they can ask questions about the overall thesis rather than Slide 3.  Quality control questions now serve their legitimate function: If you missed your opportunity to address a tricky point during Part 1, audience members can correct your oversight in Part 2. The pet peeves still make a spectacle of themselves, but at least the audience walks away knowing your thesis. 

A few months ago, I saw Dan Klein use a similar format, though he made Part 1 shorter and Part 2 longer.  This Wednesday, I tried it at the Public Choice Seminar.  It could easily have been the best academic seminar I've ever done.  The approach makes so much sense, it makes me wonder if the standard approach is ever better.

My best guess: The standard approach might be better for graduate students.  Suppose you're a public speaking novice.  Interruptions are an opportunity to improve your performance in real time.  Toastmasters clubs, for example, appoint an "Ah-Counter" to clap whenever the speaker says "ah" or "um."  Alternately, suppose you know less about your topic than several experts in the audience.  Their interruptions are an opportunity to prevent you from derailing your own talk with errors and irrelevancies.  The background assumption here, though, is that the speaker and the audience both constructively treat the seminar as training.  Shredding graduate students helps no one.

Comments on my seminar system are good, but field experiments are better.  Have I piqued your interest?  Try my approach and tell me how it works.


Comments and Sharing






COMMENTS (25 to date)
Jody writes:

That's SOP in engineering talks at conferences. 20-25 minutes for the presentation followed by 5-10 minutes for questions.

You still horrible questions, but you get to give the presentation the way you intended.

Fabio Rojas writes:

The interrupathon is specific to econ programs. What you describe (deliver talk, then have questions) is completely normal in the rest of academia.

Michael Makovi writes:

I have a suggested improvement for this:

"3. However, the speaker retains the option to ask the audience questions during Part 1. If the speaker sees a lot of confused faces, he can query, "Are you familiar with the efficiency case for Pigovian taxation?" and adjust his presentation accordingly."

Instead of the speaker having to look for quizzical looks, let the audience ask questions during the talk, but ONLY questions that clarify things they simply don't understand. Explain that in-talk Q&A is only for essential clarification while post-talk Q&A is for further elaboration.

If anyone abuses this and asks the wrong kind of question, they will be subject to justified social shaming.

I was just at a talk where the speaker kept using a certain acronym I wasn't familiar with. If I hadn't been allowed to ask him what the acronym meant, I'd have had to put on a really quizzical look and hope he noticed.

Tyler Cowen writes:

I find this an oddly non-Caplanian post. Maybe other speakers aren't, on average, as good as you and deserve to be interrupted rather rapidly.

Fabio Rojas writes:

Tyler, no they don't!!! Interruptathons are lame and are about ego. If they speaker is lame, you will have more than enough time to dismember the argument later.

My thoughts in detail:
https://orgtheory.wordpress.com/2010/03/25/econ-rules/

blink writes:

It sounds like the problem is poor colleagues. Presumably, the listeners have invited the speaker, yet insist on detracting from the public good they have collectively procured.

How about an audience agreement? Impose a tax on questions during the first X minutes of a presentation so that people internalize the cost of spoiling the commons. This is also more flexible than your command-and-control policy that simply prohibits questions.

Alternatively, there is the "Chicago" style in which everyone actually reads the paper ahead of time. Then the entire talk consists of questions and rebuttal.

Vera writes:

The rules for economic seminars may be the same almost everywhere (questions allowed through the seminar) but the reality varies wildly with department culture. When there is a culture of asking "good" questions (which doesn't work as an explicit rule but is just fine as a norm, since most people do in fact know the difference!) and saving others for the end, things work beautifully.

On the other hand, changing culture is a lot harder than setting up some rules that, when inevitably enforced imperfectly, get closer to this ideal than the status quo...

david writes:

I think Cowen is onto something: if you're a really great speaker, it's on you to move beyond formal seminars and into organizing your own research networking.

JLV writes:

Actually, this practice (interruptions) ably contributes to Econ being relatively insular. Other researchers outside econ who cover similar topics are simply unwilling to present in front of economists after being rightly horrified at the way they act in seminars.

Ben writes:

Interesting. I can confirm Fabio's statement (hi Fab!) for biology: I have never seen a bio seminar interrupted by questions except extremely brief clarification questions (i.e., "what is the x axis here?").

Jonathan Goff writes:

Bryan,
I have to echo Jody's and Fabio's points about this sort of interrupting behavior not seeming to be standard in other fields. The approach you suggest of having Q&A at the end seems to be the standard elsewhere (in my case various engineering disciplines). Other than in conference panels I've never se en much in the way of interruptions.

Jon

Robinson writes:

I agree with other commenters that in other academic fields, the "no-interruptions, then questions at the end" format is standard. It certainly is in my own (genomics), and as soon as you started describing the interruption-based format all I could think was "That sounds terrible!"

Jared writes:

Bryan - as several have already said, you're reinventing the wheel here. Before I came to academia, nobody did during regular business/corporate/political presentations what economists do during seminars. And in academia, only economists do this. Have you considered, maybe, just maybe, that the problem is just with economists? Because that's what my spouse suggested, and I think she's entirely correct. Seminars work fine; economists are broken.

David Friedman writes:

I prefer what I think of as the Chicago style seminar. The participants read the paper. The speaker has a short time, fifteen or twenty minutes, in which to talk about his paper and need not take questions. Since the participants have read it, he isn't trying to present the paper. Thereafter it's open season for questions and responses. The closest thing I have observed to multi-mind thinking in real time.

My favorite example of the superiority of this approach was a Law and Econ seminar at Stanford, which follows a watered down version of the Chicago rules—the speaker, in practice, is often allowed as much as half an hour, sometimes more. Before the speaker arrived, I mentioned to someone else that there was an error in the paper, equations in which a payment was made but not received, and correcting it reversed the sign implied by that part of the argument. One of the other people had also spotted the error. A few minutes into the seminar, Ken Arrow raised his hand to point out the same error.

This was not the first time the paper had been given. But I presume it was the first time it had been given under rules that resulted in a significant number of people actually reading the paper. If you first encounter a bunch of math as the presenter goes through slides, you are quite likely not to notice mistakes.

Philemon writes:

Since we are sharing experiences...

In my own department (a philosophy department), we let the speaker present for up to 50-55 minutes, take a 5-10 minutes break, and all reconvene for q/a. Very brief purely clarificatory questions are very rarely fielded during the presentation stage, but the chair reserves the right to shut them down. For q/a, interested parties raise their hands for new and substantive questions/comments, and a finger for a quick follow up to someone else's question/comment. But once you do a follow up, then if your name was already down for a full question, you go to the back of the queue. It's the chair's job to keep track and give the go ahead to questioners. It's ok to quickly follow up one's own full question/comment after the speaker has already answered, but chair always reserve the right to shut a particular back and forth down in the interest of time so that others may join the conversation. I won't say it's anywhere near a perfect system (I think it's probably too time intensive), but so far, we have had many excellent presentations and discussions. The downside is that duds last just as long.

My favorite multiple presenter workshop format however, is for all papers to be circulated beforehand, with each assigned to specific commentators. At session itself, the assigned commentator will begin by briefly summarizing the key points of the paper and raise the first questions/comments, followed by a brief rejoinder by the paper writer, followed by open discussion. No one gets to read his or her own paper. It's really intense and fruitful when people actually all come to the workshop having read each others' papers beforehand. But seldom easy to arrange.

Toby writes:

I am also partial to this (Chicago) approach, the question I wonder about however is what rules you should have for which group of individuals given the purpose of an academic seminar (from whose perspective though?). If few people have the time to read the paper, then such rules will probably attract very few participants and possibly a disproportionate number of those with comparatively more time available. How valuable will those comments be to the speaker compared to the alternative where different rules exist?

Tracy W writes:

I'm in the energy analysis business, and we do seminars on our work, and take questions throughout. We cut off the people making very long questions (aka speeches) but otherwise it works reasonably well.

Nick writes:
Can't speakers deflect the pointless interruptions to make the talk train run on time? It's tough. You can't just say, "Please, only good questions."

I had a professor who put off all questions to the end of a lecture, but once in a while when someone would interrupt with a hand up, would call on them, let them ask their question, and reply (something like): "There is no such thing as a stupid question, only stupid people who feel it is appropriate to waste the time of the 40 people gathered here by not waiting for the appropriate time to ask their question. Now where was I...."

He would probably get in trouble for that kind of thing today, but it was very effective at stopping interruptions.

EclectEcon writes:

Like several of the others, I prefer the Chicago style of seminar wherein attendees are expected to have read the paper in advance. The questions are usually better, and the seminars are intellectually exciting and challenging. In these seminars, people who haven't read the paper in advance are expected not to ask questions until near the end of the seminar. I sat in on Fogel's economic history workshops at Chicago and much preferred that style to the "sit and listen" seminars elsewhere.

Michael Sullivan writes:

It's been a long time since I was in academia but in educational seminars in professional settings, what you propose is the standard. Presenter may encourage participation and questions at their option, but normally there is a presentation phase, controlled by the presenter. then a question period. For longer talks, there may be multiple sections, each with a presentation and question phase. Unsolicited questions that are not very quickly answerable are deferred to the question period.

The idea that one would give a seminar intended to present a concept/paper (as opposed to a talk where the presenter is being taught or tested, such as a oral exam or thesis defense) where the standard culture is that the presenter is not expected to control the meeting, and defer most/all questions to the end seems crazy to me.

Jameson writes:

I wanted to join the chorus of other academics asking what is wrong with economists. In mathematics, it is a pretty rare seminar that gets interrupted by such annoying questions. The occasional question in the middle is usually somewhat useful to help people understand what is going on, and otherwise all questions are saved until the end.

Borrowed_Username writes:

The answer is simple:
Have some kind of chat room or reddit-like thread open where people can post questions and you can see them on a screen. Between slides look down, if there's something important that you're not already going to answer, answer it, if not, move on.

Musca writes:

Perhaps a Robin Hanson-style analysis makes sense here - question periods are information-gathering opportunities for some, signaling opportunities for others. In academia and corporate life, the proportions of each depend on the specific topic, discipline, and temperaments of the participants. Written questions could indeed reduce signaling opportunities to zero, therefore reducing presenter annoyance.

Maximum Liberty writes:

I had a professor in college who started a lower division undergrad class with a prohibition on any questions until the first six weeks of class were done. He explicitly said, "Until then, you don't know enough to ask questions that are worth the time of the rest of the class." You could always ask him after class, of course. True to his word, after six weeks, he started reserving time for questions at the end of class. He started that day with, "Remember that, about six weeks ago, I told you ..." He was right.

Seth Green writes:

I was reminded of this post when I went to a talk at my university's econ department this afternoon. The model was clear and transparent, the empirical work very well thought out and exciting, and the interruptions were just endless. in the poli sci department I'm in, there's less of that.

Comments for this entry have been closed
Return to top