Bryan Caplan  

Letting Students Drop a Question: A Big Mistake

PRINT
Goldberger Explains Why My Ref... I Don't Understand Mark Thoma...

On many exams, professors give students e.g. 4 questions, then say "Answer any three." The point, I suppose, is to:

1. Avoid penalizing students for random blind spots.

2. Create safety valve for badly-written questions.

In practice, though, it seems like giving students a choice of which questions to answer makes exams a noisier measure of student knowledge. The problem: Empirically, there is a very low correlation between "knowledge of the subject" and "ability to identify the easiest questions." So bad students often hone in on the easy questions and get full credit, while good students often try to do the hard questions and fail.

It's particularly bad when some questions are so easy that every student who attempts them gets 100%. Then students who fail to notice how easy the question is (and given time pressure, it's easy not to notice that a question is easy until you try to answer it) get much lower scores largely because of luck.

In any case, if you're worried about problems (1) and (2), there's an easier solution. For (1), just give a large number of smaller questions. Then random blind spots won't matter much. And for (2), just grade on a curve. Then the occasionally badly-written question won't lead to low average performance. (Again, this works better if you have lots of small questions instead of a few big questions).

Anyone care to defend the "drop a question" approach?


Comments and Sharing





COMMENTS (23 to date)
Jonathan Palmer writes:

First - Why would a good student hone in a harder question? As a good student, I have enough sense about me to logically choose the question that I am going to answer the best, to get the most credit. It seems very illogical to do anything different.

Now my question, if you are worried about random blind spots, what is the solution if the class material is not well suited to more frequent, shorter questions?

Stephen writes:

Mr. Caplan, I think that you forget that what question is 'easiest' is entirely subjective and dependent on the individual knowledge of the student. Perhaps when we are comparing two math problems, we can determine that one question is more difficult than the other. But when we talk about other fields such as history, the question of what question is easiest is subjective. For instance, I took a final in a history class that dealt with European diplomacy in the nineteenth century. On this exam we were given the choice between two lengthy questions, one dealing with European imperialism in Africa and the other dealing with Bismarck's alliance system. I wouldn't argue that one question was easier than the other, but rather the material was presented to me better in one and I could articulate my position better.

Furthermore, I challenge the assertion that good students will attempt tougher questions and fail while bad students will attempt easier questions and succeed. A student does not have good grades just because they are knowledgeable of facts, but because they know how to play the 'game.' To put this in utility terms, it makes no sense for a good student to attempt a 'tougher' question unless they feel like they can receive the same or more credit than they would for answering the 'easier' question.

-Stephen

Zach Wendling writes:

Letting a student drop a question is one way of giving an examinee a sense of control over an otherwise stressful situation. In one respect, this might alleviate anxiety -- in another, it may exacerbate it (Did I choose the right one?).

It might also give the professor yet another out when it comes to post-exam complaints: "Don't blame me, you chose the questions you would answer."

Peter Twieg writes:

I have to concur with Jonathan - I don't think good students are particularly likely to tackle hard problems for the sake of tackling hard problems. I don't see doing so as particularly virtuous, unless you think that students should be rewarded not based on whether they actually fulfill the demands of the question but how much they can show off in doing so.

But one alternate method which is commonly employed would be to have the students try to answer all the questions, and then simply ignore the lowest score and grade him/her based on the best three. This largely retains the advantages of the "3 of 4" paradigm and wouldn't penalize people for choosing tougher sets of questions.

agent00yak writes:

In defense: You want to see if the student can present an in depth answer to a question and time for test taking is limited.

Your point is more generally applied to making tests "simpler" to avoid measuring test taking ability. However, this isn't easy as there are almost always ways to take a test more efficiently. Your solution may even be counter-productive. A lot of smaller questions might actually be easier for a good test taker as they have even more options to allocate their time efficiently. Reducing the total number of questions could simplify the test even though it adds the variable of "choice".

Besides, you can always curve the questions. The grader can always be instructed to grade the "easier" question more harshly.

Ben Kalafut writes:

In some fields, e.g. physics, the reducting the "size" of the question can't be done without making the test meaningless.

Moreover, "curving" the exams is inappropriate in such a subject where there exists a fairly objective standard of understanding.

A meaningful physics exam problem almost always requires the student to be "clever"; the student might understand the concept rather well but not immediately see the application to this problem or that. In that case, allowing the student to drop a problem is a safety valve accounting for the non-straightforward nature of the subject.

This is perhaps a stronger argument for the take-home exam, but too many students in service classes at megauniversities can't handle that kind of responsibility honestly.

drobviousso writes:

In defense: I'm in favor of these kind of questions in advanced classes where students could likely have a variety of backgrounds. Both EE's and Software Engineers will probably have to understand how the network stack works, but EE's will be more focused on the hardware and lower layers while the SE will be more focused on the upper layers. I don't know econ education well enough to come up with an example for you there.

As a student, I think I was generally pretty good, and would never, ever, ever have taken the harder question 'just because I'm a good student'. In fact, I'd consider the ability to identify an easy question part of being a good student. This is a skill that may or may not make you a good economist/software engineer/manager, etc, but there's easier ways to find out if a student has this skill.

Alex J. writes:

I think Bryan's argument is true as far as it goes. Sometimes, the teacher wants the student to write a long-ish essay. This is obviously in conflict with a large number of small questions.

grant writes:

I think it would be interesting to conduct a series of exams where some number of questions can be dropped from each, and then give a final exam consisting solely of the questions that each person dropped from the previous exams. Or, if you don't have the time and ability to make individualized exams, set an exam that has the most frequently dropped questions on it.

Ironman writes:

Just out of curiosity, how many students end up answering the same three questions when given the option to drop one of four given questions?

Acad Ronin writes:

It's a bother, but you can standardize the scores on each question (i.e., subtract the mean and divide by the standard deviation). That brings all the grades to the same metric, units of standard deviation. That adjusts nicely for difficulty and means that you are summing appropriatly. If you simply sum across the questions done, you are letting variance decide the weighting. I assign a number of very short quizes (5-10 T/F or multiple choice questions on the readings and the last lecture) a semester, and permit students to drop almost half. Standardizing is the only way to make aggregating fair.

wintercow20 writes:

I think you are on target. At times in the past I have given students an optional extra credit question. You get credit for getting it right, but lose credit for getting it wrong.

rubemode writes:

Brian,

My econ prof had a combined use tactic. Her exams were like 7 global questions, broken down into parts a,b,c, etc. Then you were told to pick 6. The tough part was each a,b,c of each question had ranges of difficulty. So to get the easy a & b parts was mixed in with a tough c, so it wasn't always easy to determine which of the 7 global questions were the easier 6. You might actually take a question that had three medium parts, to avoid a question that had two slam dunk parts and one you had no idea on.

liberty writes:

I will defend.

I like a bit of flexibility; it makes the test more personalized. Tests can be so one-size-fits-all. Students have different backgrounds, different strengths, and different focuses.

When you use "pick 3" (or "drop one", which is a bit less defensible perhaps than "pick 3 and if you do all 4 I will grade the first 3") then students can hone in on the ones that interest them more, and do a better overall job on the group.

Why have them spread themselves thin, when they can feel more confident, focused and happy by picking their favorite 3, instead of being stuck in a meat-grinder forcing them to do exactly the same as every other student?

dave smith writes:

I think knowing things is important.

But knowing what you don't know is sometimes just as important. Allowing a student to drop a question adds this dimension to what your test is measuring. They must decide which question they would have the most difficulty.

This is a iteration of what I tell my students: if you walked out of the exam thinking you did well and bombed, you are in worse shape than the student who know he did badly and bombed.

Dan writes:

When the drop-one strategy is employed, it is usually because the test questions are tricky, confusingly worded, or can take a long time to answer. I see no clear advantage or disadvantage to students compared to grading on a curve - which is the other strategy in this situation.

I can defend this position as it has advantages for instructors, three to save time and two to improve quality:
1. As Zach Wendling wrote above: this is a way to eliminate student complaints about test question wording, saving instructor and class time. (I have seen instuctors cut off all discussion this way.)
2. This also eliminates student complaints about not having enough time to complete the test.
3. Instructors spend less work on developing the test questions, since each student will be selecting the easiest questions.
4. Instructors can analyze which questions are skipped and try to re-work or replace them - improving future tests.
5. Instructors can put in tricky questions to reward students able to intuit the solution (which may be desired).

Bryan,

I would appreciate it if you would post a follow-up with details on "Empirically, there is a very low correlation between 'knowledge of the subject' and 'ability to identify the easiest questions.'"

I assume you have some specific research in mind and I'm curious about it.

Because my experience is different. Granted, my "experience" is subject to confirmation bias and other memory biases, but I find that better-prepared students tend to avoid the hardest questions while less-well-prepared students tend to try them with greater frequency. A choice of questions helps do, therefore, what you want a test, primarily, to do: identify the students who know more.

And it seems intuitively plausible: if you are poorly prepared, virtually all of the questions seem difficult; if you are well-prepared, a lot of them should seem easy and the more challenging questions should stick out.

Finally, in my experience, students overwhelmingly prefer tests with choice. It seems to be a relatively low-cost yet ethical way to help them feel a little better.

Craig

floccina writes:

This shows that we do not test what a person should know becuase it is important for him in life (my guess is that this would make the tests to easy for the class to be considered rigerous). Instead we test to see who is smarter/more diligent.

We need a separation of teaching/educating and testing/credentialing.

Since tests in themselves are stupid, the drop a question (or questions) makes it better. You'd probably be horrified to know that many disciplines give the questions to the students before the exam. Since most testing in the social sciences and humanities is a worthless brain dump, it makes more sense to have students focus on a question and come prepared with a better answer.

Overall I think this works differently in economics, math and the sciences. In history, for example, it's completely impossible to get 100% on all the questions anyway.

I'd still say that grading on the curve is the stupiest innovation in academic history. If someone is such a bad teacher that they can't get their students to understand the material then they should punish themselves not make pretend grades.

Tom Crispin writes:

Some real-world examples:

Suppose you are teaching a history course with a large supplemental reading list. You require answers to any three of four short essay questions. Each question can be best answered by material from different of the supplemental readings, but you don't expect your students to have read all the supplemental material.

Alternately, you give an open-book take-home final with seven qeustions in an advanced math course. The students are told that answering the first five correctly will get an "A" on the final; the other two (much harder questions) are for extra credit and can compensate for earlier, poorer, grades. What you are actually looking for is the "A" student who works them anyway, for the love of the subject.

Steve Miller writes:

I let students drop a question because it helps me learn precisely which questions are the easiest. Then I try to make them harder the next semester. We haven't all been teaching for decades, you know!

Glen writes:

I've used the "drop a question" approach many times on exams. While I can't say I've collected statistics systematically, my general impression is that you're wrong about the lack of correlation between knowledge of subject and ability to identify easier questions. On the contrary, I often see the duller students choosing the harder questions.

When I give this kind of exam, I tell students that if they answer more than the specified number of questions, they must indicate which answers they want graded. Surprisingly often, I see students (usually the poorer ones) choose to exclude an answer that would have gotten them more points than at least one of those they chose to have graded. I take this as an indication that while they may know the answer to a given question, they lack confidence in their knowledge. I don't mind punishing the lack of confidence.

Bob Hawkins writes:

The tests I've taken like this, were tricks. Of the four questions, three could be done by the average-to-good student but would take some time and effort. One could be done quickly and easily if you had insight or had done work beyond that assigned in class, otherwise not at all.

Your reward for reading and remembering the paper from which the trick question was cribbed, is you now have the time for 2.5 questions, to do 2.0 remaining questions.

Comments for this entry have been closed
Return to top