Arnold Kling  

Reliability of Survey Data

PRINT
Discover Your Inner Literalist... High School Curriculum...

Tom Slee writes,


what I get from the Netflix prize is that there are probably significant limits to recommender systems. Even the smartest don't do a whole lot better than the simple approaches, and a lot of work is required to eke out even a little more actual information from the morass of data.

This is from a much longer post, that I got to via Brad DeLong from Tyler Cowen.

The gist of it is that Netflix offered a prize to someone who can come up with a better algorithm for predicting movies that people will like. Slee gives all sorts of examples of suspect data, including users who have rated thousands of movies.

Slee and Cowen see this as a sign that recommender systems will fail. I totally disagree. My guess is that statistical models that recommend movies will succeed extremely well, but not when they are based on survey data.

Suppose that instead of ratings, you asked consumers to vote with dollars. For any movie, a consumer who owns the movie can post an "ask" price and a consumer who does not own the movie can post a "bid" price. To ensure that these prices are real, every once in a while, you would fill all the orders for a particular customer--that is, you would buy the consumers' DVD's at her asking prices and sell her DVD's at her bid prices.

My guess is that this approach would generate better data than Netflix's current process. Perhaps I am wrong about that. But the point is, if you had good data, meaning data that is based on revealed preference rather than survey ratings, my guess is that recommender systems would be quite powerful. It's the garbage data, not the concept of statistically-based recommendations, that limits the ability of the Netflix system.

The larger point, of course, is that subjective survey data tends to be garbage. Which is one of my issues with happiness research.


Comments and Sharing


CATEGORIES: Economic Methods



COMMENTS (5 to date)
Dan Weber writes:

Assuming that getting people to "bid" on movies would be good for coming up with a total quality metric for movies, that's not what Netflix wants.

Some people like Waterworld. Some people don't like On Golden Pond. The point is to recommend movies to people based on what other movies they have liked or not liked.

awoolf writes:

If Netflix only allowed people who rented a movie to recommend it, that would solve the problem. The only cost is that Netflix would have to base its recommendations on fewer votes (I wouldn't be able to recommend Citizen Kane since I haven't rented it from them).

testcase writes:

Your proposed solution suffers from the same problem as your criticism of surveys of attitudes on America you mention in the linked article - the "revealed preference" is simply not measuring what you believe it is.

The fact that people outside the US have negative attitudes about US foreign policy or have problems with US popular culture in no way conflicts with peoples' desire to live here. A preference to live somewhere is not contradictory to a negative opinion of how that country behaves internationally.

In much the same way the question of what movies one would recommend is different than the question of what movies one would wish to own. For instance, I would highly recommend Schindler's List but I have no desire to own it simply because I do not see myself repeatedly watching it. Since Netflix is in the business of movie rentals (typically one viewing) not movie sales (for multiple viewings) I don't see you proposal as much use to them.

I think this all points to a much larger problem. Many of these decisions are more complicated than revealed preference would suggest. In the end, statistical methods which can't include why one has a preference for certain movies (or countries) are of very little value in looking at these more complex behaviors.

Bruce G Charlton writes:

Re: Happiness research - would it work, in a national survey for example, if you offered a prize (or let people bet) on the average level of national happiness?

So - if you guessed closely enough the answer of the survey, you would get a prize (or you could bet on the answer, and the odds could evolve, maybe).

I haven't really thought this through - but might it work?

Horatio writes:

I've always thought Netflix could use more graduations in its rating system. They should move to 10 or even 20 stars in order to let people really make distinctions between films they believe are very good and those they believe are true works of art.

Comments for this entry have been closed
Return to top