Arnold Kling  

Measurement Without Measurement

PRINT
Lighten Up on Happiness, Arnol... Economic Nationalism...

Bryan writes


But Arnold's prediction and the happiness literature are actually compatible.

I am not going to be trapped into talking about happiness research as if it makes behavioral predictions. Happiness research cannot make behavioral predictions at all. It consists of taking meaningless surveys, and the most it can do is make predictions about the "findings" of other meaningless surveys.

There is a classic paper in economic methodology by Nobel Laureate Tjalling Koopmans called "Measurement without Theory," which is an indictment of the "leading indicators" approach to macroeconomic statistical analysis.

I call happiness research "measurement without measurement," because the researchers are not really measuring anything. If you ask somebody to rate their happiness on a scale, you have no idea what the answer means. Are respondents reporting a feeling, or an evaluation of how they think they ought to feel? Are they reporting something instantaneous, or something that also combines looking backward and/or forward? Are they attempting to report their feeling/evaluation relative to themselves at some other time in life? Relative to other people at some time in their lives?

There is a classic problem in economics with comparing interpersonal utility. Bernie Saffran used to say that you could never disprove the hypothesis that all of the world's resources should be devoted to giving him ice cream. How do we know that the marginal utility to him of the millionth scoop of ice cream isn't higher than the marginal utility of one bread crumb to everyone else?

To my knowledge, happiness research has not solved this methodological problem. You have no more basis for saying that "Married people are happier than single people" than you do for claiming that Bernie's marginal utility from the millionth scoop of ice cream exceeds my marginal utility from the first bread crumb.

Happiness research is fascinating for the same reason that astrology is fascinating. And it is equally scientific.


Comments and Sharing


CATEGORIES: Economic Methods



COMMENTS (10 to date)
Constant writes:

Thank you. Finally somebody is pointing out the emperor has no clothes.

Mikael writes:

Well, first off Bryan was right in his comment. There's nothing in happinness research that contradicts why not more Layardians move to sub-saharan Africa. As Bryan noted, we compare ourselves to different reference points.

When I was an undergraduate I thought that being accepted to the Ph.D program would make me soo happy. Well, here I am, and I'm not really more happy than before. Making this step has also implied changing my reference point. I'm also of course aware of the fact that getting my first published papers will make me shift my reference point another step, and so on.

So, your example that would somehow show the contradiction was not a good one. The relative income hypothesis / reference points etcetera is perfectly in line with revealed behavior that Layardians do not move to sub-saharan Africa.

However, I also think that there is a big difference from this mere fact from going to policy implications from Happiness reserach á Layard. (which tend to be very leftist of course, but they disregard from all the negative dynamic effects from this)

I also clearly think that there are great problems with survey measures. When do we actually sit down to fill in a survey? Its not when I'm euphoric at least, probably I would take time to answer such a survey on an occasion when I'm quite bored. What does that do to my reported happiness?

Also, buying stuff is usually seen a giving a brief upward jump in my happiness. But then after a while that new car won't make me any happier than before (which I think is very true). But, current happiness research assumes away this fact that getting a higher absolute income will make it possible to buy more of those short-term material "happiness highs". Thus, if we in some way could measure accumulated self-reported happiness, I think that we actually would get a relationship of also higher absolute (not only relative) income with higher acculumated happiness.

This won't show up in the surveys currently used in the field. And it would seriously change the policy implications that Layardians try to make from the current research findings. In the future perhaps we could use some instrument tracking brain activity over some period of time that could test this.

Finally, I think the very negative attitute by many economists towards survey data and self-reported data is a bit troublesome. Sure, there are extremely many caveats here (especially in such things as self-reported happiness, health, satisfaction etc.), but I would argue that these research fields are also very aware of this, and at least spends a lot of time and effort in trying to understand these problems (see e.g. the CVM literature).

This is unfortunately something that often is lacking in other fields. I get the feeling that researchers that get their data from national accounts and so on sometimes treats them as given from god, despite that we all know about enormous measurement errors also with this type of data (especially in macro studies).

In introudction to econometrics students are teached about severe biases due to measurement errors, primarily of course if it is about the regressors. But then, how much time do the average researcher spend on this problem? Its being taught, but how well practiced?

I concur that we should be very, very cautious in interpreting extremely uncertain happiness data, but, the same cautious attitute would also be nice to see from "revealed preference" economists to a larger extent.

conchis writes:

I too am sceptical of interpersonal comparisons of happiness. However:

(a) A fair chunk of the happiness research doesn't make any assumptions about the interpersonal comparability of happiness anyway, as it is concerned with comparing the same individuals over time (See e.g. the excellent paper by Ferrer-i-Carbonell and Frijters (2004) "Does Methodology Matter for the Estimates of the Determinant of Happiness in the EJ). The basis for saying that "married people are happier than single people" is that the same individuals tend to become happier when they get married (and unhappier if they get divorced/widowed);

(b) That you can't disprove Saffran's contention doesn't make it any more likely to be true. It's bollocks, and we all know it is. The problem here is essentially one of measurement error. That can be a big problem, but don't make it out to be bigger than it is. For one thing, measurement error in the dependent variable doesn't bias your co-efficients. For another, happiness comparisons within cultures are likely to have less measurement error than than comparisons across cultures.

Will Wilkinson writes:

Arnold, I think you're swinging for the fence, and missing, when you could connect if you swung for a double.

The surveys aren't simply useless. There are very rough, and not very reliable instruments--but not useless. People who report themselves happier, whatever they mean by it, do tend to be in better health, etc. There is 3rd person corroboration. People tend to answer in the same neighborhood when given the test again. Some FRMI studies show that answering "very happy" is correlated with greater dopaminergic system activity, etc. This is enough to ensure that the surveys track something.

My problem with the surveys is that I think that the scales renorm over time--the goalposts move--and so the stability of the trends in rreported happiness is not a function of the stability of average happiness, but a function of the process by which we report. This doesn't rule out the surveys giving us meaningful rough correlations at any given time (wealthier/happier; healthier/happier; etc.) but it does make the surveys useless for identifying with any precision trends over time, or between very different cultures.

There is no way out of relying on self-reports in psychology, and they have taught us a lot. When people who say they feel very stressed out turn out to have high cortisol levels, that means something. A lot of times you simply can't tell what bits of the brain do unless people tell you what they are feeling at the same time as you are looking at which bits are active. People are not VERY reliable about reporting how they are feeling, but they are reliable ENOUGH to pin down meaningful correlations between brain activity, hormone levels, etc. And this provides a basis for deeper study of the phsyical processes that do not depend so heavily on self-reports. I'd like scientists to stop jacking around with the surveys and get serious about teasing out the complex physical composition of good feelings. Good science is better than bad science.

If you're willing to think of utility as some kind of subjective affective state, then you need to get over your beef against interpersonal comparisons. Subjective states aren't magic. They are caused by real physical processes, and variation in the intensity of these states is determined by variation in the physical processes. Anyone with an FRMI can prove that ice cream puts Saffran in just about the same physical state as it would put anyone else. Real feelings are cardinal and comparable, and there's just no way around that.

Jim Bim writes:

I'm happy when Arnold and Bryan get in to the fisticuffs. :-) Seriously, I agree that happiness research involves a lot of bogusness. Interesting to talk and read about happiness, but how the hell do you measure it?

Zac writes:

First, I will second Will's comments. Very good.

Now, Arnold needs to read over what he's writing and use a little introspection. Forget about fMRIs and neurology for a second, and read this over:

You have no more basis for saying that "Married people are happier than single people" than you do for claiming that Bernie's marginal utility from the millionth scoop of ice cream exceeds my marginal utility from the first bread crumb.

Clearly they don't teach common sense at MIT, and this is further evidence that just because someone has generally sensible, libertarian viewpoints doesn't mean there aren't at least a few issues where they are completely idiotic.

You should really be ashamed of yourself Arnold, and I mean that. There's just no nice way to put it.

Dan Landau writes:

Saying happiness research, which uses subjective data, has some value, is not sufficient to justify happiness research in economics. What the psychologists want to do with it is another matter.

If the data in happiness research can’t compete with GDP data, with all its problems, then why should economists care about happiness findings? If we could say,”People are happier with less income and thus better off with less income.” That would be proposition that should matter to economists. But happiness data are much too weak and subjective to duke it out with GDP per capita. So the fact that such data is not worthless doesn’t make it of value to economists.

Tom West writes:

Unlike, for example, straight mathematics, economists often promote policies that have *significant* effects on our lives. They do so because they believe that they can make people's lives "better". Since "better" is inextricably linked with "happier", it makes sense that economicists actually understand *why* they are promoting policies. It's sort of like doctors promoting particular lifestyles with only a vague idea of what makes for healthier individuals.

If economists would stick to simply modeling behaviour of economic systems instead of envangelizing certain political objectives, I'm certain that happiness research would be a lot less important to economists.

And don't get me started on revealed preference as a substitute for happiness research...

conchis writes:

Dan Landau reckons that "happiness data are much too weak and subjective to duke it out with GDP per capita".

To quote Tukey (1962): "Far better an approximate answer to the right question, which is often vague, that an exact answer to the wrong question, which can always be made precise."

GDP is a pretty abysmal measure of actual welfare. The fact that measured GDP may be a better measure of actual GDP than self-reported happiness is a measure of actual happiness is kind of beside the point.

I was under the impression that economics was -- to put it mildly -- a little on the "soft" side where being a science is concerned anyway. So why should it bother us automatically that there are fuzzinesses where happiness is concerned? Do I detect a bit of physics envy in the air?

Comments for this entry have been closed
Return to top