The Significance of Changes in the Gender Happiness Gap

Last week, Betsey Stevenson and I finished a paper highlighting “The Paradox of Declining Female Happiness”. In short, we show that women have become less happy, relative to men. You can read the paper here (and subsequent discussion by Alex Tabarrok, Steve Levitt, or the NY Times). The paper is about documenting the trends. Interpretation is for you, dear reader. And from the 700 comments generated at the NY Times, it appears that there is plenty of interpreting to do.

Additionally, we have upset the linguists. From Language Log:

OK, everybody, take a deep breath and listen: THERE IS NO HAPPINESS GAP!

Every year since 1972, the General Social Survey has been asking a big demographically-balanced sample of American men and women "Taken all together, how would you say things are these days? Are you a) very happy, b) pretty happy, c) not too happy."

Neither in 1972 nor in 2006 was there any statistically significant difference between men and women in the distribution of their responses! And in both 1972 and 2006, the proportion of women who said "very happy" was a little bit higher than the proportion of men who gave that response (though again, in neither year was the difference distinguishable from chance fluctuations).

So what is everyone talking about? Well, some economists fit a complicated statistical model (called an "ordered probit") to the whole sequence of survey results from 1972 to 2006, and this analysis suggests that women have become a little tiny bit less happy relative to men over that whole time period. But the effect is so small that you can't actually see it in the statistical analysis for any one year; the effect is much smaller than the amount of year-to-year jiggle. That's true even through the General Social Survey involves a huge sample, much bigger than is normally used for opinion polls: 4,500 people in 2006.

Read more of this criticism here, here and here. (Thanks to Mark Thoma and Steve Levitt for a heads-up.)

So what is going on here?

• A misunderstanding. I suspect that the claim that happiness did not significantly change from 1972-2006 comes from the fact that we did not include stars when reporting the implied gender gaps in Table 1 of our paper. Thus, the claim that

the ordered probit analysis found that the "Gender happiness gap" was not statistically significant, either in 1972 or in 2006, even at the 0.10 level

is simply untrue. Here’s the relevant part of Table 1, which is an ordered probit regression, of happiness on time trends by gender:

The right way to test for whether women were, on average, happier at the start of the sample is to look at the “Female dummy”, which is clearly significant. The right way to ask whether this gender gap has changed is to look at the difference in trends, which is also clearly significant. The last two rows are regression-based predicted values, so we didn’t think we should put stars next to these numbers.

•  Statistical mischief: When you want to make a result go away, throw away enough data, and a result will become insignificant. For instance pooling all of the data gives us a useful 46,303 observations. Analyze any specific year, and you are left with only 1,500-3,000 data points. Even so, let’s analyze only data from 1972 and 2006:
  • %Very happy = 28.7 + 3.1*Female +1.6*(Year2006) – 2.4*(Female in 2006)
  • %Not too happy = 18.1 -3.2*Female – 5.5*(Year2006) + 4.1*(Female in 2006)

In the first case, no coefficients are statistically significant, and in the latter, all are. In both cases, the estimates say that women were once a fair bit happier than men, and this is no longer true. Comparing this regression with those in our paper, we simply learn that a smaller sample yields similar estimates, but they are less likely to be statistically significant.

• Looking for a masterpiece, when we are doing collage. Sometimes studying social phenomena is hard, and one draws on many data sources to put together a collage of evidence. Our paper finds declining happiness among women relative to men in: the General Social Survey (n=46,303 from 1972-2006); the Virginia Slims Poll (n=26,701 from 1972-2000); among U.S. 12^th graders (Monitoring the Future; n=433,906 from 1976-2005); in the United Kingdom (British Household Panel Study data from 1991-2004; n=121,135); in Europe (the Eurobarometer analysis has n=636,400 from 1973-2002, covering 15 countries), and across developed countries (the International Social Survey Program contains surveys 35 countries from 1991-2001 yielding n=97,462). The only dataset that does not yield clear results of a decline in women’s happiness relative to men’s is the World Values Survey, and even there, the data do not speak clearly.

Let me try to give a particularly transparent description of the data, simply splitting the GSS data into two periods, 1972-1989 v. 1990-2006. There was a clear gender happiness gap in the earlier period (34.3% of women were very happy v. 31.8% of men). This difference is clearly statistically significant (t=4.1). In the later period, 30.9% of women were very happy, compared with 31.1% of men. This recent gender happiness gap is insignificant (t=-0.3). The decline in the share of women who were very happy (34.3% v. 30.9%) is clearly significant (t=5.9), while the corresponding changes for men were not (t=-1.1). The decline in the share of women who were very happy relative to men is also significant (t=-3.1). Analyzing the share who are “not too happy” yields a roughly similar pattern (but in reverse): an insignificant “unhappiness gap” in the earlier period, but a significant gap emerged in the latter period. Interestingly, the “unhappiness gap” emerged because as men became less likely to be unhappy, as women’s unhappiness remained largely stable. The ordered probit is a regression technique that allows one to make these happiness and unhappiness comparisons all at the same time; these regressions tell us that there was a gender happiness gap favoring women in the earlier period, and it now favors men. For the regression-heads, if your library subscribes can download the GSS data from the ICPSR here. I’ll post some stata code in the comments.

This post only deals with whether the effects we describe in the paper are statistically significant. The other complaint is that our results are too small to matter. Later today, I’ll turn to how we think about whether these are large or small effects.

[Written jointly with my coauthor Betsey Stevenson]

UPDATE: See discussion of "economic significance" here.