Teens Love Guitar Hero, Part Two
November 4th, 2008(This is the second of two parts of a series on the “Teens, Video Games, and Civics” put out by the non-partisan Pew Internet & American Life Project. Part one was about the general findings of the report. Part two, in true data nerd form, will be about the methodology of the survey.)
After a two-week hiatus and feeling refreshed, I’m back. If you’ve been following the blog, part one of this series might be a little rusty. If you need to refresh your memory, feel free. (It’s not so important, though, since this post takes a different tack.) And sorry for the delay. My life has been pretty crazy lately - especially with my new job that I start this week - and I’m really mulling over a lot of formatting changes for the blog, which I will definitely keep you, ever-faithful reader, informed on.
So, what did we learn last time? Gamers aren’t shut-ins, most teens play games, and game franchises like Guitar Hero and Halo are arguably bigger than juggernauts like Madden NFL. Is there more to learn? Well, I considered going over the data in even more detail than what I did last time, but in retrospect I think I did a fair job covering most of the highlights.
Digging Deeper
But what about how the survey was actually conducted? Just how reliable is it? Many studies have an agenda behind them, and thus their methods are often less transparent than I would like. However, Pew is supposedly non-partisan and actually surprisingly open about their methods. They also employ some surveying techniques - some new, some not-so-new - to help get better surveys.
Regressing
One technique used in the Pew Report is logistic regression. This is just like linear regression, but with a logarithmic curve. A linear regression simply says that if you plot data as points on a graph, the regression will will give you the line that best approximates those points. In logistic regression, you try and fit the points to the best logarithmic curve you can find instead. That just means that instead of trying to see a linear relationship of two variables to one another, you’re trying to see a logarithmic connection; in other words, you’re guessing your data are mostly gathered around 0 or a neutral result, with a few really crazy outliers. Interested readers can check out this link to see why statisticians prefer logistic to linear regression (mostly for technical reasons, as it turns out).
There’s a problem, though. Long-time readers of this blog already know I’m not a fan of regression. Why? If you’ve read Nassim Taleb’s book “The Black Swan” (which I covered here in an earlier post), you’ll see he derides such simplistic models like those used for regression. If people want to see a logarithmic relationship between two data points, feel free to put them in a table or chart for people to look at and conjecture. But basing your conclusions on what, to me, is a sketchy theory is not something I condone.
That’s not to say regression has no place. Some things show a clear linear (or non-linear) relationship, like temperature vs. season, or age vs. life expectancy. Using regression for these is fine. But I don’t think complex psychological and social phenomena like gaming can in good conscience be captured by a simple regression model. Plus, such models try to “control” for most variables to study the relationship of two variables at a time. Not only is such analysis oversimplistic, but it also involves some fairly large assumptions to be of any use.
In lieu of this, I largely left out most of the regression-based findings of the survey, many of which had to do with the relation between gamers and civic-mindedness.
Over-compensating
Another common problem these surveys have (which they share with political polls) is that they try to compensate for the the inevitable sampling problems they have. It’s a fact that more educated, richer people answer more surveys. And a lot more women answer surveys than men. These are obvious deficiencies. However, Pew (and others) attempt to address this by over-weighting answers by under-represented groups and under-weighting answers by the over-represented.
Is this tweaking fair? I wouldn’t think so. Sounds like another oversimplistic fudge to me. It would be better to simply randomly sample answers from the over-represented to bring them down to the levels of the least-represented. This will greatly reduce the data set and maybe add back in a lot of volatility and noise. So be it, I say. Your survey is only as good as the representativeness of your sample. Merely trying to compensate by adding weights is making up and distorting answers you don’t have. It’s just a guess at that point.
Pollsters have the same problem and they attempt to solve it in the same way - by adding weights to groups like “likely voters”. Perhaps this explains the some of the gross failures of pollsters from time to time (like their infamous misfire in the Democratic New Hampshire primary this year). The fact of the matter is that different political polls on the same topic at the same time often have results outside the overlap of each poll’s margin of error. There’s no explanation for that except bad sampling or differences in assumptions or methodology. No matter which cause, it’s a damning conclusion. Let the data set speak for itself.
What About All Those Cell Phones?
A big polling problem is that a lot of people (especially younger folks like myself) are dropping their land lines and going mobile phone only. Survey companies, however, have strong restrictions on automatic calling of cell phones, among other concerns. (For example, many cell phone users are reached in places inappropriate for a survey, like on the road.)
For many reasons, then, it’s a lot harder to survey someone on a cell phone. This is a problem, and Pew admits they mostly only carry out land-line surveys. This is mitigated by the fact that they were only trying to call families with children (who are much more likely to have land lines), but it’s still an issue.
On the Other Hand
Let’s not be too negative here. There are also positive things Pew has done while conducting this survey. (Or should I say Princeton Survey Research International, the people who actually ran the survey itself.)
To combat problems with unlisted numbers and the like, Princeton used blocks of the first eight numbers (area code + exchange + first two numbers of the last four) that had a lot of listed residential numbers and added the last two numbers on randomly. That gives you a pretty nice base of random likely households to call.
Princeton also filters out a staggering amount of the huge number of calls they make for various reasons. They leave out business numbers, faxes, modems, cell phones (oddly enough), busy signals, answering machines, people that don’t speak English, and so forth. Out of 112,882 calls, only 1,102 resulted in a completed survey. That’s mind-boggling to me. They made over a hundred thousand calls and got a less than one percent response rate. Talk about looking for a needle in a haystack. I appreciate the tremendous amount of legwork that goes into data like these, but again it also makes me worry about the data being unrepresentative.
Conclusion
“Teens, Video Games, and Civics” had some interesting findings based on data that were probably collected and published in a non-partisan fashion. However, I have serious reservations about their sampling and weighting methods. Adjusting data is never something I take lightly. And leaving out all families that don’t speak English or only use cell phones seems like a significant problem.
Of course, the oversimplification their logistic regression model introduces is something else I also have an issue with. The fact that they get so few hits on these surveys in the first place (only 40% of people they contact cooperate, and 92% of those are screened out afterwards) also raises issues with sampling and self-selection.
This is no idle worry. People they survey are subject to a well-known self-serving bias, where they see themselves as more capable and more often correct than others. (Like how most people think they drive better than average, a statistical impossibility.) So this poll also relies on dodgy self-reporting, especially on sensitive topics like parents reading game ratings before they buy them. How many parents are going to admit they never look at game ratings, even in a survey?
Therefore, despite the fact this survey is non-partisan (and probably more unbiased and accurate because of it), you should take all these findings with a healthy grain of salt. In fact, I’d say that applies to any telephone survey conducted in this manner, including a lot of political polls. For political matters, I’m more partial to the prediction market site Intrade, as I often mention. However, I’m currently losing a bundle on my all-out McCain bet on their play money market. Again, it just goes to show how bad we are at predicting the future, by polls or any other method. It could’ve been worse, though: after all, I could’ve lost billions betting on Volkswagen…
| | | del.icio.us |