Taking Surveys Seriously? (II)

            In the previous post, I looked at whether it’s possible to code the answers to open-ended survey questions in such a way as to flag respondents who perhaps weren’t taking the survey seriously. My primary concerns were whether these outbursts of respondent hostility can affect our inferences, but also whether we can use characteristics of respondents or of particular questions to predict when these outbursts will occur. I think looking at these hostile responses is an important reminder of just how many different stories there are within a single survey. If social scientists want to generalise about these stories, then I think we ought to acknowledge that some individuals in our dataset don’t want to be generalised about. Frustration is one way to gauge this.

            The basic idea is straightforward. Sometimes when filling out an online registration form, there are questions that you don’t feel like answering. If the website won’t let you through without putting something in the box, you might put something absurd there just to move on. Sometimes the question is so absurd that you feel like responding in kind – rumour has it that even children applying for tourist visas to the US are required to be asked whether they committed war crimes in WWII. These kinds of responses are interesting. Just as important is whether they signal that the whole set of answers from that respondent should be treated with suspicion.

            After coding the 2010 CCES (admittedly it was difficult to be systematic – see previous post), the first thing I want to think about is the validity of the measure. If the variable is capturing something real, then I would expect that it is correlated with other variables that tap into the same broad underlying personality features. Specifically, outbursts of bored hostility should occur more amongst the dis-interested than the interested. This is indeed what we find; our ‘Not Serious’ variable ranging from 0 to 3 depending on how many flippant write-in answers a respondent gave, and chi-square tests of its co-variation with political interest are highly significant (39.46, p=0.000). This is an encouraging start from a validity perspective.

            The second thing I want to think about is who is most likely to respond to surveys with aggravated boredom. My first hypothesis is that it’s people who are most comfortable expressing that they feel like their time is being wasted. More specifically, people who feel like they are high enough on the social hierarchy that they can safely express their frurstration even in a strange setting. In the contemporary US, I think this sense of entitlement (i use that word cautiously because I don't mean it to have negative conotations, because any respondent is entitled to be frustrated. Demographic features only help predict who is likely to feel they can safely express this in a somewhat 'official' setting) is most likely to be found amongst men, and amongst white people. Given that we’re dealing with only a small minority of respondents, I’m going to look only at these broad demographic features.

            Let’s see if I’m right. Here’s bivariate tables, with chi-squared significance tests:

Sex

Race

So sex seems to work in the predicted way, but race does not. This suggests the beginnings of a pattern, and perhaps a more systematic and refined coding scheme might find more such answers and be able to start making more reliable inferences. What else, apart from the sex of the respondent, might help us predict who is likely to give these responses?

An interesting point is that these answers tend to come from moderates, not from extremists. When we tabulate political ideology (below includes ‘don’t know’ as moderate, but the same results are found when this category is excluded) we see that – perhaps because moderates tend to be disinterested – most responses come from this category.

Ideology

            In the next entry I’ll show the results of a logit predicting ‘any’ flagged responses, as well as discuss the consequences of excluding these respondents. In the meantime, I hope I’ve persuaded you that exploring these responses is an interesting way of looking at how the public react to survey research. I wish I could end on another Keith clip but until he gets his own spin-off I shall settle for this classic survey scene:

http://www.youtube.com/watch?v=G0ZZJXw4MTA

Thanks for reading.