Taking Surveys Seriously? (I)

Time Magazine: “Do you care about what you sing?”

Bob Dylan: “How can I answer that question if you’ve got the nerve to ask me?”

In this post I discuss whether we should exclude a respondent from survey analysis if their answers show that they’ve lost patience with the process. This stems from my discovery that the American
public can get pretty sassy when they’re taking political science surveys. If you ask enough questions that people don’t appreciate, they’ll get sick of giving reasonable answers.

This comes across in the Co-operate Congressional Election Study (CCES) – a US public opinion survey, exceptional both for its breadth and for its depth. Not only are over 50,000 respondents contacted, but these respondents are given the opportunity to answer some questions in an open-ended way.

And at least some respondents take advantage of the extra space to make a joke at the expense of the survey itself. Like Senator Coburn, some respondents never miss a chance to strike a blow against the survey-industrial complex. It's these write-in answers, and their causes and consequences, that I'm going to examine. There are of course other more systematic ways to look at the general topic of accurate measurement in surveys. For example, factual questions assess respondents’ political knowledge, interviewers sometimes quantify respondents’ level of engagement, and respondents’ answers to some questions can be independently validated – like those on turnout.

But perhaps write-in answers give us a perspective that these other measures do not. Perhaps other people have their own views on this approach, but I believe isolating facetious write-in answers
could be important for three reasons. First, we can perhaps use the number of flippant responses as one way of assessing how arduous the respondents are finding the survey overall. Second, we can look at whether particular questions seem to touch a nerve for multiple respondents, or whether some respondents just give flippant answers to all questions. Third, if people aren’t taking the survey seriously, should we take their responses seriously? I believe that frustration, flippancy, facetiousness, etc, are all completely reasonable reactions to survey research, and to politics – but if (for example) someone gives flippant write-in answers, and then ticks the first box on every subsequent multiple choice question, should we interpret those answers as genuine, or not? Should we include a respondent in our analysis if their answer to the question of “How often do you pray?” was “now hoping this is over”? Write-in answers give us the chance to measure aggravated boredom, actual hostility, or just ridicule, directed towards the survey instrument itself. I hope I've persuaded you that this is at least potentially worthwhile (although the irony of quantifying these qualitative examples of hostility to quantification is not lost on me). I use the 2010 CCES and examine this topic in the following way:

1) How many respondents ‘write-in’ answers that could be perceived as flippant or facetious – as not taking the survey seriously?

2) Can we predict which types of respondents have a propensity for such answers?

3) In practice, can including or excluding such respondents affect our inferences?

            The first point is what I’m going to tackle today. It looks like a relatively simple descriptive question. I went through all responses to open-ended questions, and highlighted those that were not necessarily anti-political but simply anti-survey. It’s hard to tell whether a given answer represents someone not caring about the survey, or just not caring about politics. Not caring about politics is a very different question, and decades of scholarship on low democratic participation deals with this very idea. Not caring about the survey is different.

            In order to try to get at this and only this, I only coded answers that weren’t necessarily derogatory. It’s one thing to skip a write-in question because you don’t care about politics. Even registering your disgust by claiming, for instance, that your ideal senate candidate is “any stray animal”, doesn’t necessarily imply to me that you aren’t taking the survey seriously. Nor did I
include responses like “None of your business” unless the response included not just refusal but also hostility – e.g. “NONE OF YOUR DAMN BUSINESS”. I mostly look for answers that seem to impart no judgment at all except a sort of aggravated boredom on the part of the respondent – but I welcome suggestions on how to code this better in the future.

Hopefully there would be little argument over coding some of these:

“For whom did you vote for U.S. Senator?”

             “Yo Mama”

            “Kim Jong-Il”

            “Pee Wee Herman”

Others, however, are more difficult to code. But using questions on why respondents didn’t vote, on their preferred candidates for House, Senate, and Governor, and on their party ID, we obtain the following: