Generally, when a new “public opinion survey” comes out, I am pretty cynical about it. Having had a lot of exposure to the mechanics behind such surveys, and having had a lot of opportunities over the years to talk extensively with public opinion research specialists, I can say that without any hesitation that what comes out of a survey depends greatly on what you put into a survey. Or put another way – if you ask a slanted question, you will get a slanted answer.
The latest survey out from the Public Policy Institute of California presents a great example of manipulating those being surveyed to skew the results in a particular direction. The issue is on global warming and AB 32 – but first let me walk you through the broader survey manipulation issue…
You can write a whole book (I’m sure someone has) on all of the ways to manipulate the outcome of public opinion surveys. But one way that these kinds of surveys can fall short is that do not sufficiently educate the person being questioned before asking the question – producing a meaningless response. A hypothetical example of this kind of situation would be my “Disneyland” example, where the following question is asked with no preface…
Q: Do you think that ticket prices at Disneyland should be lowered by 25%?
You can guess that overwhelmingly the response to this question in a survey would be yes.
But would the answer change if you prefaced the question with a paragraph saying, “There are many who think that the prices for tickets at Disneyland are too high, and it keeps them from attending the park. However, the Disney Company says that if they lower the ticket prices, in order to reduce their cost of running the part to make up for their losses, many key attractions may be closed on some days, and prices for food, beverages and souvenirs will be sharply increases.”
I think a lot more people, hearing both sides of the issue, even briefly, give a more educated and more valuable response – assuming that the goal is to truly understand the thoughts of the person being surveyed.
Imagine if the polling firm were trying to get you to really answer yes to the question of whether you want tickets to Disneyland to be lower in price. So they put this series of questions in the survey before asking you the ticket price question:
The Walt Disney Company, a publicly held company, makes vast profits every year, and we want your opinion about whether you approve or disapprove of some of the things on which the executives of Disney spend those profits:
Do you think it is a good idea for Disney Corp. executives to fly in multi-million dollar private jet aircraft as a regular mode of travel?
Do you think it is a good idea for Disney Corp. executives to meet for “corporate retreats” at some of most posh resorts around the world?
Do you think it is a good idea for Disney Corp. executives to get free passes to any of Disney’s Theme Parks for themselves and any of their friends?
Then ask the original question, “Do you think that ticket prices at Disneyland should be lowered by 25%?”
Think that would change the outcome? I sure do. You will have emotionally engaged the person being surveyed, and done so by asking questions that are designed to convey biased information to them before you ask a subsequent question, thus influencing their decision on that final question.
So let me now take this to a question that I looked up on the PPIC survey that caught my eye.
was, “…To address global warming, do you favor or oppose the state law that requires California to reduce its greenhouse gas emissions back to 1990 levels by the year 2020?”
Respondents came back: 66% favor, 23 oppose, 11 don’t know
So I looked to see what was asked of those surveyed immediately before this question (with the Disneyland example in mind). Guess what? Yep – set of question intended to emphasize negative impacts of global warming on those being polled before they got to the question above.
This is what you were asked before the AB 32 question, if you were surveyed:
Now I am going to name a few of the possible impacts of global warming in the future in California, and I would like you to tell me whether you are very concerned, somewhat
concerned, not too concerned, or not at all concerned about each one.
[rotate questions 19 to 22a]
19. How about increased flooding?
20. How about droughts that are more severe?
21. How about increased coastal erosion?
22a.How about wildfires that are more severe?
23. Next, to address global warming, do you favor or oppose the state law that requires California to reduce its greenhouse gas emissions back to 1990 levels by the year 2020?
See, there is the manipulation. The PPIC survey is completely lacking in any information about the negative economic impacts of AB 32.
What if instead of proceeding question 23 with the biased questions as they did, instead the PPIC prefaced the question with a balanced introduction, such as this one: rising temperatures on the planet could lead to increase flooding, coastal erosion or draughts. Limiting greenhouse gas emissions, however, will be expensive for Californians, substantially increasing the costs of services and products that you buy.
Do you think that the answers to 23 would have changed somewhat? I do.
Consider that the PPIC has come to its own conclusions when it comes to the issue of global warming, when the reality is that there is substantial disagreement on the subject, and perhaps that sheds some light on the issue. So while PPIC in their recent survey, as compared to other surveys, says that support for fighting global warming through government mandated reductions in carbon emissions is down — the reality is it is likely a lot less popular than they say.
In closing – don’t judge surveys by their results – but rather by the these big factors – whom they question, what they ask, and how they ask it.
Care to read comments, or make your own about today’s Daily Commentary?
Just click here to go to the FR Weblog, where this Commentary has its own blog post, and where you can read and make comments.