"Survey said..."

It is fairly amazing how many times you hear these words nowadays:

 

"A recent study found that..."

 

...and you can then say almost anything after those words and it sounds more "true" than if you had just "said it". But really, how many times do people actually look into that study to verify the  claims made as a result of it?

 

Was it a good, double-blind, isolated-variable, fully-randomized study, properly administered? Was the data carefully processed using the proper statistical techniques? Are the conclusions drawn from the data in line with the data? Very often, the answer is no to at least one of those questions. And the vast majority of people can't tell the difference anyhow. Another very good reason we need better and more advanced mathematics and science programs in our schools. And we need to encourage skepticism at every level.

 

And if a particular study is not based on science, then what is it based on?

 

Was it even a study?

 

Very often results from surveys or even internet polls (internet polls!) are treated with the same validity as results from a real scientific study. This is often used in cases of public opinion. Sometimes the more professional news agencies like the the BBC or NPR will have fine print at the bottom of a survey stating that it is not a scientific poll, but often there is no such disclaimer.

 

Internet polls. Yeah. They are, almost always, useless. No, they are worse than useless because they are, more often than not, completely misleading. You simply cannot get good data from an internet poll. Simply by staging such a poll you've already highly skewed the results because you're only going to get responses from people who:

 

-Have a computer capable of accessing the internet

-Are able to go online

-Find your site

-Take the time to take the poll (what is the motivation?)

-And many many more conditions, some subtle, some not

 

Any time you host a phone poll, same thing. Or a voting exit poll. All very skewed for similar reasons.

 

Back to the dreaded internet poll. What you end up with is a cloud of people who are probably white, young, middle class, and frequent sites that have something in common with the subject matter of your poll. And you also get people who are white, young, middle class, and hate your subject matter with a passion. Some subjects attract only females, others only males. Others only gays, the religious, knitters, SCUBA divers, etc, and each of those groups, like all groups, has built-in biases.

 

Often two opposing groups fight it out with the help of numerous external blogs that pipe traffic to your poll based on the ideologies of their particular stances. And for good measure, add to that some number of people who think it's fun to hijack internet polls, and web bots that do the same thing, and you have one huge mess of "data" that is completely, utterly meaningless.

 

There are very strict and specific ways a question in a scientific survey study is composed and administered so as to make it as neutral as possible. You are trying to randomize the error inherent in all questions by making the questions as clear and as "non-leading" as possible. And you often ask the same question, reworded, more than once so that you can further randomize subject response or confusion to the way a question is worded (and also, interestingly, because people often lie about things even to themselves and even in scientific studies and this is one way to catch them at it).

 

Here's an actual internet poll question from Fox News in 2009 on the subject of "Water-boarding":

 

"If there were a possibility that another major terrorist attack on the scale of the September 11 attacks could be prevented, then do you think the CIA should be allowed to use these techniques to obtain information from prisoners?"

 

Wow. This kind of question would NEVER be used in a scientific poll. It is purposely designed to lead to a specific answer. When you distill this question down, it's basically asking for you to choose between another 9/11 or letting the government use "these techniques" on prisoners. Well, shit! no brainer. Note how the first option is made ominous and graphic with carefully chosen wording: "another" ... "major" ... "terrorist attack" ... "scale" ... "September 11 attacks", linked to the second part with the word "prevented" and then the second option is watered down to "use these techniques to obtain information".

 

This question is telling you something more than it is asking you. That's not even bad science. That's no science. Oh, and this poll was not accompanied by any disclaimer.

 

A true randomized study is one where the variables to be studied are isolated, and all others are as completely randomized as possible to ensure they do not affect the results.

 

Now consider this alternate, but also biased wording of the same question:

 

"Should the people of the United States allow the Federal Government to torture citizens or non-citizens at will and hold them indefinitely without due process in order to obtain information on possible future enemy actions, information that could be obtained in another manner?"

 

It's really the same question, but restated with a different perspective. And note that it is designed to lead to opposite answer of the first question.

 

How about this one:

 

"Should the government torture prisoners, some of whom may be linked to terrorist groups and some not, to attempt to obtain information that may lead to clues of a future terrorist plot?"

 

This time the question is worded "more" neutrally, much of the bias has been removed and replaced with lots of conditional words like "may" and "attempt", although as you can see, it is just about impossible to strip all traces of political and cultural tilt from a question.

 

So, the point of this whole rant is that scientists doing real studies go to extreme measures to minimize bias and maximize randomness in any data-gathering exercise, illustrated here using the example of a survey.

 

Next time you see an internet poll, dissect the wording of the questions and attempt to guess what they are trying to get you to say. Because most all of them are trying to bias their own polls. That's the whole reason they stage polls. And you can be sure they'll be using the results to further promote some cause.