POLLS, SCHMOLLS

“There are three kinds of lies: lies, damned lies, and statistics.”
–Benjamin Disraeli

SURVEY SAYS...
A survey is an attempt to understand trends, activities, or beliefs of people by asking them questions. Since one cannot realistically ask every single person on earth or in even in a country (unless you live in Liechtenstein or Andorra), a "sample" is taken, that is, a smaller group of people is taken to represent the whole. From this information you count different responses to a question and calculate how many chose each of a given response to a query. This technique is also called polling.

The more people polled, the more representative the poll is said to be, as it is closer to the ideal of asking every single person of the targeted group - a city, a political party, a gender, an age group, etc. The more variety of people within that group that you survey, the more representative the poll is considered as well. Polling is a shortcut to finding out what the whole of a target group thinks on an issue or does, thus giving a sort of indicator on trends, ideas, and such.

A representative group is one that has a broader range of people that more accurately represent the whole. For example, polling fans of classical music should include young and old, male and female, urban and rural, those who listen only to recordings and those who go to concerts, and connoisseurs of all styles of classical, not simply a single type such as romantic or baroque. The sample surveyed needs to be large enough to avoid too much error.

For example, a survey is taken to ask how often people listen to a given radio program. If you ask a small group, then even a small number of unrepresentative folks will change the numbers by a significant amount. Say you ask 100 people how often they listen, and 20 say "never." That looks like 20% of the population never listen to the radio. But if you ask 5,000 and only 40 say "never" then you've doubled the people who reject the radio, but the ratio has dropped to .8%! By asking more people, you reduce the chance that the sample is skewed too heavily by a non representative group you happened to run into. The closer you get to the full number of humanity you are trying to find out about, the smaller the chance is for error.

This range of mistakes is called the "margin of error" and it represents an effort to estimate approximately how many people are unrepresentative of the whole, answered in error, or simply lied to you. For most polls, the margin is 3-5%, but as we'll see, this can be much larger. Any time a poll is read, the margin of error should be subtracted from the totals to see the more reliable results. For example, if a poll says 52% of the public will vote for Floont and 48% will vote for Yoont, and the margin of error is listed at 4%, the poll tells you almost nothing - there's only 4% difference between these two candidates, so the poll can't predict who will win at all at least 90% of the time.

A BRIEF HISTORY
Polling has been done since 1824, when two American newspapers, the Harrisburg Pennsylvanian and the Raleigh Star, organized “show votes” to determine the political preferences of voters prior to the presidential election of that year. This poll showed a clear lead for Andrew Jackson over John Quincy Adams (335 votes to 169). Polling of this type was done in larger cities, and grew in popularity but were not nationwide until 1916 when Literary Digest sent out 2.3 million reader survey cards with the magazine and counted the results. With this method, they accurately predicted four presidential elections.

George Gallup in 1936 began polling with a more scientific approach than Literary Digest, taking a sample of different voters, rather than the readership - largely Republican - of a single magazine. Literary Digest picked Alf Landon to win, and Gallup picked Franklin D. Roosevelt. The more scientific method was more accurate - Gallup even took his smaller sample and predicted how Literary Digest's survey would go and was accurate.

Gallup started a subsidiary polling service in Great Britain and in 1945 accurately predicted the defeat of Winston Churchill. By the 1950s, polling had spread to most Democracies, with organizations attempting to predict events or understand trends by asking what they believe to be a representative group of the public. These polls were done primarily face to face by knocking on doors and asking questions, but this is rarely done today, using telephone calls, mailed surveys, and internet surveys as a cheaper alternative. Today in the United States there are dozens of polling organizations, each one trying to find out information for whoever hires them.

The idea of polls, predicting elections, and finding out trends has taken the US and other nations by storm. Companies, politicians, and entertainment rely on surveys, studies, demographic research, and polls to try to find out what the public thinks and is likely to do. Even unrelated events are used to predict outcomes. One such indicator was the Washington Redskins NFL football team. For 15 straight presidential elections, if the Redskins have lost their last home game prior to the election, the incumbent party has lost the White House. When they have won, the incumbent has stayed in power.

Sounds pretty straight forward right? You ask people how they'll vote or what they do, and you add up the numbers and if you get enough, you'll get a pretty clear picture of how things work, right? Tell that to Thomas Dewey. In the 1948 US presidential election, the Gallup organization's poll indicated that Thomas Dewey would win the presidency by 5-15% of the vote. When the dust settled and all the chads were counted, Dewey's opponent Harry Truman had won by almost 5%. It stands today as the most famous example of polling gone wrong, of failure. Newspapers, confident in Gallup's ability to predict elections and the polling techniques printed newspapers that indicated Dewey's success (see picture above).

In the 2004 US Presidential election, exit polling showed a 3% victory for Senator John Kerry. Exit polling is done by asking people who are leaving voting areas how they voted. The results are totaled and from that a general trend is calculated. President George Bush won reelection by 2.5% of the vote. According to at least one study, the chances of properly executed exit polling being off by this much is close to 1 in 1,000,000 - a pretty tiny number.

Also in 2004, the Washington Redskins lost their October 31st game, their last such match before the national elections. According to this superstitious indicator, Senator Kerry should have won.

WHAT WENT WRONG?
Polling should be a scientific endeavor, a method that generates predictable, statistically accurate results, right? After 180 years, we should be pretty good at this, one should think. So what happened, how is it that polling sometimes, even increasingly, is inaccurate?

There are three possibilities:

1. The polling was done incorrectly
2. These were freak accidents that statistically are rare but can happen
3. Someone tampered with the results, while the polling was accurate.

Any of these three are plausible outcomes, and can certainly happen. For example, November 21, 2004, the Ukraine had a runoff election between Prime Minister Yanukovych and challenger Viktor Yushchenko. The exit polls had a clear victory for Yushchenko by as much as 8%, but Yanukovych won by 2.7% of the vote. Public outcry ensued, and the election was found to have been fraudulent, a fix by the Prime Minister (Yushchenko was also poisoned with dioxin at one point during the struggle). In 2002, Saddam Hussein held an election in Iraq, in which he won with a 100% turnout of voters, up from 99.96% from the last vote, yet Hussein did not enjoy such popular support as even the most virulent anti-war radical would admit.

These are clear results of fraud, of the polling being more accurate than the election. This happens regularly in dictatorships, where to keep the pretense of democracy up, an election is held (such as the one in Iraq) and the results are either decided beforehand, or fixed to represent what the leadership wants after.

And while statistics are generally predictable and can result in reliable information, there's always a chance that something can go wrong. Even in the 2004 exit survey example above, there was an admitted 1/1 millionth chance it could go wrong (a number most statisticians would dispute is absurdly small). The very existence of a margin of error is an admission that surveys can be inaccurate and fail to result in the predicted result accurately.

But it is when pollsters get it wrong because they used poor methodology that we'll examine most closely here, because it is the most common and likely reason for poor results. For example, using exit polls to determine who will win has been recently less reliable a metric. Kevin Drum points out the last few national elections and how inaccurate exit polling has been:

• In 1988, the exit polling favored George Bush by just 0.6%, but he won by 7.7%.
• In 1992, the exit polling data favored Bill Clinton by 12.8%, but he won by only 5.6%.
• In 1996, the exit polling data favored Bill Clinton by 14.7%, but he won by only 8.5%
• In 2000, the exit polling data favored Al Gore by 2.3%, but George Bush won.

This makes the polls off by as much as 7.1%, far over the margin of error. So what happened, why were these and other survey results so wrong?

POLL HOLES
The first area where there can be problems is the fact that polls miss some people by their very nature. To a certain degree, this simply cannot be avoided, and is a calculated risk in the effort. For example, one area that is beginning to grow significantly as a hole for polling is the increasing use of cell phones. Most people in America own or have access to a cell phone. Many people are dropping their old-fashioned "land line" telephone and using their cell phones exclusively. This is a rather large hole in the polls, telephone efforts to poll will simply not reach a large and growing portion of the population, as it is illegal to call cell phones unsolicited.

A smaller hole that can take effect is that some people have more than one phone number. This is increased by the fact that cell phones are being polled more, which means that someone can be called at their land line, and their cell phone, and then another phone in the house might ring, which gives that person three times the input on a poll as a person with only one phone. Since the calling is anonymous and typically random, the result is skewed with no way of knowing that it has been.

Another significant hole is the fact that many people simply don't care to talk to pollsters. If you are like me, you find pollsters annoying, intrusive, boring, and time-wasting. They particularly call when you are busy at some other thing, such as lunch. There are people who enjoy answering polls, either they like the opportunity to sound off or they feel as if they are doing something important or their civic duty. Some are lonely and simply appreciate any call or input, and some think their opinion is so significant that the pollster is being done a favor by asking them any questions at all. But many people simply don't care for the call, and hang up either rudely or politely without participating.

In fact, some people have come to distrust or dislike pollsters due to the results that come out or the way they are used. The theory is that opinionated people are most likely to vote and will be likely to answer a survey, but those same people might be opinionated about surveys and their validity as well.

Caller ID, as noted above, makes it easier for people who are less inclined to be polled to screen out the entire process and never be surveyed at all. When you see someone you don't care to answer on the ID, that call goes unanswered. And since polling can be annoying and intrusive, and rarely is a welcome call to most people, that's a significant hole in the polls.

Although this is covered more below in Methodology, not everyone who is polled for an election, for example, actually votes. Randomly calling households and getting their opinion on an upcoming election can be interesting, but if you can't guarantee that person will vote, all you've done is get an opinion, not anything useful for predicting anything. The person you just polled might not even be a citizen or old enough to vote. Even when asked if someone votes, they are unlikely to admit they didn't vote last time or that they won't now. They might even intend to vote, then change their minds and stay home later. This creates a hole in the polls as well, making them significantly less accurate.

And finally, the most enormous hole in the polls is the sheep effect. People will sometimes at least answer polls they way they feel that they ought to, or the way that most people do (see below in Uncooperative Voters). Instead of getting their real opinion, you get what they think the pollster wants to hear, or what most people are saying. Sometimes the poll will simply get what society most pressures someone to say. For example, someone might be a horrific bigot, but since this is frowned on my civilized society, a surveyed person is unlikely to admit this. Sexual activity polls get another result: people will brag about their experience, exploits, and ideas.

UNCOOPERATIVE VOTERS
There are four kinds of ways that voters can cause trouble for polling and accurate prediction of events or elections, and these are all caused by polling. Had polls not been taken, these effects would likely not have taken place.

The first is the Bandwagon Effect, which is when voters see who is doing better and decide they want to jump in and be part of the big win, or decide that if that many people think someone is best, they must know something. Although difficult to quantify, researchers Irwin & van Holsteyn showed it can happen in a study done in 2000 (no link available).

The second is the Underdog Effect. There are not a few people who will always pull for the guy that's doing the worst in a contest. Some people are said to vote for whoever is doing poorest out of sympathy, rebellion against the majority, or just bullheaded opposition. This is also difficult to quantify and I know of no empirical evidence of it occurring (just anecdotal - stories, people saying they did so). The news media tends to mention this effect in many elections nevertheless.

The third effect is Tactical Voting. This kind of voting is done in congressional or parliamentary elections, to try to sway the larger picture of a body rather than being interested in a specific candidate - choosing a government rather than one person. This can take effect in several ways, such as registering as a different party and voting for a weaker candidate, but in terms of poll-influenced election, it can take effect this way (from Wikipedia):

An example can be found in the United Kingdom general election, 1997, then Cabinet Minister, Michael Portillo's constituency of Enfield was believed to be a safe seat but opinion polls showed the Labour candidate Stephen Twigg steadily gaining support, which may have prompted undecided voters or supporters of other parties to support Twigg in order to remove Portillo.

And the final way polls influence voting is the Boomerang Effect. This is when a candidate polls so strongly and seems so likely to win, that people will vote for a second choice as a protest or because they are uncomfortable with anyone winning by too much - or simply not voting at all, because the outcome is so sure. This can actually end up with the opposing candidate winning if it is too exaggerated.

If polling had not taken place, none of these effects would take place either. The very act of publicizing the results of a poll result inevitably in a shift in the results because of these effects.

METHODOLOGY
The method used to collect data for a poll can make a significant difference to its outcome. Poor methodology especially, but even good methodology can still be problematic. Polling relies on a random, large, and representative sample of the target group - say, voters in the state of Virginia. To find out what people think or are likely to do, you have to ask each person polled the same questions, so that answers can be tallied up. If you simply allowed free-form essay answers, then the results would be up to the interpretation of the pollsters (and be much harder to collect).

Questions?
But this in its self can cause problems: which questions are asked can result in an inaccurate or even worthless poll. If a poll presumes a certain outcome or type of voter, it will lack other issues. For example, if a poll asks you to rank your top 10 concerns and lacks 5 of the most important issues for you, then it's not getting a real sample of concerns from all the respondents. It's simply getting what they think about what you care to ask them. Instead of learning the public's top concerns, you learn how people think about your top concerns, or at least the ones you care to ask about.

Questions can change results simply based on how they are worded, as well as how they are ordered. For example, look at the difference between these two questions:

Do you support the murder of unborn babies?
Do you support a woman's right to choose?

Each of these questions clearly shoves the respondent in one direction or the other. The polled person of course is not helplessly compelled to answer, but this can make a difference in how some respond. Ideally, the question should not bias or nudge people to one conclusion or another, the question should be as clear, easy to understand, and neutral as possible:

Do you support abortion in any form?

And the question should not require more data to respond than is given:

Do you support abortion?

What kind? When? In what circumstances? Of what creatures? The question given here is difficult to answer because there are too many unknown variables. But it is typical of a poll question, and without being more precise can result in skewed results. Someone who supports abortion only to save a mother's life might answer yes, which makes it seem like people support abortion on demand in a greater number than is accurate.

Outside Influence
Media can influence polling as well. While someone ordinarily might be most concerned about might take second place after a media blitz of news about a certain topic. For example, in 1979 someone might consider the inflation their primary concern, then the film China Syndrome is a huge movie hit and media focus, they might shift it instead to nuclear power and the dangers of plant failures, at least temporarily. This would be reflected in polls, but might represent an artificial spike in concern or focus.

Push Polls
However, there are poor methodologies that can be used. One of the more infamous methods is the "push poll," where questions are used to try to nudge a surveyed person to a conclusion. For example, a pollster might call and ask a person to rank candidates by their position on illegal immigration or taxes, thus planting the seeds of a thought in the mind of the polled person. Often the poll is not intended to collect information at all, it's simply a screen to advertise against a candidate by bringing up facts about them that are not known.

In the 2000 election, it has been alleged (without evidence or proof) that voters were called and asked "Would you be more likely or less likely to vote for John McCain for president if you knew he had fathered an illegitimate black child?" McCain has fathered no such child, as far as anyone knows. If accurate, this would be a particularly loathsome example of push polling - attempting to bias voters against McCain.

But even if a poll isn't an attempt to bias voters, asking questions in a particular order can change how voters respond, by setting up question by question a final question that might be answered differently based on previous points brought up.

Who You Gonna Call?
While the ideal is to get a representative sample of people for the most accurate results, this is not always done. Demographic studies are done of cities and guides are printed with neighborhoods, ethnic backgrounds, and other data broken down, and phone numbers listed in each. Cities are more liberal than conservative, while rural areas tend to be more conservative. Thus, by choosing where to call, the poll can be shifted in one direction or another, or the information can be used to attempt to be more balanced and broad in it's coverage.

What Time is it?
When the poll is done makes a difference. Telephone or door-to-door polls suffer from this effect - or benefit from it. Polling homes during the day will tend to result in a more leftward tilt while polling later in the day will tend toward more conservative, studies have shown. Polling after a catastrophic or heavy media event can change the results as well, as shown above.

Respond!
Response rates are difficult for pollsters. According to several studies, getting 50% of the attempts to respond is a good result. But when you try 1000 people and only 500 answer, you've gotten a pretty small sample of your target group even though you tried to get more. This can exaggerate errors or statistical anomalies, and cost a lot more money than a business cares to spend. But if you simply add more calls to try to fill out the numbers, you've gone beyond your original sample, and this might cause as many problems as not getting enough.

The Net
Internet surveys are notoriously inaccurate and not of any use. The biggest problem is that it is impossible to keep people from voting more than once. Not only can they use different computers to vote more than once, but a clever or capable user is able to eliminated the cookie put on their system to identify their voting and do so again. Further, since websites are passive - they wait til someone visits rather than going out and getting voters - the results will reflect not people in general, but only people who go to that website.

Polls, due to the holes in them and methodological problems, are increasingly difficult to trust or rely on for information. This, more than anything else, explains why exit polls are so untrustworthy in the last 18 years. Asking people who they voted for as they leave polls will not only miss people for whom this is private or who are in a hurry, but can be very misleading depending on where and when you ask. The 2004 exit polls were done only in some areas and according to reports heavily tilted toward female voters. Some polls were said to be very selective in who they asked, only picking some people out of the crowd based on the pollster's decision and whim. This is not going to be very random or representative, but might be easy to use to mislead.

DELIBERATE SLANT
While most pollsters attempt to find the best and most accurate information, this is not always the case. In some cases, polls are done, using methodologies pointed out above, to find information that is skewed to give a certain result. Pollsters agree that post-debate polls are misleading and poorly done because of when the calls are done and who they target. Some newspapers have in the past polled only their immediate area, which results in a Literary Digest effect mentioned above - you'll get great results, but only of people of a certain mindset.

Los Angeles Times, for example, did a poll in which they asked - admittedly - mostly Democratic Party voters and fewer Republican Party voters. The results skewed in the favor of the Democratic Party, to the amazement of no one. In their news story, the data was not reported, but it was online.

Another technique that is used is to take an otherwise carefully crafted poll and report only parts of the poll or compress results. Lets say there's fictitious poll is taken:

Do you believe America will win in Iraq?
Yes 60%
No 30%

Are we winning now?
Yes 12%
No 80%

Will we win soon?
Yes 55%
No 40%

And this poll is reported as saying 80% of Americans think Iraq is a failure! Technically, this poll notes that 80% think things are going poorly, but overall, most Americans think the war is a great success. Alternately, if the poll is reported 60% of Americans call Iraq a success! Technically this is also correct, but it fails to note the huge majority that thinks things are awful there right now.

Choosing which poll to report makes a difference as well. An initiative measure to prevent illegal immigrants from receiving benefits or public services in the State of California called proposition187 was reported by Ann Coulter as having been supported by a majority of Hispanics. But the exit polls showed that most Hispanics voted against it. Each side of this argument had a poll that they pointed to that supported their position. By emphasizing one or another, specific results can be reported, and the truth may be one, or neither.

But why do this, why take a poll to get specific results or target people who will not accurately represent the opinions of the general public. What is the intent?

POLLING AS NEWS
One of the reasons that someone might skew an opinion poll is to try to use it as a lever to shift public opinion or to print something as news. This can be done several ways, but the easiest way is to report the results in a misleading manner. As shown above with the fake Iraq poll, the reported results can be technically accurate, but ultimately misleading. Such a result can change how people view a topic because of the Bandwagon Effect.

Using headlines can emphasize this technique. A poll was taken of Americans about Social Security reform that President Bush suggested. The poll as reported in the New York Times said that the majority of people rejected the Bush plan for reforming Social Security! But reading closer, you learned something interesting. When people were asked if they supported the "Bush plan" for social security reform, about 60% of the respondants replied no. But when asked if they supported a plan that privatized part of the social security benefits - exactly what the "Bush plan" did - they responded by almost 60% that they would. Which got the headline? The rejection of the Bush plan.

One of reasons polls are taken by newspapers is not simply to gather information, but to be able to print some dead story or opinion of the paper on the front page instead of the editorial section. Say there's nothing actually going on in the news about FISA wiretapping. By taking the right kind of poll about the topic, the news can write about this "new story" with a report on the poll that was taken.

If the newspaper has a particular opinion, they can run a poll about this topic then write the story all about how people in the poll thought about something, pushing it as news, when it is simply reporting on the opinion of the people - with bits of information that supports this opinion.

The use of polling and it's power over politics and public opinion has actually begun to trouble major pollsters such as the Gallup organization.

POLLING AS KINGMAKER
A Pennsylvania State researcher named J. Michael Hogan, professor of speech communication has written three books on polls and their influence on the public. Hogan has conducted extensive research on public opinion and polling, political campaigns and social movements, and presidential rhetoric. His interest in election polls focuses on how they affect the conduct of political campaigns, the speeches and strategy of the candidates, and media coverage. Professor Hogan has this to say about polls:

"Polls influence the quality and character of political campaigns, and their role in campaign coverage has changed dramatically. The number of polls has increased substantially in every election since 1980 and the media organizations that sponsor them are more interested in news about the ‘horserace’ than in the substance of the campaign.

"This portrays a very cynical view of the electorate. The danger in all of this is that it puts the American public in the role of spectators rather than active participants. This media obsession with polls–and I believe it is an obsession–also reflects a cynical view of the electoral process as a mere game rather than an important debate over the nation’s future. The proliferation of polls artificially affects media interpretations of political campaigns in ways that, in my opinion are bad for the political system, the presidential campaigns, and public discussion."

"Today’s polls are not serving the important democratic function of assessing public opinion for the guidance of policymakers, as George Gallup envisioned in the early days of polling."

Professor Hogan notes some of the problems I've outlined above, and is concerned about how quickly and often polls are taken and reported on.

"What concerns me most is how polls supplant the substance of the campaign by focusing on who’s ahead rather than on the differences among candidates. Poll results create a tendency for reporters to look for good things in the campaign that is ahead in the polls and bad things in the campaign that is behind. Polls have become what media scholars call a framing device. A bad poll frames the story of a campaign in negative terms. A positive poll frames the news story in a positive light."

In the 1940's there was a congressional study done on polls, concerned about the Bandwagon effect on elections and whether they ought to be banned. Certainly financial backing can heavily be influenced by polling, as big business and contributors try their best to be on the winning side so their money isn't wasted and hopefully to gain influence on elected officials.

Pollsters have tremendous influence, and can shape not only the public's perception of a topic, but public policy as spineless nervous congressmen try to please their constituency or not be too far away from the public on key issues. Polls can give these men and women a view of the public's ideas that may or may not be accurate.

SO?
In my opinion, polls should for the most part be simply ignored. The only kind of survey that truly matters is at the ballot box in an election. Then when votes are counted they get the entire body that is involved and the votes are final rather than opinion that can change from day to day. There's no way to bias the outcome with techniques, reporting, and how things are framed - it's a yes or no vote that's final.

Reading and being encouraged by or fearing polls is pointless and foolish - it might shift completely the next day and might be utterly unrepresentative of how people really think. Especially approval rating polls are completely valueless. Your approval of your best friend might vary a great deal from day to day or hour to hour, but that doesn't mean you hate that person all the sudden. You might simply be really upset at what they just did or said - but in the long run, they're still your best friend.

Imagine an approval rating taken from marriages? The hour after a huge screaming fight over the washcloth and where it's proper to hang it at the sink one might have a pretty low approval rating from the spouse. Does that mean the marriage is over? That you despise this person? That you consider their overall ideas and goals poor or wrong? Certainly having public policy or voting shaped by current popularity or annoyance is a poor way to run government.

Polls, schmolls.

*UPDATE: A classic example of a poll that cannot be trusted due to methodology is the Lancet poll on Iraqi deaths (pdf file) that was mentioned a lot a year or so past. The New York Times ran an article on this, and it became a classic liberal meme: 98,000 more Iraqis died as a result of the invasion to depose Hussein! The study did a study of the number of deaths in the 14-month buildup period before the invasion, then conducted surveys to estimate how many died in the 14 months during and after the invasion of Iraq by the Coalition of the Willing.

We estimate there were 98,000 extra deaths (95% CI 8000-194 000) during the post-war period.

An article on Slate asks for readers to look at that sentance a bit more closely.

Readers who are accustomed to perusing statistical documents know what the set of numbers in the parentheses means. For the other 99.9 percent of you, I'll spell it out in plain English—which, disturbingly, the study never does. It means that the authors are 95 percent confident that the war-caused deaths totaled some number between 8,000 and 194,000. (The number cited in plain language—98,000—is roughly at the halfway point in this absurdly vast range.)

This is a bit different than a margin of error of around 3-5%. This is like saying someone won between 4 and 96% of the vote, then averaging it out to 50%. But the study was pushed by the men who did the work as accurate, hyping their 98,000 deaths number (rounded to 100,000):

Gilbert Burnham, one of the co-authors, told the International Herald Tribune (for a story reprinted in today's New York Times), "We're quite sure that the estimate of 100,000 is a conservative estimate."

But why is this number so wildly uncertain? Because the methodology was poor. 33 equally-sized neighborhoods across Iraq were randomly selected and surveyed in September 2004, interviewing 30 households in each neighborhood. They asked how many people in each household died, of what causes, during the 14 months before the U.S. invasion—and how many died, of what, in the 17 months since the war began. This small sample (990 households in a nation of over 26,000,000 people) was then extrapolated to represent the entire country and presumed to be accurate. However, one of the randomly selected cities interviewed was Fallujah, one of the cities that saw the heaviest fighting and bombing in Iraq.

There were other problems. The survey team simply could not visit some of the randomly chosen clusters; the roads were blocked off, in some cases by coalition checkpoints. So the team picked other, more accessible areas that had received similar amounts of damage. But it's unclear how they made this calculation. In any case, the detour destroyed the survey's randomness; the results are inherently tainted. In other cases, the team didn't find enough people in a cluster to interview, so they expanded the survey to an adjoining cluster. Again, at that point, the survey was no longer random, and so the results are suspect.

Another problem is the estimate used for deaths before Hussein was deposed. The research team assumed 5/1000. But according to quite comprehensive data collected by the United Nations, Iraq's mortality rate from 1980-85 was 8.1 per 1,000. From 1985-90, the years leading up to the 1991 Gulf War, the rate declined to 6.8 per 1,000. Due to the massacre of the Kurds and the starvation and want from Hussein's handling of the embargo and sanctions, the numbers were at least 6.8/1000.

The website Iraq Body Count is a bit more careful in their research, and they estimate that between 14,181 and 16,312 Iraqi civilians have died as a result of the war in that time period. In other words, the Lancet study was wholly worthless as a survey and resulted in numbers that could not be trusted. Why did they do it in such a manner? Judging by their insistance on the average of 100,000 one suspects that it was to make things look worse - vastly worse - than they really are.

So what polls can be trusted? As I said before, the safest choice is to simply disregard them all, at least until you can study how the poll was taken, what questions were asked, and of whom. My rule of thumb is that a poll that says the opposite of the ideology of the polling group is more trustworthy than one that fits it. If Right For Life publishes a poll that says 60% of Americans oppose abortion, it's suspect. But if the New York Times publishes a poll that says 60% of Americans think they should face criminal charges for printing classified information that hurts the War on Terror, that's a bit more plausible and likely.

Essays