The credibility of surveys: Tatad’s reply
By John Nery
Philippine Daily Inquirer
Ex-Senator Kit Tatad wants an “intelligent debate” on the uses of surveys. I am happy to oblige.
In reply to my Feb. 23 column mocking his new-found anxiety over the perfidy of political pollsters, he wrote me a lengthy letter; I am reprinting it below, albeit with some editing in (missing commas, etc.) and some editing out (three less-important paragraphs, etc., to give me space, about 1,500 characters worth, for my response). But I have refrained from inserting my answers into his letter, reserving them for the end.
His letter. “On Feb. 17 … I presented a paper on our ‘fatally flawed political surveys.’ It was an updated recap of what I had been saying since the pre-campaign polling began, long before I became a candidate, long before anyone ever filed a certificate of candidacy for the May elections, long before the official campaign rolled off.
“These were some of my points: 1) that our local pollsters have been using quota sampling and face-to-face interviewing long after these have been junked by reputable pollsters in the US, where opinion polling originated; 2) that these have produced ‘unrepresentative samples’ that could not possibly produce any good results; 3) that the basic information about each survey—who sponsored it, who did it, what samples were used, what questions were asked, in what order were they asked, what is the margin of error, etc.—all of which should be published with every survey result, has never been published; 4) that politicians are allowed to ask their own questions in these surveys for P100,000 per question, on top of a P300,000 subscription fee; 5) that the media have routinely published the results without any critical analysis; 6) that the surveys have shaped voter preferences, even without further inputs.
“My presentation offered more than enough room for an intelligent debate, in case of disagreement. Yet, instead of pointing out any errors in my brief, you chose to aim at my person, which is far from perfect, even without the added burden of imaginary misadventures and questionable quotes. Sad, to say the least.
“There are at least 30 countries in the world today … where one may not publish the results of a political survey within a certain period before an election, unless all the basic information about the survey is also published … And the most reputable pollsters maintain that no pre-election (or pre-campaign) survey may be taken at face value to predict the outcome of any election. Some outstanding examples:
“1. In 1932, Literary Digest, the leading US pollster, predicted that Alf Landon would defeat President Franklin Delano Roosevelt, who was running for a third term. FDR won, and the Digest folded up not long thereafter.
“2. In 1948, America’s leading pollsters—George Gallup, Archibald Crossley and Elmo Roper—predicted President Harry Truman would be overwhelmed by Thomas Dewey. Thus the election day headline screamed: ‘Dewey defeats Truman.’ But Truman won. The pollsters were investigated by the US Congress and the Social Science Research Council later.
“3. On Jan. 8, 2008, eleven pollsters predicted Barack Obama winning the Democratic primary in New Hampshire. But Hillary Clinton won …
“5. On May 11, 2004, in Metro Manila, an SWS exit poll showed President Gloria Macapagal Arroyo leading Fernando Poe Jr. 31 to 23 percent. The official count, however, gave FPJ 36.67 percent of the votes to GMA’s 26.46. In principle no exit poll should make such a mistake. SWS was never investigated by any council or Congress. Nor has SWS ever publicly apologized or explained why it erred. Was that error so trivial or is public memory so short that SWS should once again be polling voter preferences for any election, as though its credibility had never been tarnished?”
My reply. I have space only for four rebuttals: three factual, one practical.
It is simply not true that SWS never apologized for or explained the 2004 exit poll, or that it was never investigated. The former senator can read the results of the independent investigation, which were disclosed in an October 2004 news conference, on the SWS website (check under 2004). The culprit, in the case of Metro Manila, was the substantial number of “no answers,” caused most likely by the heavy rains between 3 and 6 p.m. on election day.
It is not true that the most famous Chicago Tribune headline of all time was caused by faulty polling; indeed, Roper had stopped polling altogether, because he thought Dewey was the runaway winner. The reason the Tribune got it wrong was because a newspaper strike had forced the paper to go to press several hours earlier than usual, and because its Washington, DC correspondent, staunchly Republican, called the election wrong. No polls were involved in the making of that headline.
It is also not true that the Literary Digest was the “leading US pollster” of its time. It was a popular magazine that polled its own readers on their presidential preferences. In 1936 (not 1932), it predicted FDR would lose his bid for a second (not third) term, based on what its readers, more affluent than most Americans in the depth of the Great Depression, preferred. It was a non-scientific survey (and did away with face-to-face interviewing too!).
In his latest column, SWS’ Mahar Mangahas explained why face-to-face interviews are necessary in the Philippine setting. (It’s a no-brainer, really.) My question for Mr. Tatad: If SWS and Pulse use “unrepresentative samples,” how is it possible that the results of most senatorial and all presidential and vice presidential contests they have tracked confirm their findings?