Tester's up by 5--no, wait a minute--Rehberg's up by 10, ARGH! Making Sense of Conflicting Polls

Within 48 hours, we have seen two different firms release conflicting polls in the Montana Senate race. On May 1, Public Policy Polling (PPP)--a Democratic polling company--released a poll indicating that Jon Tester maintained a five point lead over Congressman Rehberg, 48-43. The lead was just within the margin of error of 3.2%. Nate Silver at The New York Times has analyzed polling firms and their biases. He actually found that PPP's polls lean slightly toward Republicans.

On May 3, Rasmussen Reports--a polling firm that generally has a pro-Republican tilt in its polling--released another poll with completely different results. This poll shows a ten point lead for Congressman Rehberg--53-43--the largest lead we've seen in this race to date. This lead however is also just inside the margin of error, which was 5 percent in this poll.

How do we make sense of these conflicting polls?

Let's start with a couple of fundamental points. First, the PPP poll was conducted between April 26-29 and included 934 Montana voters. The Rasmussen poll was conducted on May 2 and included only 450 likely Montana voters. That's why we see the different margin of errors--the larger the sample size, the lower the margin of error.

Second, let's talk about the margin of error and what that means. In the PPP poll, Senator Tester's support could range between 51.2 and 44.8. Congressman Rehberg's support could be as high as 46.2 and as low as 39.8 percent. In the second poll by Rasmussen, Senator Tester and Congressman Rehberg could both be tied at 48 percent. In either case, the leads by both are within the margin of error--so the results are not quite as out of line as one might expect just by looking at the head to head matchups reported by the polling firms.

But let's dig a bit deeper. One of the hardest things to figure out in the polling world is who will actually show up to vote. Forecasting turnout is about as hard as forecasting the weather because there are so many variables at work and the instruments we use to measure intent are subject to social desirability biases. If you ask a person if they intend to vote, most likely will give you the socially desirable answer: "Sure, I plan to vote". The problem is about 80 to 85% of voters will answer yes--and we know that turnout generally hovers between 50 and 60 percent. In other words, a bunch of folks who say they will vote simply don't.

Pollsters have lots of ways to measure turnout, and the differences in measuring turnout can have consequences for the final polling results that are reported.

How does Rasmusen and PPP differ in their turnout screeing questions? According to an e-mail exchange I had this morning with Tom Jensen at PPP, his organization calls folks who have voted in one of the last three general elections.Rasmussen, however, polls "likely voters". What's a likely voter? Rasmussen asks several screening questions, including the respondent's voting history, their interest in the election, and their likely voting intentions. This is a much more vigorous screening process designed to weed out folks who may not actually show up on election day.

PPP's process likely yields a "liberal" definition of turnout and Rasmussen's a "conservative". I use those quotes deliberately. PPP process might include folks in the sample who are less committed to voting than the Rasmussen poll. Demographically, Democrats usually have the turnout deck stacked against them relative to Republicans. Folks who are poorer, less educated, and not white are less likely to vote than those who are richer, more educated, and white. In short, those who are more likely to vote for Democrats are also those who are less likely to vote.

In short, the difference in results MIGHT be a factor of how each polling firm choses to define a voter. And there is no one "right" way. A generous interpretation of these conflicting polls is the higher the turnout on election day, the better chance Senator Tester has at getting reelected. Lower turnout, on the other hand, will likely benefit Congressman Rehberg.

Final take away message: Read polls carefully, examine the methodology section thoroughly, and go beyond the first page of the press release if you really want to understand why polls conflict.

Other Important Notes:


Both Senator Tester and Crossroads GPS launched some new ads in the past week. Senator Tester's ad, a nice positive bio spot about is propensity to fly Montana meat to DC with him on the plane, is right below.



The Crossroads GPS ad is below. It covers no new ground, focusing on Tester's votes on healthcare, cap and trade, and the federal budget--again, mentioning that Senator Tester voted 97% of the time with President Obama (see my previous analysis on voting here).