Tester's up by 5--no, wait a minute--Rehberg's up by 10, ARGH! Making Sense of Conflicting Polls

Within 48 hours, we have seen two different firms release conflicting polls in the Montana Senate race. On May 1, Public Policy Polling (PPP)--a Democratic polling company--released a poll indicating that Jon Tester maintained a five point lead over Congressman Rehberg, 48-43. The lead was just within the margin of error of 3.2%. Nate Silver at The New York Times has analyzed polling firms and their biases. He actually found that PPP's polls lean slightly toward Republicans.

On May 3, Rasmussen Reports--a polling firm that generally has a pro-Republican tilt in its polling--released another poll with completely different results. This poll shows a ten point lead for Congressman Rehberg--53-43--the largest lead we've seen in this race to date. This lead however is also just inside the margin of error, which was 5 percent in this poll.

How do we make sense of these conflicting polls?

Let's start with a couple of fundamental points. First, the PPP poll was conducted between April 26-29 and included 934 Montana voters. The Rasmussen poll was conducted on May 2 and included only 450 likely Montana voters. That's why we see the different margin of errors--the larger the sample size, the lower the margin of error.

Second, let's talk about the margin of error and what that means. In the PPP poll, Senator Tester's support could range between 51.2 and 44.8. Congressman Rehberg's support could be as high as 46.2 and as low as 39.8 percent. In the second poll by Rasmussen, Senator Tester and Congressman Rehberg could both be tied at 48 percent. In either case, the leads by both are within the margin of error--so the results are not quite as out of line as one might expect just by looking at the head to head matchups reported by the polling firms.

But let's dig a bit deeper. One of the hardest things to figure out in the polling world is who will actually show up to vote. Forecasting turnout is about as hard as forecasting the weather because there are so many variables at work and the instruments we use to measure intent are subject to social desirability biases. If you ask a person if they intend to vote, most likely will give you the socially desirable answer: "Sure, I plan to vote". The problem is about 80 to 85% of voters will answer yes--and we know that turnout generally hovers between 50 and 60 percent. In other words, a bunch of folks who say they will vote simply don't.

Pollsters have lots of ways to measure turnout, and the differences in measuring turnout can have consequences for the final polling results that are reported.

How does Rasmusen and PPP differ in their turnout screeing questions? According to an e-mail exchange I had this morning with Tom Jensen at PPP, his organization calls folks who have voted in one of the last three general elections.Rasmussen, however, polls "likely voters". What's a likely voter? Rasmussen asks several screening questions, including the respondent's voting history, their interest in the election, and their likely voting intentions. This is a much more vigorous screening process designed to weed out folks who may not actually show up on election day.

PPP's process likely yields a "liberal" definition of turnout and Rasmussen's a "conservative". I use those quotes deliberately. PPP process might include folks in the sample who are less committed to voting than the Rasmussen poll. Demographically, Democrats usually have the turnout deck stacked against them relative to Republicans. Folks who are poorer, less educated, and not white are less likely to vote than those who are richer, more educated, and white. In short, those who are more likely to vote for Democrats are also those who are less likely to vote.

In short, the difference in results MIGHT be a factor of how each polling firm choses to define a voter. And there is no one "right" way. A generous interpretation of these conflicting polls is the higher the turnout on election day, the better chance Senator Tester has at getting reelected. Lower turnout, on the other hand, will likely benefit Congressman Rehberg.

Final take away message: Read polls carefully, examine the methodology section thoroughly, and go beyond the first page of the press release if you really want to understand why polls conflict.

Other Important Notes:


Both Senator Tester and Crossroads GPS launched some new ads in the past week. Senator Tester's ad, a nice positive bio spot about is propensity to fly Montana meat to DC with him on the plane, is right below.



The Crossroads GPS ad is below. It covers no new ground, focusing on Tester's votes on healthcare, cap and trade, and the federal budget--again, mentioning that Senator Tester voted 97% of the time with President Obama (see my previous analysis on voting here).

Baucus, a Poll, and High on the Hog--Oh my!

Lots of developments in Montana in the past 48 hours.

First, Senator Max Baucus recently announced that he's running for re-election in 2014 and has purchased $25,000 in radio ads to be aired...wait...THIS WEEK in several markets throughout the state. Read the story here. More on this below.

Second, a new poll is out in the Montana Race that is actually taking place this year shows...wait...A Rehberg lead within the margin of error. Again, the races is: too close to call. Here's that Rasmussen poll (with all the qualifiers from my previous posts about Rasmussen).

Third, AP had an article in Montana papers two days ago slamming both Senator Tester and Congressman Rehberg for living "high on the hog" by staying in fancy hotels and eating expensive meals, which are picked up by their campaign accounts. Read that piece here.

Whew! Where to start? I think we'll start with Montana's senior senator, Max Baucus. According to news accounts, the ad stresses his role in the payroll tax extension and features a testimonial from a prominent Republican businessman.

In an earlier post, I asked whether the Tester television ads were put up too early--and came to the conclusion that they were not. Well, I can certainly say that if Senator Baucus is putting up radio ads in 2012 for an election in 2014, that can only signal trouble. Not that it is a secret...Senator Baucus' approval ratings have been on downward slide since 2008. Last year's Billings Gazette poll found his approval rating at only 38 percent, certainly not encouraging for someone hoping to get reelected. Clearly, the radio ad is meant to stem that downward spiral and help Montanans remember why they've elected him for so long: he has political clout, seniority, and is often the pivotal voter in the Senate. His central position politically is only accentuated by Nebraska Democratic Senator Ben Nelson's impending retirement. Even if Democrats lose the majority, Republicans will not gain a filibuster proof majority--and will be looking to cut deals with conservative Democrats like Baucus.

All of that said, I'm shocked--shocked--that Senator Baucus would insert his reelection into the conversation during an already competitive Senate election. I'm not sure how this benefits Senator Tester--it merely detracts from his media message and his efforts. I'm puzzling through this one and coming up empty.

On the Tester-Rehberg poll, there's not much to be said. The race is close--we already knew that. Perhaps the one thing worth mentioning is this is the first poll taken after the emergence of two Libertarian candidates. And, interestingly, these candidates poll about 6 percent of the vote. Of course, with a margin of error at 4.5 percent, support for a third party candidate could be as low as 1.5 percent or as high as 10.5 percent. But note that only THREE percent of voters did not express a clear choice. Lots of money will be spent trying to help those voters figure things out (and lots of money will be spent making sure those who have already made up their minds get out and vote). Another Montana blogger, James Conner, crunched the numbers in this year's Tester-Rehberg polling and compared them to the Burns-Tester race in 2006. He concludes that the patterns might suggest some concern about Senator Tester's reelection. You can read his analysis here.

Finally, the AP piece on campaign expenditures and reimbursements. The piece essential puts a pox on both Republican and Democratic houses, suggesting neither candidate is the frugal, average guy they purport to be. I guess, as an observer of politics, my reaction was "meh". When individuals travel on business, they get reimbursed for their travel and meals. What's all the fuss about? If a candidate travels on campaign business, they get their expenses reimbursed. Sometimes some expensive meals might happen at a fundraiser. Sometimes the fundraiser is held at a nice hotel to encourage big donors, and you have to pay a large amount for a room in a hotel that is in a good location.

The fuss is, of course, is that both Rehberg and Tester represent Montana. And Montana is a poor state that is not too kind about others making too much money or looking like they have gotten too far ahead of the Joneses. This is especially true when it seems that public figures are profiting from their public position. In another state, this type of story might never have been noticed or even written. But in a state--and in a race--where that "one of us" connection is so important and a contestable point between the candidates, this article seems to suggest that neither Senator Tester nor Congressman Rehberg are as "one of us" as they would have us believe.

Is that a problem? Politically, it might be. While it doesn't bother me much that a member of Congress might have a nice dinner at fundraiser once in a while or stay at a nice hotel, I can see why it might upset folks who can barely make ends meet. And even if though these bills and reimbursements are standard operating procedure in campaigns (and, from my limited experience, they are), one thing that both candidates might do to avoid negative press in the future is to apply the government travel and meals reimbursement limits to their campaign travel, too. At the very least, this would avoid a story like the one that appeared in the paper this morning, which draws attention away from their core themes and messages. In the end, however, the story serves to increase the cynicism of voters and as someone who studies Congress, I'm not sure we need to be doing much of that right now.