Lessons in Statistics: The New MSU-Billings Poll

I saw on my Twitter feed that MSU-Billings has come out with it's latest Montana public opinion poll. You can read the article about the poll findings here and go look at the full report here. I take some issue with how this poll was reported and how some folks are talking about it in the Twitter-verse.

First, let's talk gay marriage. The poll results indicated that a plurality of Montanans favor gay marriage, with 46.6 in favor and 42 percent opposed. The problem with this statement is the margin of error in the poll is 5 percentage points. This means a majority of Montanans might actually support gay marriage (5+46.6 is 51.6 percent) OR less than a plurality support it (46.6-5 = 41.2). So, is not accurate to say that a plurality of Montanans support gay marriage. It would be better to say that opinion on gay marriage is mixed and that support for gay marriage is statistically tied with opposition to it.

Second, let's talk about job approval ratings. Montana Cowgirl, one of the Montana state politics blogs I follow, sent a Tweet saying that Steve Daines is the most unpopular politician in the state. Here are the job approval numbers in the poll:

Steve Bullock, Approval: 53 percent.
Jon Tester, Approval: 44.7 percent
Max Baucus, Approval: 44.1 percent
Steve Daines, Approval: 39 percent

Remember that margin of error? We can confidently say that Steve Bullock is more popular than Steve Daines because the Bullock's approval rating could be as low as 49 percent or as high as 58 percent. Bullock might be as popular as Tester and Baucus, and Steve Daines might be as popular as Tester and Baucus. Or Steve Daines might really be the most unpopular elected official (39-5 = 34). But we cannot confidently, based upon these results, say that Steve Daines is the most unpopular elected official with the large margin of error.

What is most interesting, however, the number of folks who are undecided about Daines and Bullock relative to the senators:

Baucus, undecided: 16.9
Tester, undecided: 15.9
Daines, undecided: 37
Bullock, undecided: 33

What this tells me, and I can stay this with statistical confidence, is that Steve Bullock and Steve Daines are the least KNOWN statewide elected officials. And, look at the disapproval numbers:

Bullock, dispprove: 14
Daines, disapprove: 24
Tester, disapprove: 39.4
Baucus, disapprove: 39

Looked at this way, I could say with confidence that more Montanans who can express an opinion on on Tester and Max Baucus neither clearly approve or disapprove of the job they are doing. I can also say, however, that among those who express an opinion on Daines' job that more approve of the job he's doing than disapprove (39-5 = 34, and 24+5=29). He and Bullock are the only ones according to these numbers who have a statistically discernible pro-job approval rating outside the margin of error.

Clearly, among those Montanans who express an opinion on Steve Bullock really like the job he's doing. I suspect that part of that has to do with a) Bullock's not associated with the mess and Washington and b) the legislature has gone home, so he's not associated with any divisive doings in Helena at the moment. Main lesson: It's good to be the Governor when Washington is blowing up.

All three of our federal officeholders likely have depressed job approval numbers because of the government shutdown and the debt crisis. What will be interesting to watch is whether Daines, in particular, can increase his approval ratings among those who are currently undecided about him.

How Not to Abuse Statistics

This past weekend, former Speaker of the Montana House of Representatives wrote a guest column that appeared in the Bozeman Chronicle. Read it here.

As a social scientist, I stress to my students the importance of using statistics correctly when developing an argument. To use statistics properly and to use them well, it is incumbent upon the person making the argument to place those statistics in their proper context.

What is notable in Mr. Sales' column is how he uses context when it serves his argument, and ignores context when it does not. I should make it clear that Mr. Sales is not alone in doing this; folks on both the right and the left are guilty of cherry picking statistics.

In any case, I wrote the following response which appeared today in the Chronicle:

To the Editor:

I read with interest Mr. Sales’ guest column yesterday. Although Mr. Sales provides a lot of information suggesting that the state of Montana is profligate, he fails to put those figures into their proper context. One example makes the general point: Mr. Sales makes much out of the fact that 780 state employees earn more than $100,000, a number that has increased by 278 since 2005. This is not justifiable, he claims, given that Montana ranks in the bottom 25 percent in personal income.

Mr. Sales fails to explain, however, who exactly receives such “outrageously” high pay. A quick perusal of the one hundred highest paid employees in the state (a search readily available at the Montana Policy Institute’s website at http://www.opengovtmt.org) reveals that nearly all are doctors, psychiatrists, lawyers, judges, and top administrators—some of whom have worked for the state for more than two decades. The accurate yardstick with which to judge these salaries are national or state medians for these positions—not the median salary of all Montanans. These positions require extensive education and experience, and the amount Montana provides in compensation is not out of line with private sector salaries. In fact, these state salaries are likely much lower than the amount such professionals could obtain in private practice. Many likely serve the state of Montana out of a sense of civic duty and pride.

I do not disagree that Montana faces some tough budgetary choices; however, factual arguments divorced from their proper context certainly do not return us to the “values that [have] served us so well.”

Context matters. Indeed, it is everything.