Monday, September 18, 2006

Monitor Poll Flawed

Update: Since this post, my claim that the poll was flawed has been backed up by every poll released. While only one of 4 polls since has shown Hodes in the lead (with one showing a tie), the Monitor poll is an outlier by more than 5 points in each direction. For posts on the subsequent polls, see here and here.

The Concord Monitor is reporting a poll that doesn't look great for Paul Hodes. Here's what they tell us about the methodology:
Bradley led Shea-Porter 56 to 31 percent, according to a Monitor poll of 300 likely voters in the First Congressional District (with a 6 percent margin of error). In a poll of 300 likely Second District voters, Bass, a six-term incumbent, led Hodes 55 to 30 percent. Libertarian Ken Blevens was supported by 1 percent of the vote.
Research 2000 used randomly generated telephone numbers to interview 600 likely voters. Those interviewed - 180 Democrats (30 percent), 192 Republicans (32 percent) and 228 voters who identified themselves as independents (38 percent) - reflect voter registration numbers statewide. The interviews were divided evenly between the two congressional districts, and they included 294 men and 306 women.
First, we have to note that that's a pretty huge margin of error.

Next, I have some questions, drawing on a Mystery Pollster post from 2005 in which 19 pollsters discuss their criteria for evaluating a poll's reliability.

The partisan breakdown matches statewide, but does it reflect the differing partisan breakdown of the two districts, or does it see each district as essentially a subset of the whole state?

They were doing random digit dialing, but from what source of numbers? What type of voter list were they working from? Did they know anything about the voting histories of the people they were calling? They claim these were likely voters, but what kind of likely voter model was it?

What was the age distribution? (In particular, it's very easy to overrepresent older voters.) The geographic distribution? Race and ethnicity are probably somewhat less of an issue in New Hampshire than in many states, but nonetheless important, so what about them?

This poll was apparently done in two nights, in contrast to the poll done by Anzalone Liszt Research for the Hodes campaign, which called the same number of people (but all in the Second District) over a period of a week; as Jeff Liszt says in the Mystery Pollster post, "Very large samples taken in one or two nights sometimes raise a red flag because of the implications for the poll's call-back procedures."

In short, we just don't know enough about this poll to really evaluate it, though the large margin of error and the short time period in which it was done suggest that it was done on the cheap. So...if you want to know where I'll be over the next few weeks, the answer is probably going to be "out canvassing," because that's clearly an important part of this race. But while I'm certainly motivated to work like we really need to make up this differential, I'm not going to be losing any sleep over these results.

It's also worth noting, as The Yankee Doodler does, Bass's poor showing in the primary. At the same time, as I think about the fact that this poll was done in the two days following the primary, Bass may well have been benefitting from a related media boost. Because he had a contested primary, Bass was much more the focus of television and print coverage of the primaries, so this poll was done during a mini-blitz of media for Bass that was not equaled by coverage of the unopposed Hodes.


Blogger IndyNH said...

I think it is a huge mistake to shoot at these polls, it makes us look naïve. Every independent poll has almost exactly the same numbers while only our internal numbers have this as a close race.

Let’s admit the truth: if the election were tomorrow Charlie Bass would win by 20 + points. I hate it, but it is true. The good news is that the election is not tomorrow.

I am sure the Bass people read this and other blogs. Let’s be sure they know we are not a bunch of idiots who think just because we write something on the Internet that we believe it is true.

What will frighten them more is for us to admit the current reality and yet keep working hard day in and out. That will make them wonder exactly what we might know that they don’t.

10:42 AM  
Blogger MissLaura said...

Well, considering that this morning as I sat here volunteering at Hodes headquarters, a woman came in and said that she'd been polled on the race, only the caller had asked whether she supported Bass or Hodds, and was surprised when she corrected the pronunciation of Hodes, I feel that it's legitimate to question the polling techniques being used in this race.

12:40 PM  
Blogger trendite said...

I completely disagree with the statement that Hodes would lose if the election was held today. Isn't that what they said about Shea-Porter? In a year like this, the definition of "likely voter" is fuzzy at best.

Besides that, the fact that Charlie has not released his internal poll tells us that it was not good news for him -- not nearly as good for him as the recent Venture 2000 poll (that is the name right?). I guess the question is, if these numbers are so pat, and so reproducible, why won't Charlie release his internal poll?

My honest-to-god guess is that sampling of "likely voters" is everything in this election... we are possibly looking at a 1994 situation here where what "likely midterm voter" means determines the election...

And I sure would like to see that Bass internal poll...

6:18 PM  
Blogger IndyNH said...

The poll in question is not a Bass poll or a Hodes poll, I expect them to be biased. The one making the news right now was done by the Concord Monitor (see 'Pretty grim' poll results for Hodes campaign). And the one before that was from UNH. The interim one we put out and any the Bassmaster might trot out are of course going to be suspect.

In my view, we can pull these polls and samples apart all we want, but we are better off recognizing that we are down and start acting accordingly. I don’t understand why it is a good idea to fool ourselves that anything other than that is true. I do not think that other people will read this blog and get excited or turned off to Hodes based on what we say. They need to hear from Paul himself. We should help him get his message out and not try to spin an obvious cow’s ear into a silk purse.

8:21 PM  
Blogger MissLaura said...

You're absolutely right, indy, that the way to campaign is as if you're down, and that's certainly what everyone associated with the Hodes campaign I've talked to is doing.

At the same time, I think there are very valid and important critiques to be made of polls like this one, which essentially apply a public opinion polling model to electoral polling - and that's not a reasonable leap to make. I also don't know how much you know about polling, but you need to look at things like margin of error, and this poll has a huge one.

So I'm arguing for two things simultaneously. It's incredibly hard to defeat an incumbent. It will take every ounce of effort everyone associated in any way with Paul Hodes can produce to do that. That said, I just don't buy this poll, for reasons having as much to do with my social science training as with my partisanship.

3:02 AM  
Blogger Chris Quincy said...

So...2 independent polls show that Hodes has no name recognition, hasn't consolidated his base, and down by a huge margin that no sampling error could possibly account for...

...but you're staking your assertion that Hodes is in good shape on the results of a phantom Bass poll that no one's seen and no one actually knows exists?

Do you realize how weird that sounds?

6:44 PM  
Blogger MissLaura said...

Heh. Yes, I can see where it would look odd. But that's part of why I tried to explain the methodological issues simply and clearly - because this isn't stuff that's totally transparent to the average reader.

In fact, though, in addition to the methodological reasons this most recent poll simply isn't worth paying attention to, the "phantom Bass poll" is crucial here. You can bet that either Bass or the NRCC has polled this race, and if their results made them happy, we'd know about it. And we actually do have evidence of such a poll, as noted in the third item of this Swing State Project post:

7:57 PM  
Blogger Keener said...

Bass's team has actually admitted that they are sitting on a poll, I'll find the reference.

It's results may be unknown, but it's not phantom.

You make good points, Chris, but I think Laura is right here.

I actually worried somewhat about the last UNH poll. If you look at my blog, you'll see that I assumed it was not a sampling error. I assumed the Dem/Repub mix shifted based on self-identification. I threw cold water on it, but I didn't actually attack the sample.

The internal Hodes poll though did make me think twice. You don't get a firm like that to shred their reputation as a publicity's a real poll, and the discrepancy has to be dealt with. My guess is we'll see much stronger polls coing out of the DCCC near-term, and from the UNH poll in three weeks or so.

So come back then and we'll talk.

9:25 PM  

Post a Comment

<< Home