Melman: How polling organizations’ behind-the-scenes decisions affect visible poll results

I never intended to be a poll commentator. In fact, I’m more of a supporter of polls.

Nate Cohn of the New York Times is one of the most famous members of our profession, careful, thoughtful, creative and transparent, and on Monday nights he writes my crap about the latest polls. He also answered my questions.

Mr. Cohn’s general conclusion from a poll conducted by The Times and Siena College in six battleground states is consistent with my argument here a few weeks ago: The presidential election is on a knife’s edge. is in danger of.

But I would like to use two specific findings to highlight the importance of the decisions polling organizations make behind the scenes.

The first is the difference caused by sampling and weighting.

The Times publishes results for all registered voters and two audiences of likely voters. How do pollsters define likely voters, whether they sample likely or likely voters, and whether and when they survey all registered voters? or how to weight the various tiers, etc., but there is often little difference.

In fact, in four out of six state polls, the difference between registered voters and likely voters is less than one point between President Biden and President Donald Trump. In the fifth poll (Wisconsin), the gap is slightly larger by 3 points, with Mr. Biden leading by 2 points among registered voters overall, but by 1 point among likely voters.

And then there’s Michigan.

In the Great Lakes state, President Biden is 7 points behind registered voters, but ahead by 1 point among likely voters, effectively giving him an 8 point lead.

Some news outlets report on results for registered voters, while others focus on people who are likely to vote. Both accurately reflect public opinion polls, but different sampling decisions can tell very different stories.

There’s nothing sinister here. I don’t completely believe them (nor does Cohn), but the Michigan results suggest that behind-the-scenes “technical” decisions can have a big impact on polls. It shows that.

Another backroom decision involves coding responses to open-ended questions.

The Times-Siena poll asked, “What is the most important issue in deciding your vote this November?” Voters were interviewed and answered verbatim, faithfully recorded in their own words.

To make these words meaningful, we need to classify them according to some rules. Therefore, answers such as “I can’t make ends meet every month,” “Food costs are too high,” and “Inflation” are all included and counted under the heading “Inflation and the cost of living.”

After analyzing these data, Cohn attributes the president’s political woes in part to his support for Israel in its war against Hamas. “About 13% of voters who voted for Biden last time but do not plan to vote again said Biden’s foreign policy and the war in Gaza were the most important issues in their vote.”

A headline writer who wasn’t paying much attention changed the headline to, “13 percent of voters who switched support from Biden cite Biden’s Gaza policy.”

Cohn’s explanation is accurate, but a bit slippery and the headline inaccurate.

To reach his conclusion and generate the 13 percent, Cohn combined two different categories of open-ended responses. A much smaller group (only 2%) specifically mentioned the Middle East, Israel, Palestine, and Gaza.

The larger group cited some aspect of general or specific foreign policy as a top priority, but did not mention Gaza or related terms.

Cohn said some of those people made vague references to “stopping wars” (such as), but about which wars (Gaza, Ukraine, Myanmar, Sudan, the Maghreb, Somalia, Syria, to name a few). It is said that he did not explicitly state that.

However, most of those who mentioned foreign policy did not mention war. It is unclear which aspects of Biden’s foreign policy were their focus.

In other words, Mr. Cohn added a very small group that mentioned the Middle East to a large group that made no mention of Israel, Palestinians, or Gaza, as if they were saying the same thing.

Furthermore, an unspecified majority of those at the center of the Hamas conflict believed that the president was giving insufficient support to Israel.

Mr. Cohn would no doubt answer that these people are more sympathetic to Palestinians than Israelis, but as I have argued here before, that is fundamentally the wrong question. For many people, “whom you sympathize with” means “whom you sympathize with”, which is quite different from who you support.

However, the important point here is that the decision to mix two seemingly disparate categories of responses had a significant impact on the analysis and interpretation of the data.

We rarely get to see exactly what’s going on in the backrooms (or more precisely, the laptops) of pollsters, but it can have a huge impact on the visible results.

Mr. Mellman is a pollster and president of the Mellman Group, a political consulting firm. He is also the president of the Israeli Democratic Party majority.