To understand what went wrong with the Crain’s/TDL/ABC “poll,” it’s best to look back at one of the most famous polling misses of all time. Ahead of the 1936 presidential election, the popular Literary Digest magazine sponsored a national survey of the contest between President Franklin D. Roosevelt and challenger Alf Landon. After conducting one of the largest samples in history, the survey projected Landon to win 57 percent of the popular vote and 370 electoral votes.
On Election Day, Landon received less than 37 percent of the popular vote and eight electoral votes. It turns out that the type of people who read Literary Digest and responded to the survey weren’t reflective of the electorate.
Today, an explosion of “polling” that claims to measure public opinions exists all over the internet. Previously the sole domain of entities like Pew, Gallup and academic institutions, we now see new surveys coming out almost every day, with varying methodologies and professionalism. Many in the media report on these click-driving surveys, regardless of how they were conducted.
Good public opinion research should be as representative as possible. A representative sample attempts to provide every potential respondent with the same chance of entering the sample. A representative poll of Chicago-area residents must give people in every community the same opportunity to take the survey.
But the data collection method in the Crain’s/TDL/ABC “poll” fails to meet that standard. The survey’s sponsors reached out to their personal networks—especially political operatives and insiders—and those reading their stories directly. That gave some well-connected people (like us) a higher likelihood of entering the sample than millions of others. This meant their disproportionately white and male network of political insiders and business executives were more likely to take the poll.
There are certainly deficiencies in modern polling. The areas where pollsters are working to address problems, including achieving full coverage of the electorate, declining response rates, omitted variables and the incorporation of multiple modes of data collection, have been discussed in detail elsewhere. But the Crain’s/TDL/ABC survey fails to correct for any of the problems present in public opinion research today—if anything, it makes them worse and creates new ones.
The massive design flaws in the survey are only part of the problem. The widespread media coverage of the index treated this instrument with the same seriousness as it would one from Pew, Gallup or another academic institution with a long record of measuring public opinion.
Reporters who have frequently requested significant assurances from us (methodology, crosstabs, etc.) for our proprietary surveys before reporting on any data we made public did not live up to their own rules. But the editors at Crain’s, The Daily Line and ABC7 have apparently decided not to release demographic crosstabs for the Index. A full understanding of how this survey was weighted and balanced would lead to greater confidence in its findings. These media outlets should live up to the standards for transparency and accountability that they would demand of any politician they cover.
What should media outlets do? As Crain’s does with other people’s polls, we suggest they consider the following:
● Does this survey reflect the audience it claims to reflect—and is it characterized properly?
● Does the survey’s methodology match up w/ the best practices of data collection/weighting as set by opinion-research associations like AAPOR and other experts?
● If the findings challenge conventional wisdom, what might be leading to that? Sometimes surprising findings are harbingers of a change in attitudes. Other times, they are just wrong.
Chicago is our home, and it is one of the great cities of the world. Chicagoans deserve a fair and professional look at how its people view the city and its leaders. Hopefully, next time, these outlets will be more responsible and provide a poll that all of us can appreciate with confidence.
Brian Stryker is a partner at ALG Research, a Democratic campaign polling firm, where he manages the firm’s Chicago office and Midwestern practice. Jason McGrath is a senior vice president at GBAO Strategies, where he conducts research and provides advice to progressive organizations and Democratic candidates. He is also an adjunct professor at the Harris School of Public Policy at the University of Chicago.