When Nate Silver at FiveThirtyEight released the 2020 pollster ratings, he wrote an accompanying article saying that live-caller polls are no longer the gold standard.
Here at Cygnal, we’re glad Nate Silver finally agrees with us. More than five years ago, Cygnal pioneered what’s now called “multi-mode sampling” in an attempt to reach more voters where and how they like to communicate while maintaining the same high standards of results we’ve held since our founding. In FiveThirtyEight’s polling review, Silver and his team found that hybrid polling produced the least amount of error of any polls during the 2020 cycle.
Pollster ratings can change over time but identifying future trends and best practices for providing quality polling will always be our mission, and the next section details that it absolutely paid off.
Cygnal’s 2020 Boasts a Stellar Record
According to the objective data underlying FiveThirtyEight’s ratings, Cygnal held its position atop an impressive group of the best performing pollsters of the last cycle according to the metrics that matter. We called 92% of all races correctly and it wasn’t just getting the winner right. Of all pollsters that had 10 or more polls analyzed, Cygnal was the only Republican pollster with absolutely no races–that’s right, 0 races–with an error outside the margin of error. Additionally, Cygnal predicted the correct winner in 95% of our pre-election polls.
If you are looking for a pollster that tells you both who is leading and how far ahead they are, you only have to look to our last cycle where the proven multi-mode polling methodology helped us provide accurate information. You can compare our results to our competitors using the chart below, which compares Cygnal’s 2020 cycle to other Republican pollsters with the best comprehensive record on races called correctly and misses outside the margin of error.
Why Did the Letter Rating Drop?
With the best record of any Republican polling firm, we would expect Cygnal to maintain one of the highest letter grade ratings of all pollsters. If you’ve followed Cygnal’s growth over the last few cycles, you might be surprised to see our letter grade rating at an A/B instead of an A. Rest assured, the letter grade reflects a change in FiveThirtyEight’s rating methodology and not a deterioration of our quality.
Now that FiveThirtyEight has dropped the bonus for live polling, their rating matrix instead places a lot of weight on being a member of a professional polling organization, like AAPOR or NCPP. However, in order to comply with the standards of these organizations, pollsters must hand over a lot of confidential sample data. While we agree with the purpose behind AAPOR and NCPP’s emphasis on transparency, publicly releasing source data would place our client’s strategic advantage at risk. Thus, the change in FiveThirtyEight’s letter grade matrix benefits private pollsters who have syndicated research with media outlets, which have more flexibility to publicly release their data sets.
While professional membership is an easy way to get a better pollster rating, it does not make for better polls on its own. It’s the equivalent to saying that the prescription pad a doctor uses is as important as the medication and dosage prescribed. One is clearly more predictive of the outcome than the other. The chart below shows pollsters grouped by their FiveThirtyEight letter rating and whether they were marked as an AAPOR or NCPP member by how many races were missed outside the margin of error.
Once you leave the A rating, polls that had their grade inflated by professional membership produced worse polling overall than non-members. You may recall that Cygnal had 0% of our polls outside this range but in the similar A/B range non-members significantly outperformed those who received a grade bump due to registering with one of these groups.
We believe that, like our turnout frames, grades are only as good as the variables you put in. Private pollsters are at a disadvantage because of this but we pride ourselves on the fact that your data will remain private and that our polling record and constant methodology innovation will provide potent strategic advantages.
So with that, congrats to the Cygnal team for being the most accurate Republican polling firm in the nation for the 2020 cycle based on the metrics that matter–accuracy and margin of error!