Quality over Quantity

qulaity-over-quantityAs the time of planning for the school year ends, I’m struck by a common question asked by superintendents. This question often comes from a fearful place — fear of data, fear of misinterpretation and fear of unwarranted criticism. While I enjoy alleviating these fears, I do worry about how common they are. This is a question many of us ask when confronted with the results from a survey or poll:

Can we trust the data if everyone did not participate?

Before I answer (yes, for those who just cannot wait), let me remind you of an iconic story from the 1948 U.S. presidential election. On November 3, 1948, the Chicago Tribune ran a front-page story with the headline declaring “Dewey Defeats Truman.” In a famous photo taken after the story ran, a beaming president-elect Truman is seen holding up the paper with its embarrassingly erroneous headline.

How did the Chicago Tribune get the election outcome so wrong? While the editors trusted a result predicated on a low participation rate, this was not the fatal flaw. In fact, their expert pollsters and consultants could have trusted the responses of these few voters if they had been representative of the voting population[1]. That is the key concept superintendents often forget, and therein stems their fear of survey results.

Representativeness is not a trivial concept. It’s what makes a result valid and trustworthy — even if participation rates are low. Considering representative samples can be very small, and unrepresentative samples can be extremely large, including demographic questions on a survey is essential to certifying its validity. These demographics must reflect the population parameters of the school district where the survey is being deployed.

In Massachusetts, for example, it may be necessary to ask the participant’s primary language, whereas surveys launched in Texas may ask how long the participant has lived in the school district. Superintendents often know the population composition of their constituent groups, so including these demographic questions on our surveys allows us to determine if they are participating.

So, why the fear?

Since superintendents often don’t ask demographic questions, they are criticized — fairly or not — for not providing a chance for everyone to participate. And while many of us take those end-of-survey questions for granted, those demographic inquiries about gender, income and number of children are often the most important.

If we forget to ask demographic questions, then we don’t know who’s participating. This leads to an incomplete community profile and less accurate data to interpret, which is the same fatal flaw that allowed Truman to beam and the Chicago Tribune editors to cringe.

Whether the survey asks respondents who’d they prefer to be president, or if their district should adjust school start times, when it comes to participation rates it’s always better to have quality over quantity.

[1] By “voting population,” I am referring to the citizens who do vote as its own population. I am not assuming, nor should anyone, that the voting population is representative of United States citizens.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s