Author Archives: Dr. Russ

About Dr. Russ

Senior Director of Research

More Objective Than An Ink Blot

Posted on by
Reply

Creating and analyzing open-response or unstructured survey questions is as much an art as a science.

This type of question serves two main purposes.  For the survey researcher, it is a gateway to the thoughts of respondents without the influence of researcher-provided options.  For the communications and outreach folks, it provides a vehicle for improving engagement with community stakeholders.  The dynamics between these two perspectives create a tension that affects both the question development and the response analysis.

The researcher is generally interested in specifics — List three things you like about product X.  The open response question specifies exactly what the responses should address ( given that it is open, however, there is no guarantee that respondents will comply).  This narrowing of the range of responses makes the task of coding and analyzing the responses comparatively reliable and efficient.

Being more interested in building engagement and trust, the communications and outreach folks tend to prefer more general, less structured questions — Tell us about your most recent experience with X.  Since respondents will feel less constrained in what they can address, their range of responses will be broader.  Thus, more categories are needed to capture the respondents’ views, making  the task of coding and analyzing responses more difficult and less efficient.

When designing a survey, it’s important to consider the purpose of an open-ended question. Is it purely for research interest, building engagement and trust, or both?  Will the data be used to augment the responses to other structured survey questions or to find anecdotal stories and evidence?  What is the tolerance for reliability and validity of the resulting data?  Answers to these questions will help survey designers construct more targeted, valuable open-response questions.

How Technology is Improving Open-Ended Questions

Posted on by
Reply

Quality surveys are intended to enhance communication and engagement between surveyor and participants. One of the best ways to create a quality survey is to include “open-ended” or “unstructured” questions, which allow respondents to write their own thoughts, rather than choosing from among responses provided by the researcher.  However, while this question type makes respondents feel that their honest feedback is valued, it also requires extra work for the people tasked with collecting and coding those responses into usable data.

All of that is changing with the advent of Natural Language Processing, computerized programs with the ability to code written language, thereby greatly reducing the cost and labor associated with human reading and coding of responses. While this technology is becoming more prevalent in research, there are still issues to be addressed, including the need to train the program to recognize appropriate answer categories and to ensure that the coded answers make sense.  Survey specialists, therefore, are not yet obsolete.

Automated technology holds great promise for K12 Insight.  Our research department is testing several Natural Language Processing programs to determine how they may bring additional value to our clients. We’ll keep you updated on our progress.

Don’t Measure the Individual by Measuring the Climate

Posted on by
Reply

In a previous post, I talked about the tendency of many school districts to use climate surveys as an evaluation tool, often to assess the performance of individuals, such as principals, assistant principals, guidance counselors or teachers.  Climate surveys are designed to understand whether the general conditions within a school promote a sound learning environment and, if they don’t, to identify areas for further investigation.  A climate survey is too blunt to detect the subtle actions of an individual.

A 360 Degree Feedback Survey is a more appropriate tool for evaluating individuals because the focus of the survey (e.g., a school principal) is held constant, with the perspectives of different groups measured and compared.  The survey includes a comprehensive set of items to capture the full range of expected performance functions, with teachers, office staff, other principals, assistant principals, parents, students and administrators rating the principal on each of those functions. The result is comprehensive feedback regarding whether or not the principal is fulfilling relevant duties.

Keep in mind that stakeholders will have varying perspectives and expectations, based upon their own relationship with the survey subject.  Staff might expect the principal to solve personnel problems, while the superintendent expects enforcement of district policies and parents expect the principal to uphold education quality.  A 360 survey is sensitive to all these variables.

Although survey tools can support an evaluation, caution should be taken to ensure they are not the only data used.  The true value of the 360 survey may not be recognized when it’s solely used as an evaluation or performance appraisal tool; however, the feedback it provides will help the individual improve.  Handled properly, this information can have a positive impact on overall performance of both the individual and the organization.

An Individual is Not the Climate

Posted on by
1

Wow, it’s November already.  That means that many school districts are starting to think about their school site climate surveys.  As I work with district staff to develop and customize these surveys for their particular needs, I’m finding that a critical conversation revolves around the purpose of a climate survey.

I mentioned in a previous post that a common mistake is trying to measure everything in a very detailed fashion, as opposed to being satisfied with broad indicators of the education environment.  Another misconception is that the climate survey is an evaluative tool.  Attempts to make the climate survey into an evaluation tool involve an attempt either to rate specific individuals or to use the survey as a type of self-assessment.

Let’s consider the climate survey as an evaluation of individuals.  When this occurs, individuals within the school — typically the principal, assistant principals and/or guidance counselors — are identified by name.  Questions are asked about how well the named individual performs various aspects of the position; i.e., how well does Assistant Principal Jones effectively address disciplinary problems?  While this approach may provide feedback on perceptions of the individual’s performance, it results in two possible outcomes that are counter to the purposes of a climate survey.

First, when rating an individual, rather than the broader climate, ratings may be influenced by personal feelings toward that person.  For example, Assistant Principal Jones may be well-liked and therefore rated highly, despite rampant student behavior problems.

Second, individual ratings run counter to the goal of a climate survey, which is to develop trust and cooperation among the school’s stakeholders.  If individuals are rated poorly, a natural reaction is for them to become defensive.  And defensive posturing stifles dialogue, reducing the possibility of future cooperation and improvement.

Now let’s consider self-assessment questions and their problems for a survey.  These items are usually stated in an “I” format.  Not all “I” questions are inappropriate for a climate survey, but these items can be problematic when they address positive behaviors that are expected of the respondent.  Because people want to appear positive, they will report that they do everything that they “should” do.  For example, what teacher would not agree with, “I help my students when they have trouble in my class”?  Virtually everyone will agree with these types of questions, regardless of whether they actually do the described behavior.

The results are unrealistically high ratings that provide no useful information for improving the school climate.

The bottom line is that climate survey items should avoid ratings of individuals, either singled out in the evaluative sense or in the self-assessment sense.  I’ll speak in a future post about appropriate methods for obtaining and using these more individualized types of ratings.

The Climate Survey — Accident of Evolution or Adaptive Tool?

Posted on by
Reply

It is no secret that the climate of a school directly impacts how well teachers teach and students learn.  If the climate is one in which students and teachers are concerned about their safety, lack necessary equipment, or are otherwise uncomfortable, their focus will be on those concerns rather than on academic achievement.  When school leaders want to address those types of issues, they need to understand how staff and students experience the school climate.

Typically, school administrators go about studying the school climate by conducting a CLIMATE SURVEY. Yes, bold and caps are warranted because this is a monster survey. It includes a large number of questions meant to address various details and nuances specific to the school climate. It evolves by committee, with district administrators designing it with input from school staff and parents.  Problems tend to arise because everyone on the committee wants to measure everything.  Parts of other surveys, such as safety or engagement, often find their way into the climate survey.  The result is a survey covering a swath of topics a mile wide and a level of detail a mile deep.

Who wants to take a survey like that?  Respondents get fatigued and drop out before finishing, while potential respondents don’t even try after hearing how long and confusing the survey is.  Naturally, the response rate ends up being disappointingly low, and district administrators don’t get the information they need.

Fortunately, there is a better way. The K12 Insight staff has created an “adaptive” climate survey.  With this approach, we do not ask everyone to answer every single question.  Instead, there is a set of core questions that address the broad areas of school climate like academic preparation, safety, student support, etc.  There is one question for each of these core items, and each respondent answers up to 8 core questions.  Districts then use these responses to track school climate quality from year-to-year.  If respondents give low ratings in one of these areas, the survey will adapt, asking additional questions to probe more deeply into specific areas.  For instance, if a staff member indicates that safety is an issue, additional questions are presented to understand the reasons for that perception.  This approach drastically reduces the length of a climate survey, while providing essential data to track information relevant to improvement efforts.

There is a trade-off with this approach. Since survey participants do not answer all possible questions, some detail is lost. The benefit, however, is that participation will be higher because the survey is more manageable.  And increased survey participation helps boost parents and broader community engagement in the overall district dialogue.