METHODS READINGS AND ASSIGNMENTS REVISED CALENDAR HERE

EDF 5481-2 METHODS OF EDUCATIONAL RESEARCH  FALL 2017 GUIDE 5: A SURVEY RESEARCH TIMETABLE SUSAN CAROL LOSH

OVERVIEW

In survey research, the investigator asks verbal questions (either aloud or on paper or screen) and either an interviewer or the respondent records the answers.

If you do these two steps, you are conducting a survey, even if you call it a "standardized test," a language assessment, a "personality profile" or some other high-falutin' name, or if you embed a questionnaire inside your study. Public opinion surveys or polls are simply one special kind of survey, a survey that seeks to generalize its results to a well-defined population.This means that no matter what name the research is called, if you ask questions and record the answers, the same rules for good question construction apply, and the data are subject to the same methodological problems as most other surveys are.

Be aware that surveys "try to hit a moving target". People change their knowledge, beliefs, and attitude responses, in part because of the reactivity that occurs as a result of being studied. In terms of reactivity, surveys typically rank below experiments, but above field research, such as ethnographies or more secondary collections, such as content analysis. And, of course, people change their beliefs and attitudes as they learn new things, obtain new information, or as a result of persuasion campaigns. Think how sad it would be if we did not.

Everyone must be concerned about sampling issues. I elaborate upon them here because excellent sampling has been a hallmark potential strength of general public opinion surveys in the recent past. However, sampling directly addresses external validity, and thus is a feature of any kind of study design. Who wants to spend months doing a study, only to conclude that generalizations can only be made to a particular group of students taking an online course and no one else?

Most scholars are more ambitious than that.

Because it typically deals with naturalistic variables, there is less internal validity in a survey than in an experiment but possibly as much as a quasi-experiment. Statistical control, in which one measures as many intervening and confounding variables as possible, then statistically control these, typically substitutes for experimental control. Public opinion surveys generally define their populations very specifically and take careful samples from their populations, thus usually have had very good, sometimes excellent, external validity, although internal validity or causal connections can be difficult to establish.

Unfortunately, response rates have fallen sharply in recent years. Survey researchers have gone from face to face surveys (very expensive) to random digit dial surveys, to including cell phones to online surveys and probably everything in between. It is often unknown how respondents differ from non-respondents and external validity can be jeopardized accordingly.

Sampling error, which we hope is truly random, and which lies behind most attempts to measure variation from survey to surve, is only ONE form of error. Instrumentation presents many problems as can administration errors.

Review the rules on causal inference and establishing independent or dependent variables in nonexperimental research HERE. Think about these rules when you study variables included as independent variables in a report that you read. Be sure that these independent variables are plausible.

Of course, a scholar can embed an experiment within a survey, and randomly assign who gets which questionnaire version. Professional survey researchers do so all the time. Often we call these "split-ballot" studies.

Consider what demographics "stand for" in survey research. Variables such as "age," "gender" or "ethnicity" are useful because they condition life experiences. By the same token, these are sometimes confounded variables that are "proxies" for multiple sets of experiences. Older people have different options, have had different life experiences, and are treated differently than younger people. People with different educational levels have had differential access to knowledge, and college students have experienced a whole different culture than those who never went to college. Thus, think about what each demographic stands for and see if the researcher could have directly included those variables in the study. (For example, if you believe that a key with "age" is experience handling money, such as wages, checking accounts, or credit, were questions related to wages, checking accounts and credit included in the survey, in addition to "years of age.")

FOR EXAMPLE: I've worked with several surveys about disentangling confounded effects. OPTIONAL: Check out the study I co-authored with my students in the September 2003 Skeptical Inquirer ("Sam Savage" wrote the synopsis; reprinted several places online). We found that most of the variation due to "level of education" was due to something else entirely: either correlates of education level (e.g., age or gender) or specific products of education (e.g., the total number of science classes.)

To repeat: "Educational level" is a highly confounded variable.

A second study compared age and generational effects, which are totally confounded in a cross-sectional survey at one point in time.  Virtually all the knowledge effects typically assumed to be due to age are due to generation (birth cohort) instead.* This study, which examines science knowledge in the American adult general public was published in Martin W. Bauer, Rajesh Shukla and Nick Allum (eds.(2012), The Culture of Science - How does the Public relate to Science across the Globe? NY: Routledge: 55-75   OPTIONAL: it can be accessed online HERE. In yet another study, I found that generation influences access to and use of information technology but age generally does not.

*It is, of course, nice to know as one ages that generation plays an important role and we can hold on to our "minds" as we grow older.

There are several steps to conducting a survey. The most elaboration to those steps related to collecting data and the study design is in Guide 1, leaving data management and statistics to your other methods courses.

REVIEW: ASSIGNMENT 1: CONSTRUCTING A RESEARCH PROBLEM

CHECK FOR DIFFERENT TYPES OF SAMPLES AT THE SAMPLING WEB SITE, CLICK HERE!

1. Define the population. Make the definition clear enough so that there is absolutely no question about who is or is not in the population.

• GOOD EXAMPLE: All students registered for at least one credit hour at Florida State University Fall 2017.

• BAD EXAMPLE: All Florida State University students.

3. Locate or construct the most thorough list of the population as possible. This will become the "sampling frame." If a complete list is not possible, see if sampling stages can be created, and then completely enumerate all elements at each stage.

4. At about the same time, the researcher must decide on the type of administration for the survey: self-administered (mail, group, or, increasingly, Internet) or interviewer-administered. Some types of samples, such as simple random samples, can be done with straightforward logistics by telephone (Random Digit Dial or RDD) or online but are often difficult to do in other circumstances (such as area in-person surveys). The increased use of cell phones rather than landlines has complicated RDD. Except for single institutions (a business or a university, for example), it is often difficult to get a good list of email addresses. Remember that virtually all WEB surveys are self-administered. If the population is not very literate or relatively accustomed to being online a lot, an Internet survey is a very bad idea.

My recent experiences suggest that online survey response tend to be poor, even with a prestigeous sponsor or incentives (e.g., a "lottery" for a gift card). However, in some instances they can be better than random digit dialing.

Furthermore, incentives such as lottery-type drawings, e.g., for a gift card, generally do not increase response rate although a small guaranteed incentive (even a "bright, shiny quarter") can.

5. Make the final decision on the type of sample to draw. Try for a probability sample. Accept that under certain circumstances this may not be possible. Decide whether in fact the researcher is taking the entirety of a small population instead (a census).

6. Draw the sample. If appropriate, send out an introductory letter or email. Include inducements (shiny new quarter, Barnes and Noble gift certificate!) if feasible.

Human subjects regulations at FSU specify either that everyone sampled receives the incentive or is entered into some kind of lottery to be awarded the incentive.
 
 

SAMPLING COUNTS IN EXPERIMENTS TOO! EVEN IF THERE IS HIGH INTERNAL VALIDITY, ONE CAN'T LEGITIMATELY GENERALIZE IF THE SAMPLE IS POOR (LOW EXTERNAL VALIDITY). A BIG PROBLEM NOW IS VERY LOW RESPONSE RATES IN SOME SAMPLES.

I bet you receive requests at least once a week in your email: Internet surveys are proliferating. Currently, well over 95 percent of the United State population is "online" from home (counting smart phone access and tablets). This is NOT randomly distributed. The "digital divide" still exists. Single women, minorities, the less-educated all have lower online access. Wealthy agencies, such as Stanford, can afford to give respondents selected in other ways their own Internet connections and laptops so that the whole sample can respond. Web surveys clearly work better with highly literate and motivated respondents. Thus, they become similar to other self-administered questionnaires, such as mail surveys.

Response rates are also similar to mail surveys and are generally low but they can be high (see above for motivation). These days they can be higher than for telephone surveys. However response bias is still relatively unknown.

Build in safeguards so that respondents can only answer the survey once. I usually try this with any online survey I receive to assess how professional the survey is. Imagine my shock when the FSU Presidential Search Committee constructed both its surveys and its "Comments" submissions in 2014 so that the same person could submit multiples--and judging from the responses, many people did.

Also remember that the researcher should choose the respondents. If she or he "advertises" and takes whoever answers, there's not a whole lot of difference between that survey and dialing in to one of Entertainment Tonight's 900 numbers. Both are self-selected samples.

Presently, WEB surveys appear to work best with subsamples of the population who are (1) already online (preferably from a computer rather than a smart mobile phone--slightly easier to write), (2) who are interested the topic, and with (3) with whom the researchers generally stay in touch (e.g., a class, a club) to remind and prompt.
 

WE TEND TO ONLY THINK OF FORMAL INTERVIEW SITUATIONS AS "SURVEYS." BUT DOCTORS, LAWYERS, OR THERAPISTS WHO ASK PATIENTS OR CLIENTS QUESTIONS ALSO SHOULD USE CAREFUL QUESTION CONSTRUCTION AND INTERVIEWING TECHNIQUES. AND "STANDARDIZED TESTS" ARE JUST AS SUSCEPTIBLE TO QUESTION FORMAT RESPONSE SET PROBLEMS AS SURVEYS AND NEED TO FOLLOW RULES FOR GOOD QUESTION CONSTRUCTION.

 

FOR SOME RULES OF GOOD QUESTION CONSTRUCTION, CLICK HERE! FOR A QUESTIONNAIRE EXAMPLE, CLICK HERE!

The same is true for mail or online surveys. These should be short--and pilot tested to see just how long they take. Long surveys depress response rates and introduce probably unmeasurable biases.

Young, mobile adults are more likely to be "cell phone only." Until the last few years, researchers could not [knowingly] random dial cell phone exchanges because many plans charge subscribers for in-coming minutes and it is currently illegal to charge people to take part in a survey. We are now getting more data (and surveys) from cell phone users. If the individual is on land-line or cell, the researcher faces a thicket of answering machines, caller id, voice mail and other privacy oriented options. According to the Director of the UVA Survey, these depress response rates, especially caller ID. Concerns about "hacking" and identity theft are adding to response rate problems.

With telephone surveys, try everything possible to increase the response rates:

• NO ROBO-CALLS OR ROBO-SURVEYS. (Their response rates are even lower, often in the single digits.)
• Personable interviewers.
• CALL BACKS.
• Refusal conversions.
• And don't be in too much of a hurry, no "3 day surveys" if at all possible.

People dislike writing by hand. This includes people with doctorates. If the questionnaire has many open-ended questions, use an interviewer. Otherwise the sample response rate will be low and the open-ended item response even lower. Concentrate on closed questions for self-administered surveys when possible.

General public participants don't like doing a LOT of typing online either and most aren't used to writing long papers that way (even if the investigator is).

If using a self-administered survey, keep page clutter to a minimum. Use a lot of white space. Don't use complicated charts for people to complete.

ALWAYS PILOT TEST THE QUESTIONNAIRE ALOUD!

Among other things, always a good idea to see if people know ANYTHING (or have even heard of) the topic.

Do this even if the questionnaire will be self-administered.

Pilot test the questionnaire on respondents who are comparable to the selected population on key issues such as education, gender, and ethnicity.

Revise the questionnaire. If needed, do a second pilot test. This is no place to cut corners!

I am out of the University of Michigan Survey Research Center training. Our questionnaires tend to have  a more casual, "chatty" feel to them which we believe encourages responses and relaxes respondents.

The very worst example of the University of Michigan style that I have ever heard--which was only partially presented as a joke--went as follows:

"A lot of people think about suicide. How about you? How many times have YOU thought about killing yourself?" (Not at all was not a response category.)

OUCH. The idea of introducing the "A lot of people..." is so that the respondent will not feel deviant and to legitimate both the question and the respondent's answer. This is supposed to help with questions in sensitive areas, such as drinking, illegal drug use, sexuality, birth control, and failing to keep to your diet.

And remember, little kids (e.g., 5th grade on down) are not very good at writing, and that includes self-administered surveys...
 
 

STEP FIVE: INTERVIEWER SELECTION AND TRAINING (IF APPROPRIATE)

 

When was the last time someone asked you what you really thought, cared about what you said, and LISTENED to the answer?

However, as you might guess, a survey that uses interviewers is more expensive. The researcher must also budget time to train interviewers, and interviewers should be periodically monitored to ensure that they do a good job. Don't use ROBO-CALLS or ROBO-INTERVIEWERS. Response rates are the worst.

The General Social Survey conducts in-person (usually in household) interviews with at least 3000 people every two years. Each (recent) survey costs about $3,000,000.

Nearly all professional telephone surveys these days are done via CATI: Computer Assisted Telephone Interviewing. CATI has many improvements over the old paper and pencil questionnaires. In person interviews are now often done with a laptop that has the questionnaire loaded on it:

• Virtually all questions receive answers, because the program will not advance to the next question until an answer is filled in. Interviewer omissions are kept to an absolute minimum.

• The program handles all skip patterns (for example, a section in which questions are only asked to teachers, diabetics, or some other special group) and much more flawlessly than an interviewer can.

• Virtually all CATI programs provide for verbatim open-ended answers. Some will even record voices to make sure the complete answer is recorded.

• Virtually all CATI programs allow interviewers to change answers if they hit the wrong button or a respondent changes his or her mind on an answer.

• Except for coding open-ended questions, coding is eliminated as a separate step because when a survey is complete, the closed question answers immediately enter a data file.

Newer hardware/software packages include voice recognition software, although the current quality varies enormously across companies.

Good interviewers are:

• Interested in others
• Friendly
• Empathetic; able to view the world through the respondent's eyes
• Tolerant and nonjudgmental, because they will hear a lot!
• Objective
• Bright--but genius level is not required
• Firm (because "ramblers" should be discouraged)

• Sounds like a telemarketer or salesperson. They turn respondents off--FAST. No "fast-talkers!"
• Wants to use the interview as a forum for THEIR OWN beliefs
• Has literacy problems. Remember they must write down complete answers to open-ended questions.
• Dislikes certain groups of people.
• Loses their objectivity so that interviews take way too long and stray from the topic.
• Saids "I'm shy."
• "Too" empathetic. Interviewers are not psychological counselors. Also, if interviewers create too much rapport, respondents tend to become withdrawn because the interviewer comes off too much like a "friend."

Have a solid interviewer manual--also read in advance.

Have interviewers practice in teams and practice role-playing respondents.

Have interviewers practice saying in to the mirror (for in-person interviews:) "I'd like to come in and talk with you about this" OR (for telephone interviews:) "The interview will only take about 10 minutes; here's the first question."

MONITOR ALL INTERVIEWS! Check call on in-person interviews. Spot monitor on telephone interviews.
 

FOR ONE INTERVIEWER'S MANUAL CLICK HERE!

THESE STEPS ARE FOR LATER COURSES:

 

STEP SIX: CODING AND DATA ENTRY

Not necessary for most CATI programs. Researcher receives usb or other storage device with data. If data must be manually entered, select a random sample of cases and re-enter to validate. If large numbers of errors (over 5 percent) are found, consider checking entire database.

 

STEP SEVEN: INDEX CONSTRUCTION (if appropriate)

 

STEP EIGHT: UNIVARIATE DESCRIPTIVE STATISTICS

 

STEP NINE: MORE COMPLEX MULTIVARIATE STATISTICS

 

STEP TEN: REPORT WRITING AND DISSEMINATION

METHODS READINGS AND ASSIGNMENTS

OVERVIEW

Susan Carol Losh
October 3 2017
This page was built with Netscape Composer