Survey Designs

Surveys & Questionnaires

Functions of Quantitative Designs
• Purpose of...
–Experimental Research is to draw conclusions about cause and effect
–Comparative Research is to examine relationships between variables
–Descriptive (Survey) Research is to summarize characteristics of a particular sample or population


Types of Surveys

Cross-Sectional –data is collected once from a variety of different groups

Longitudinal –data is collected over several points in time from a single group (better, but can be hard to do because people drop out of the study)
–Trend Studies–survey similar samples at different points in time (compromise between cross-sectional and longitudinal because it's not the same group, but at least they're more likely to have similar characteristics)
• I could sample the EDPS 6030 cohorts every year
–Cohort Studies--survey a different sample from the same group at different points in time
• I could sample 4 students from this group of EDPS 6030 students, with different students in each sample (hoping that the different samples are similar because at least they were from the same group, but problem is you never look at the original 4 people again)
–Panel Studies--survey the same sample at different points in time
• I could sample the current entire group of EDPS 6030 students every year


Survey Research Methodology

• What is the focus (unit of analysis) of the survey?
–Focus should be interesting and important enough for individuals to want to respond
–Objectives should be clearly defined, and all questions should relate to the objectives

• Who made up the sample?
–Target Populations vs. Actual Populations
–Demographic Information
–Sampling Response Rates - this is really going to depend upon how they survey is administered and the following items -->

• How is the survey administered?
– Direct, aka – In Person (ex. carrying clipboards around in the mall - people see you coming & avoid you)
– Mail
– Telephone - no call lists and people not answering is making this less effective
– Internet (These are becoming more and more effective. They're fast, easy, convenient.)


• What is explained to the subjects in the cover letter?
- The cover letter should answer the questions: who, what, how long, why, when
-It shouldn't bias them, but it should give them enough information t0 make them willing to spend the time to do it.

• Individual Survey Items
–Should be as unambiguous as possible
–Should be short and well-organized
–Should not ask more than one question at a time
–Should not contain biasing or overly technical terms
–Should not use leading questions
–Should avoid using double negatives
–Should not overuse contingency questions
-Reverse Scoring - when you always word your questions in the affirmative, they might start giving lazy answers, but if you reverse and write some in the negative, you can better ensure that you're getting real responses

Survey Item Types

Closed-ended
–Advantages
• Consistent and unambiguous responses
• Easier to score and interpret
• More likely to elicit responses
–Disadvantages
• Limits amount of “data”received
• More difficult to construct good questions
• More questions are usually required

Open-ended
–Advantages
• Allows more freedom in individual responses
• Easier to construct
–Disadvantages
• More difficult to score and interpret
• May elicit more ambiguous responses
• Less consistency in individuals’ response rates and response contents

Analyses of Survey Data: Closed-Ended Questions

• Descriptive Statistics
–Percentages
–Mean, Median, Mode
–Frequency Distributions

• Comparative Statistics
–Correlations (r, R)
–Chi Square (χ 2) -- this tells you if the distribution of responses different than what you would expect from chance (ex. the Pepsi challenge - if there are 3 choices of cola, you would expect Pepsi to get 33.3% of the votes simply by chance - the Chi Square tells you if the results were different than this)

• Inferential Statistics
–T-tests (t)
–ANOVAs (F)
–MANOVAs(F)
(if a survey starts using these kinds of results, you want to ask for a p value to back up their data and their claims)

Analyses of Survey Data: Open-Ended Questions

• Use of Qualitative Analysis Methods
–Content Analysis - claims are backed up by participant quotes because you can't give statistical data
–Triangulation - support is needed for this claim from at least 3 different sources of data
–Emergent Patterns
–Emergent Hypotheses

Validity and Reliability
• Pretesting - try out your survey before sending it to everyone
• Reliability Checks - someone needs to check how reliably the data was coded or entered into a program (like Xcel)
• Validity Checks - make sure your survey is valid
–Internal Validity - Make sure questions aren't biased, mode of conducting survey is effective, etc.
–External Validity - Am I going to be able to generalize my results beyond those who answer it?


QUIZ

http://www.iteaconnect.org/TAA/PDFs/Gallupsurvey.pdf
http://www.iteaconnect.org/TAA/PDFs/GallupQuestions2004.pdf
In light of what you just learned, examine the questions used by the Gallup Poll.
Are they clear? Well-written? Free from bias?
If you do think there are problems with any of the questions, indicate what the problems are and then rewrite the questions to address those problems.