Survey Designs

Surveys & Questionnaires

Functions of Quantitative Designs
• Purpose of...
–Experimental Research is to draw conclusions about cause and effect
–Comparative Research is to examine relationships between variables
–Descriptive (Survey) Research is to summarize characteristics of a particular sample or population


Types of Surveys

Cross-Sectional –data is collected once from a variety of different groups

Longitudinal –data is collected over several points in time from a single group (better, but can be hard to do because people drop out of the study)
–Trend Studies–survey similar samples at different points in time (compromise between cross-sectional and longitudinal because it's not the same group, but at least they're more likely to have similar characteristics)
• I could sample the EDPS 6030 cohorts every year
–Cohort Studies--survey a different sample from the same group at different points in time
• I could sample 4 students from this group of EDPS 6030 students, with different students in each sample (hoping that the different samples are similar because at least they were from the same group, but problem is you never look at the original 4 people again)
–Panel Studies--survey the same sample at different points in time
• I could sample the current entire group of EDPS 6030 students every year


Survey Research Methodology

• What is the focus (unit of analysis) of the survey?
–Focus should be interesting and important enough for individuals to want to respond
–Objectives should be clearly defined, and all questions should relate to the objectives

• Who made up the sample?
–Target Populations vs. Actual Populations
–Demographic Information
–Sampling Response Rates - this is really going to depend upon how they survey is administered and the following items -->

• How is the survey administered?
– Direct, aka – In Person (ex. carrying clipboards around in the mall - people see you coming & avoid you)
– Mail
– Telephone - no call lists and people not answering is making this less effective
– Internet (These are becoming more and more effective. They're fast, easy, convenient.)


• What is explained to the subjects in the cover letter?
- The cover letter should answer the questions: who, what, how long, why, when
-It shouldn't bias them, but it should give them enough information t0 make them willing to spend the time to do it.

• Individual Survey Items
–Should be as unambiguous as possible
–Should be short and well-organized
–Should not ask more than one question at a time
–Should not contain biasing or overly technical terms
–Should not use leading questions
–Should avoid using double negatives
–Should not overuse contingency questions
-Reverse Scoring - when you always word your questions in the affirmative, they might start giving lazy answers, but if you reverse and write some in the negative, you can better ensure that you're getting real responses

Survey Item Types

Closed-ended
–Advantages
• Consistent and unambiguous responses
• Easier to score and interpret
• More likely to elicit responses
–Disadvantages
• Limits amount of “data”received
• More difficult to construct good questions
• More questions are usually required

Open-ended
–Advantages
• Allows more freedom in individual responses
• Easier to construct
–Disadvantages
• More difficult to score and interpret
• May elicit more ambiguous responses
• Less consistency in individuals’ response rates and response contents

Analyses of Survey Data: Closed-Ended Questions

• Descriptive Statistics
–Percentages
–Mean, Median, Mode
–Frequency Distributions

• Comparative Statistics
–Correlations (r, R)
–Chi Square (χ 2) -- this tells you if the distribution of responses different than what you would expect from chance (ex. the Pepsi challenge - if there are 3 choices of cola, you would expect Pepsi to get 33.3% of the votes simply by chance - the Chi Square tells you if the results were different than this)

• Inferential Statistics
–T-tests (t)
–ANOVAs (F)
–MANOVAs(F)
(if a survey starts using these kinds of results, you want to ask for a p value to back up their data and their claims)

Analyses of Survey Data: Open-Ended Questions

• Use of Qualitative Analysis Methods
–Content Analysis - claims are backed up by participant quotes because you can't give statistical data
–Triangulation - support is needed for this claim from at least 3 different sources of data
–Emergent Patterns
–Emergent Hypotheses

Validity and Reliability
• Pretesting - try out your survey before sending it to everyone
• Reliability Checks - someone needs to check how reliably the data was coded or entered into a program (like Xcel)
• Validity Checks - make sure your survey is valid
–Internal Validity - Make sure questions aren't biased, mode of conducting survey is effective, etc.
–External Validity - Am I going to be able to generalize my results beyond those who answer it?


QUIZ

http://www.iteaconnect.org/TAA/PDFs/Gallupsurvey.pdf
http://www.iteaconnect.org/TAA/PDFs/GallupQuestions2004.pdf
In light of what you just learned, examine the questions used by the Gallup Poll.
Are they clear? Well-written? Free from bias?
If you do think there are problems with any of the questions, indicate what the problems are and then rewrite the questions to address those problems.

Correlational Designs

Correlational Designs

Correlational Designs
–Used to measure relationships between two or more variables (r)
• Explanation
• Prediction
–No conclusions about cause and effect may be drawn

More and more researchers are making causal conclusions inappropriately.
They are making causal conclusions from correlational data, which you can NOT do.

Analyzing Data
• Correlation Coefficients
– “r”can range from -1 to +1
– Negative correlation = as one variable decreases, other increases
The negative doesn't mean the correlation is any less strong, it simply goes in the other direction.
ex. Typing skill and typing errors
– Positive correlation = as one variable increases, other also increases
ex. Standardized test scores and GPA
– Zero correlation = no relationship between the two variables

Plotting Correlational Data --Scatterplots
The more scattered the dots are on the graph, the closer the correlation coefficient gets to zero

Interpreting Correlations
• Correlation coefficient “r”
–Ranges from -1 to 1
–Indicates whether relationship is positive or negative
–Statistical significance (p-value) depends on size of relationship and size of sample

Magnitude of r Interpretation
Look at the absolute value of r to determine the magnitude of r
So what does r mean, anyway?
.00 to .40 weak relationship
.41 to .60 moderate relationship
.61 to .80 strong relationship
.81 or above very strong relationship.

More on CorrelationalDesigns
• Predicting from Multiple Variables
–Can compute several individual correlation coefficients to produce a correlation matrix (a table showing how the different variables correlate)
• Or can conduct a Multiple Regression Analysis - puts all the variables together for one coefficient (R)
–Yields coefficient R, which can be interpreted similarly to the simple correlation, r
–Also yields R2(coefficient of determination)
Coefficient of determination = Effect size estimate for correlational studies = how much of the result we can explain by the effect of all the other variables
ex. Reading Comprehension has many variables. R2 = How much of the reading comprehension can we explain through these other factors?

Imagine trying to dissect the concept “Reading Comprehension.” It is made up of several related factors, such as:
•Fluency
•Intrinsic Motivation
•Verbal IQ
•Working Memory Capacity
•Background Knowledge
If we sum up the portion of those components that uniquely overlaps with reading comprehension, we can explain a big part of reading comprehension. That is essentially what R2 in a multiple regression does.
So R2tells you, given all of the factors we’ve entered into the equation, how much of Reading Comprehension can be explained by those factors.

Other Uses of Correlational Data
• Structural Equation Modeling (aka. SEM, Path Analysis, Hierarchical, Stepwise) –maps out relationships among several variables
–Instead of lumping everything into a multiple regression, we can put them into a structural equation model; Allows researchers to see the “big picture”as well as relationships among individual variables
• Factor Analysis –how do individual variables or items in a measure combine to create “mega-
variables”
–E.g. Several items on a questionnaire might relate to your interest in a topic. Instead of treating each item as an individual variable, we combine them as one “factor”
•Path Modeling - shows how all of the factors correlated together AND how they work together to predict supportive computer use (looks like boxes with arrows going every which way showing correlational values between each combination)