Sampling & Measurement

Sampling & Measurement
I. Sampling
A. Samples vs. Populations
B. Sampling Methods
1. Quantitative Methods
2. Qualitative Methods
C. Issues in Sampling
II. Measurement
A. Measurement, Evaluation, & Assessment
B. Types of Educational Measures
C. Interpreting Data
D. Evaluating Measures

Samples vs. Populations
• Sample= group of people participating in your study
• Population= group of people to whom you want to generalize your results
–Target Population - the population you are trying to represent with your research findings
–Accessible Population - the population you are actually able to get a sample from *may or may not be the same as your target population - the closer it matches your target population, the better

Two Types of Sampling
1. Probability Sampling (aka Simple Random Sampling, aka Straight Random Sample)= take a random selection of individuals from our population, such that each individual has an equal chance of being selected for participation in the study.
2. Non-Probability Sampling (aka Non-Random Sample)= individuals are selected from the population in such a way that not everyone has an equal chance of being selected for participation in the study. Not totally random.

Probability Sampling Methods:
1. Stratified Random Sampling= select subsets of the population to participate in the study in the same proportion as they appear in the population
e.g., 400 teachers in Salt Lake area schools, 275 are female and 125 are male
I decide to sample 40% of Salt Lake area teachers. My sample contains:
40% * 400 teachers = 160 total teachers in sample
40% * 275 female teachers = 110 females in sample
40% * 125 male teachers = 50 males in sample

2. Clustered random sample= select existing groups of participants instead of creating subgroups
e.g., Instead of randomly selecting individuals in correct proportions, I randomly select groups of individuals. So now I randomly select some schools in Salt Lake area district, and all teachers in those selected schools participate in my study. But, I must ensure that those groups selected are representative of my population as a whole.

3. Two-Stage Random Sampling= combines methods 1 and 2; in stage 1, existing groups are
randomly selected; in stage 2, individuals from those groups are randomly selected
e.g., Instead of randomly selecting individuals in correct proportions, I randomly select groups of individuals, then randomly select individuals from those groups
Stage 1: I randomly select some schools in Salt Lake area district.
Stage 2: From each selected school, I randomly select a subset of teachers to participate in the study

*If you don't have a really good reason for controlling your sample, it's probably better to just do a simple random sample. You can't control for every characteristic, so it's often best just to be random.

Non-Probability Sampling Methods:
1. Systematic Sampling= every nth individual in a population is selected for participation in the study
e.g., I take an alphabetical list of all teachers in Salt Lake area schools, and select every 3rd individual from that list for participation in my study. Here, 3 is my sampling interval
sampling interval = population size / desired sample size
e.g., sampling interval = 400 teachers / 160 teachers (or 40%) =2.5
sampling ratio = proportion of individuals in population selected for sample
e.g., sampling ratio = 160/400 = .4 or 40%

2. Convenience Sampling = select from a group of individuals who are conveniently available to be participants in your study
e.g., I go into schools at lunchtime and give surveys to those teachers who can be found in the teachers’ lounge
Potential Problem:
Sample is likely to be biased –are those teachers in the lounge at lunchtime likely to be different from those who aren’t?
This type of sampling should be avoided if possible.

3. Purposive Sampling= researchers use past knowledge or own judgment to select a sample that he/she thinks is representative of the population
e.g., I decide to just give my survey to teachers who are also currently enrolled in the EDPS 6030, because I *think* they are representative of the population of Salt Lake area teachers
Potential problem: Researchers may be biased about what they believe is representative of a population, or they may be just plain wrong.
Be very cautious of this kind of sampling!

Sampling in Qualitative Research
• Purposive Sampling
• Case Analysis (aka Case Study)
–Typical - the prototype, the typical example
–Extreme - the unusually extreme example
–Critical - highlights the characteristics you want to study
• Maximum Variation - you are representing the extremes of your examples (some less than typical, some typical, some extreme)
• Snowball Sampling - you select some people for your sample, then ask the to get some people to participate, then they get some people to participate, etc.

Sampling and Validity
1. What size sample is appropriate?
Descriptive => 100 subjects
Correlational=> 50 subjects
Experimental => 30 subjects per group* (You will often see less than that.)
Causal-Comparative => 30 subjects per group*
But if groups are tightly controlled, less (e.g., 15 per group) may be OK.

2. How generalizable is the sample?
external validity= the results should be generalizable beyond the conditions of the individual study
a. Population generalizability= extent to which the sample represents the population of interest
b. Ecological generalizability= degree to which the results can be extended to other settings or conditions

What is Measurement?
• Measurement - the collection of data, the gathering of information
• Evaluation - making a decision based on the information
• Where does assessment fit in? - both measurement and evaluation are lumped together

What kind of scale is the measurement based on?
• Nominal - categorical (qualitative variables)
• Ordinal - rank order, no other information.
eg. 1st, 2nd, and 3rd place but no details about the distance between 1st and 2nd or 2nd and 3rd
• Interval - we do know the distance between the results - there is NO absolute zero
• Ratio - we do know the distance between the results - there IS an absolute zero

Types of Educational Measures
• Cognitive vs. Non-Cognitive
cognitive - interested in the cognitive processes involved
non-cognitive - ex. opionion - not cognitively based
• Commercial vs. Non-Commercial
commercial - developed by a company - tried and tested, standardized, generalized
non-commercial - developed by the researcher - tailored for your own needs
• Direct vs. Indirect
direct - getting our information directly from the participants
indirect - getting our information from somewhere besides the participants

Sample Cognitive Measures
• Standardized Tests
–Achievement Tests - tests things already learned
–Aptitude Tests - tests potential for future learning
• Behavioral Measures
–Naming Time
–Response Time
–Reading Time
• Wpm
• Eyetracking Measures

Non-Cognitive Measures
• Surveys & Questionnaires
• Observations
• Interviews

How is an individual’s score interpreted?
1. Norm-referenced instruments= an individual’s score is based on comparison with peers (e.g.,
percentile rank, age/grade equivalents, grading on curve, etc.)
2. Criterion-referenced instruments= an individual’s score is based on some predetermined standard (e.g., raw score)

Interpreting Data
Different Ways to Present Scores:
1. Raw Score= number of items answered correctly, number of times behavior is tallied, etc.
2. Derived Scores= scores changed into a more meaningful unit
a. age/grade equivalent scores= for a given score, tell what age/grade score usually falls in
b. percentile rank= ranking of score compared to all other individuals who took the test
c. standard scores (aka Z Scores) = indicates how far scores are from a reference point; usually best to use in research - allows you to compare scores from two totally different scales

Important Characteristics of Measures
• Objectivity
• Usability - Can I use it? Am I able to interpret the data I get from it?
• Validity - Does the measure actually measure what it's supposed to measure?
• Reliability - Do I get consistent measurements over time?