Review of Terms:
Random Selection/Sampling vs. Random Assignment
Random Selection = How do I get my sample? All people in your population have an equal chance of being selected
Random Assignment = Once you have your sample, how do you assign them to a group?
Internal Validity vs. External Validity
Internal Validity = control of the experiment
External Validity = generalizability of your experiment
You want to have the correct balance between the two
Independent, Dependent, & Extraneous Variables
Independent Variable = What you are manipulating
Dependent Variable = depends on the independent
Extraneous Variables = the things that mess everything up
Between vs. Within Subjects Variables
Between subjects variable = looking for a difference between subjects - they don't get the same experience in both groups - but you need to make sure that both groups are as similar as possible to confirm that the only differences between groups is your independent variable
Within subjects variable = Every individual gets both experiences in the experiment - ex. Pre vs. Post - Within is more ideal because you know your groups will be consistent for both independent variables
Factorial Design = measures the impact of more than one independent variable at once
Benefit: you can see if there is an interaction between the different independent variables
(No more than three independent variables, otherwise it gets too complicated and you can't tell what variables are interacting with which variables very clearly)
Experimental vs. Quasi-Experimental Designs
"True" Experimental Designs = involve random assignment to conditions manipulated by experimenter
Quasi-Experimental Designs = involve comparisions of groups in pre-selected conditions or groups - I design the study before I collect the data
Causal Comparative Designs = are ex post facto quasi-experimental designs; They involve comparison of pre-selected conditions/groups after the fact
Time-Series Design
Measure dependent variable a lot over time
Experimental Designs, cont.
Single Subject Designs
-Like mini longitudinal experimental designs, on individuals or small groups of individuals
-Similar to pretest/posttest designs; examines DV before and after IV
-Used when it doesn’t make sense to pool effects across individuals - ex. When working with children with special needs, the specific behaviors and needs of one child are not the same as others' - But tracking that one child over time may help develop strategies to help that specific child - you're not trying to generalize your findings, you're just trying to help that one individual
--Special populations
--Focus on success of specific interventions with specific individuals
Possible Single-Subject Designs
A-B Design = baseline, followed by intervention
A = Baseline
B = Intervention
But what happens after the intervention is removed? Does behavior go back to baseline?
A-B-A Withdrawal Design = baseline, followed by intervention, concluded with baseline
When the intervention is removed, does behavior go back to baseline?
Ethical issue: is it OK to intervene but then leave subjects back at baseline behavior, especially if we know that the intervention is needed?
A-B-A-B Design = one step further; instead of leaving subjects back at baseline, present intervention again (more ethical)
Multiple Baselines Designs = alternative to ABAB design; used when it’s not ethical to leave the subject at the baseline condition and when measures on multiple Dependent Variables (DVs) are taken.
-Taking baselines for multiple behaviors at same time – whether it’s one behavior in multiple individuals, multiple behaviors in one individual, one type of behavior in one individual in multiple settings, etc.
-Difficult to use, because must ensure that multiple DVs aren’t related to one another
Issues to Consider: Internal Validity
Is Treatment Standardized?
-# Variables Changing Across Conditions ? - Is the behavior changing because of my intervention, or is there another explanation?
Condition Length?
-Degree and Speed of Change? - You want to show a stable trend. Do you have enough data points to see a stable trend in the experiment results? - The "magic" number is 7 measurements to see a stable trend
Experimenter Effects?
-Because this is a one-on-one experiment, the experimenter is likely to have an impact on the individual
Practical Significance of Results?
-Practical significance is looked at more than statistical significance because of the single-subject design. - ie. Did it help the student improve?
Single Subject Designs and External Validity
It is very difficult to generalize the results of these designs beyond the sample studied – WHY?
-Because it was only designed for one person for a specific purpose. It is not necessarily meant to be generalized.
Thus, it is VERY IMPORTANT for this type of research to be replicated before strong conclusions are drawn.
*We now have all the information we need for quiz 3 if we want to take it now.
Wednesday, February 27, 2008
Wednesday, February 20, 2008
Experimental Design
Experimental Designs
What makes experimental designs so great? CONTROL!!!
• In experimental designs, researchers have control over:
–Independent Variables (what is manipulated; whatever you are comparing, such as traiditonal vs. online technology)
–Dependent Variables (what is being measured, how are they operational? How is it being measured?)
–Extraneous Variables (all of the things I can't control; things that have impact on my dependent variable, such as dog color blindness)
Internal Validity and Control
• History (some uncontrolled event, such as a fire alarm)
• Selection (how do you select your subject)
• Maturation (individuals change over time)
• Testing (does just giving a test somehow influence what the subjects think about the Independent Variable)
• Instrumentation (something could be wrong with the test)
• Treatment Replications
• Attrition (mortality, or losing subjects during the experiment)
• Regression to the Mean
• Diffusion of Treatment - (does the Independent Variable from one group share info to another group) Does your independent variable in one group bleed into another group? One group tells the other group what happens in the experiment, or they talk about it and share opinions and thoughts that skew the results
• Experimenter Effects - (is the experimenter overly nice or overly mean to subjects; or if your tester is a cute girl and your subjects are 13 yr old boys, then they subjects do whatever the tester wants you to do)
How the behavior, personality, looks of the experimenter affect how the subjects react to and participate in the experiment
• Subject Effects - in self-report data, the attitude of the subject or the reason they are there can affect how they participate in the experiment - why did they participate? Does it take a certain kind of person to actually participate? Does this affect your results?
•Social Desirability Bias - the participant alters answers so they pain the picture of themselves that they want
External Validity and Control
• So more control ≈ higher levels of internal validity...
–Note, this is what allows us to make cause/effect conclusions about experimental data
• But what happens to external validity? - External validity lessens the more you control because more control means you are getting farther and farther away from the real world
*It's like a see-saw. You have to find the right balance between external validity (less control) and internal validity (more control)
Important Terms
• Randomization
–Random Selection - How do you get the participants for your study in the first place?
–vs. Random Assignment - What do you do with the participants once you've already gotten them? How do you assign them within the experiment?
• Within Subjects - Every participant gets every level of the independent variable - ex. in a pretest/post test design, every participant takes both the pre and post tests - this is always going to be better if you can make it work because everyone participates in everything and there is no chance that different groups are going to have different results because of extraneous variables
vs. Between Subjects Variables and Designs - One group gets level A of the independent variable, but a different group gets level B of the independent variable - ex. experimental vs. control group designs OR men vs. women design - You do run the risk that there is some extraneous variable that exists in one group, but not the other; there might be something fundamentally different between groups that affects your results
• Controlling for Confounds
–Holding variables constant
–Building variables into design
–Matching
Pretest/Posttest Designs
• Single Group Posttest-Only Design
– Expose 1 group to the IV and then measure the DV (posttest) once
– Example: I decide to test whether using puppets to “read” books with young children to help them learn different sound combinations. – What’s the problem? - You don't have anything to compare to, no baseline to start at. There is no way you can say the puppets affected anything if you don't know what the results of the test were before using the puppets. To make it even better, you should have a control group that takes a pre and post test without the puppets to prove it was the puppets that made the difference in test scores and not just the fact that they took the test twice.
Single Group Pretest Posttest
• For a single group, give them a pretest (DV), then the IV, then a posttest (DV again)
• Example, in the “puppet experiment,”I give students a pre-reading PA quiz, then read to them with the puppet, then give them a post-reading PA quiz
Nonequivalent Groups Pretest Posttest: Quasi-Experimental
• Involves 2 groups (experimental and control); both get pretest (DV), then only the exper. Gp. is exposed to the IV, and both gps. get the posttest (DV)
*Pre-existing groups
• Between Subjects Designs= each group of subjects receives a different level of the IV
–Advantage: often more practical that within subjects designs
–Disadvantage: are differences due to groups? Or to the IV?
–Use of Matching
And what you’ve all been waiting for...“TRUE”Experimental Designs
• Randomized Groups PrettestPosttest Design
– Individuals are randomly assigned to either exper. Or control grp.; both groups receive the pretest (DV), then only the exper.
Gp. Is exposed to the IV, and then both gps. Receive the posttest (DV)
– Random assignment gives control over group differences, and pretest allows for a baseline measure in both groups
Experimental Designs with more than 1 Independent Variable (IV)
• Factorial Designs= measure the impact of 2 or more IVs on the Dependent Variable (DV)
(ex. test whether puppets help develop reading skills, AND whether it helps English Language Learners more)
–Advantages
• Allow researchers to determine whether effects are consistent across subject characteristics
• Allow researchers to assess interactions
–Disadvantages
• Can get messy if there are too many IVs because every IV you add changes the "personality" of the experiment - NEVER HAVE MORE THAN 3 IVs, otherwise it gets too messy
Sample Between Subjects Factorial Design
Analyses of Exp. Designs (t and F)
• T-test (t) --used to assess the impact of two levels of one IV on a single DV
• ANOVA (F) –used to assess impact of one or more IVs on a single DV
• MANOVA –used to assess impact of one or more IVs on multiple DVs
• ANCOVA –used to assess impact of one or more IVs on a single DV after removing effects of a variable that might be correlated to DV (e.g., age, gender, aptitude, achievement, etc.)
• MANCOVA –used to assess impact of one or more IVs on multiple DVs after removing effects of a variable that might be correlated to DV (e.g., age, gender, aptitude, achievement, etc.)
What makes experimental designs so great? CONTROL!!!
• In experimental designs, researchers have control over:
–Independent Variables (what is manipulated; whatever you are comparing, such as traiditonal vs. online technology)
–Dependent Variables (what is being measured, how are they operational? How is it being measured?)
–Extraneous Variables (all of the things I can't control; things that have impact on my dependent variable, such as dog color blindness)
Internal Validity and Control
• History (some uncontrolled event, such as a fire alarm)
• Selection (how do you select your subject)
• Maturation (individuals change over time)
• Testing (does just giving a test somehow influence what the subjects think about the Independent Variable)
• Instrumentation (something could be wrong with the test)
• Treatment Replications
• Attrition (mortality, or losing subjects during the experiment)
• Regression to the Mean
• Diffusion of Treatment - (does the Independent Variable from one group share info to another group) Does your independent variable in one group bleed into another group? One group tells the other group what happens in the experiment, or they talk about it and share opinions and thoughts that skew the results
• Experimenter Effects - (is the experimenter overly nice or overly mean to subjects; or if your tester is a cute girl and your subjects are 13 yr old boys, then they subjects do whatever the tester wants you to do)
How the behavior, personality, looks of the experimenter affect how the subjects react to and participate in the experiment
• Subject Effects - in self-report data, the attitude of the subject or the reason they are there can affect how they participate in the experiment - why did they participate? Does it take a certain kind of person to actually participate? Does this affect your results?
•Social Desirability Bias - the participant alters answers so they pain the picture of themselves that they want
External Validity and Control
• So more control ≈ higher levels of internal validity...
–Note, this is what allows us to make cause/effect conclusions about experimental data
• But what happens to external validity? - External validity lessens the more you control because more control means you are getting farther and farther away from the real world
*It's like a see-saw. You have to find the right balance between external validity (less control) and internal validity (more control)
Important Terms
• Randomization
–Random Selection - How do you get the participants for your study in the first place?
–vs. Random Assignment - What do you do with the participants once you've already gotten them? How do you assign them within the experiment?
• Within Subjects - Every participant gets every level of the independent variable - ex. in a pretest/post test design, every participant takes both the pre and post tests - this is always going to be better if you can make it work because everyone participates in everything and there is no chance that different groups are going to have different results because of extraneous variables
vs. Between Subjects Variables and Designs - One group gets level A of the independent variable, but a different group gets level B of the independent variable - ex. experimental vs. control group designs OR men vs. women design - You do run the risk that there is some extraneous variable that exists in one group, but not the other; there might be something fundamentally different between groups that affects your results
• Controlling for Confounds
–Holding variables constant
–Building variables into design
–Matching
Pretest/Posttest Designs
• Single Group Posttest-Only Design
– Expose 1 group to the IV and then measure the DV (posttest) once
– Example: I decide to test whether using puppets to “read” books with young children to help them learn different sound combinations. – What’s the problem? - You don't have anything to compare to, no baseline to start at. There is no way you can say the puppets affected anything if you don't know what the results of the test were before using the puppets. To make it even better, you should have a control group that takes a pre and post test without the puppets to prove it was the puppets that made the difference in test scores and not just the fact that they took the test twice.
Single Group Pretest Posttest
• For a single group, give them a pretest (DV), then the IV, then a posttest (DV again)
• Example, in the “puppet experiment,”I give students a pre-reading PA quiz, then read to them with the puppet, then give them a post-reading PA quiz
Nonequivalent Groups Pretest Posttest: Quasi-Experimental
• Involves 2 groups (experimental and control); both get pretest (DV), then only the exper. Gp. is exposed to the IV, and both gps. get the posttest (DV)
*Pre-existing groups
• Between Subjects Designs= each group of subjects receives a different level of the IV
–Advantage: often more practical that within subjects designs
–Disadvantage: are differences due to groups? Or to the IV?
–Use of Matching
And what you’ve all been waiting for...“TRUE”Experimental Designs
• Randomized Groups PrettestPosttest Design
– Individuals are randomly assigned to either exper. Or control grp.; both groups receive the pretest (DV), then only the exper.
Gp. Is exposed to the IV, and then both gps. Receive the posttest (DV)
– Random assignment gives control over group differences, and pretest allows for a baseline measure in both groups
Experimental Designs with more than 1 Independent Variable (IV)
• Factorial Designs= measure the impact of 2 or more IVs on the Dependent Variable (DV)
(ex. test whether puppets help develop reading skills, AND whether it helps English Language Learners more)
–Advantages
• Allow researchers to determine whether effects are consistent across subject characteristics
• Allow researchers to assess interactions
–Disadvantages
• Can get messy if there are too many IVs because every IV you add changes the "personality" of the experiment - NEVER HAVE MORE THAN 3 IVs, otherwise it gets too messy
Sample Between Subjects Factorial Design
Analyses of Exp. Designs (t and F)
• T-test (t) --used to assess the impact of two levels of one IV on a single DV
• ANOVA (F) –used to assess impact of one or more IVs on a single DV
• MANOVA –used to assess impact of one or more IVs on multiple DVs
• ANCOVA –used to assess impact of one or more IVs on a single DV after removing effects of a variable that might be correlated to DV (e.g., age, gender, aptitude, achievement, etc.)
• MANCOVA –used to assess impact of one or more IVs on multiple DVs after removing effects of a variable that might be correlated to DV (e.g., age, gender, aptitude, achievement, etc.)
Wednesday, February 6, 2008
Meta-Analysis
Quiz 1 Review
Alpha = .05 = probability that you're wrong. (Probability that there is no effect when you say there is one.)
Constitutive and Operational Definitions are essentially the same thing, but Constitutive is elaborating on terms, and Operational is clarifying your measuring.
Meta-Analysis
And Other Methods of Research Synthesis...
Levels of Research Synthesis
• Literature Reviews
--allows for in depth discussion of individual studies’findings and their theoretical implications
--no weight given to statistics in the discussion
• Numerical Reviews (“Vote Counting”) - Every significant effect casts a vote one way or the other - yes the data supports the hypothesis or no it doesn't - Doesn't look at any other factors, weight effect size, sample size, etc.
–Studies that are statistically significant “cast votes”for effects
–Statistical significance depends on many factors
• Meta-Synthesis
–Used in qualitative research
–It is NOT collection and aggregation of research results
–It is “the bringing together and breaking down of findings, examining them, discovering the
essential features, and, in some way, combining them into a transformed whole”
(Schreiber et al., 1997, p. 314)
- It tries to decontextualize the findings
-One important issue in this is selection criteria: must be published in a peer reviewed journal b/c at least they've been criticized and reviewed; date; participants; methods used; are researchers working in teams or individually (typically teams are better b/c there are people to bounce ideas off of throughout this long, tedious process - a system of checks and ballances)
-Then you must code and categorize all the contextual details of each study to make it simplified
-Audit Trail is necessary - listing the order and description of the coding and categorizing you did so others can follow it and replicate it
-Triangulation - come at your conclusions from at least 3 different perspectives
Meta-Analysis
• Origins of Meta-Analysis - farming!
• Definition – Statistical technique that enables the results from a number of studies to be combined to determine the average effect size of a given IV
–Supposedly more objective than narrative literature review b/c they care about the findings and the effect size (Cohen's d) as will as significant effect (p-value)
N= Sample Size
• Advantages
–Use stats to organize and extract info
–Eliminate selection bias (???)
–Makes use of all info in a study (???)
–Detects mediating characteristics
• Limitations
–No guide for implementation
–Time-limited - the analysis is already dated by the time it is even started
–No real rigid methodological rules
–Only as good as the studies it is based on
Meta-Analysis Methodology
• Research Focus - What is the question to be answered?
• Sampling - which studies will be used
–Inclusion Criteria? - must be peer-reviewed; dates; methodology; sample characteristics within those studies
• Classifying and Coding - easier b/c Meta-Analysis is quantitative rather than qualitative
• Role of the “Audit Trail” - this is usually only mentioned in qualitative research, but it is just as applicable and important for quantitative studies
• Data Analysis
–Significance vs. Effect size (Cohen's d - the number) vs. Effect Magnitude (small, medium, or large?)
–Comparison to Zero Effects
–Positive vs. Negative Effect Sizes
• Interpretation and Beyond
–Raises Researchers Consciousness?
–Highlights Gaps in Field Knowledge Base - Can identify what is really missing in this area of research
–Motivates Future Research
–Implications for Practice?
Alpha = .05 = probability that you're wrong. (Probability that there is no effect when you say there is one.)
Constitutive and Operational Definitions are essentially the same thing, but Constitutive is elaborating on terms, and Operational is clarifying your measuring.
Meta-Analysis
And Other Methods of Research Synthesis...
Levels of Research Synthesis
• Literature Reviews
--allows for in depth discussion of individual studies’findings and their theoretical implications
--no weight given to statistics in the discussion
• Numerical Reviews (“Vote Counting”) - Every significant effect casts a vote one way or the other - yes the data supports the hypothesis or no it doesn't - Doesn't look at any other factors, weight effect size, sample size, etc.
–Studies that are statistically significant “cast votes”for effects
–Statistical significance depends on many factors
• Meta-Synthesis
–Used in qualitative research
–It is NOT collection and aggregation of research results
–It is “the bringing together and breaking down of findings, examining them, discovering the
essential features, and, in some way, combining them into a transformed whole”
(Schreiber et al., 1997, p. 314)
- It tries to decontextualize the findings
-One important issue in this is selection criteria: must be published in a peer reviewed journal b/c at least they've been criticized and reviewed; date; participants; methods used; are researchers working in teams or individually (typically teams are better b/c there are people to bounce ideas off of throughout this long, tedious process - a system of checks and ballances)
-Then you must code and categorize all the contextual details of each study to make it simplified
-Audit Trail is necessary - listing the order and description of the coding and categorizing you did so others can follow it and replicate it
-Triangulation - come at your conclusions from at least 3 different perspectives
Meta-Analysis
• Origins of Meta-Analysis - farming!
• Definition – Statistical technique that enables the results from a number of studies to be combined to determine the average effect size of a given IV
–Supposedly more objective than narrative literature review b/c they care about the findings and the effect size (Cohen's d) as will as significant effect (p-value)
N= Sample Size
• Advantages
–Use stats to organize and extract info
–Eliminate selection bias (???)
–Makes use of all info in a study (???)
–Detects mediating characteristics
• Limitations
–No guide for implementation
–Time-limited - the analysis is already dated by the time it is even started
–No real rigid methodological rules
–Only as good as the studies it is based on
Meta-Analysis Methodology
• Research Focus - What is the question to be answered?
• Sampling - which studies will be used
–Inclusion Criteria? - must be peer-reviewed; dates; methodology; sample characteristics within those studies
• Classifying and Coding - easier b/c Meta-Analysis is quantitative rather than qualitative
• Role of the “Audit Trail” - this is usually only mentioned in qualitative research, but it is just as applicable and important for quantitative studies
• Data Analysis
–Significance vs. Effect size (Cohen's d - the number) vs. Effect Magnitude (small, medium, or large?)
–Comparison to Zero Effects
–Positive vs. Negative Effect Sizes
• Interpretation and Beyond
–Raises Researchers Consciousness?
–Highlights Gaps in Field Knowledge Base - Can identify what is really missing in this area of research
–Motivates Future Research
–Implications for Practice?
Wednesday, January 30, 2008
Validity & Statistics
Important Characteristics of Measures
• Validity
• Reliability
• Objectivity
• Usability
Validity vs. Reliability
Validity= appropriateness, correctness, meaningfulness, and usefulness of inferences made about the instruments used in a study
Reliability= consistency of scores obtained, across individuals, administrators, and sets of items
Relationship Between Reliability and Validity
Suppose I have a faulty measuring tape and I use it to measure each student’s height.
On the other hand, if I have a correctly printed measuring tape...
My tool is invalid, but it’s still reliable.
My tool is both valid and reliable.
Something can be valid & reliable.
Something can be invalid but reliable.
But if something is unreliable, it is always invalid.
Types of Validity
• Content Validity -
• Criterion Validity -
–Predictive Validity - Ability of the measure to predict future performance
–Concurrent Validity -
• Convergent vs. DiscriminantValidity
Convergent Validity - they are trying to show that one measure is showing the same thing as another measure
Discriminant Validity - showing one measure is actually showing something quite different than another measure
• Construct Validity -
• Internal Validity - How well is your study designed?
Threats to Internal Validity:
Subject characteristics
Mortality threat (attrition)
Location
Instrumentation
Data Collectors
Testing
History
Maturation
Attitude of subjects
Regression threat
Implementation
Ways That Threats to Internal Validity Can be Minimized:
a. Standardized study conditions - The "Bus Test" - If you walked out the door and got hit by a bus, someone else could pick up right where you left off with your research.
b. Obtain more information on individuals in the sample
c. Obtain more information about details of study
d. Choice of appropriate design
Reliability Checks
• Test-Retest (aka Stability) - Tests have consistent results
• Equivalent Forms - Multiple forms of the same test - If one individual takes both forms of the test, the scores should be highly correlated
• Internal Consistency
–Split-half - compare 1/2 of the items on the test to the other 1/2 to ensure that all items on the test are reliable - NEVER compare the 1st half to the last half of the items because fatigue or not completing the test can greatly affect the answers on the 2nd half; instead you could compare odd #s to even #s
–KuderRichardson
–ChronbachAlpha
• Inter-Rater (Agreement)
Analyzing Data
Frequency Polygon aka Frequency Distribution
Normal Distribution
Descriptive Statistics = describe a sample
Inferential Statistics = describe a sample, and are inferred to a larger (target) population
•Measures of Central Tendency:
–Mean = statistical average - the best, most stable measure of central tendency
–Median = middle score
–Mode = most frequent score
• Measures of Variability
–Range = highest score minus the lowest score
–Standard deviation = average deviation from the mean
–Standard error of measurement = range in which “true score”is likely to fall
Standardized scores (or z-scores) = transform raw scores into standard deviation units on the normal distribution; z = (raw score –mean) / stand. dev.
Correlational Data -- plotted on scatterplots
• Correlation Coefficients
–“r”can range from -1 to +1
–Negative correlation = as one variable decreases, other increases (r is close to -1)
–Positive correlation = as one variable increases, other also increases (r is close to +1)
–Zero correlation = no relationship between the two variables (the closer r is to 0, the less correlation there is)
*You cannot imply causation from correlation.*
Hypothesis Testing
Null Hypothesis (H0) = set up to state that there is no effect
Alternative Hypothesis (H1) = set up to state that there is an effect
These two hypotheses must be:
• Mutually Exclusive - they can't overlap - either there is no effect or there is an effect
• Exhaustive
Test by determining by doing statistics to determine probability that the result was due to
chance:
• If probability that the result was due to chance <> 5%, the null hypothesis cannot be rejected
• 5% level => alpha level => .05
So, a researcher wants the probability (p) that their results were due to chance to be less than 5% (0.05).
If p is <>
If p is > 0.05, there is a non-significant effect.
If my null hypothesis is true, but I reject the null, that is a Type I Error.
If my null hypothesis is true, and I fail to reject the null, that is a correct decision.
If my null hypothesis is false, and I reject the null, that is a correct decision.
If my null hypothesis is false, and I fail to reject the null, that is a Type II Error.
This is the one I want! I will do anything I can to increase my POWER to get this result.
Ways Researchers May try to Increase Likelihood of Rejecting Null Hypothesis:
• Increase sample size.
• Control for extraneous variables (confounds).
• Increase the strength of the treatment.
• Use a one-tailed test when justifiable.
How do you know how “big”an effect really is?
• Effect Sizes = an estimate of the magnitude of an effect between two groups or variables
–Cohen’s d - an estimate of effect size
–η2(eta-squared) or partial η2
–Coefficient of determination (R2)
Interpreting Cohen’s d:
Small d <.2 (statistically significant, but not practically significant)
Medium .3 < d < .5
Large d > .5
NEXT WEEK
• Moving into different Research Designs
–Everybody read:
• Kavalearticle• Rosen & Solomon article
–Starting with Meta-Analyses
• I’ll discuss Kavalearticle
• Staci, Randy, and Katie will lead discussion on Rosen & Solomon article
• Initial Article Analyses are Due
–Use guidelines on Initial Analysis handout–Consider “What you Know to Ask So Far”
–Turn in your review and a complete copy of the article you reviewed
1st Quiz is due Tuesday at midnight.
• Validity
• Reliability
• Objectivity
• Usability
Validity vs. Reliability
Validity= appropriateness, correctness, meaningfulness, and usefulness of inferences made about the instruments used in a study
Reliability= consistency of scores obtained, across individuals, administrators, and sets of items
Relationship Between Reliability and Validity
Suppose I have a faulty measuring tape and I use it to measure each student’s height.
On the other hand, if I have a correctly printed measuring tape...
My tool is invalid, but it’s still reliable.
My tool is both valid and reliable.
Something can be valid & reliable.
Something can be invalid but reliable.
But if something is unreliable, it is always invalid.
Types of Validity
• Content Validity -
• Criterion Validity -
–Predictive Validity - Ability of the measure to predict future performance
–Concurrent Validity -
• Convergent vs. DiscriminantValidity
Convergent Validity - they are trying to show that one measure is showing the same thing as another measure
Discriminant Validity - showing one measure is actually showing something quite different than another measure
• Construct Validity -
• Internal Validity - How well is your study designed?
Threats to Internal Validity:
Subject characteristics
Mortality threat (attrition)
Location
Instrumentation
Data Collectors
Testing
History
Maturation
Attitude of subjects
Regression threat
Implementation
Ways That Threats to Internal Validity Can be Minimized:
a. Standardized study conditions - The "Bus Test" - If you walked out the door and got hit by a bus, someone else could pick up right where you left off with your research.
b. Obtain more information on individuals in the sample
c. Obtain more information about details of study
d. Choice of appropriate design
Reliability Checks
• Test-Retest (aka Stability) - Tests have consistent results
• Equivalent Forms - Multiple forms of the same test - If one individual takes both forms of the test, the scores should be highly correlated
• Internal Consistency
–Split-half - compare 1/2 of the items on the test to the other 1/2 to ensure that all items on the test are reliable - NEVER compare the 1st half to the last half of the items because fatigue or not completing the test can greatly affect the answers on the 2nd half; instead you could compare odd #s to even #s
–KuderRichardson
–ChronbachAlpha
• Inter-Rater (Agreement)
Analyzing Data
Frequency Polygon aka Frequency Distribution
Normal Distribution
Descriptive Statistics = describe a sample
Inferential Statistics = describe a sample, and are inferred to a larger (target) population
•Measures of Central Tendency:
–Mean = statistical average - the best, most stable measure of central tendency
–Median = middle score
–Mode = most frequent score
• Measures of Variability
–Range = highest score minus the lowest score
–Standard deviation = average deviation from the mean
–Standard error of measurement = range in which “true score”is likely to fall
Standardized scores (or z-scores) = transform raw scores into standard deviation units on the normal distribution; z = (raw score –mean) / stand. dev.
Correlational Data -- plotted on scatterplots
• Correlation Coefficients
–“r”can range from -1 to +1
–Negative correlation = as one variable decreases, other increases (r is close to -1)
–Positive correlation = as one variable increases, other also increases (r is close to +1)
–Zero correlation = no relationship between the two variables (the closer r is to 0, the less correlation there is)
*You cannot imply causation from correlation.*
Hypothesis Testing
Null Hypothesis (H0) = set up to state that there is no effect
Alternative Hypothesis (H1) = set up to state that there is an effect
These two hypotheses must be:
• Mutually Exclusive - they can't overlap - either there is no effect or there is an effect
• Exhaustive
Test by determining by doing statistics to determine probability that the result was due to
chance:
• If probability that the result was due to chance <> 5%, the null hypothesis cannot be rejected
• 5% level => alpha level => .05
So, a researcher wants the probability (p) that their results were due to chance to be less than 5% (0.05).
If p is <>
If p is > 0.05, there is a non-significant effect.
If my null hypothesis is true, but I reject the null, that is a Type I Error.
If my null hypothesis is true, and I fail to reject the null, that is a correct decision.
If my null hypothesis is false, and I reject the null, that is a correct decision.
If my null hypothesis is false, and I fail to reject the null, that is a Type II Error.
This is the one I want! I will do anything I can to increase my POWER to get this result.
Ways Researchers May try to Increase Likelihood of Rejecting Null Hypothesis:
• Increase sample size.
• Control for extraneous variables (confounds).
• Increase the strength of the treatment.
• Use a one-tailed test when justifiable.
How do you know how “big”an effect really is?
• Effect Sizes = an estimate of the magnitude of an effect between two groups or variables
–Cohen’s d - an estimate of effect size
–η2(eta-squared) or partial η2
–Coefficient of determination (R2)
Interpreting Cohen’s d:
Small d <.2 (statistically significant, but not practically significant)
Medium .3 < d < .5
Large d > .5
NEXT WEEK
• Moving into different Research Designs
–Everybody read:
• Kavalearticle• Rosen & Solomon article
–Starting with Meta-Analyses
• I’ll discuss Kavalearticle
• Staci, Randy, and Katie will lead discussion on Rosen & Solomon article
• Initial Article Analyses are Due
–Use guidelines on Initial Analysis handout–Consider “What you Know to Ask So Far”
–Turn in your review and a complete copy of the article you reviewed
1st Quiz is due Tuesday at midnight.
Wednesday, January 23, 2008
Sampling & Measurement
Sampling & Measurement
I. Sampling
A. Samples vs. Populations
B. Sampling Methods
1. Quantitative Methods
2. Qualitative Methods
C. Issues in Sampling
II. Measurement
A. Measurement, Evaluation, & Assessment
B. Types of Educational Measures
C. Interpreting Data
D. Evaluating Measures
Samples vs. Populations
• Sample= group of people participating in your study
• Population= group of people to whom you want to generalize your results
–Target Population - the population you are trying to represent with your research findings
–Accessible Population - the population you are actually able to get a sample from *may or may not be the same as your target population - the closer it matches your target population, the better
Two Types of Sampling
1. Probability Sampling (aka Simple Random Sampling, aka Straight Random Sample)= take a random selection of individuals from our population, such that each individual has an equal chance of being selected for participation in the study.
2. Non-Probability Sampling (aka Non-Random Sample)= individuals are selected from the population in such a way that not everyone has an equal chance of being selected for participation in the study. Not totally random.
Probability Sampling Methods:
1. Stratified Random Sampling= select subsets of the population to participate in the study in the same proportion as they appear in the population
e.g., 400 teachers in Salt Lake area schools, 275 are female and 125 are male
I decide to sample 40% of Salt Lake area teachers. My sample contains:
40% * 400 teachers = 160 total teachers in sample
40% * 275 female teachers = 110 females in sample
40% * 125 male teachers = 50 males in sample
2. Clustered random sample= select existing groups of participants instead of creating subgroups
e.g., Instead of randomly selecting individuals in correct proportions, I randomly select groups of individuals. So now I randomly select some schools in Salt Lake area district, and all teachers in those selected schools participate in my study. But, I must ensure that those groups selected are representative of my population as a whole.
3. Two-Stage Random Sampling= combines methods 1 and 2; in stage 1, existing groups are
randomly selected; in stage 2, individuals from those groups are randomly selected
e.g., Instead of randomly selecting individuals in correct proportions, I randomly select groups of individuals, then randomly select individuals from those groups
Stage 1: I randomly select some schools in Salt Lake area district.
Stage 2: From each selected school, I randomly select a subset of teachers to participate in the study
*If you don't have a really good reason for controlling your sample, it's probably better to just do a simple random sample. You can't control for every characteristic, so it's often best just to be random.
Non-Probability Sampling Methods:
1. Systematic Sampling= every nth individual in a population is selected for participation in the study
e.g., I take an alphabetical list of all teachers in Salt Lake area schools, and select every 3rd individual from that list for participation in my study. Here, 3 is my sampling interval
sampling interval = population size / desired sample size
e.g., sampling interval = 400 teachers / 160 teachers (or 40%) =2.5
sampling ratio = proportion of individuals in population selected for sample
e.g., sampling ratio = 160/400 = .4 or 40%
2. Convenience Sampling = select from a group of individuals who are conveniently available to be participants in your study
e.g., I go into schools at lunchtime and give surveys to those teachers who can be found in the teachers’ lounge
Potential Problem:
Sample is likely to be biased –are those teachers in the lounge at lunchtime likely to be different from those who aren’t?
This type of sampling should be avoided if possible.
3. Purposive Sampling= researchers use past knowledge or own judgment to select a sample that he/she thinks is representative of the population
e.g., I decide to just give my survey to teachers who are also currently enrolled in the EDPS 6030, because I *think* they are representative of the population of Salt Lake area teachers
Potential problem: Researchers may be biased about what they believe is representative of a population, or they may be just plain wrong.
Be very cautious of this kind of sampling!
Sampling in Qualitative Research
• Purposive Sampling
• Case Analysis (aka Case Study)
–Typical - the prototype, the typical example
–Extreme - the unusually extreme example
–Critical - highlights the characteristics you want to study
• Maximum Variation - you are representing the extremes of your examples (some less than typical, some typical, some extreme)
• Snowball Sampling - you select some people for your sample, then ask the to get some people to participate, then they get some people to participate, etc.
Sampling and Validity
1. What size sample is appropriate?
Descriptive => 100 subjects
Correlational=> 50 subjects
Experimental => 30 subjects per group* (You will often see less than that.)
Causal-Comparative => 30 subjects per group*
But if groups are tightly controlled, less (e.g., 15 per group) may be OK.
2. How generalizable is the sample?
external validity= the results should be generalizable beyond the conditions of the individual study
a. Population generalizability= extent to which the sample represents the population of interest
b. Ecological generalizability= degree to which the results can be extended to other settings or conditions
What is Measurement?
• Measurement - the collection of data, the gathering of information
• Evaluation - making a decision based on the information
• Where does assessment fit in? - both measurement and evaluation are lumped together
What kind of scale is the measurement based on?
• Nominal - categorical (qualitative variables)
• Ordinal - rank order, no other information.
eg. 1st, 2nd, and 3rd place but no details about the distance between 1st and 2nd or 2nd and 3rd
• Interval - we do know the distance between the results - there is NO absolute zero
• Ratio - we do know the distance between the results - there IS an absolute zero
Types of Educational Measures
• Cognitive vs. Non-Cognitive
cognitive - interested in the cognitive processes involved
non-cognitive - ex. opionion - not cognitively based
• Commercial vs. Non-Commercial
commercial - developed by a company - tried and tested, standardized, generalized
non-commercial - developed by the researcher - tailored for your own needs
• Direct vs. Indirect
direct - getting our information directly from the participants
indirect - getting our information from somewhere besides the participants
Sample Cognitive Measures
• Standardized Tests
–Achievement Tests - tests things already learned
–Aptitude Tests - tests potential for future learning
• Behavioral Measures
–Naming Time
–Response Time
–Reading Time
• Wpm
• Eyetracking Measures
Non-Cognitive Measures
• Surveys & Questionnaires
• Observations
• Interviews
How is an individual’s score interpreted?
1. Norm-referenced instruments= an individual’s score is based on comparison with peers (e.g.,
percentile rank, age/grade equivalents, grading on curve, etc.)
2. Criterion-referenced instruments= an individual’s score is based on some predetermined standard (e.g., raw score)
Interpreting Data
Different Ways to Present Scores:
1. Raw Score= number of items answered correctly, number of times behavior is tallied, etc.
2. Derived Scores= scores changed into a more meaningful unit
a. age/grade equivalent scores= for a given score, tell what age/grade score usually falls in
b. percentile rank= ranking of score compared to all other individuals who took the test
c. standard scores (aka Z Scores) = indicates how far scores are from a reference point; usually best to use in research - allows you to compare scores from two totally different scales
Important Characteristics of Measures
• Objectivity
• Usability - Can I use it? Am I able to interpret the data I get from it?
• Validity - Does the measure actually measure what it's supposed to measure?
• Reliability - Do I get consistent measurements over time?
I. Sampling
A. Samples vs. Populations
B. Sampling Methods
1. Quantitative Methods
2. Qualitative Methods
C. Issues in Sampling
II. Measurement
A. Measurement, Evaluation, & Assessment
B. Types of Educational Measures
C. Interpreting Data
D. Evaluating Measures
Samples vs. Populations
• Sample= group of people participating in your study
• Population= group of people to whom you want to generalize your results
–Target Population - the population you are trying to represent with your research findings
–Accessible Population - the population you are actually able to get a sample from *may or may not be the same as your target population - the closer it matches your target population, the better
Two Types of Sampling
1. Probability Sampling (aka Simple Random Sampling, aka Straight Random Sample)= take a random selection of individuals from our population, such that each individual has an equal chance of being selected for participation in the study.
2. Non-Probability Sampling (aka Non-Random Sample)= individuals are selected from the population in such a way that not everyone has an equal chance of being selected for participation in the study. Not totally random.
Probability Sampling Methods:
1. Stratified Random Sampling= select subsets of the population to participate in the study in the same proportion as they appear in the population
e.g., 400 teachers in Salt Lake area schools, 275 are female and 125 are male
I decide to sample 40% of Salt Lake area teachers. My sample contains:
40% * 400 teachers = 160 total teachers in sample
40% * 275 female teachers = 110 females in sample
40% * 125 male teachers = 50 males in sample
2. Clustered random sample= select existing groups of participants instead of creating subgroups
e.g., Instead of randomly selecting individuals in correct proportions, I randomly select groups of individuals. So now I randomly select some schools in Salt Lake area district, and all teachers in those selected schools participate in my study. But, I must ensure that those groups selected are representative of my population as a whole.
3. Two-Stage Random Sampling= combines methods 1 and 2; in stage 1, existing groups are
randomly selected; in stage 2, individuals from those groups are randomly selected
e.g., Instead of randomly selecting individuals in correct proportions, I randomly select groups of individuals, then randomly select individuals from those groups
Stage 1: I randomly select some schools in Salt Lake area district.
Stage 2: From each selected school, I randomly select a subset of teachers to participate in the study
*If you don't have a really good reason for controlling your sample, it's probably better to just do a simple random sample. You can't control for every characteristic, so it's often best just to be random.
Non-Probability Sampling Methods:
1. Systematic Sampling= every nth individual in a population is selected for participation in the study
e.g., I take an alphabetical list of all teachers in Salt Lake area schools, and select every 3rd individual from that list for participation in my study. Here, 3 is my sampling interval
sampling interval = population size / desired sample size
e.g., sampling interval = 400 teachers / 160 teachers (or 40%) =2.5
sampling ratio = proportion of individuals in population selected for sample
e.g., sampling ratio = 160/400 = .4 or 40%
2. Convenience Sampling = select from a group of individuals who are conveniently available to be participants in your study
e.g., I go into schools at lunchtime and give surveys to those teachers who can be found in the teachers’ lounge
Potential Problem:
Sample is likely to be biased –are those teachers in the lounge at lunchtime likely to be different from those who aren’t?
This type of sampling should be avoided if possible.
3. Purposive Sampling= researchers use past knowledge or own judgment to select a sample that he/she thinks is representative of the population
e.g., I decide to just give my survey to teachers who are also currently enrolled in the EDPS 6030, because I *think* they are representative of the population of Salt Lake area teachers
Potential problem: Researchers may be biased about what they believe is representative of a population, or they may be just plain wrong.
Be very cautious of this kind of sampling!
Sampling in Qualitative Research
• Purposive Sampling
• Case Analysis (aka Case Study)
–Typical - the prototype, the typical example
–Extreme - the unusually extreme example
–Critical - highlights the characteristics you want to study
• Maximum Variation - you are representing the extremes of your examples (some less than typical, some typical, some extreme)
• Snowball Sampling - you select some people for your sample, then ask the to get some people to participate, then they get some people to participate, etc.
Sampling and Validity
1. What size sample is appropriate?
Descriptive => 100 subjects
Correlational=> 50 subjects
Experimental => 30 subjects per group* (You will often see less than that.)
Causal-Comparative => 30 subjects per group*
But if groups are tightly controlled, less (e.g., 15 per group) may be OK.
2. How generalizable is the sample?
external validity= the results should be generalizable beyond the conditions of the individual study
a. Population generalizability= extent to which the sample represents the population of interest
b. Ecological generalizability= degree to which the results can be extended to other settings or conditions
What is Measurement?
• Measurement - the collection of data, the gathering of information
• Evaluation - making a decision based on the information
• Where does assessment fit in? - both measurement and evaluation are lumped together
What kind of scale is the measurement based on?
• Nominal - categorical (qualitative variables)
• Ordinal - rank order, no other information.
eg. 1st, 2nd, and 3rd place but no details about the distance between 1st and 2nd or 2nd and 3rd
• Interval - we do know the distance between the results - there is NO absolute zero
• Ratio - we do know the distance between the results - there IS an absolute zero
Types of Educational Measures
• Cognitive vs. Non-Cognitive
cognitive - interested in the cognitive processes involved
non-cognitive - ex. opionion - not cognitively based
• Commercial vs. Non-Commercial
commercial - developed by a company - tried and tested, standardized, generalized
non-commercial - developed by the researcher - tailored for your own needs
• Direct vs. Indirect
direct - getting our information directly from the participants
indirect - getting our information from somewhere besides the participants
Sample Cognitive Measures
• Standardized Tests
–Achievement Tests - tests things already learned
–Aptitude Tests - tests potential for future learning
• Behavioral Measures
–Naming Time
–Response Time
–Reading Time
• Wpm
• Eyetracking Measures
Non-Cognitive Measures
• Surveys & Questionnaires
• Observations
• Interviews
How is an individual’s score interpreted?
1. Norm-referenced instruments= an individual’s score is based on comparison with peers (e.g.,
percentile rank, age/grade equivalents, grading on curve, etc.)
2. Criterion-referenced instruments= an individual’s score is based on some predetermined standard (e.g., raw score)
Interpreting Data
Different Ways to Present Scores:
1. Raw Score= number of items answered correctly, number of times behavior is tallied, etc.
2. Derived Scores= scores changed into a more meaningful unit
a. age/grade equivalent scores= for a given score, tell what age/grade score usually falls in
b. percentile rank= ranking of score compared to all other individuals who took the test
c. standard scores (aka Z Scores) = indicates how far scores are from a reference point; usually best to use in research - allows you to compare scores from two totally different scales
Important Characteristics of Measures
• Objectivity
• Usability - Can I use it? Am I able to interpret the data I get from it?
• Validity - Does the measure actually measure what it's supposed to measure?
• Reliability - Do I get consistent measurements over time?
Wednesday, January 16, 2008
Research Questions and Variables and Hypotheses
Research Questions & Variables and Hypotheses
I.What is a researchable question?
II. Characteristics of researchable questions
III. Research Variables
A. Quantitative vs. Categorical
B. Independent vs. Dependent
IV. Hypotheses
A. Quantitative vs. Qualitative
V. Identifying Research Articles
Research Problems vs. Research Questions
Research Problem:a problem to be solved, area of concern, general question, etc.
e.g. We want to increase use of technology in K-3 classrooms in Utah.
Research Question:a clarification of the research problem, which is the focus of the research and drives the methodology chosen
e.g. Does integration of technology into teaching in grades K-3 lead to higher standardized achievement test scores than traditional teaching methods alone?
Researchable Research Questions
•Where do they come from?
–Experimenter interests
–Application issues
–Replication issues
•Do they focus on product or process? Or neither?
•Are they researchable? Unresearchable?
Researchable vs. Un-Researchable Questions
Researchable Questions–contain empirical referents
Empirical Referent –something that can be observed and/or quantified in some way
e.g., The Pepsi Challenge –which soda do people prefer more? Coca-Cola or Pepsi?
Un-Researchable Questions–contain no empirical referents, involve value judgments
e.g., Should prayer be allowed in schools?
Essential Characteristics of Good Research Questions:
1. They are feasible.
2. They are clear.
a. Conceptual or Constitutive definition = all terms in the question must be well-defined and understood
ex. In the question, "Does technology in K-3 schools improve standardized test scores over traditional teaching methods?" What counts as technology? What kind of K-3 schools? Which tests? What are "traditional" teaching methods? - You must be very clear about these terms.
b. Operational definition = specify how the dependent variable will be measured
3. They are significant.
4. They are ethical.
a. Protect participants from harm.
b. Ensure confidentiality.
c. Should subjects be deceived?
Variables: Quantitative vs. Categorical
1. Quantitative Variables
a.Continuous
b.Discontinuous (Discrete)
2. Categorical Variables - not quantifiable - you can't attach a number to it
Can look for relationships among:
1. Two Quantitative Variables - ex. height and weight
2. Two Categorical Variables - ex. religion and political affiliation
3. A Quantitative and Categorical Variable - ex. age and occupation (unless they make age categorical)
Variables: Independent vs. Dependent
1. Independent Variable= variable that affects the dependent variable, or is the effect of interest - This is what you are manipulating in the experiment.
a.Manipulated
b.Selected
2. Dependent Variable= dependent on the independent variable, or what is being measured
3. Extraneous variable (aka. confound) = uncontrolled factors affecting the dependent variable
*Dependent variables and extraneous variables are separate variables. They can't, in theory, be both.
Quantitative Research Hypotheses
•They should be stated in declarative form.
•They should be based on facts/research/theory. - can't just be based on a hunch
•They should be testable.
•They should be clear and concise.
•If possible, they should be directional - Take a stand! Predict the effect is going to be in a certain direction.
Qualitative Research Questions
•They are written about a central phenomenon instead of a prediction.
•They should be:
–Not too general…not too specific
–Amenable to change as data collection progresses - you may start with one idea and one direction, but move in a new direction based on new things you learn from your research - be open to whatever you might discover
--Unbiased by the researcher’s assumptions or hoped findings
Identifying Research Articles
1. What type of source is it?
–Primary Source–original research article
–Secondary Source–reviews, summarizes, or discusses research conducted by others
–Tertiary Source–summary of a basic topic, rather than summaries of individual studies
2. Is it peer reviewed?
–Refereed journals
•Editors vs. Reviewers
•Blind Reviews
•Level of journal in field
–Non-refereed journals
Why peer review?
•Importance of verification before dissemination
–Once the media disseminates information, it is hard to undo the damage
•ex. Scientist arguing autism a result of MMR vaccine never published his results in a scientific journal
•ex. Claim of first human baby clone was based only on the company’s statement
–Greater the significance of the finding, the more important it is to ensure that the finding is valid
Is peer review an insurance policy?
•Not exactly –some fraudulent (or incorrect) claims may still make it through to publication
ex. –Korean scientist who fabricated data supporting the landmark claim in 2004 that he created the world's first stem cells from a cloned human embryo. - His false data was in a peer review journal, but it was still fraudulent.
•Peer review is another source of information for:
–Funding Allocation
–Quality of Research / Publication in Scientific Journals
–Quality of Research Institutions (both on department and university levels)
–Policy Decisions
Where to find research articles:
Marriot Library - ERIC Ebsco
Make sure it has:
Method
Participants
Measure or Instruments
Prodecures
I.What is a researchable question?
II. Characteristics of researchable questions
III. Research Variables
A. Quantitative vs. Categorical
B. Independent vs. Dependent
IV. Hypotheses
A. Quantitative vs. Qualitative
V. Identifying Research Articles
Research Problems vs. Research Questions
Research Problem:a problem to be solved, area of concern, general question, etc.
e.g. We want to increase use of technology in K-3 classrooms in Utah.
Research Question:a clarification of the research problem, which is the focus of the research and drives the methodology chosen
e.g. Does integration of technology into teaching in grades K-3 lead to higher standardized achievement test scores than traditional teaching methods alone?
Researchable Research Questions
•Where do they come from?
–Experimenter interests
–Application issues
–Replication issues
•Do they focus on product or process? Or neither?
•Are they researchable? Unresearchable?
Researchable vs. Un-Researchable Questions
Researchable Questions–contain empirical referents
Empirical Referent –something that can be observed and/or quantified in some way
e.g., The Pepsi Challenge –which soda do people prefer more? Coca-Cola or Pepsi?
Un-Researchable Questions–contain no empirical referents, involve value judgments
e.g., Should prayer be allowed in schools?
Essential Characteristics of Good Research Questions:
1. They are feasible.
2. They are clear.
a. Conceptual or Constitutive definition = all terms in the question must be well-defined and understood
ex. In the question, "Does technology in K-3 schools improve standardized test scores over traditional teaching methods?" What counts as technology? What kind of K-3 schools? Which tests? What are "traditional" teaching methods? - You must be very clear about these terms.
b. Operational definition = specify how the dependent variable will be measured
3. They are significant.
4. They are ethical.
a. Protect participants from harm.
b. Ensure confidentiality.
c. Should subjects be deceived?
Variables: Quantitative vs. Categorical
1. Quantitative Variables
a.Continuous
b.Discontinuous (Discrete)
2. Categorical Variables - not quantifiable - you can't attach a number to it
Can look for relationships among:
1. Two Quantitative Variables - ex. height and weight
2. Two Categorical Variables - ex. religion and political affiliation
3. A Quantitative and Categorical Variable - ex. age and occupation (unless they make age categorical)
Variables: Independent vs. Dependent
1. Independent Variable= variable that affects the dependent variable, or is the effect of interest - This is what you are manipulating in the experiment.
a.Manipulated
b.Selected
2. Dependent Variable= dependent on the independent variable, or what is being measured
3. Extraneous variable (aka. confound) = uncontrolled factors affecting the dependent variable
*Dependent variables and extraneous variables are separate variables. They can't, in theory, be both.
Quantitative Research Hypotheses
•They should be stated in declarative form.
•They should be based on facts/research/theory. - can't just be based on a hunch
•They should be testable.
•They should be clear and concise.
•If possible, they should be directional - Take a stand! Predict the effect is going to be in a certain direction.
Qualitative Research Questions
•They are written about a central phenomenon instead of a prediction.
•They should be:
–Not too general…not too specific
–Amenable to change as data collection progresses - you may start with one idea and one direction, but move in a new direction based on new things you learn from your research - be open to whatever you might discover
--Unbiased by the researcher’s assumptions or hoped findings
Identifying Research Articles
1. What type of source is it?
–Primary Source–original research article
–Secondary Source–reviews, summarizes, or discusses research conducted by others
–Tertiary Source–summary of a basic topic, rather than summaries of individual studies
2. Is it peer reviewed?
–Refereed journals
•Editors vs. Reviewers
•Blind Reviews
•Level of journal in field
–Non-refereed journals
Why peer review?
•Importance of verification before dissemination
–Once the media disseminates information, it is hard to undo the damage
•ex. Scientist arguing autism a result of MMR vaccine never published his results in a scientific journal
•ex. Claim of first human baby clone was based only on the company’s statement
–Greater the significance of the finding, the more important it is to ensure that the finding is valid
Is peer review an insurance policy?
•Not exactly –some fraudulent (or incorrect) claims may still make it through to publication
ex. –Korean scientist who fabricated data supporting the landmark claim in 2004 that he created the world's first stem cells from a cloned human embryo. - His false data was in a peer review journal, but it was still fraudulent.
•Peer review is another source of information for:
–Funding Allocation
–Quality of Research / Publication in Scientific Journals
–Quality of Research Institutions (both on department and university levels)
–Policy Decisions
Where to find research articles:
Marriot Library - ERIC Ebsco
Make sure it has:
Method
Participants
Measure or Instruments
Prodecures
Wednesday, January 9, 2008
First Day of Class
Print out the quizzes and see if you can answer the questions using your notes, lecture, reading, etc. Don't work with others, but feel free to take notes and discuss based upon those questions before taking it online.
Next week meet in the MBH computer lab (rm. 108) for class.
I. Theme is technology in education
II.
III.
IV.
Why is research important?
textbooks don't know everything
programs of instruction - we keep changing programs based on new research
(ie. phonics vs. whole lang.)
who conducts research?
No Child Left Behind Act - scientifically based evidence (very narrow)
How do we know what we think we know?
personal experiences
tradition - also known as tenacity
appeal to authority - ask an expert
a priori knowledge - a hunch, intuition
reason
inductive - specific to general - after we collect the data and try to interpret and apply it
deductive - general to specific - before we ever collect the data
Science as a "Knowing"
Research = systematic process of gathering, interpreting, and reporting information
In this class we're going to focus on the methods and results of a research project (rather than just the intro. and discussion about it). Has a researcher really found what he says he has found? Did he really do what he says he did?
It's up to the consumer to figure out what the implications of a research article will be. It's up to the teacher, for example, to interpret the research and decide how it will affect their instruction in the classroom.
Basic vs. Applied Research
Action vs. Theoretical Research
Are some research findings more easily applied than others?
Qualitative vs. Quantitative Research
Quantitative -
posivitist orientation (the truth is out there, and I can find it)
world is a reality made up of facts to be discovered
balck and white, relationships between varieables, cause and effects
detached and objective researchers
goal is to generalize beyond the experiment
Qualitative -
interpretivist/constructivist orientation (there are multiple realities and multiple truths)
world is made up of multiple realities that are socially constructed by individual views of a
situation
understand situations and events from the viewpoints of participants involved
researchers are immersed in situations being studied
Don't necessarity try to generalize beyond the situation because the research is so tied to
the context
Perhaps the best research model is a mix between qualitative and quantitative
Characteristics of both qualitative and quantitative studies:
Research creates info. that should be shard publicly. (whether it's perfect or not, it should be shared - that's the point)
Research findings should be replicable.
Research is used to refute or support claims - NOT to PROVE them. (You can't prove anything and close the book on it. Someone can always come along and refute your claims.)
Researchers should take care to control for errors and biases.
Research findings are limited in their generalizability.
Research should be analyzed and critiqued carefully.
The Scientific Method
1.Problem
2. Clarify
3. Determine the information needed and how to get it
4. Organize the information
5. Interpret the results
This is easier to apply to quantitative than to qualitative
Method of Scientific Inquiry
Objectivity
Control of bias
Willingness to alter beliefs
Verification (through replication)
Induction (generlize beyond specific causes)
Precision (details: timing, order, subject, etc.)
Truth - always working toward it, but will never prove it
How does this apply to research on technology in education?
•Research on process - focus on understanding (psychology)
•Research on product - focus on application (education)
Next week meet in the MBH computer lab (rm. 108) for class.
I. Theme is technology in education
II.
III.
IV.
Why is research important?
textbooks don't know everything
programs of instruction - we keep changing programs based on new research
(ie. phonics vs. whole lang.)
who conducts research?
No Child Left Behind Act - scientifically based evidence (very narrow)
How do we know what we think we know?
personal experiences
tradition - also known as tenacity
appeal to authority - ask an expert
a priori knowledge - a hunch, intuition
reason
inductive - specific to general - after we collect the data and try to interpret and apply it
deductive - general to specific - before we ever collect the data
Science as a "Knowing"
Research = systematic process of gathering, interpreting, and reporting information
In this class we're going to focus on the methods and results of a research project (rather than just the intro. and discussion about it). Has a researcher really found what he says he has found? Did he really do what he says he did?
It's up to the consumer to figure out what the implications of a research article will be. It's up to the teacher, for example, to interpret the research and decide how it will affect their instruction in the classroom.
Basic vs. Applied Research
Action vs. Theoretical Research
Are some research findings more easily applied than others?
Qualitative vs. Quantitative Research
Quantitative -
posivitist orientation (the truth is out there, and I can find it)
world is a reality made up of facts to be discovered
balck and white, relationships between varieables, cause and effects
detached and objective researchers
goal is to generalize beyond the experiment
Qualitative -
interpretivist/constructivist orientation (there are multiple realities and multiple truths)
world is made up of multiple realities that are socially constructed by individual views of a
situation
understand situations and events from the viewpoints of participants involved
researchers are immersed in situations being studied
Don't necessarity try to generalize beyond the situation because the research is so tied to
the context
Perhaps the best research model is a mix between qualitative and quantitative
Characteristics of both qualitative and quantitative studies:
Research creates info. that should be shard publicly. (whether it's perfect or not, it should be shared - that's the point)
Research findings should be replicable.
Research is used to refute or support claims - NOT to PROVE them. (You can't prove anything and close the book on it. Someone can always come along and refute your claims.)
Researchers should take care to control for errors and biases.
Research findings are limited in their generalizability.
Research should be analyzed and critiqued carefully.
The Scientific Method
1.Problem
2. Clarify
3. Determine the information needed and how to get it
4. Organize the information
5. Interpret the results
This is easier to apply to quantitative than to qualitative
Method of Scientific Inquiry
Objectivity
Control of bias
Willingness to alter beliefs
Verification (through replication)
Induction (generlize beyond specific causes)
Precision (details: timing, order, subject, etc.)
Truth - always working toward it, but will never prove it
How does this apply to research on technology in education?
•Research on process - focus on understanding (psychology)
•Research on product - focus on application (education)
Subscribe to:
Posts (Atom)
