Review of Terms:
Random Selection/Sampling vs. Random Assignment
Random Selection = How do I get my sample? All people in your population have an equal chance of being selected
Random Assignment = Once you have your sample, how do you assign them to a group?
Internal Validity vs. External Validity
Internal Validity = control of the experiment
External Validity = generalizability of your experiment
You want to have the correct balance between the two
Independent, Dependent, & Extraneous Variables
Independent Variable = What you are manipulating
Dependent Variable = depends on the independent
Extraneous Variables = the things that mess everything up
Between vs. Within Subjects Variables
Between subjects variable = looking for a difference between subjects - they don't get the same experience in both groups - but you need to make sure that both groups are as similar as possible to confirm that the only differences between groups is your independent variable
Within subjects variable = Every individual gets both experiences in the experiment - ex. Pre vs. Post - Within is more ideal because you know your groups will be consistent for both independent variables
Factorial Design = measures the impact of more than one independent variable at once
Benefit: you can see if there is an interaction between the different independent variables
(No more than three independent variables, otherwise it gets too complicated and you can't tell what variables are interacting with which variables very clearly)
Experimental vs. Quasi-Experimental Designs
"True" Experimental Designs = involve random assignment to conditions manipulated by experimenter
Quasi-Experimental Designs = involve comparisions of groups in pre-selected conditions or groups - I design the study before I collect the data
Causal Comparative Designs = are ex post facto quasi-experimental designs; They involve comparison of pre-selected conditions/groups after the fact
Time-Series Design
Measure dependent variable a lot over time
Experimental Designs, cont.
Single Subject Designs
-Like mini longitudinal experimental designs, on individuals or small groups of individuals
-Similar to pretest/posttest designs; examines DV before and after IV
-Used when it doesn’t make sense to pool effects across individuals - ex. When working with children with special needs, the specific behaviors and needs of one child are not the same as others' - But tracking that one child over time may help develop strategies to help that specific child - you're not trying to generalize your findings, you're just trying to help that one individual
--Special populations
--Focus on success of specific interventions with specific individuals
Possible Single-Subject Designs
A-B Design = baseline, followed by intervention
A = Baseline
B = Intervention
But what happens after the intervention is removed? Does behavior go back to baseline?
A-B-A Withdrawal Design = baseline, followed by intervention, concluded with baseline
When the intervention is removed, does behavior go back to baseline?
Ethical issue: is it OK to intervene but then leave subjects back at baseline behavior, especially if we know that the intervention is needed?
A-B-A-B Design = one step further; instead of leaving subjects back at baseline, present intervention again (more ethical)
Multiple Baselines Designs = alternative to ABAB design; used when it’s not ethical to leave the subject at the baseline condition and when measures on multiple Dependent Variables (DVs) are taken.
-Taking baselines for multiple behaviors at same time – whether it’s one behavior in multiple individuals, multiple behaviors in one individual, one type of behavior in one individual in multiple settings, etc.
-Difficult to use, because must ensure that multiple DVs aren’t related to one another
Issues to Consider: Internal Validity
Is Treatment Standardized?
-# Variables Changing Across Conditions ? - Is the behavior changing because of my intervention, or is there another explanation?
Condition Length?
-Degree and Speed of Change? - You want to show a stable trend. Do you have enough data points to see a stable trend in the experiment results? - The "magic" number is 7 measurements to see a stable trend
Experimenter Effects?
-Because this is a one-on-one experiment, the experimenter is likely to have an impact on the individual
Practical Significance of Results?
-Practical significance is looked at more than statistical significance because of the single-subject design. - ie. Did it help the student improve?
Single Subject Designs and External Validity
It is very difficult to generalize the results of these designs beyond the sample studied – WHY?
-Because it was only designed for one person for a specific purpose. It is not necessarily meant to be generalized.
Thus, it is VERY IMPORTANT for this type of research to be replicated before strong conclusions are drawn.
*We now have all the information we need for quiz 3 if we want to take it now.
Wednesday, February 27, 2008
Wednesday, February 20, 2008
Experimental Design
Experimental Designs
What makes experimental designs so great? CONTROL!!!
• In experimental designs, researchers have control over:
–Independent Variables (what is manipulated; whatever you are comparing, such as traiditonal vs. online technology)
–Dependent Variables (what is being measured, how are they operational? How is it being measured?)
–Extraneous Variables (all of the things I can't control; things that have impact on my dependent variable, such as dog color blindness)
Internal Validity and Control
• History (some uncontrolled event, such as a fire alarm)
• Selection (how do you select your subject)
• Maturation (individuals change over time)
• Testing (does just giving a test somehow influence what the subjects think about the Independent Variable)
• Instrumentation (something could be wrong with the test)
• Treatment Replications
• Attrition (mortality, or losing subjects during the experiment)
• Regression to the Mean
• Diffusion of Treatment - (does the Independent Variable from one group share info to another group) Does your independent variable in one group bleed into another group? One group tells the other group what happens in the experiment, or they talk about it and share opinions and thoughts that skew the results
• Experimenter Effects - (is the experimenter overly nice or overly mean to subjects; or if your tester is a cute girl and your subjects are 13 yr old boys, then they subjects do whatever the tester wants you to do)
How the behavior, personality, looks of the experimenter affect how the subjects react to and participate in the experiment
• Subject Effects - in self-report data, the attitude of the subject or the reason they are there can affect how they participate in the experiment - why did they participate? Does it take a certain kind of person to actually participate? Does this affect your results?
•Social Desirability Bias - the participant alters answers so they pain the picture of themselves that they want
External Validity and Control
• So more control ≈ higher levels of internal validity...
–Note, this is what allows us to make cause/effect conclusions about experimental data
• But what happens to external validity? - External validity lessens the more you control because more control means you are getting farther and farther away from the real world
*It's like a see-saw. You have to find the right balance between external validity (less control) and internal validity (more control)
Important Terms
• Randomization
–Random Selection - How do you get the participants for your study in the first place?
–vs. Random Assignment - What do you do with the participants once you've already gotten them? How do you assign them within the experiment?
• Within Subjects - Every participant gets every level of the independent variable - ex. in a pretest/post test design, every participant takes both the pre and post tests - this is always going to be better if you can make it work because everyone participates in everything and there is no chance that different groups are going to have different results because of extraneous variables
vs. Between Subjects Variables and Designs - One group gets level A of the independent variable, but a different group gets level B of the independent variable - ex. experimental vs. control group designs OR men vs. women design - You do run the risk that there is some extraneous variable that exists in one group, but not the other; there might be something fundamentally different between groups that affects your results
• Controlling for Confounds
–Holding variables constant
–Building variables into design
–Matching
Pretest/Posttest Designs
• Single Group Posttest-Only Design
– Expose 1 group to the IV and then measure the DV (posttest) once
– Example: I decide to test whether using puppets to “read” books with young children to help them learn different sound combinations. – What’s the problem? - You don't have anything to compare to, no baseline to start at. There is no way you can say the puppets affected anything if you don't know what the results of the test were before using the puppets. To make it even better, you should have a control group that takes a pre and post test without the puppets to prove it was the puppets that made the difference in test scores and not just the fact that they took the test twice.
Single Group Pretest Posttest
• For a single group, give them a pretest (DV), then the IV, then a posttest (DV again)
• Example, in the “puppet experiment,”I give students a pre-reading PA quiz, then read to them with the puppet, then give them a post-reading PA quiz
Nonequivalent Groups Pretest Posttest: Quasi-Experimental
• Involves 2 groups (experimental and control); both get pretest (DV), then only the exper. Gp. is exposed to the IV, and both gps. get the posttest (DV)
*Pre-existing groups
• Between Subjects Designs= each group of subjects receives a different level of the IV
–Advantage: often more practical that within subjects designs
–Disadvantage: are differences due to groups? Or to the IV?
–Use of Matching
And what you’ve all been waiting for...“TRUE”Experimental Designs
• Randomized Groups PrettestPosttest Design
– Individuals are randomly assigned to either exper. Or control grp.; both groups receive the pretest (DV), then only the exper.
Gp. Is exposed to the IV, and then both gps. Receive the posttest (DV)
– Random assignment gives control over group differences, and pretest allows for a baseline measure in both groups
Experimental Designs with more than 1 Independent Variable (IV)
• Factorial Designs= measure the impact of 2 or more IVs on the Dependent Variable (DV)
(ex. test whether puppets help develop reading skills, AND whether it helps English Language Learners more)
–Advantages
• Allow researchers to determine whether effects are consistent across subject characteristics
• Allow researchers to assess interactions
–Disadvantages
• Can get messy if there are too many IVs because every IV you add changes the "personality" of the experiment - NEVER HAVE MORE THAN 3 IVs, otherwise it gets too messy
Sample Between Subjects Factorial Design
Analyses of Exp. Designs (t and F)
• T-test (t) --used to assess the impact of two levels of one IV on a single DV
• ANOVA (F) –used to assess impact of one or more IVs on a single DV
• MANOVA –used to assess impact of one or more IVs on multiple DVs
• ANCOVA –used to assess impact of one or more IVs on a single DV after removing effects of a variable that might be correlated to DV (e.g., age, gender, aptitude, achievement, etc.)
• MANCOVA –used to assess impact of one or more IVs on multiple DVs after removing effects of a variable that might be correlated to DV (e.g., age, gender, aptitude, achievement, etc.)
What makes experimental designs so great? CONTROL!!!
• In experimental designs, researchers have control over:
–Independent Variables (what is manipulated; whatever you are comparing, such as traiditonal vs. online technology)
–Dependent Variables (what is being measured, how are they operational? How is it being measured?)
–Extraneous Variables (all of the things I can't control; things that have impact on my dependent variable, such as dog color blindness)
Internal Validity and Control
• History (some uncontrolled event, such as a fire alarm)
• Selection (how do you select your subject)
• Maturation (individuals change over time)
• Testing (does just giving a test somehow influence what the subjects think about the Independent Variable)
• Instrumentation (something could be wrong with the test)
• Treatment Replications
• Attrition (mortality, or losing subjects during the experiment)
• Regression to the Mean
• Diffusion of Treatment - (does the Independent Variable from one group share info to another group) Does your independent variable in one group bleed into another group? One group tells the other group what happens in the experiment, or they talk about it and share opinions and thoughts that skew the results
• Experimenter Effects - (is the experimenter overly nice or overly mean to subjects; or if your tester is a cute girl and your subjects are 13 yr old boys, then they subjects do whatever the tester wants you to do)
How the behavior, personality, looks of the experimenter affect how the subjects react to and participate in the experiment
• Subject Effects - in self-report data, the attitude of the subject or the reason they are there can affect how they participate in the experiment - why did they participate? Does it take a certain kind of person to actually participate? Does this affect your results?
•Social Desirability Bias - the participant alters answers so they pain the picture of themselves that they want
External Validity and Control
• So more control ≈ higher levels of internal validity...
–Note, this is what allows us to make cause/effect conclusions about experimental data
• But what happens to external validity? - External validity lessens the more you control because more control means you are getting farther and farther away from the real world
*It's like a see-saw. You have to find the right balance between external validity (less control) and internal validity (more control)
Important Terms
• Randomization
–Random Selection - How do you get the participants for your study in the first place?
–vs. Random Assignment - What do you do with the participants once you've already gotten them? How do you assign them within the experiment?
• Within Subjects - Every participant gets every level of the independent variable - ex. in a pretest/post test design, every participant takes both the pre and post tests - this is always going to be better if you can make it work because everyone participates in everything and there is no chance that different groups are going to have different results because of extraneous variables
vs. Between Subjects Variables and Designs - One group gets level A of the independent variable, but a different group gets level B of the independent variable - ex. experimental vs. control group designs OR men vs. women design - You do run the risk that there is some extraneous variable that exists in one group, but not the other; there might be something fundamentally different between groups that affects your results
• Controlling for Confounds
–Holding variables constant
–Building variables into design
–Matching
Pretest/Posttest Designs
• Single Group Posttest-Only Design
– Expose 1 group to the IV and then measure the DV (posttest) once
– Example: I decide to test whether using puppets to “read” books with young children to help them learn different sound combinations. – What’s the problem? - You don't have anything to compare to, no baseline to start at. There is no way you can say the puppets affected anything if you don't know what the results of the test were before using the puppets. To make it even better, you should have a control group that takes a pre and post test without the puppets to prove it was the puppets that made the difference in test scores and not just the fact that they took the test twice.
Single Group Pretest Posttest
• For a single group, give them a pretest (DV), then the IV, then a posttest (DV again)
• Example, in the “puppet experiment,”I give students a pre-reading PA quiz, then read to them with the puppet, then give them a post-reading PA quiz
Nonequivalent Groups Pretest Posttest: Quasi-Experimental
• Involves 2 groups (experimental and control); both get pretest (DV), then only the exper. Gp. is exposed to the IV, and both gps. get the posttest (DV)
*Pre-existing groups
• Between Subjects Designs= each group of subjects receives a different level of the IV
–Advantage: often more practical that within subjects designs
–Disadvantage: are differences due to groups? Or to the IV?
–Use of Matching
And what you’ve all been waiting for...“TRUE”Experimental Designs
• Randomized Groups PrettestPosttest Design
– Individuals are randomly assigned to either exper. Or control grp.; both groups receive the pretest (DV), then only the exper.
Gp. Is exposed to the IV, and then both gps. Receive the posttest (DV)
– Random assignment gives control over group differences, and pretest allows for a baseline measure in both groups
Experimental Designs with more than 1 Independent Variable (IV)
• Factorial Designs= measure the impact of 2 or more IVs on the Dependent Variable (DV)
(ex. test whether puppets help develop reading skills, AND whether it helps English Language Learners more)
–Advantages
• Allow researchers to determine whether effects are consistent across subject characteristics
• Allow researchers to assess interactions
–Disadvantages
• Can get messy if there are too many IVs because every IV you add changes the "personality" of the experiment - NEVER HAVE MORE THAN 3 IVs, otherwise it gets too messy
Sample Between Subjects Factorial Design
Analyses of Exp. Designs (t and F)
• T-test (t) --used to assess the impact of two levels of one IV on a single DV
• ANOVA (F) –used to assess impact of one or more IVs on a single DV
• MANOVA –used to assess impact of one or more IVs on multiple DVs
• ANCOVA –used to assess impact of one or more IVs on a single DV after removing effects of a variable that might be correlated to DV (e.g., age, gender, aptitude, achievement, etc.)
• MANCOVA –used to assess impact of one or more IVs on multiple DVs after removing effects of a variable that might be correlated to DV (e.g., age, gender, aptitude, achievement, etc.)
Wednesday, February 6, 2008
Meta-Analysis
Quiz 1 Review
Alpha = .05 = probability that you're wrong. (Probability that there is no effect when you say there is one.)
Constitutive and Operational Definitions are essentially the same thing, but Constitutive is elaborating on terms, and Operational is clarifying your measuring.
Meta-Analysis
And Other Methods of Research Synthesis...
Levels of Research Synthesis
• Literature Reviews
--allows for in depth discussion of individual studies’findings and their theoretical implications
--no weight given to statistics in the discussion
• Numerical Reviews (“Vote Counting”) - Every significant effect casts a vote one way or the other - yes the data supports the hypothesis or no it doesn't - Doesn't look at any other factors, weight effect size, sample size, etc.
–Studies that are statistically significant “cast votes”for effects
–Statistical significance depends on many factors
• Meta-Synthesis
–Used in qualitative research
–It is NOT collection and aggregation of research results
–It is “the bringing together and breaking down of findings, examining them, discovering the
essential features, and, in some way, combining them into a transformed whole”
(Schreiber et al., 1997, p. 314)
- It tries to decontextualize the findings
-One important issue in this is selection criteria: must be published in a peer reviewed journal b/c at least they've been criticized and reviewed; date; participants; methods used; are researchers working in teams or individually (typically teams are better b/c there are people to bounce ideas off of throughout this long, tedious process - a system of checks and ballances)
-Then you must code and categorize all the contextual details of each study to make it simplified
-Audit Trail is necessary - listing the order and description of the coding and categorizing you did so others can follow it and replicate it
-Triangulation - come at your conclusions from at least 3 different perspectives
Meta-Analysis
• Origins of Meta-Analysis - farming!
• Definition – Statistical technique that enables the results from a number of studies to be combined to determine the average effect size of a given IV
–Supposedly more objective than narrative literature review b/c they care about the findings and the effect size (Cohen's d) as will as significant effect (p-value)
N= Sample Size
• Advantages
–Use stats to organize and extract info
–Eliminate selection bias (???)
–Makes use of all info in a study (???)
–Detects mediating characteristics
• Limitations
–No guide for implementation
–Time-limited - the analysis is already dated by the time it is even started
–No real rigid methodological rules
–Only as good as the studies it is based on
Meta-Analysis Methodology
• Research Focus - What is the question to be answered?
• Sampling - which studies will be used
–Inclusion Criteria? - must be peer-reviewed; dates; methodology; sample characteristics within those studies
• Classifying and Coding - easier b/c Meta-Analysis is quantitative rather than qualitative
• Role of the “Audit Trail” - this is usually only mentioned in qualitative research, but it is just as applicable and important for quantitative studies
• Data Analysis
–Significance vs. Effect size (Cohen's d - the number) vs. Effect Magnitude (small, medium, or large?)
–Comparison to Zero Effects
–Positive vs. Negative Effect Sizes
• Interpretation and Beyond
–Raises Researchers Consciousness?
–Highlights Gaps in Field Knowledge Base - Can identify what is really missing in this area of research
–Motivates Future Research
–Implications for Practice?
Alpha = .05 = probability that you're wrong. (Probability that there is no effect when you say there is one.)
Constitutive and Operational Definitions are essentially the same thing, but Constitutive is elaborating on terms, and Operational is clarifying your measuring.
Meta-Analysis
And Other Methods of Research Synthesis...
Levels of Research Synthesis
• Literature Reviews
--allows for in depth discussion of individual studies’findings and their theoretical implications
--no weight given to statistics in the discussion
• Numerical Reviews (“Vote Counting”) - Every significant effect casts a vote one way or the other - yes the data supports the hypothesis or no it doesn't - Doesn't look at any other factors, weight effect size, sample size, etc.
–Studies that are statistically significant “cast votes”for effects
–Statistical significance depends on many factors
• Meta-Synthesis
–Used in qualitative research
–It is NOT collection and aggregation of research results
–It is “the bringing together and breaking down of findings, examining them, discovering the
essential features, and, in some way, combining them into a transformed whole”
(Schreiber et al., 1997, p. 314)
- It tries to decontextualize the findings
-One important issue in this is selection criteria: must be published in a peer reviewed journal b/c at least they've been criticized and reviewed; date; participants; methods used; are researchers working in teams or individually (typically teams are better b/c there are people to bounce ideas off of throughout this long, tedious process - a system of checks and ballances)
-Then you must code and categorize all the contextual details of each study to make it simplified
-Audit Trail is necessary - listing the order and description of the coding and categorizing you did so others can follow it and replicate it
-Triangulation - come at your conclusions from at least 3 different perspectives
Meta-Analysis
• Origins of Meta-Analysis - farming!
• Definition – Statistical technique that enables the results from a number of studies to be combined to determine the average effect size of a given IV
–Supposedly more objective than narrative literature review b/c they care about the findings and the effect size (Cohen's d) as will as significant effect (p-value)
N= Sample Size
• Advantages
–Use stats to organize and extract info
–Eliminate selection bias (???)
–Makes use of all info in a study (???)
–Detects mediating characteristics
• Limitations
–No guide for implementation
–Time-limited - the analysis is already dated by the time it is even started
–No real rigid methodological rules
–Only as good as the studies it is based on
Meta-Analysis Methodology
• Research Focus - What is the question to be answered?
• Sampling - which studies will be used
–Inclusion Criteria? - must be peer-reviewed; dates; methodology; sample characteristics within those studies
• Classifying and Coding - easier b/c Meta-Analysis is quantitative rather than qualitative
• Role of the “Audit Trail” - this is usually only mentioned in qualitative research, but it is just as applicable and important for quantitative studies
• Data Analysis
–Significance vs. Effect size (Cohen's d - the number) vs. Effect Magnitude (small, medium, or large?)
–Comparison to Zero Effects
–Positive vs. Negative Effect Sizes
• Interpretation and Beyond
–Raises Researchers Consciousness?
–Highlights Gaps in Field Knowledge Base - Can identify what is really missing in this area of research
–Motivates Future Research
–Implications for Practice?
Subscribe to:
Posts (Atom)
