Meta-Analysis

Quiz 1 Review
Alpha = .05 = probability that you're wrong. (Probability that there is no effect when you say there is one.)

Constitutive and Operational Definitions are essentially the same thing, but Constitutive is elaborating on terms, and Operational is clarifying your measuring.


Meta-Analysis
And Other Methods of Research Synthesis...

Levels of Research Synthesis
• Literature Reviews
--allows for in depth discussion of individual studies’findings and their theoretical implications
--no weight given to statistics in the discussion
• Numerical Reviews (“Vote Counting”) - Every significant effect casts a vote one way or the other - yes the data supports the hypothesis or no it doesn't - Doesn't look at any other factors, weight effect size, sample size, etc.
–Studies that are statistically significant “cast votes”for effects
–Statistical significance depends on many factors

• Meta-Synthesis
–Used in qualitative research
–It is NOT collection and aggregation of research results
–It is “the bringing together and breaking down of findings, examining them, discovering the
essential features, and, in some way, combining them into a transformed whole”
(Schreiber et al., 1997, p. 314)
- It tries to decontextualize the findings
-One important issue in this is selection criteria: must be published in a peer reviewed journal b/c at least they've been criticized and reviewed; date; participants; methods used; are researchers working in teams or individually (typically teams are better b/c there are people to bounce ideas off of throughout this long, tedious process - a system of checks and ballances)
-Then you must code and categorize all the contextual details of each study to make it simplified
-Audit Trail is necessary - listing the order and description of the coding and categorizing you did so others can follow it and replicate it
-Triangulation - come at your conclusions from at least 3 different perspectives


Meta-Analysis
• Origins of Meta-Analysis - farming!
• Definition – Statistical technique that enables the results from a number of studies to be combined to determine the average effect size of a given IV
–Supposedly more objective than narrative literature review b/c they care about the findings and the effect size (Cohen's d) as will as significant effect (p-value)

N= Sample Size

• Advantages
–Use stats to organize and extract info
–Eliminate selection bias (???)
–Makes use of all info in a study (???)
–Detects mediating characteristics

• Limitations
–No guide for implementation
–Time-limited - the analysis is already dated by the time it is even started
–No real rigid methodological rules
–Only as good as the studies it is based on

Meta-Analysis Methodology
• Research Focus - What is the question to be answered?
• Sampling - which studies will be used
–Inclusion Criteria? - must be peer-reviewed; dates; methodology; sample characteristics within those studies
• Classifying and Coding - easier b/c Meta-Analysis is quantitative rather than qualitative
• Role of the “Audit Trail” - this is usually only mentioned in qualitative research, but it is just as applicable and important for quantitative studies
• Data Analysis
–Significance vs. Effect size (Cohen's d - the number) vs. Effect Magnitude (small, medium, or large?)
–Comparison to Zero Effects
–Positive vs. Negative Effect Sizes
• Interpretation and Beyond
–Raises Researchers Consciousness?
–Highlights Gaps in Field Knowledge Base - Can identify what is really missing in this area of research
–Motivates Future Research
–Implications for Practice?