Final Paper & Poster

Final Paper:
Bring a hard copy of the article you reviewed.
Bring a hard copy of your original review.
Bring a hard copy of your new revised review.
Just paper clip them together.
More than 5 pages, less than 10.

Action Research Poster:
Come up with a question you have based on the article you read.
Answer a question that might help you in your teaching practice.
You do this type of thing in class anyway, this is just a little more formal.
Use the page given in class to plan your action research.

Create a poster displaying your action research project:
Make it on a tri-fold
You do not actually have to do the project, just plan it and explain what you think will happen.
See the sample page given in class.

Program Evaluation

Program Evaluation
Program Evaluation can take components of any or multiple of the designs discussed earlier this semester into one giant research project.

ACCOUNTABILITY
Assessment
Program Evaluation
Outcomes Assessment
Evidence
Formative
Summative

Both Program and Outcome Accountability are important
• You could have a program that does everything it says it’s going to do, but it doesn’t produce any outcomes
– D.A.R.E. is an example of this problem
• Or you could have a program that doesn’t do what it says it’s going to do, but there are still changes in outcomes
– So then how do you know what the changes in outcomes are due to –the program, or some extraneous variable?
Structures of the Program –how is it organized, what infrastructures are in place to support it?
Domains, Standards/Goals and Competencies of the Program
Process of the Program

PROGRAM ELEMENTS
• STRUCTURE
– Definition of the program
• "the program is a developmental educational program...”
• “This program is an integral part...”
• The program includes...”– Rationale for the program
• Importance of the program
• Reasons why students need what the program offers
– Assumptions of the program
• Identify and briefly describe the premise upon which a program rests
– Ethical and legal practice– Roles

PROGRAM STRUCTURAL ELEMENTS
Resources
–Human
–Financial
–Political
• Program Components
–Training
–Student Interventions
–Support for System

DOMAINS
Domains identify conceptual aspects of focus or priorities for the program.
The identification of domains:
1. Provides a description of areas of importance
2. Provides a structure to organize goals and competencies
3. All assessments should be related to the identified domains of the program

STANDARDS/GOALS
Standards define specific desired student results to be achieved by a certain time,
i.e. by the time the student leaves school or by the time that the student finishes the grade level.
They are stated in terms of knowledge, attitudes, skills that have been identified as necessary to the education of the student.
They are written in global terms for each domain.

COMPETENCIES and INDICATORS
Competencies are descriptions of student behaviors (knowledge, skills, attitudes) that are linked to stated goals for their education. They are based on the assumption that the behaviors can be transferred from a learning situation to a real-life situation.

Competencies are:
1. organized into goal content areas
2. arranged developmentally
3. measurable

Components of Program Evaluation
• Process vs. Outcome Evaluation
– Formative = evaluation of process
– Summative = evaluation of final product

• Stakeholders --anyone with a vested interest in the program
– Designers, evaluators, teachers, students, etc.

Program Evaluation: The Assessment Loop
• Components of Assessment
– Measurement
– Evaluation

• What kinds of evidence can be used in Program Evaluation assessments?
– Portfolios, performances, outside judges, observations, local tests, standardized tests, self- assessments, interviews, focus groups, student work (In qualitative research you want to have at least 3 different kinds of evidence to triangulate your results.)
– Nature of evidence depends on nature of question

SCHOOL PROGRAM EVALUATION
Program Evaluation is a cyclical process, in which the outcome of each cycle informs the content and objectives of the following cycle.
Note that both Process (e.g., Action Plan) and Product (e.g. Assessments) are important components of every cycle

Steps in Each Cycle of Program Evaluation:
•Identify Program Goals
•Conduct Needs Assessment
•Set Program Priorities
•Identify Target Groups
•Write Objectives for Activities
•Evaluate the Degree to Which Objectives Were Met
•Re-Set Program Priorities with the Established Program Goals

Observations and Interviews

Observations and Interviews

Interviews
•Who is your interviewee?
•Design the interview protocol.
•Where will the interview take place?
•Wording of the Questions - No leading or biasing questions; open-ended; clear & understandable, unambiguous, not double-barrelled, no two-part questions, no double negatives; ask in a respectful way
•Memorize the questions.
•Obtain consent before starting.
•Demonstrate Respect. - For individual, culture, community, etc.
•Develop rapport
•Stick to protocol as much as possible
•Record comments faithfully.
•Use recorder or videotape if possible.

Interview Types
Structured or Semi-Structured
“Verbal questionnaires”More Formal
Specific protocol prepared in advance
Designed to elicit specific information
Informal Interviews
More informal, more like a conversationStill involve planning
Move from non-threatening to threatening questionsDesigned to determine more general views of respondent

Types of Interview Questions
Demographic Questions
Knowledge Questions
Experience or Behavior Questions
Opinion/Value/Feeling Questions
Sensory Questions

Interview Questioning Tips
Ask questions in different ways
Ask for clarification
Vary control of flow of information
Avoid leading questions
Ask one question at a time
Don’t interrupt

Observation Process
1. What is the research question?
2. Where is the observation taking place?.
3. Who, what, when, and how long?
4. What is the role of the observer?

4. What is the observer’s role?
•Participant Observation
Complete Participant - posing as part of the group being observed - participates just like every other person in the group
Participant-as-Observer - participates in all the group's activities except for a few differences (ie. participates in a class but doesn't turn in the assignments)
Non-Participant Observation -
Complete Observer - the observer is not involved in the group activities and is a complete outsider
Observer-as-Participant - everyone in the group knows that the person is not part of the group, but they participate to some degree - more of an outsider

Minimize Bias
Collecting Observation/Interview Data
•Note Taking Methods
Field Jottings - quick notes
Field Diary - more in-depth information
Field Log - a chronology of events that occur during the observational process
•Descriptive vs. Reflective Notes
Describe subjects, settings, events, activities, observer behaviors
Reflect on analysis, method, ethics, frame of mind, points of clarification
•Data Coding Schemes

Sample Field Notes Form
Research Question:
Date/Time:
Setting:
Descriptive Notes:
Reflective Notes:

Sample Interview Protocol
Research Question:
Time of Interview:Date:
Setting:Interviewer:
Interviewee:
Position of Interviewee (if relevant):
(Briefly describe project to interviewee and obtain consent.)
Sample Interview Questions:
1. What has been your role in the incident?
Interviewee’s Response:
Interviewer’s Reflective Comments:
2. What has happened since the event that you have been involved in?
Interviewee’s Response:
Interviewer’s Reflective Comments:
3. What has been the impact on the community of this incident?
Interviewee’s Response:
Interviewer’s Reflective Comments:
4. To whom should we talk to find out more information about the incident?
Interviewee’s Response:
Interviewer’s Reflective Comments:
(Thank individual for participating in this interview. Assure him/her of confidentiality of responses and potential future interviews.)

Quiz 7 –last question
As a class, come up with a focus topic for a one-question interview related to the use of technology in education.
Availability of technology in the workplace
Next, write the question. Make sure it is clear, unbiased, uses respectful terminology, etc., and that it gets at your focus topic.
How does the availability of technology influence your productivity?
For your answer to the last question on quiz 7, you should interview one person, and record both descriptive (transcript) and reflective notes for this interview.

Survey Designs

Surveys & Questionnaires

Functions of Quantitative Designs
• Purpose of...
–Experimental Research is to draw conclusions about cause and effect
–Comparative Research is to examine relationships between variables
–Descriptive (Survey) Research is to summarize characteristics of a particular sample or population


Types of Surveys

Cross-Sectional –data is collected once from a variety of different groups

Longitudinal –data is collected over several points in time from a single group (better, but can be hard to do because people drop out of the study)
–Trend Studies–survey similar samples at different points in time (compromise between cross-sectional and longitudinal because it's not the same group, but at least they're more likely to have similar characteristics)
• I could sample the EDPS 6030 cohorts every year
–Cohort Studies--survey a different sample from the same group at different points in time
• I could sample 4 students from this group of EDPS 6030 students, with different students in each sample (hoping that the different samples are similar because at least they were from the same group, but problem is you never look at the original 4 people again)
–Panel Studies--survey the same sample at different points in time
• I could sample the current entire group of EDPS 6030 students every year


Survey Research Methodology

• What is the focus (unit of analysis) of the survey?
–Focus should be interesting and important enough for individuals to want to respond
–Objectives should be clearly defined, and all questions should relate to the objectives

• Who made up the sample?
–Target Populations vs. Actual Populations
–Demographic Information
–Sampling Response Rates - this is really going to depend upon how they survey is administered and the following items -->

• How is the survey administered?
– Direct, aka – In Person (ex. carrying clipboards around in the mall - people see you coming & avoid you)
– Mail
– Telephone - no call lists and people not answering is making this less effective
– Internet (These are becoming more and more effective. They're fast, easy, convenient.)


• What is explained to the subjects in the cover letter?
- The cover letter should answer the questions: who, what, how long, why, when
-It shouldn't bias them, but it should give them enough information t0 make them willing to spend the time to do it.

• Individual Survey Items
–Should be as unambiguous as possible
–Should be short and well-organized
–Should not ask more than one question at a time
–Should not contain biasing or overly technical terms
–Should not use leading questions
–Should avoid using double negatives
–Should not overuse contingency questions
-Reverse Scoring - when you always word your questions in the affirmative, they might start giving lazy answers, but if you reverse and write some in the negative, you can better ensure that you're getting real responses

Survey Item Types

Closed-ended
–Advantages
• Consistent and unambiguous responses
• Easier to score and interpret
• More likely to elicit responses
–Disadvantages
• Limits amount of “data”received
• More difficult to construct good questions
• More questions are usually required

Open-ended
–Advantages
• Allows more freedom in individual responses
• Easier to construct
–Disadvantages
• More difficult to score and interpret
• May elicit more ambiguous responses
• Less consistency in individuals’ response rates and response contents

Analyses of Survey Data: Closed-Ended Questions

• Descriptive Statistics
–Percentages
–Mean, Median, Mode
–Frequency Distributions

• Comparative Statistics
–Correlations (r, R)
–Chi Square (χ 2) -- this tells you if the distribution of responses different than what you would expect from chance (ex. the Pepsi challenge - if there are 3 choices of cola, you would expect Pepsi to get 33.3% of the votes simply by chance - the Chi Square tells you if the results were different than this)

• Inferential Statistics
–T-tests (t)
–ANOVAs (F)
–MANOVAs(F)
(if a survey starts using these kinds of results, you want to ask for a p value to back up their data and their claims)

Analyses of Survey Data: Open-Ended Questions

• Use of Qualitative Analysis Methods
–Content Analysis - claims are backed up by participant quotes because you can't give statistical data
–Triangulation - support is needed for this claim from at least 3 different sources of data
–Emergent Patterns
–Emergent Hypotheses

Validity and Reliability
• Pretesting - try out your survey before sending it to everyone
• Reliability Checks - someone needs to check how reliably the data was coded or entered into a program (like Xcel)
• Validity Checks - make sure your survey is valid
–Internal Validity - Make sure questions aren't biased, mode of conducting survey is effective, etc.
–External Validity - Am I going to be able to generalize my results beyond those who answer it?


QUIZ

http://www.iteaconnect.org/TAA/PDFs/Gallupsurvey.pdf
http://www.iteaconnect.org/TAA/PDFs/GallupQuestions2004.pdf
In light of what you just learned, examine the questions used by the Gallup Poll.
Are they clear? Well-written? Free from bias?
If you do think there are problems with any of the questions, indicate what the problems are and then rewrite the questions to address those problems.

Correlational Designs

Correlational Designs

Correlational Designs
–Used to measure relationships between two or more variables (r)
• Explanation
• Prediction
–No conclusions about cause and effect may be drawn

More and more researchers are making causal conclusions inappropriately.
They are making causal conclusions from correlational data, which you can NOT do.

Analyzing Data
• Correlation Coefficients
– “r”can range from -1 to +1
– Negative correlation = as one variable decreases, other increases
The negative doesn't mean the correlation is any less strong, it simply goes in the other direction.
ex. Typing skill and typing errors
– Positive correlation = as one variable increases, other also increases
ex. Standardized test scores and GPA
– Zero correlation = no relationship between the two variables

Plotting Correlational Data --Scatterplots
The more scattered the dots are on the graph, the closer the correlation coefficient gets to zero

Interpreting Correlations
• Correlation coefficient “r”
–Ranges from -1 to 1
–Indicates whether relationship is positive or negative
–Statistical significance (p-value) depends on size of relationship and size of sample

Magnitude of r Interpretation
Look at the absolute value of r to determine the magnitude of r
So what does r mean, anyway?
.00 to .40 weak relationship
.41 to .60 moderate relationship
.61 to .80 strong relationship
.81 or above very strong relationship.

More on CorrelationalDesigns
• Predicting from Multiple Variables
–Can compute several individual correlation coefficients to produce a correlation matrix (a table showing how the different variables correlate)
• Or can conduct a Multiple Regression Analysis - puts all the variables together for one coefficient (R)
–Yields coefficient R, which can be interpreted similarly to the simple correlation, r
–Also yields R2(coefficient of determination)
Coefficient of determination = Effect size estimate for correlational studies = how much of the result we can explain by the effect of all the other variables
ex. Reading Comprehension has many variables. R2 = How much of the reading comprehension can we explain through these other factors?

Imagine trying to dissect the concept “Reading Comprehension.” It is made up of several related factors, such as:
•Fluency
•Intrinsic Motivation
•Verbal IQ
•Working Memory Capacity
•Background Knowledge
If we sum up the portion of those components that uniquely overlaps with reading comprehension, we can explain a big part of reading comprehension. That is essentially what R2 in a multiple regression does.
So R2tells you, given all of the factors we’ve entered into the equation, how much of Reading Comprehension can be explained by those factors.

Other Uses of Correlational Data
• Structural Equation Modeling (aka. SEM, Path Analysis, Hierarchical, Stepwise) –maps out relationships among several variables
–Instead of lumping everything into a multiple regression, we can put them into a structural equation model; Allows researchers to see the “big picture”as well as relationships among individual variables
• Factor Analysis –how do individual variables or items in a measure combine to create “mega-
variables”
–E.g. Several items on a questionnaire might relate to your interest in a topic. Instead of treating each item as an individual variable, we combine them as one “factor”
•Path Modeling - shows how all of the factors correlated together AND how they work together to predict supportive computer use (looks like boxes with arrows going every which way showing correlational values between each combination)

Other Comparative Designs

Review of Terms:

Random Selection/Sampling vs. Random Assignment
Random Selection = How do I get my sample? All people in your population have an equal chance of being selected
Random Assignment = Once you have your sample, how do you assign them to a group?

Internal Validity vs. External Validity
Internal Validity = control of the experiment
External Validity = generalizability of your experiment
You want to have the correct balance between the two

Independent, Dependent, & Extraneous Variables
Independent Variable = What you are manipulating
Dependent Variable = depends on the independent
Extraneous Variables = the things that mess everything up

Between vs. Within Subjects Variables
Between subjects variable = looking for a difference between subjects - they don't get the same experience in both groups - but you need to make sure that both groups are as similar as possible to confirm that the only differences between groups is your independent variable
Within subjects variable = Every individual gets both experiences in the experiment - ex. Pre vs. Post - Within is more ideal because you know your groups will be consistent for both independent variables

Factorial Design = measures the impact of more than one independent variable at once
Benefit: you can see if there is an interaction between the different independent variables
(No more than three independent variables, otherwise it gets too complicated and you can't tell what variables are interacting with which variables very clearly)

Experimental vs. Quasi-Experimental Designs
"True" Experimental Designs = involve random assignment to conditions manipulated by experimenter
Quasi-Experimental Designs = involve comparisions of groups in pre-selected conditions or groups - I design the study before I collect the data
Causal Comparative Designs = are ex post facto quasi-experimental designs; They involve comparison of pre-selected conditions/groups after the fact

Time-Series Design
Measure dependent variable a lot over time

Experimental Designs, cont.
Single Subject Designs
-Like mini longitudinal experimental designs, on individuals or small groups of individuals
-Similar to pretest/posttest designs; examines DV before and after IV
-Used when it doesn’t make sense to pool effects across individuals - ex. When working with children with special needs, the specific behaviors and needs of one child are not the same as others' - But tracking that one child over time may help develop strategies to help that specific child - you're not trying to generalize your findings, you're just trying to help that one individual
--Special populations
--Focus on success of specific interventions with specific individuals

Possible Single-Subject Designs
A-B Design = baseline, followed by intervention
A = Baseline
B = Intervention
But what happens after the intervention is removed? Does behavior go back to baseline?

A-B-A Withdrawal Design = baseline, followed by intervention, concluded with baseline
When the intervention is removed, does behavior go back to baseline?
Ethical issue: is it OK to intervene but then leave subjects back at baseline behavior, especially if we know that the intervention is needed?

A-B-A-B Design = one step further; instead of leaving subjects back at baseline, present intervention again (more ethical)

Multiple Baselines Designs = alternative to ABAB design; used when it’s not ethical to leave the subject at the baseline condition and when measures on multiple Dependent Variables (DVs) are taken.
-Taking baselines for multiple behaviors at same time – whether it’s one behavior in multiple individuals, multiple behaviors in one individual, one type of behavior in one individual in multiple settings, etc.
-Difficult to use, because must ensure that multiple DVs aren’t related to one another

Issues to Consider: Internal Validity
Is Treatment Standardized?
-# Variables Changing Across Conditions ? - Is the behavior changing because of my intervention, or is there another explanation?
Condition Length?
-Degree and Speed of Change? - You want to show a stable trend. Do you have enough data points to see a stable trend in the experiment results? - The "magic" number is 7 measurements to see a stable trend
Experimenter Effects?
-Because this is a one-on-one experiment, the experimenter is likely to have an impact on the individual
Practical Significance of Results?
-Practical significance is looked at more than statistical significance because of the single-subject design. - ie. Did it help the student improve?

Single Subject Designs and External Validity
It is very difficult to generalize the results of these designs beyond the sample studied – WHY?
-Because it was only designed for one person for a specific purpose. It is not necessarily meant to be generalized.
Thus, it is VERY IMPORTANT for this type of research to be replicated before strong conclusions are drawn.

*We now have all the information we need for quiz 3 if we want to take it now.

Experimental Design

Experimental Designs

What makes experimental designs so great? CONTROL!!!
• In experimental designs, researchers have control over:
–Independent Variables (what is manipulated; whatever you are comparing, such as traiditonal vs. online technology)
–Dependent Variables (what is being measured, how are they operational? How is it being measured?)
–Extraneous Variables (all of the things I can't control; things that have impact on my dependent variable, such as dog color blindness)

Internal Validity and Control
• History (some uncontrolled event, such as a fire alarm)
• Selection (how do you select your subject)
• Maturation (individuals change over time)
• Testing (does just giving a test somehow influence what the subjects think about the Independent Variable)
• Instrumentation (something could be wrong with the test)
• Treatment Replications
• Attrition (mortality, or losing subjects during the experiment)
• Regression to the Mean
• Diffusion of Treatment - (does the Independent Variable from one group share info to another group) Does your independent variable in one group bleed into another group? One group tells the other group what happens in the experiment, or they talk about it and share opinions and thoughts that skew the results
• Experimenter Effects - (is the experimenter overly nice or overly mean to subjects; or if your tester is a cute girl and your subjects are 13 yr old boys, then they subjects do whatever the tester wants you to do)
How the behavior, personality, looks of the experimenter affect how the subjects react to and participate in the experiment
• Subject Effects - in self-report data, the attitude of the subject or the reason they are there can affect how they participate in the experiment - why did they participate? Does it take a certain kind of person to actually participate? Does this affect your results?
•Social Desirability Bias - the participant alters answers so they pain the picture of themselves that they want

External Validity and Control
• So more control ≈ higher levels of internal validity...
–Note, this is what allows us to make cause/effect conclusions about experimental data
• But what happens to external validity? - External validity lessens the more you control because more control means you are getting farther and farther away from the real world
*It's like a see-saw. You have to find the right balance between external validity (less control) and internal validity (more control)

Important Terms
• Randomization
–Random Selection - How do you get the participants for your study in the first place?
–vs. Random Assignment - What do you do with the participants once you've already gotten them? How do you assign them within the experiment?

• Within Subjects - Every participant gets every level of the independent variable - ex. in a pretest/post test design, every participant takes both the pre and post tests - this is always going to be better if you can make it work because everyone participates in everything and there is no chance that different groups are going to have different results because of extraneous variables
vs. Between Subjects Variables and Designs - One group gets level A of the independent variable, but a different group gets level B of the independent variable - ex. experimental vs. control group designs OR men vs. women design - You do run the risk that there is some extraneous variable that exists in one group, but not the other; there might be something fundamentally different between groups that affects your results

• Controlling for Confounds
–Holding variables constant
–Building variables into design
–Matching

Pretest/Posttest Designs
• Single Group Posttest-Only Design
– Expose 1 group to the IV and then measure the DV (posttest) once
– Example: I decide to test whether using puppets to “read” books with young children to help them learn different sound combinations. – What’s the problem? - You don't have anything to compare to, no baseline to start at. There is no way you can say the puppets affected anything if you don't know what the results of the test were before using the puppets. To make it even better, you should have a control group that takes a pre and post test without the puppets to prove it was the puppets that made the difference in test scores and not just the fact that they took the test twice.

Single Group Pretest Posttest
• For a single group, give them a pretest (DV), then the IV, then a posttest (DV again)
• Example, in the “puppet experiment,”I give students a pre-reading PA quiz, then read to them with the puppet, then give them a post-reading PA quiz

Nonequivalent Groups Pretest Posttest: Quasi-Experimental
• Involves 2 groups (experimental and control); both get pretest (DV), then only the exper. Gp. is exposed to the IV, and both gps. get the posttest (DV)
*Pre-existing groups

• Between Subjects Designs= each group of subjects receives a different level of the IV
–Advantage: often more practical that within subjects designs
–Disadvantage: are differences due to groups? Or to the IV?
–Use of Matching

And what you’ve all been waiting for...“TRUE”Experimental Designs
• Randomized Groups PrettestPosttest Design
– Individuals are randomly assigned to either exper. Or control grp.; both groups receive the pretest (DV), then only the exper.
Gp. Is exposed to the IV, and then both gps. Receive the posttest (DV)
– Random assignment gives control over group differences, and pretest allows for a baseline measure in both groups

Experimental Designs with more than 1 Independent Variable (IV)
• Factorial Designs= measure the impact of 2 or more IVs on the Dependent Variable (DV)
(ex. test whether puppets help develop reading skills, AND whether it helps English Language Learners more)
–Advantages
• Allow researchers to determine whether effects are consistent across subject characteristics
• Allow researchers to assess interactions
–Disadvantages
• Can get messy if there are too many IVs because every IV you add changes the "personality" of the experiment - NEVER HAVE MORE THAN 3 IVs, otherwise it gets too messy
Sample Between Subjects Factorial Design

Analyses of Exp. Designs (t and F)
• T-test (t) --used to assess the impact of two levels of one IV on a single DV
• ANOVA (F) –used to assess impact of one or more IVs on a single DV
• MANOVA –used to assess impact of one or more IVs on multiple DVs
• ANCOVA –used to assess impact of one or more IVs on a single DV after removing effects of a variable that might be correlated to DV (e.g., age, gender, aptitude, achievement, etc.)
• MANCOVA –used to assess impact of one or more IVs on multiple DVs after removing effects of a variable that might be correlated to DV (e.g., age, gender, aptitude, achievement, etc.)

Meta-Analysis

Quiz 1 Review
Alpha = .05 = probability that you're wrong. (Probability that there is no effect when you say there is one.)

Constitutive and Operational Definitions are essentially the same thing, but Constitutive is elaborating on terms, and Operational is clarifying your measuring.


Meta-Analysis
And Other Methods of Research Synthesis...

Levels of Research Synthesis
• Literature Reviews
--allows for in depth discussion of individual studies’findings and their theoretical implications
--no weight given to statistics in the discussion
• Numerical Reviews (“Vote Counting”) - Every significant effect casts a vote one way or the other - yes the data supports the hypothesis or no it doesn't - Doesn't look at any other factors, weight effect size, sample size, etc.
–Studies that are statistically significant “cast votes”for effects
–Statistical significance depends on many factors

• Meta-Synthesis
–Used in qualitative research
–It is NOT collection and aggregation of research results
–It is “the bringing together and breaking down of findings, examining them, discovering the
essential features, and, in some way, combining them into a transformed whole”
(Schreiber et al., 1997, p. 314)
- It tries to decontextualize the findings
-One important issue in this is selection criteria: must be published in a peer reviewed journal b/c at least they've been criticized and reviewed; date; participants; methods used; are researchers working in teams or individually (typically teams are better b/c there are people to bounce ideas off of throughout this long, tedious process - a system of checks and ballances)
-Then you must code and categorize all the contextual details of each study to make it simplified
-Audit Trail is necessary - listing the order and description of the coding and categorizing you did so others can follow it and replicate it
-Triangulation - come at your conclusions from at least 3 different perspectives


Meta-Analysis
• Origins of Meta-Analysis - farming!
• Definition – Statistical technique that enables the results from a number of studies to be combined to determine the average effect size of a given IV
–Supposedly more objective than narrative literature review b/c they care about the findings and the effect size (Cohen's d) as will as significant effect (p-value)

N= Sample Size

• Advantages
–Use stats to organize and extract info
–Eliminate selection bias (???)
–Makes use of all info in a study (???)
–Detects mediating characteristics

• Limitations
–No guide for implementation
–Time-limited - the analysis is already dated by the time it is even started
–No real rigid methodological rules
–Only as good as the studies it is based on

Meta-Analysis Methodology
• Research Focus - What is the question to be answered?
• Sampling - which studies will be used
–Inclusion Criteria? - must be peer-reviewed; dates; methodology; sample characteristics within those studies
• Classifying and Coding - easier b/c Meta-Analysis is quantitative rather than qualitative
• Role of the “Audit Trail” - this is usually only mentioned in qualitative research, but it is just as applicable and important for quantitative studies
• Data Analysis
–Significance vs. Effect size (Cohen's d - the number) vs. Effect Magnitude (small, medium, or large?)
–Comparison to Zero Effects
–Positive vs. Negative Effect Sizes
• Interpretation and Beyond
–Raises Researchers Consciousness?
–Highlights Gaps in Field Knowledge Base - Can identify what is really missing in this area of research
–Motivates Future Research
–Implications for Practice?

Validity & Statistics

Important Characteristics of Measures
• Validity
• Reliability
• Objectivity
• Usability

Validity vs. Reliability
Validity= appropriateness, correctness, meaningfulness, and usefulness of inferences made about the instruments used in a study
Reliability= consistency of scores obtained, across individuals, administrators, and sets of items

Relationship Between Reliability and Validity
Suppose I have a faulty measuring tape and I use it to measure each student’s height.
On the other hand, if I have a correctly printed measuring tape...
My tool is invalid, but it’s still reliable.
My tool is both valid and reliable.

Something can be valid & reliable.
Something can be invalid but reliable.
But if something is unreliable, it is always invalid.

Types of Validity
• Content Validity -
• Criterion Validity -
–Predictive Validity - Ability of the measure to predict future performance
–Concurrent Validity -
• Convergent vs. DiscriminantValidity
Convergent Validity - they are trying to show that one measure is showing the same thing as another measure
Discriminant Validity - showing one measure is actually showing something quite different than another measure
• Construct Validity -
• Internal Validity - How well is your study designed?

Threats to Internal Validity:
Subject characteristics
Mortality threat (attrition)
Location
Instrumentation
Data Collectors
Testing
History
Maturation
Attitude of subjects
Regression threat
Implementation

Ways That Threats to Internal Validity Can be Minimized:
a. Standardized study conditions - The "Bus Test" - If you walked out the door and got hit by a bus, someone else could pick up right where you left off with your research.
b. Obtain more information on individuals in the sample
c. Obtain more information about details of study
d. Choice of appropriate design

Reliability Checks
• Test-Retest (aka Stability) - Tests have consistent results
• Equivalent Forms - Multiple forms of the same test - If one individual takes both forms of the test, the scores should be highly correlated
• Internal Consistency
–Split-half - compare 1/2 of the items on the test to the other 1/2 to ensure that all items on the test are reliable - NEVER compare the 1st half to the last half of the items because fatigue or not completing the test can greatly affect the answers on the 2nd half; instead you could compare odd #s to even #s
–KuderRichardson
–ChronbachAlpha
• Inter-Rater (Agreement)

Analyzing Data
Frequency Polygon aka Frequency Distribution
Normal Distribution
Descriptive Statistics = describe a sample
Inferential Statistics = describe a sample, and are inferred to a larger (target) population

•Measures of Central Tendency:
–Mean = statistical average - the best, most stable measure of central tendency
–Median = middle score
–Mode = most frequent score

• Measures of Variability
–Range = highest score minus the lowest score
–Standard deviation = average deviation from the mean
–Standard error of measurement = range in which “true score”is likely to fall
Standardized scores (or z-scores) = transform raw scores into standard deviation units on the normal distribution; z = (raw score –mean) / stand. dev.

Correlational Data -- plotted on scatterplots
• Correlation Coefficients
–“r”can range from -1 to +1
–Negative correlation = as one variable decreases, other increases (r is close to -1)
–Positive correlation = as one variable increases, other also increases (r is close to +1)
–Zero correlation = no relationship between the two variables (the closer r is to 0, the less correlation there is)
*You cannot imply causation from correlation.*

Hypothesis Testing
Null Hypothesis (H0) = set up to state that there is no effect
Alternative Hypothesis (H1) = set up to state that there is an effect
These two hypotheses must be:
• Mutually Exclusive - they can't overlap - either there is no effect or there is an effect
• Exhaustive

Test by determining by doing statistics to determine probability that the result was due to
chance:
• If probability that the result was due to chance <> 5%, the null hypothesis cannot be rejected
• 5% level => alpha level => .05
So, a researcher wants the probability (p) that their results were due to chance to be less than 5% (0.05).
If p is <>
If p is > 0.05, there is a non-significant effect.

If my null hypothesis is true, but I reject the null, that is a Type I Error.
If my null hypothesis is true, and I fail to reject the null, that is a correct decision.
If my null hypothesis is false, and I reject the null, that is a correct decision.
If my null hypothesis is false, and I fail to reject the null, that is a Type II Error.

This is the one I want! I will do anything I can to increase my POWER to get this result.

Ways Researchers May try to Increase Likelihood of Rejecting Null Hypothesis:
• Increase sample size.
• Control for extraneous variables (confounds).
• Increase the strength of the treatment.
• Use a one-tailed test when justifiable.

How do you know how “big”an effect really is?
• Effect Sizes = an estimate of the magnitude of an effect between two groups or variables
–Cohen’s d - an estimate of effect size
–η2(eta-squared) or partial η2
–Coefficient of determination (R2)

Interpreting Cohen’s d:
Small d <.2 (statistically significant, but not practically significant)
Medium .3 < d < .5
Large d > .5

NEXT WEEK
• Moving into different Research Designs
–Everybody read:
• Kavalearticle• Rosen & Solomon article
–Starting with Meta-Analyses
• I’ll discuss Kavalearticle
• Staci, Randy, and Katie will lead discussion on Rosen & Solomon article
• Initial Article Analyses are Due
–Use guidelines on Initial Analysis handout–Consider “What you Know to Ask So Far”
–Turn in your review and a complete copy of the article you reviewed
1st Quiz is due Tuesday at midnight.

Sampling & Measurement

Sampling & Measurement
I. Sampling
A. Samples vs. Populations
B. Sampling Methods
1. Quantitative Methods
2. Qualitative Methods
C. Issues in Sampling
II. Measurement
A. Measurement, Evaluation, & Assessment
B. Types of Educational Measures
C. Interpreting Data
D. Evaluating Measures

Samples vs. Populations
• Sample= group of people participating in your study
• Population= group of people to whom you want to generalize your results
–Target Population - the population you are trying to represent with your research findings
–Accessible Population - the population you are actually able to get a sample from *may or may not be the same as your target population - the closer it matches your target population, the better

Two Types of Sampling
1. Probability Sampling (aka Simple Random Sampling, aka Straight Random Sample)= take a random selection of individuals from our population, such that each individual has an equal chance of being selected for participation in the study.
2. Non-Probability Sampling (aka Non-Random Sample)= individuals are selected from the population in such a way that not everyone has an equal chance of being selected for participation in the study. Not totally random.

Probability Sampling Methods:
1. Stratified Random Sampling= select subsets of the population to participate in the study in the same proportion as they appear in the population
e.g., 400 teachers in Salt Lake area schools, 275 are female and 125 are male
I decide to sample 40% of Salt Lake area teachers. My sample contains:
40% * 400 teachers = 160 total teachers in sample
40% * 275 female teachers = 110 females in sample
40% * 125 male teachers = 50 males in sample

2. Clustered random sample= select existing groups of participants instead of creating subgroups
e.g., Instead of randomly selecting individuals in correct proportions, I randomly select groups of individuals. So now I randomly select some schools in Salt Lake area district, and all teachers in those selected schools participate in my study. But, I must ensure that those groups selected are representative of my population as a whole.

3. Two-Stage Random Sampling= combines methods 1 and 2; in stage 1, existing groups are
randomly selected; in stage 2, individuals from those groups are randomly selected
e.g., Instead of randomly selecting individuals in correct proportions, I randomly select groups of individuals, then randomly select individuals from those groups
Stage 1: I randomly select some schools in Salt Lake area district.
Stage 2: From each selected school, I randomly select a subset of teachers to participate in the study

*If you don't have a really good reason for controlling your sample, it's probably better to just do a simple random sample. You can't control for every characteristic, so it's often best just to be random.

Non-Probability Sampling Methods:
1. Systematic Sampling= every nth individual in a population is selected for participation in the study
e.g., I take an alphabetical list of all teachers in Salt Lake area schools, and select every 3rd individual from that list for participation in my study. Here, 3 is my sampling interval
sampling interval = population size / desired sample size
e.g., sampling interval = 400 teachers / 160 teachers (or 40%) =2.5
sampling ratio = proportion of individuals in population selected for sample
e.g., sampling ratio = 160/400 = .4 or 40%

2. Convenience Sampling = select from a group of individuals who are conveniently available to be participants in your study
e.g., I go into schools at lunchtime and give surveys to those teachers who can be found in the teachers’ lounge
Potential Problem:
Sample is likely to be biased –are those teachers in the lounge at lunchtime likely to be different from those who aren’t?
This type of sampling should be avoided if possible.

3. Purposive Sampling= researchers use past knowledge or own judgment to select a sample that he/she thinks is representative of the population
e.g., I decide to just give my survey to teachers who are also currently enrolled in the EDPS 6030, because I *think* they are representative of the population of Salt Lake area teachers
Potential problem: Researchers may be biased about what they believe is representative of a population, or they may be just plain wrong.
Be very cautious of this kind of sampling!

Sampling in Qualitative Research
• Purposive Sampling
• Case Analysis (aka Case Study)
–Typical - the prototype, the typical example
–Extreme - the unusually extreme example
–Critical - highlights the characteristics you want to study
• Maximum Variation - you are representing the extremes of your examples (some less than typical, some typical, some extreme)
• Snowball Sampling - you select some people for your sample, then ask the to get some people to participate, then they get some people to participate, etc.

Sampling and Validity
1. What size sample is appropriate?
Descriptive => 100 subjects
Correlational=> 50 subjects
Experimental => 30 subjects per group* (You will often see less than that.)
Causal-Comparative => 30 subjects per group*
But if groups are tightly controlled, less (e.g., 15 per group) may be OK.

2. How generalizable is the sample?
external validity= the results should be generalizable beyond the conditions of the individual study
a. Population generalizability= extent to which the sample represents the population of interest
b. Ecological generalizability= degree to which the results can be extended to other settings or conditions

What is Measurement?
• Measurement - the collection of data, the gathering of information
• Evaluation - making a decision based on the information
• Where does assessment fit in? - both measurement and evaluation are lumped together

What kind of scale is the measurement based on?
• Nominal - categorical (qualitative variables)
• Ordinal - rank order, no other information.
eg. 1st, 2nd, and 3rd place but no details about the distance between 1st and 2nd or 2nd and 3rd
• Interval - we do know the distance between the results - there is NO absolute zero
• Ratio - we do know the distance between the results - there IS an absolute zero

Types of Educational Measures
• Cognitive vs. Non-Cognitive
cognitive - interested in the cognitive processes involved
non-cognitive - ex. opionion - not cognitively based
• Commercial vs. Non-Commercial
commercial - developed by a company - tried and tested, standardized, generalized
non-commercial - developed by the researcher - tailored for your own needs
• Direct vs. Indirect
direct - getting our information directly from the participants
indirect - getting our information from somewhere besides the participants

Sample Cognitive Measures
• Standardized Tests
–Achievement Tests - tests things already learned
–Aptitude Tests - tests potential for future learning
• Behavioral Measures
–Naming Time
–Response Time
–Reading Time
• Wpm
• Eyetracking Measures

Non-Cognitive Measures
• Surveys & Questionnaires
• Observations
• Interviews

How is an individual’s score interpreted?
1. Norm-referenced instruments= an individual’s score is based on comparison with peers (e.g.,
percentile rank, age/grade equivalents, grading on curve, etc.)
2. Criterion-referenced instruments= an individual’s score is based on some predetermined standard (e.g., raw score)

Interpreting Data
Different Ways to Present Scores:
1. Raw Score= number of items answered correctly, number of times behavior is tallied, etc.
2. Derived Scores= scores changed into a more meaningful unit
a. age/grade equivalent scores= for a given score, tell what age/grade score usually falls in
b. percentile rank= ranking of score compared to all other individuals who took the test
c. standard scores (aka Z Scores) = indicates how far scores are from a reference point; usually best to use in research - allows you to compare scores from two totally different scales

Important Characteristics of Measures
• Objectivity
• Usability - Can I use it? Am I able to interpret the data I get from it?
• Validity - Does the measure actually measure what it's supposed to measure?
• Reliability - Do I get consistent measurements over time?

Research Questions and Variables and Hypotheses

Research Questions & Variables and Hypotheses
I.What is a researchable question?
II. Characteristics of researchable questions
III. Research Variables
A. Quantitative vs. Categorical
B. Independent vs. Dependent
IV. Hypotheses
A. Quantitative vs. Qualitative
V. Identifying Research Articles


Research Problems vs. Research Questions
Research Problem:a problem to be solved, area of concern, general question, etc.
e.g. We want to increase use of technology in K-3 classrooms in Utah.
Research Question:a clarification of the research problem, which is the focus of the research and drives the methodology chosen
e.g. Does integration of technology into teaching in grades K-3 lead to higher standardized achievement test scores than traditional teaching methods alone?

Researchable Research Questions
•Where do they come from?
–Experimenter interests
–Application issues
–Replication issues
•Do they focus on product or process? Or neither?
•Are they researchable? Unresearchable?

Researchable vs. Un-Researchable Questions
Researchable Questions–contain empirical referents
Empirical Referent –something that can be observed and/or quantified in some way
e.g., The Pepsi Challenge –which soda do people prefer more? Coca-Cola or Pepsi?
Un-Researchable Questions–contain no empirical referents, involve value judgments
e.g., Should prayer be allowed in schools?

Essential Characteristics of Good Research Questions:
1. They are feasible.
2. They are clear.
a. Conceptual or Constitutive definition = all terms in the question must be well-defined and understood
ex. In the question, "Does technology in K-3 schools improve standardized test scores over traditional teaching methods?" What counts as technology? What kind of K-3 schools? Which tests? What are "traditional" teaching methods? - You must be very clear about these terms.
b. Operational definition = specify how the dependent variable will be measured
3. They are significant.
4. They are ethical.
a. Protect participants from harm.
b. Ensure confidentiality.
c. Should subjects be deceived?

Variables: Quantitative vs. Categorical
1. Quantitative Variables
a.Continuous
b.Discontinuous (Discrete)
2. Categorical Variables - not quantifiable - you can't attach a number to it

Can look for relationships among:
1. Two Quantitative Variables - ex. height and weight
2. Two Categorical Variables - ex. religion and political affiliation
3. A Quantitative and Categorical Variable - ex. age and occupation (unless they make age categorical)

Variables: Independent vs. Dependent
1. Independent Variable= variable that affects the dependent variable, or is the effect of interest - This is what you are manipulating in the experiment.
a.Manipulated
b.Selected
2. Dependent Variable= dependent on the independent variable, or what is being measured
3. Extraneous variable (aka. confound) = uncontrolled factors affecting the dependent variable

*Dependent variables and extraneous variables are separate variables. They can't, in theory, be both.

Quantitative Research Hypotheses
•They should be stated in declarative form.
•They should be based on facts/research/theory. - can't just be based on a hunch
•They should be testable.
•They should be clear and concise.
•If possible, they should be directional - Take a stand! Predict the effect is going to be in a certain direction.

Qualitative Research Questions
•They are written about a central phenomenon instead of a prediction.
•They should be:
–Not too general…not too specific
–Amenable to change as data collection progresses - you may start with one idea and one direction, but move in a new direction based on new things you learn from your research - be open to whatever you might discover
--Unbiased by the researcher’s assumptions or hoped findings


Identifying Research Articles
1. What type of source is it?
–Primary Source–original research article
–Secondary Source–reviews, summarizes, or discusses research conducted by others
–Tertiary Source–summary of a basic topic, rather than summaries of individual studies
2. Is it peer reviewed?
–Refereed journals
•Editors vs. Reviewers
•Blind Reviews
•Level of journal in field
–Non-refereed journals

Why peer review?
•Importance of verification before dissemination
–Once the media disseminates information, it is hard to undo the damage
•ex. Scientist arguing autism a result of MMR vaccine never published his results in a scientific journal
•ex. Claim of first human baby clone was based only on the company’s statement
–Greater the significance of the finding, the more important it is to ensure that the finding is valid

Is peer review an insurance policy?
•Not exactly –some fraudulent (or incorrect) claims may still make it through to publication
ex. –Korean scientist who fabricated data supporting the landmark claim in 2004 that he created the world's first stem cells from a cloned human embryo. - His false data was in a peer review journal, but it was still fraudulent.
•Peer review is another source of information for:
–Funding Allocation
–Quality of Research / Publication in Scientific Journals
–Quality of Research Institutions (both on department and university levels)
–Policy Decisions

Where to find research articles:
Marriot Library - ERIC Ebsco
Make sure it has:
Method
Participants
Measure or Instruments
Prodecures

First Day of Class

Print out the quizzes and see if you can answer the questions using your notes, lecture, reading, etc. Don't work with others, but feel free to take notes and discuss based upon those questions before taking it online.

Next week meet in the MBH computer lab (rm. 108) for class.


I. Theme is technology in education
II.
III.
IV.

Why is research important?
textbooks don't know everything
programs of instruction - we keep changing programs based on new research
(ie. phonics vs. whole lang.)
who conducts research?
No Child Left Behind Act - scientifically based evidence (very narrow)

How do we know what we think we know?
personal experiences
tradition - also known as tenacity
appeal to authority - ask an expert
a priori knowledge - a hunch, intuition
reason
inductive - specific to general - after we collect the data and try to interpret and apply it
deductive - general to specific - before we ever collect the data

Science as a "Knowing"
Research = systematic process of gathering, interpreting, and reporting information

In this class we're going to focus on the methods and results of a research project (rather than just the intro. and discussion about it). Has a researcher really found what he says he has found? Did he really do what he says he did?

It's up to the consumer to figure out what the implications of a research article will be. It's up to the teacher, for example, to interpret the research and decide how it will affect their instruction in the classroom.
Basic vs. Applied Research
Action vs. Theoretical Research

Are some research findings more easily applied than others?
Qualitative vs. Quantitative Research

Quantitative -
posivitist orientation (the truth is out there, and I can find it)
world is a reality made up of facts to be discovered
balck and white, relationships between varieables, cause and effects
detached and objective researchers
goal is to generalize beyond the experiment

Qualitative -
interpretivist/constructivist orientation (there are multiple realities and multiple truths)
world is made up of multiple realities that are socially constructed by individual views of a
situation
understand situations and events from the viewpoints of participants involved
researchers are immersed in situations being studied
Don't necessarity try to generalize beyond the situation because the research is so tied to
the context

Perhaps the best research model is a mix between qualitative and quantitative

Characteristics of both qualitative and quantitative studies:
Research creates info. that should be shard publicly. (whether it's perfect or not, it should be shared - that's the point)
Research findings should be replicable.
Research is used to refute or support claims - NOT to PROVE them. (You can't prove anything and close the book on it. Someone can always come along and refute your claims.)
Researchers should take care to control for errors and biases.
Research findings are limited in their generalizability.
Research should be analyzed and critiqued carefully.

The Scientific Method
1.Problem
2. Clarify
3. Determine the information needed and how to get it
4. Organize the information
5. Interpret the results

This is easier to apply to quantitative than to qualitative

Method of Scientific Inquiry
Objectivity
Control of bias
Willingness to alter beliefs
Verification (through replication)
Induction (generlize beyond specific causes)
Precision (details: timing, order, subject, etc.)

Truth - always working toward it, but will never prove it

How does this apply to research on technology in education?
•Research on process - focus on understanding (psychology)
•Research on product - focus on application (education)