Final Paper:
Bring a hard copy of the article you reviewed.
Bring a hard copy of your original review.
Bring a hard copy of your new revised review.
Just paper clip them together.
More than 5 pages, less than 10.
Action Research Poster:
Come up with a question you have based on the article you read.
Answer a question that might help you in your teaching practice.
You do this type of thing in class anyway, this is just a little more formal.
Use the page given in class to plan your action research.
Create a poster displaying your action research project:
Make it on a tri-fold
You do not actually have to do the project, just plan it and explain what you think will happen.
See the sample page given in class.
Wednesday, April 16, 2008
Program Evaluation
Program Evaluation
Program Evaluation can take components of any or multiple of the designs discussed earlier this semester into one giant research project.
ACCOUNTABILITY
Assessment
Program Evaluation
Outcomes Assessment
Evidence
Formative
Summative
Both Program and Outcome Accountability are important
• You could have a program that does everything it says it’s going to do, but it doesn’t produce any outcomes
– D.A.R.E. is an example of this problem
• Or you could have a program that doesn’t do what it says it’s going to do, but there are still changes in outcomes
– So then how do you know what the changes in outcomes are due to –the program, or some extraneous variable?
Structures of the Program –how is it organized, what infrastructures are in place to support it?
Domains, Standards/Goals and Competencies of the Program
Process of the Program
PROGRAM ELEMENTS
• STRUCTURE
– Definition of the program
• "the program is a developmental educational program...”
• “This program is an integral part...”
• The program includes...”– Rationale for the program
• Importance of the program
• Reasons why students need what the program offers
– Assumptions of the program
• Identify and briefly describe the premise upon which a program rests
– Ethical and legal practice– Roles
PROGRAM STRUCTURAL ELEMENTS
Resources
–Human
–Financial
–Political
• Program Components
–Training
–Student Interventions
–Support for System
DOMAINS
Domains identify conceptual aspects of focus or priorities for the program.
The identification of domains:
1. Provides a description of areas of importance
2. Provides a structure to organize goals and competencies
3. All assessments should be related to the identified domains of the program
STANDARDS/GOALS
Standards define specific desired student results to be achieved by a certain time,
i.e. by the time the student leaves school or by the time that the student finishes the grade level.
They are stated in terms of knowledge, attitudes, skills that have been identified as necessary to the education of the student.
They are written in global terms for each domain.
COMPETENCIES and INDICATORS
Competencies are descriptions of student behaviors (knowledge, skills, attitudes) that are linked to stated goals for their education. They are based on the assumption that the behaviors can be transferred from a learning situation to a real-life situation.
Competencies are:
1. organized into goal content areas
2. arranged developmentally
3. measurable
Components of Program Evaluation
• Process vs. Outcome Evaluation
– Formative = evaluation of process
– Summative = evaluation of final product
• Stakeholders --anyone with a vested interest in the program
– Designers, evaluators, teachers, students, etc.
Program Evaluation: The Assessment Loop
• Components of Assessment
– Measurement
– Evaluation
• What kinds of evidence can be used in Program Evaluation assessments?
– Portfolios, performances, outside judges, observations, local tests, standardized tests, self- assessments, interviews, focus groups, student work (In qualitative research you want to have at least 3 different kinds of evidence to triangulate your results.)
– Nature of evidence depends on nature of question
SCHOOL PROGRAM EVALUATION
Program Evaluation is a cyclical process, in which the outcome of each cycle informs the content and objectives of the following cycle.
Note that both Process (e.g., Action Plan) and Product (e.g. Assessments) are important components of every cycle
Steps in Each Cycle of Program Evaluation:
•Identify Program Goals
•Conduct Needs Assessment
•Set Program Priorities
•Identify Target Groups
•Write Objectives for Activities
•Evaluate the Degree to Which Objectives Were Met
•Re-Set Program Priorities with the Established Program Goals
Program Evaluation can take components of any or multiple of the designs discussed earlier this semester into one giant research project.
ACCOUNTABILITY
Assessment
Program Evaluation
Outcomes Assessment
Evidence
Formative
Summative
Both Program and Outcome Accountability are important
• You could have a program that does everything it says it’s going to do, but it doesn’t produce any outcomes
– D.A.R.E. is an example of this problem
• Or you could have a program that doesn’t do what it says it’s going to do, but there are still changes in outcomes
– So then how do you know what the changes in outcomes are due to –the program, or some extraneous variable?
Structures of the Program –how is it organized, what infrastructures are in place to support it?
Domains, Standards/Goals and Competencies of the Program
Process of the Program
PROGRAM ELEMENTS
• STRUCTURE
– Definition of the program
• "the program is a developmental educational program...”
• “This program is an integral part...”
• The program includes...”– Rationale for the program
• Importance of the program
• Reasons why students need what the program offers
– Assumptions of the program
• Identify and briefly describe the premise upon which a program rests
– Ethical and legal practice– Roles
PROGRAM STRUCTURAL ELEMENTS
Resources
–Human
–Financial
–Political
• Program Components
–Training
–Student Interventions
–Support for System
DOMAINS
Domains identify conceptual aspects of focus or priorities for the program.
The identification of domains:
1. Provides a description of areas of importance
2. Provides a structure to organize goals and competencies
3. All assessments should be related to the identified domains of the program
STANDARDS/GOALS
Standards define specific desired student results to be achieved by a certain time,
i.e. by the time the student leaves school or by the time that the student finishes the grade level.
They are stated in terms of knowledge, attitudes, skills that have been identified as necessary to the education of the student.
They are written in global terms for each domain.
COMPETENCIES and INDICATORS
Competencies are descriptions of student behaviors (knowledge, skills, attitudes) that are linked to stated goals for their education. They are based on the assumption that the behaviors can be transferred from a learning situation to a real-life situation.
Competencies are:
1. organized into goal content areas
2. arranged developmentally
3. measurable
Components of Program Evaluation
• Process vs. Outcome Evaluation
– Formative = evaluation of process
– Summative = evaluation of final product
• Stakeholders --anyone with a vested interest in the program
– Designers, evaluators, teachers, students, etc.
Program Evaluation: The Assessment Loop
• Components of Assessment
– Measurement
– Evaluation
• What kinds of evidence can be used in Program Evaluation assessments?
– Portfolios, performances, outside judges, observations, local tests, standardized tests, self- assessments, interviews, focus groups, student work (In qualitative research you want to have at least 3 different kinds of evidence to triangulate your results.)
– Nature of evidence depends on nature of question
SCHOOL PROGRAM EVALUATION
Program Evaluation is a cyclical process, in which the outcome of each cycle informs the content and objectives of the following cycle.
Note that both Process (e.g., Action Plan) and Product (e.g. Assessments) are important components of every cycle
Steps in Each Cycle of Program Evaluation:
•Identify Program Goals
•Conduct Needs Assessment
•Set Program Priorities
•Identify Target Groups
•Write Objectives for Activities
•Evaluate the Degree to Which Objectives Were Met
•Re-Set Program Priorities with the Established Program Goals
Wednesday, April 2, 2008
Observations and Interviews
Observations and Interviews
Interviews
•Who is your interviewee?
•Design the interview protocol.
•Where will the interview take place?
•Wording of the Questions - No leading or biasing questions; open-ended; clear & understandable, unambiguous, not double-barrelled, no two-part questions, no double negatives; ask in a respectful way
•Memorize the questions.
•Obtain consent before starting.
•Demonstrate Respect. - For individual, culture, community, etc.
•Develop rapport
•Stick to protocol as much as possible
•Record comments faithfully.
•Use recorder or videotape if possible.
Interview Types
Structured or Semi-Structured
“Verbal questionnaires”More Formal
Specific protocol prepared in advance
Designed to elicit specific information
Informal Interviews
More informal, more like a conversationStill involve planning
Move from non-threatening to threatening questionsDesigned to determine more general views of respondent
Types of Interview Questions
Demographic Questions
Knowledge Questions
Experience or Behavior Questions
Opinion/Value/Feeling Questions
Sensory Questions
Interview Questioning Tips
Ask questions in different ways
Ask for clarification
Vary control of flow of information
Avoid leading questions
Ask one question at a time
Don’t interrupt
Observation Process
1. What is the research question?
2. Where is the observation taking place?.
3. Who, what, when, and how long?
4. What is the role of the observer?
4. What is the observer’s role?
•Participant Observation
Complete Participant - posing as part of the group being observed - participates just like every other person in the group
Participant-as-Observer - participates in all the group's activities except for a few differences (ie. participates in a class but doesn't turn in the assignments)
Non-Participant Observation -
Complete Observer - the observer is not involved in the group activities and is a complete outsider
Observer-as-Participant - everyone in the group knows that the person is not part of the group, but they participate to some degree - more of an outsider
Minimize Bias
Collecting Observation/Interview Data
•Note Taking Methods
Field Jottings - quick notes
Field Diary - more in-depth information
Field Log - a chronology of events that occur during the observational process
•Descriptive vs. Reflective Notes
Describe subjects, settings, events, activities, observer behaviors
Reflect on analysis, method, ethics, frame of mind, points of clarification
•Data Coding Schemes
Sample Field Notes Form
Research Question:
Date/Time:
Setting:
Descriptive Notes:
Reflective Notes:
Sample Interview Protocol
Research Question:
Time of Interview:Date:
Setting:Interviewer:
Interviewee:
Position of Interviewee (if relevant):
(Briefly describe project to interviewee and obtain consent.)
Sample Interview Questions:
1. What has been your role in the incident?
Interviewee’s Response:
Interviewer’s Reflective Comments:
2. What has happened since the event that you have been involved in?
Interviewee’s Response:
Interviewer’s Reflective Comments:
3. What has been the impact on the community of this incident?
Interviewee’s Response:
Interviewer’s Reflective Comments:
4. To whom should we talk to find out more information about the incident?
Interviewee’s Response:
Interviewer’s Reflective Comments:
(Thank individual for participating in this interview. Assure him/her of confidentiality of responses and potential future interviews.)
Quiz 7 –last question
As a class, come up with a focus topic for a one-question interview related to the use of technology in education.
Availability of technology in the workplace
Next, write the question. Make sure it is clear, unbiased, uses respectful terminology, etc., and that it gets at your focus topic.
How does the availability of technology influence your productivity?
For your answer to the last question on quiz 7, you should interview one person, and record both descriptive (transcript) and reflective notes for this interview.
Interviews
•Who is your interviewee?
•Design the interview protocol.
•Where will the interview take place?
•Wording of the Questions - No leading or biasing questions; open-ended; clear & understandable, unambiguous, not double-barrelled, no two-part questions, no double negatives; ask in a respectful way
•Memorize the questions.
•Obtain consent before starting.
•Demonstrate Respect. - For individual, culture, community, etc.
•Develop rapport
•Stick to protocol as much as possible
•Record comments faithfully.
•Use recorder or videotape if possible.
Interview Types
Structured or Semi-Structured
“Verbal questionnaires”More Formal
Specific protocol prepared in advance
Designed to elicit specific information
Informal Interviews
More informal, more like a conversationStill involve planning
Move from non-threatening to threatening questionsDesigned to determine more general views of respondent
Types of Interview Questions
Demographic Questions
Knowledge Questions
Experience or Behavior Questions
Opinion/Value/Feeling Questions
Sensory Questions
Interview Questioning Tips
Ask questions in different ways
Ask for clarification
Vary control of flow of information
Avoid leading questions
Ask one question at a time
Don’t interrupt
Observation Process
1. What is the research question?
2. Where is the observation taking place?.
3. Who, what, when, and how long?
4. What is the role of the observer?
4. What is the observer’s role?
•Participant Observation
Complete Participant - posing as part of the group being observed - participates just like every other person in the group
Participant-as-Observer - participates in all the group's activities except for a few differences (ie. participates in a class but doesn't turn in the assignments)
Non-Participant Observation -
Complete Observer - the observer is not involved in the group activities and is a complete outsider
Observer-as-Participant - everyone in the group knows that the person is not part of the group, but they participate to some degree - more of an outsider
Minimize Bias
Collecting Observation/Interview Data
•Note Taking Methods
Field Jottings - quick notes
Field Diary - more in-depth information
Field Log - a chronology of events that occur during the observational process
•Descriptive vs. Reflective Notes
Describe subjects, settings, events, activities, observer behaviors
Reflect on analysis, method, ethics, frame of mind, points of clarification
•Data Coding Schemes
Sample Field Notes Form
Research Question:
Date/Time:
Setting:
Descriptive Notes:
Reflective Notes:
Sample Interview Protocol
Research Question:
Time of Interview:Date:
Setting:Interviewer:
Interviewee:
Position of Interviewee (if relevant):
(Briefly describe project to interviewee and obtain consent.)
Sample Interview Questions:
1. What has been your role in the incident?
Interviewee’s Response:
Interviewer’s Reflective Comments:
2. What has happened since the event that you have been involved in?
Interviewee’s Response:
Interviewer’s Reflective Comments:
3. What has been the impact on the community of this incident?
Interviewee’s Response:
Interviewer’s Reflective Comments:
4. To whom should we talk to find out more information about the incident?
Interviewee’s Response:
Interviewer’s Reflective Comments:
(Thank individual for participating in this interview. Assure him/her of confidentiality of responses and potential future interviews.)
Quiz 7 –last question
As a class, come up with a focus topic for a one-question interview related to the use of technology in education.
Availability of technology in the workplace
Next, write the question. Make sure it is clear, unbiased, uses respectful terminology, etc., and that it gets at your focus topic.
How does the availability of technology influence your productivity?
For your answer to the last question on quiz 7, you should interview one person, and record both descriptive (transcript) and reflective notes for this interview.
Wednesday, March 12, 2008
Survey Designs
Surveys & Questionnaires
Functions of Quantitative Designs
• Purpose of...
–Experimental Research is to draw conclusions about cause and effect
–Comparative Research is to examine relationships between variables
–Descriptive (Survey) Research is to summarize characteristics of a particular sample or population
Types of Surveys
Cross-Sectional –data is collected once from a variety of different groups
Longitudinal –data is collected over several points in time from a single group (better, but can be hard to do because people drop out of the study)
–Trend Studies–survey similar samples at different points in time (compromise between cross-sectional and longitudinal because it's not the same group, but at least they're more likely to have similar characteristics)
• I could sample the EDPS 6030 cohorts every year
–Cohort Studies--survey a different sample from the same group at different points in time
• I could sample 4 students from this group of EDPS 6030 students, with different students in each sample (hoping that the different samples are similar because at least they were from the same group, but problem is you never look at the original 4 people again)
–Panel Studies--survey the same sample at different points in time
• I could sample the current entire group of EDPS 6030 students every year
Survey Research Methodology
• What is the focus (unit of analysis) of the survey?
–Focus should be interesting and important enough for individuals to want to respond
–Objectives should be clearly defined, and all questions should relate to the objectives
• Who made up the sample?
–Target Populations vs. Actual Populations
–Demographic Information
–Sampling Response Rates - this is really going to depend upon how they survey is administered and the following items -->
• How is the survey administered?
– Direct, aka – In Person (ex. carrying clipboards around in the mall - people see you coming & avoid you)
– Telephone - no call lists and people not answering is making this less effective
– Internet (These are becoming more and more effective. They're fast, easy, convenient.)
• What is explained to the subjects in the cover letter?
- The cover letter should answer the questions: who, what, how long, why, when
-It shouldn't bias them, but it should give them enough information t0 make them willing to spend the time to do it.
• Individual Survey Items
–Should be as unambiguous as possible
–Should be short and well-organized
–Should not ask more than one question at a time
–Should not contain biasing or overly technical terms
–Should not use leading questions
–Should avoid using double negatives
–Should not overuse contingency questions
-Reverse Scoring - when you always word your questions in the affirmative, they might start giving lazy answers, but if you reverse and write some in the negative, you can better ensure that you're getting real responses
Survey Item Types
Closed-ended
–Advantages
• Consistent and unambiguous responses
• Easier to score and interpret
• More likely to elicit responses
–Disadvantages
• Limits amount of “data”received
• More difficult to construct good questions
• More questions are usually required
Open-ended
–Advantages
• Allows more freedom in individual responses
• Easier to construct
–Disadvantages
• More difficult to score and interpret
• May elicit more ambiguous responses
• Less consistency in individuals’ response rates and response contents
Analyses of Survey Data: Closed-Ended Questions
• Descriptive Statistics
–Percentages
–Mean, Median, Mode
–Frequency Distributions
• Comparative Statistics
–Correlations (r, R)
–Chi Square (χ 2) -- this tells you if the distribution of responses different than what you would expect from chance (ex. the Pepsi challenge - if there are 3 choices of cola, you would expect Pepsi to get 33.3% of the votes simply by chance - the Chi Square tells you if the results were different than this)
• Inferential Statistics
–T-tests (t)
–ANOVAs (F)
–MANOVAs(F)
(if a survey starts using these kinds of results, you want to ask for a p value to back up their data and their claims)
Analyses of Survey Data: Open-Ended Questions
• Use of Qualitative Analysis Methods
–Content Analysis - claims are backed up by participant quotes because you can't give statistical data
–Triangulation - support is needed for this claim from at least 3 different sources of data
–Emergent Patterns
–Emergent Hypotheses
Validity and Reliability
• Pretesting - try out your survey before sending it to everyone
• Reliability Checks - someone needs to check how reliably the data was coded or entered into a program (like Xcel)
• Validity Checks - make sure your survey is valid
–Internal Validity - Make sure questions aren't biased, mode of conducting survey is effective, etc.
–External Validity - Am I going to be able to generalize my results beyond those who answer it?
QUIZ
http://www.iteaconnect.org/TAA/PDFs/Gallupsurvey.pdf
http://www.iteaconnect.org/TAA/PDFs/GallupQuestions2004.pdf
In light of what you just learned, examine the questions used by the Gallup Poll.
Are they clear? Well-written? Free from bias?
If you do think there are problems with any of the questions, indicate what the problems are and then rewrite the questions to address those problems.
Wednesday, March 5, 2008
Correlational Designs
Correlational Designs
Correlational Designs
–Used to measure relationships between two or more variables (r)
• Explanation
• Prediction
–No conclusions about cause and effect may be drawn
More and more researchers are making causal conclusions inappropriately.
They are making causal conclusions from correlational data, which you can NOT do.
Analyzing Data
• Correlation Coefficients
– “r”can range from -1 to +1
– Negative correlation = as one variable decreases, other increases
The negative doesn't mean the correlation is any less strong, it simply goes in the other direction.
ex. Typing skill and typing errors
– Positive correlation = as one variable increases, other also increases
ex. Standardized test scores and GPA
– Zero correlation = no relationship between the two variables
Plotting Correlational Data --Scatterplots
The more scattered the dots are on the graph, the closer the correlation coefficient gets to zero
Interpreting Correlations
• Correlation coefficient “r”
–Ranges from -1 to 1
–Indicates whether relationship is positive or negative
–Statistical significance (p-value) depends on size of relationship and size of sample
Magnitude of r Interpretation
Look at the absolute value of r to determine the magnitude of r
So what does r mean, anyway?
.00 to .40 weak relationship
.41 to .60 moderate relationship
.61 to .80 strong relationship
.81 or above very strong relationship.
More on CorrelationalDesigns
• Predicting from Multiple Variables
–Can compute several individual correlation coefficients to produce a correlation matrix (a table showing how the different variables correlate)
• Or can conduct a Multiple Regression Analysis - puts all the variables together for one coefficient (R)
–Yields coefficient R, which can be interpreted similarly to the simple correlation, r
–Also yields R2(coefficient of determination)
Coefficient of determination = Effect size estimate for correlational studies = how much of the result we can explain by the effect of all the other variables
ex. Reading Comprehension has many variables. R2 = How much of the reading comprehension can we explain through these other factors?
Imagine trying to dissect the concept “Reading Comprehension.” It is made up of several related factors, such as:
•Fluency
•Intrinsic Motivation
•Verbal IQ
•Working Memory Capacity
•Background Knowledge
If we sum up the portion of those components that uniquely overlaps with reading comprehension, we can explain a big part of reading comprehension. That is essentially what R2 in a multiple regression does.
So R2tells you, given all of the factors we’ve entered into the equation, how much of Reading Comprehension can be explained by those factors.
Other Uses of Correlational Data
• Structural Equation Modeling (aka. SEM, Path Analysis, Hierarchical, Stepwise) –maps out relationships among several variables
–Instead of lumping everything into a multiple regression, we can put them into a structural equation model; Allows researchers to see the “big picture”as well as relationships among individual variables
• Factor Analysis –how do individual variables or items in a measure combine to create “mega-
variables”
–E.g. Several items on a questionnaire might relate to your interest in a topic. Instead of treating each item as an individual variable, we combine them as one “factor”
•Path Modeling - shows how all of the factors correlated together AND how they work together to predict supportive computer use (looks like boxes with arrows going every which way showing correlational values between each combination)
Correlational Designs
–Used to measure relationships between two or more variables (r)
• Explanation
• Prediction
–No conclusions about cause and effect may be drawn
More and more researchers are making causal conclusions inappropriately.
They are making causal conclusions from correlational data, which you can NOT do.
Analyzing Data
• Correlation Coefficients
– “r”can range from -1 to +1
– Negative correlation = as one variable decreases, other increases
The negative doesn't mean the correlation is any less strong, it simply goes in the other direction.
ex. Typing skill and typing errors
– Positive correlation = as one variable increases, other also increases
ex. Standardized test scores and GPA
– Zero correlation = no relationship between the two variables
Plotting Correlational Data --Scatterplots
The more scattered the dots are on the graph, the closer the correlation coefficient gets to zero
Interpreting Correlations
• Correlation coefficient “r”
–Ranges from -1 to 1
–Indicates whether relationship is positive or negative
–Statistical significance (p-value) depends on size of relationship and size of sample
Magnitude of r Interpretation
Look at the absolute value of r to determine the magnitude of r
So what does r mean, anyway?
.00 to .40 weak relationship
.41 to .60 moderate relationship
.61 to .80 strong relationship
.81 or above very strong relationship.
More on CorrelationalDesigns
• Predicting from Multiple Variables
–Can compute several individual correlation coefficients to produce a correlation matrix (a table showing how the different variables correlate)
• Or can conduct a Multiple Regression Analysis - puts all the variables together for one coefficient (R)
–Yields coefficient R, which can be interpreted similarly to the simple correlation, r
–Also yields R2(coefficient of determination)
Coefficient of determination = Effect size estimate for correlational studies = how much of the result we can explain by the effect of all the other variables
ex. Reading Comprehension has many variables. R2 = How much of the reading comprehension can we explain through these other factors?
Imagine trying to dissect the concept “Reading Comprehension.” It is made up of several related factors, such as:
•Fluency
•Intrinsic Motivation
•Verbal IQ
•Working Memory Capacity
•Background Knowledge
If we sum up the portion of those components that uniquely overlaps with reading comprehension, we can explain a big part of reading comprehension. That is essentially what R2 in a multiple regression does.
So R2tells you, given all of the factors we’ve entered into the equation, how much of Reading Comprehension can be explained by those factors.
Other Uses of Correlational Data
• Structural Equation Modeling (aka. SEM, Path Analysis, Hierarchical, Stepwise) –maps out relationships among several variables
–Instead of lumping everything into a multiple regression, we can put them into a structural equation model; Allows researchers to see the “big picture”as well as relationships among individual variables
• Factor Analysis –how do individual variables or items in a measure combine to create “mega-
variables”
–E.g. Several items on a questionnaire might relate to your interest in a topic. Instead of treating each item as an individual variable, we combine them as one “factor”
•Path Modeling - shows how all of the factors correlated together AND how they work together to predict supportive computer use (looks like boxes with arrows going every which way showing correlational values between each combination)
Wednesday, February 27, 2008
Other Comparative Designs
Review of Terms:
Random Selection/Sampling vs. Random Assignment
Random Selection = How do I get my sample? All people in your population have an equal chance of being selected
Random Assignment = Once you have your sample, how do you assign them to a group?
Internal Validity vs. External Validity
Internal Validity = control of the experiment
External Validity = generalizability of your experiment
You want to have the correct balance between the two
Independent, Dependent, & Extraneous Variables
Independent Variable = What you are manipulating
Dependent Variable = depends on the independent
Extraneous Variables = the things that mess everything up
Between vs. Within Subjects Variables
Between subjects variable = looking for a difference between subjects - they don't get the same experience in both groups - but you need to make sure that both groups are as similar as possible to confirm that the only differences between groups is your independent variable
Within subjects variable = Every individual gets both experiences in the experiment - ex. Pre vs. Post - Within is more ideal because you know your groups will be consistent for both independent variables
Factorial Design = measures the impact of more than one independent variable at once
Benefit: you can see if there is an interaction between the different independent variables
(No more than three independent variables, otherwise it gets too complicated and you can't tell what variables are interacting with which variables very clearly)
Experimental vs. Quasi-Experimental Designs
"True" Experimental Designs = involve random assignment to conditions manipulated by experimenter
Quasi-Experimental Designs = involve comparisions of groups in pre-selected conditions or groups - I design the study before I collect the data
Causal Comparative Designs = are ex post facto quasi-experimental designs; They involve comparison of pre-selected conditions/groups after the fact
Time-Series Design
Measure dependent variable a lot over time
Experimental Designs, cont.
Single Subject Designs
-Like mini longitudinal experimental designs, on individuals or small groups of individuals
-Similar to pretest/posttest designs; examines DV before and after IV
-Used when it doesn’t make sense to pool effects across individuals - ex. When working with children with special needs, the specific behaviors and needs of one child are not the same as others' - But tracking that one child over time may help develop strategies to help that specific child - you're not trying to generalize your findings, you're just trying to help that one individual
--Special populations
--Focus on success of specific interventions with specific individuals
Possible Single-Subject Designs
A-B Design = baseline, followed by intervention
A = Baseline
B = Intervention
But what happens after the intervention is removed? Does behavior go back to baseline?
A-B-A Withdrawal Design = baseline, followed by intervention, concluded with baseline
When the intervention is removed, does behavior go back to baseline?
Ethical issue: is it OK to intervene but then leave subjects back at baseline behavior, especially if we know that the intervention is needed?
A-B-A-B Design = one step further; instead of leaving subjects back at baseline, present intervention again (more ethical)
Multiple Baselines Designs = alternative to ABAB design; used when it’s not ethical to leave the subject at the baseline condition and when measures on multiple Dependent Variables (DVs) are taken.
-Taking baselines for multiple behaviors at same time – whether it’s one behavior in multiple individuals, multiple behaviors in one individual, one type of behavior in one individual in multiple settings, etc.
-Difficult to use, because must ensure that multiple DVs aren’t related to one another
Issues to Consider: Internal Validity
Is Treatment Standardized?
-# Variables Changing Across Conditions ? - Is the behavior changing because of my intervention, or is there another explanation?
Condition Length?
-Degree and Speed of Change? - You want to show a stable trend. Do you have enough data points to see a stable trend in the experiment results? - The "magic" number is 7 measurements to see a stable trend
Experimenter Effects?
-Because this is a one-on-one experiment, the experimenter is likely to have an impact on the individual
Practical Significance of Results?
-Practical significance is looked at more than statistical significance because of the single-subject design. - ie. Did it help the student improve?
Single Subject Designs and External Validity
It is very difficult to generalize the results of these designs beyond the sample studied – WHY?
-Because it was only designed for one person for a specific purpose. It is not necessarily meant to be generalized.
Thus, it is VERY IMPORTANT for this type of research to be replicated before strong conclusions are drawn.
*We now have all the information we need for quiz 3 if we want to take it now.
Random Selection/Sampling vs. Random Assignment
Random Selection = How do I get my sample? All people in your population have an equal chance of being selected
Random Assignment = Once you have your sample, how do you assign them to a group?
Internal Validity vs. External Validity
Internal Validity = control of the experiment
External Validity = generalizability of your experiment
You want to have the correct balance between the two
Independent, Dependent, & Extraneous Variables
Independent Variable = What you are manipulating
Dependent Variable = depends on the independent
Extraneous Variables = the things that mess everything up
Between vs. Within Subjects Variables
Between subjects variable = looking for a difference between subjects - they don't get the same experience in both groups - but you need to make sure that both groups are as similar as possible to confirm that the only differences between groups is your independent variable
Within subjects variable = Every individual gets both experiences in the experiment - ex. Pre vs. Post - Within is more ideal because you know your groups will be consistent for both independent variables
Factorial Design = measures the impact of more than one independent variable at once
Benefit: you can see if there is an interaction between the different independent variables
(No more than three independent variables, otherwise it gets too complicated and you can't tell what variables are interacting with which variables very clearly)
Experimental vs. Quasi-Experimental Designs
"True" Experimental Designs = involve random assignment to conditions manipulated by experimenter
Quasi-Experimental Designs = involve comparisions of groups in pre-selected conditions or groups - I design the study before I collect the data
Causal Comparative Designs = are ex post facto quasi-experimental designs; They involve comparison of pre-selected conditions/groups after the fact
Time-Series Design
Measure dependent variable a lot over time
Experimental Designs, cont.
Single Subject Designs
-Like mini longitudinal experimental designs, on individuals or small groups of individuals
-Similar to pretest/posttest designs; examines DV before and after IV
-Used when it doesn’t make sense to pool effects across individuals - ex. When working with children with special needs, the specific behaviors and needs of one child are not the same as others' - But tracking that one child over time may help develop strategies to help that specific child - you're not trying to generalize your findings, you're just trying to help that one individual
--Special populations
--Focus on success of specific interventions with specific individuals
Possible Single-Subject Designs
A-B Design = baseline, followed by intervention
A = Baseline
B = Intervention
But what happens after the intervention is removed? Does behavior go back to baseline?
A-B-A Withdrawal Design = baseline, followed by intervention, concluded with baseline
When the intervention is removed, does behavior go back to baseline?
Ethical issue: is it OK to intervene but then leave subjects back at baseline behavior, especially if we know that the intervention is needed?
A-B-A-B Design = one step further; instead of leaving subjects back at baseline, present intervention again (more ethical)
Multiple Baselines Designs = alternative to ABAB design; used when it’s not ethical to leave the subject at the baseline condition and when measures on multiple Dependent Variables (DVs) are taken.
-Taking baselines for multiple behaviors at same time – whether it’s one behavior in multiple individuals, multiple behaviors in one individual, one type of behavior in one individual in multiple settings, etc.
-Difficult to use, because must ensure that multiple DVs aren’t related to one another
Issues to Consider: Internal Validity
Is Treatment Standardized?
-# Variables Changing Across Conditions ? - Is the behavior changing because of my intervention, or is there another explanation?
Condition Length?
-Degree and Speed of Change? - You want to show a stable trend. Do you have enough data points to see a stable trend in the experiment results? - The "magic" number is 7 measurements to see a stable trend
Experimenter Effects?
-Because this is a one-on-one experiment, the experimenter is likely to have an impact on the individual
Practical Significance of Results?
-Practical significance is looked at more than statistical significance because of the single-subject design. - ie. Did it help the student improve?
Single Subject Designs and External Validity
It is very difficult to generalize the results of these designs beyond the sample studied – WHY?
-Because it was only designed for one person for a specific purpose. It is not necessarily meant to be generalized.
Thus, it is VERY IMPORTANT for this type of research to be replicated before strong conclusions are drawn.
*We now have all the information we need for quiz 3 if we want to take it now.
Wednesday, February 20, 2008
Experimental Design
Experimental Designs
What makes experimental designs so great? CONTROL!!!
• In experimental designs, researchers have control over:
–Independent Variables (what is manipulated; whatever you are comparing, such as traiditonal vs. online technology)
–Dependent Variables (what is being measured, how are they operational? How is it being measured?)
–Extraneous Variables (all of the things I can't control; things that have impact on my dependent variable, such as dog color blindness)
Internal Validity and Control
• History (some uncontrolled event, such as a fire alarm)
• Selection (how do you select your subject)
• Maturation (individuals change over time)
• Testing (does just giving a test somehow influence what the subjects think about the Independent Variable)
• Instrumentation (something could be wrong with the test)
• Treatment Replications
• Attrition (mortality, or losing subjects during the experiment)
• Regression to the Mean
• Diffusion of Treatment - (does the Independent Variable from one group share info to another group) Does your independent variable in one group bleed into another group? One group tells the other group what happens in the experiment, or they talk about it and share opinions and thoughts that skew the results
• Experimenter Effects - (is the experimenter overly nice or overly mean to subjects; or if your tester is a cute girl and your subjects are 13 yr old boys, then they subjects do whatever the tester wants you to do)
How the behavior, personality, looks of the experimenter affect how the subjects react to and participate in the experiment
• Subject Effects - in self-report data, the attitude of the subject or the reason they are there can affect how they participate in the experiment - why did they participate? Does it take a certain kind of person to actually participate? Does this affect your results?
•Social Desirability Bias - the participant alters answers so they pain the picture of themselves that they want
External Validity and Control
• So more control ≈ higher levels of internal validity...
–Note, this is what allows us to make cause/effect conclusions about experimental data
• But what happens to external validity? - External validity lessens the more you control because more control means you are getting farther and farther away from the real world
*It's like a see-saw. You have to find the right balance between external validity (less control) and internal validity (more control)
Important Terms
• Randomization
–Random Selection - How do you get the participants for your study in the first place?
–vs. Random Assignment - What do you do with the participants once you've already gotten them? How do you assign them within the experiment?
• Within Subjects - Every participant gets every level of the independent variable - ex. in a pretest/post test design, every participant takes both the pre and post tests - this is always going to be better if you can make it work because everyone participates in everything and there is no chance that different groups are going to have different results because of extraneous variables
vs. Between Subjects Variables and Designs - One group gets level A of the independent variable, but a different group gets level B of the independent variable - ex. experimental vs. control group designs OR men vs. women design - You do run the risk that there is some extraneous variable that exists in one group, but not the other; there might be something fundamentally different between groups that affects your results
• Controlling for Confounds
–Holding variables constant
–Building variables into design
–Matching
Pretest/Posttest Designs
• Single Group Posttest-Only Design
– Expose 1 group to the IV and then measure the DV (posttest) once
– Example: I decide to test whether using puppets to “read” books with young children to help them learn different sound combinations. – What’s the problem? - You don't have anything to compare to, no baseline to start at. There is no way you can say the puppets affected anything if you don't know what the results of the test were before using the puppets. To make it even better, you should have a control group that takes a pre and post test without the puppets to prove it was the puppets that made the difference in test scores and not just the fact that they took the test twice.
Single Group Pretest Posttest
• For a single group, give them a pretest (DV), then the IV, then a posttest (DV again)
• Example, in the “puppet experiment,”I give students a pre-reading PA quiz, then read to them with the puppet, then give them a post-reading PA quiz
Nonequivalent Groups Pretest Posttest: Quasi-Experimental
• Involves 2 groups (experimental and control); both get pretest (DV), then only the exper. Gp. is exposed to the IV, and both gps. get the posttest (DV)
*Pre-existing groups
• Between Subjects Designs= each group of subjects receives a different level of the IV
–Advantage: often more practical that within subjects designs
–Disadvantage: are differences due to groups? Or to the IV?
–Use of Matching
And what you’ve all been waiting for...“TRUE”Experimental Designs
• Randomized Groups PrettestPosttest Design
– Individuals are randomly assigned to either exper. Or control grp.; both groups receive the pretest (DV), then only the exper.
Gp. Is exposed to the IV, and then both gps. Receive the posttest (DV)
– Random assignment gives control over group differences, and pretest allows for a baseline measure in both groups
Experimental Designs with more than 1 Independent Variable (IV)
• Factorial Designs= measure the impact of 2 or more IVs on the Dependent Variable (DV)
(ex. test whether puppets help develop reading skills, AND whether it helps English Language Learners more)
–Advantages
• Allow researchers to determine whether effects are consistent across subject characteristics
• Allow researchers to assess interactions
–Disadvantages
• Can get messy if there are too many IVs because every IV you add changes the "personality" of the experiment - NEVER HAVE MORE THAN 3 IVs, otherwise it gets too messy
Sample Between Subjects Factorial Design
Analyses of Exp. Designs (t and F)
• T-test (t) --used to assess the impact of two levels of one IV on a single DV
• ANOVA (F) –used to assess impact of one or more IVs on a single DV
• MANOVA –used to assess impact of one or more IVs on multiple DVs
• ANCOVA –used to assess impact of one or more IVs on a single DV after removing effects of a variable that might be correlated to DV (e.g., age, gender, aptitude, achievement, etc.)
• MANCOVA –used to assess impact of one or more IVs on multiple DVs after removing effects of a variable that might be correlated to DV (e.g., age, gender, aptitude, achievement, etc.)
What makes experimental designs so great? CONTROL!!!
• In experimental designs, researchers have control over:
–Independent Variables (what is manipulated; whatever you are comparing, such as traiditonal vs. online technology)
–Dependent Variables (what is being measured, how are they operational? How is it being measured?)
–Extraneous Variables (all of the things I can't control; things that have impact on my dependent variable, such as dog color blindness)
Internal Validity and Control
• History (some uncontrolled event, such as a fire alarm)
• Selection (how do you select your subject)
• Maturation (individuals change over time)
• Testing (does just giving a test somehow influence what the subjects think about the Independent Variable)
• Instrumentation (something could be wrong with the test)
• Treatment Replications
• Attrition (mortality, or losing subjects during the experiment)
• Regression to the Mean
• Diffusion of Treatment - (does the Independent Variable from one group share info to another group) Does your independent variable in one group bleed into another group? One group tells the other group what happens in the experiment, or they talk about it and share opinions and thoughts that skew the results
• Experimenter Effects - (is the experimenter overly nice or overly mean to subjects; or if your tester is a cute girl and your subjects are 13 yr old boys, then they subjects do whatever the tester wants you to do)
How the behavior, personality, looks of the experimenter affect how the subjects react to and participate in the experiment
• Subject Effects - in self-report data, the attitude of the subject or the reason they are there can affect how they participate in the experiment - why did they participate? Does it take a certain kind of person to actually participate? Does this affect your results?
•Social Desirability Bias - the participant alters answers so they pain the picture of themselves that they want
External Validity and Control
• So more control ≈ higher levels of internal validity...
–Note, this is what allows us to make cause/effect conclusions about experimental data
• But what happens to external validity? - External validity lessens the more you control because more control means you are getting farther and farther away from the real world
*It's like a see-saw. You have to find the right balance between external validity (less control) and internal validity (more control)
Important Terms
• Randomization
–Random Selection - How do you get the participants for your study in the first place?
–vs. Random Assignment - What do you do with the participants once you've already gotten them? How do you assign them within the experiment?
• Within Subjects - Every participant gets every level of the independent variable - ex. in a pretest/post test design, every participant takes both the pre and post tests - this is always going to be better if you can make it work because everyone participates in everything and there is no chance that different groups are going to have different results because of extraneous variables
vs. Between Subjects Variables and Designs - One group gets level A of the independent variable, but a different group gets level B of the independent variable - ex. experimental vs. control group designs OR men vs. women design - You do run the risk that there is some extraneous variable that exists in one group, but not the other; there might be something fundamentally different between groups that affects your results
• Controlling for Confounds
–Holding variables constant
–Building variables into design
–Matching
Pretest/Posttest Designs
• Single Group Posttest-Only Design
– Expose 1 group to the IV and then measure the DV (posttest) once
– Example: I decide to test whether using puppets to “read” books with young children to help them learn different sound combinations. – What’s the problem? - You don't have anything to compare to, no baseline to start at. There is no way you can say the puppets affected anything if you don't know what the results of the test were before using the puppets. To make it even better, you should have a control group that takes a pre and post test without the puppets to prove it was the puppets that made the difference in test scores and not just the fact that they took the test twice.
Single Group Pretest Posttest
• For a single group, give them a pretest (DV), then the IV, then a posttest (DV again)
• Example, in the “puppet experiment,”I give students a pre-reading PA quiz, then read to them with the puppet, then give them a post-reading PA quiz
Nonequivalent Groups Pretest Posttest: Quasi-Experimental
• Involves 2 groups (experimental and control); both get pretest (DV), then only the exper. Gp. is exposed to the IV, and both gps. get the posttest (DV)
*Pre-existing groups
• Between Subjects Designs= each group of subjects receives a different level of the IV
–Advantage: often more practical that within subjects designs
–Disadvantage: are differences due to groups? Or to the IV?
–Use of Matching
And what you’ve all been waiting for...“TRUE”Experimental Designs
• Randomized Groups PrettestPosttest Design
– Individuals are randomly assigned to either exper. Or control grp.; both groups receive the pretest (DV), then only the exper.
Gp. Is exposed to the IV, and then both gps. Receive the posttest (DV)
– Random assignment gives control over group differences, and pretest allows for a baseline measure in both groups
Experimental Designs with more than 1 Independent Variable (IV)
• Factorial Designs= measure the impact of 2 or more IVs on the Dependent Variable (DV)
(ex. test whether puppets help develop reading skills, AND whether it helps English Language Learners more)
–Advantages
• Allow researchers to determine whether effects are consistent across subject characteristics
• Allow researchers to assess interactions
–Disadvantages
• Can get messy if there are too many IVs because every IV you add changes the "personality" of the experiment - NEVER HAVE MORE THAN 3 IVs, otherwise it gets too messy
Sample Between Subjects Factorial Design
Analyses of Exp. Designs (t and F)
• T-test (t) --used to assess the impact of two levels of one IV on a single DV
• ANOVA (F) –used to assess impact of one or more IVs on a single DV
• MANOVA –used to assess impact of one or more IVs on multiple DVs
• ANCOVA –used to assess impact of one or more IVs on a single DV after removing effects of a variable that might be correlated to DV (e.g., age, gender, aptitude, achievement, etc.)
• MANCOVA –used to assess impact of one or more IVs on multiple DVs after removing effects of a variable that might be correlated to DV (e.g., age, gender, aptitude, achievement, etc.)
Subscribe to:
Posts (Atom)
