Final Paper:
Bring a hard copy of the article you reviewed.
Bring a hard copy of your original review.
Bring a hard copy of your new revised review.
Just paper clip them together.
More than 5 pages, less than 10.
Action Research Poster:
Come up with a question you have based on the article you read.
Answer a question that might help you in your teaching practice.
You do this type of thing in class anyway, this is just a little more formal.
Use the page given in class to plan your action research.
Create a poster displaying your action research project:
Make it on a tri-fold
You do not actually have to do the project, just plan it and explain what you think will happen.
See the sample page given in class.
Wednesday, April 16, 2008
Program Evaluation
Program Evaluation
Program Evaluation can take components of any or multiple of the designs discussed earlier this semester into one giant research project.
ACCOUNTABILITY
Assessment
Program Evaluation
Outcomes Assessment
Evidence
Formative
Summative
Both Program and Outcome Accountability are important
• You could have a program that does everything it says it’s going to do, but it doesn’t produce any outcomes
– D.A.R.E. is an example of this problem
• Or you could have a program that doesn’t do what it says it’s going to do, but there are still changes in outcomes
– So then how do you know what the changes in outcomes are due to –the program, or some extraneous variable?
Structures of the Program –how is it organized, what infrastructures are in place to support it?
Domains, Standards/Goals and Competencies of the Program
Process of the Program
PROGRAM ELEMENTS
• STRUCTURE
– Definition of the program
• "the program is a developmental educational program...”
• “This program is an integral part...”
• The program includes...”– Rationale for the program
• Importance of the program
• Reasons why students need what the program offers
– Assumptions of the program
• Identify and briefly describe the premise upon which a program rests
– Ethical and legal practice– Roles
PROGRAM STRUCTURAL ELEMENTS
Resources
–Human
–Financial
–Political
• Program Components
–Training
–Student Interventions
–Support for System
DOMAINS
Domains identify conceptual aspects of focus or priorities for the program.
The identification of domains:
1. Provides a description of areas of importance
2. Provides a structure to organize goals and competencies
3. All assessments should be related to the identified domains of the program
STANDARDS/GOALS
Standards define specific desired student results to be achieved by a certain time,
i.e. by the time the student leaves school or by the time that the student finishes the grade level.
They are stated in terms of knowledge, attitudes, skills that have been identified as necessary to the education of the student.
They are written in global terms for each domain.
COMPETENCIES and INDICATORS
Competencies are descriptions of student behaviors (knowledge, skills, attitudes) that are linked to stated goals for their education. They are based on the assumption that the behaviors can be transferred from a learning situation to a real-life situation.
Competencies are:
1. organized into goal content areas
2. arranged developmentally
3. measurable
Components of Program Evaluation
• Process vs. Outcome Evaluation
– Formative = evaluation of process
– Summative = evaluation of final product
• Stakeholders --anyone with a vested interest in the program
– Designers, evaluators, teachers, students, etc.
Program Evaluation: The Assessment Loop
• Components of Assessment
– Measurement
– Evaluation
• What kinds of evidence can be used in Program Evaluation assessments?
– Portfolios, performances, outside judges, observations, local tests, standardized tests, self- assessments, interviews, focus groups, student work (In qualitative research you want to have at least 3 different kinds of evidence to triangulate your results.)
– Nature of evidence depends on nature of question
SCHOOL PROGRAM EVALUATION
Program Evaluation is a cyclical process, in which the outcome of each cycle informs the content and objectives of the following cycle.
Note that both Process (e.g., Action Plan) and Product (e.g. Assessments) are important components of every cycle
Steps in Each Cycle of Program Evaluation:
•Identify Program Goals
•Conduct Needs Assessment
•Set Program Priorities
•Identify Target Groups
•Write Objectives for Activities
•Evaluate the Degree to Which Objectives Were Met
•Re-Set Program Priorities with the Established Program Goals
Program Evaluation can take components of any or multiple of the designs discussed earlier this semester into one giant research project.
ACCOUNTABILITY
Assessment
Program Evaluation
Outcomes Assessment
Evidence
Formative
Summative
Both Program and Outcome Accountability are important
• You could have a program that does everything it says it’s going to do, but it doesn’t produce any outcomes
– D.A.R.E. is an example of this problem
• Or you could have a program that doesn’t do what it says it’s going to do, but there are still changes in outcomes
– So then how do you know what the changes in outcomes are due to –the program, or some extraneous variable?
Structures of the Program –how is it organized, what infrastructures are in place to support it?
Domains, Standards/Goals and Competencies of the Program
Process of the Program
PROGRAM ELEMENTS
• STRUCTURE
– Definition of the program
• "the program is a developmental educational program...”
• “This program is an integral part...”
• The program includes...”– Rationale for the program
• Importance of the program
• Reasons why students need what the program offers
– Assumptions of the program
• Identify and briefly describe the premise upon which a program rests
– Ethical and legal practice– Roles
PROGRAM STRUCTURAL ELEMENTS
Resources
–Human
–Financial
–Political
• Program Components
–Training
–Student Interventions
–Support for System
DOMAINS
Domains identify conceptual aspects of focus or priorities for the program.
The identification of domains:
1. Provides a description of areas of importance
2. Provides a structure to organize goals and competencies
3. All assessments should be related to the identified domains of the program
STANDARDS/GOALS
Standards define specific desired student results to be achieved by a certain time,
i.e. by the time the student leaves school or by the time that the student finishes the grade level.
They are stated in terms of knowledge, attitudes, skills that have been identified as necessary to the education of the student.
They are written in global terms for each domain.
COMPETENCIES and INDICATORS
Competencies are descriptions of student behaviors (knowledge, skills, attitudes) that are linked to stated goals for their education. They are based on the assumption that the behaviors can be transferred from a learning situation to a real-life situation.
Competencies are:
1. organized into goal content areas
2. arranged developmentally
3. measurable
Components of Program Evaluation
• Process vs. Outcome Evaluation
– Formative = evaluation of process
– Summative = evaluation of final product
• Stakeholders --anyone with a vested interest in the program
– Designers, evaluators, teachers, students, etc.
Program Evaluation: The Assessment Loop
• Components of Assessment
– Measurement
– Evaluation
• What kinds of evidence can be used in Program Evaluation assessments?
– Portfolios, performances, outside judges, observations, local tests, standardized tests, self- assessments, interviews, focus groups, student work (In qualitative research you want to have at least 3 different kinds of evidence to triangulate your results.)
– Nature of evidence depends on nature of question
SCHOOL PROGRAM EVALUATION
Program Evaluation is a cyclical process, in which the outcome of each cycle informs the content and objectives of the following cycle.
Note that both Process (e.g., Action Plan) and Product (e.g. Assessments) are important components of every cycle
Steps in Each Cycle of Program Evaluation:
•Identify Program Goals
•Conduct Needs Assessment
•Set Program Priorities
•Identify Target Groups
•Write Objectives for Activities
•Evaluate the Degree to Which Objectives Were Met
•Re-Set Program Priorities with the Established Program Goals
Wednesday, April 2, 2008
Observations and Interviews
Observations and Interviews
Interviews
•Who is your interviewee?
•Design the interview protocol.
•Where will the interview take place?
•Wording of the Questions - No leading or biasing questions; open-ended; clear & understandable, unambiguous, not double-barrelled, no two-part questions, no double negatives; ask in a respectful way
•Memorize the questions.
•Obtain consent before starting.
•Demonstrate Respect. - For individual, culture, community, etc.
•Develop rapport
•Stick to protocol as much as possible
•Record comments faithfully.
•Use recorder or videotape if possible.
Interview Types
Structured or Semi-Structured
“Verbal questionnaires”More Formal
Specific protocol prepared in advance
Designed to elicit specific information
Informal Interviews
More informal, more like a conversationStill involve planning
Move from non-threatening to threatening questionsDesigned to determine more general views of respondent
Types of Interview Questions
Demographic Questions
Knowledge Questions
Experience or Behavior Questions
Opinion/Value/Feeling Questions
Sensory Questions
Interview Questioning Tips
Ask questions in different ways
Ask for clarification
Vary control of flow of information
Avoid leading questions
Ask one question at a time
Don’t interrupt
Observation Process
1. What is the research question?
2. Where is the observation taking place?.
3. Who, what, when, and how long?
4. What is the role of the observer?
4. What is the observer’s role?
•Participant Observation
Complete Participant - posing as part of the group being observed - participates just like every other person in the group
Participant-as-Observer - participates in all the group's activities except for a few differences (ie. participates in a class but doesn't turn in the assignments)
Non-Participant Observation -
Complete Observer - the observer is not involved in the group activities and is a complete outsider
Observer-as-Participant - everyone in the group knows that the person is not part of the group, but they participate to some degree - more of an outsider
Minimize Bias
Collecting Observation/Interview Data
•Note Taking Methods
Field Jottings - quick notes
Field Diary - more in-depth information
Field Log - a chronology of events that occur during the observational process
•Descriptive vs. Reflective Notes
Describe subjects, settings, events, activities, observer behaviors
Reflect on analysis, method, ethics, frame of mind, points of clarification
•Data Coding Schemes
Sample Field Notes Form
Research Question:
Date/Time:
Setting:
Descriptive Notes:
Reflective Notes:
Sample Interview Protocol
Research Question:
Time of Interview:Date:
Setting:Interviewer:
Interviewee:
Position of Interviewee (if relevant):
(Briefly describe project to interviewee and obtain consent.)
Sample Interview Questions:
1. What has been your role in the incident?
Interviewee’s Response:
Interviewer’s Reflective Comments:
2. What has happened since the event that you have been involved in?
Interviewee’s Response:
Interviewer’s Reflective Comments:
3. What has been the impact on the community of this incident?
Interviewee’s Response:
Interviewer’s Reflective Comments:
4. To whom should we talk to find out more information about the incident?
Interviewee’s Response:
Interviewer’s Reflective Comments:
(Thank individual for participating in this interview. Assure him/her of confidentiality of responses and potential future interviews.)
Quiz 7 –last question
As a class, come up with a focus topic for a one-question interview related to the use of technology in education.
Availability of technology in the workplace
Next, write the question. Make sure it is clear, unbiased, uses respectful terminology, etc., and that it gets at your focus topic.
How does the availability of technology influence your productivity?
For your answer to the last question on quiz 7, you should interview one person, and record both descriptive (transcript) and reflective notes for this interview.
Interviews
•Who is your interviewee?
•Design the interview protocol.
•Where will the interview take place?
•Wording of the Questions - No leading or biasing questions; open-ended; clear & understandable, unambiguous, not double-barrelled, no two-part questions, no double negatives; ask in a respectful way
•Memorize the questions.
•Obtain consent before starting.
•Demonstrate Respect. - For individual, culture, community, etc.
•Develop rapport
•Stick to protocol as much as possible
•Record comments faithfully.
•Use recorder or videotape if possible.
Interview Types
Structured or Semi-Structured
“Verbal questionnaires”More Formal
Specific protocol prepared in advance
Designed to elicit specific information
Informal Interviews
More informal, more like a conversationStill involve planning
Move from non-threatening to threatening questionsDesigned to determine more general views of respondent
Types of Interview Questions
Demographic Questions
Knowledge Questions
Experience or Behavior Questions
Opinion/Value/Feeling Questions
Sensory Questions
Interview Questioning Tips
Ask questions in different ways
Ask for clarification
Vary control of flow of information
Avoid leading questions
Ask one question at a time
Don’t interrupt
Observation Process
1. What is the research question?
2. Where is the observation taking place?.
3. Who, what, when, and how long?
4. What is the role of the observer?
4. What is the observer’s role?
•Participant Observation
Complete Participant - posing as part of the group being observed - participates just like every other person in the group
Participant-as-Observer - participates in all the group's activities except for a few differences (ie. participates in a class but doesn't turn in the assignments)
Non-Participant Observation -
Complete Observer - the observer is not involved in the group activities and is a complete outsider
Observer-as-Participant - everyone in the group knows that the person is not part of the group, but they participate to some degree - more of an outsider
Minimize Bias
Collecting Observation/Interview Data
•Note Taking Methods
Field Jottings - quick notes
Field Diary - more in-depth information
Field Log - a chronology of events that occur during the observational process
•Descriptive vs. Reflective Notes
Describe subjects, settings, events, activities, observer behaviors
Reflect on analysis, method, ethics, frame of mind, points of clarification
•Data Coding Schemes
Sample Field Notes Form
Research Question:
Date/Time:
Setting:
Descriptive Notes:
Reflective Notes:
Sample Interview Protocol
Research Question:
Time of Interview:Date:
Setting:Interviewer:
Interviewee:
Position of Interviewee (if relevant):
(Briefly describe project to interviewee and obtain consent.)
Sample Interview Questions:
1. What has been your role in the incident?
Interviewee’s Response:
Interviewer’s Reflective Comments:
2. What has happened since the event that you have been involved in?
Interviewee’s Response:
Interviewer’s Reflective Comments:
3. What has been the impact on the community of this incident?
Interviewee’s Response:
Interviewer’s Reflective Comments:
4. To whom should we talk to find out more information about the incident?
Interviewee’s Response:
Interviewer’s Reflective Comments:
(Thank individual for participating in this interview. Assure him/her of confidentiality of responses and potential future interviews.)
Quiz 7 –last question
As a class, come up with a focus topic for a one-question interview related to the use of technology in education.
Availability of technology in the workplace
Next, write the question. Make sure it is clear, unbiased, uses respectful terminology, etc., and that it gets at your focus topic.
How does the availability of technology influence your productivity?
For your answer to the last question on quiz 7, you should interview one person, and record both descriptive (transcript) and reflective notes for this interview.
Subscribe to:
Posts (Atom)
