Week 08 - Program Evaluation Group Designs and Methods
A presentation at Heritage University @ CBC Week 08 in in Pasco, WA 99301, USA by Jacob Campbell
SOWK 460 Spring 2020 Planning: Class 08
Location: CBC Campus - SWL 208
Time: Mondays from 5:30-8:15
Week 08: 3/2/20
Topic and Content Area: Group Designs and Methods
Reading Assignment: Kapp and Anderson chapter 10
Assignments Due:
- A-02 Reading Quiz 03/02/20
- A-04a: Weekly Journal 04 03/08/20
Other Important Information: N/A
Agenda
- Checking in for the group work plan
- Key components for evaluation methods
- Threats to validity
- Types of group designs
Group Work Plan: Check-in
Follow up with how people are doing, see if there is questions around group work plan
Further Topics: Coming later this semester...
Later chapters we will be talking about…
- Qualitative designs and applications
- Consumer satisfaction
Read ahead if these are models you plan on following.
Methods for Evaluation
Any method for evaluation needs to include:
- Sample selection
- Data collection
- Analysis
- Reporting
Sampling
What kinds of sampling methods. [Whole Class Activity] Discussion regarding what types of sampling methods planning on using for groups.
Validity: How to Address Internal Validity
Classic Threats to Internal Validity (1 of 3)
- History: Events that happen outside of evaluation or contextually during the evaluation that effect the event. (Corona Virus, people being laid off… etc)
- Maturation and the passage of time: general growth that happens on it’s own. Especially true for children, but can be true for anybody.
- Testing: Pre-test effects the outcome of the post-test.
- Instrumentation: Change in the tools used to collect data during time of data collection (e.g. changing questions on pre-test/post-test)
- Statistical regression: When there are significant changes (improvement / deterioration) that is based on their extreme behavior or position prior. (Think nowhere to go but up/down)
Classic Threats to Internal Validity (2 of 3)
- Selection bias: Problems related to selection of participants (more random and larger sample better)
- Experimental mortality and attrition: Not completing the intervention or process.
- Ambiguity about the direction of causal influences: Direction of impacts and influencing conditions not clear. (Depressed causes lack of sleep or lack of sleep causes depression)
- Design contamination: change behaviors or actions because of being evaluated.
- Diffusion or imitation of treatments: looking for unique qualities which might be used by other professions (many professionals use strengths-based practice… not only ones that work in a “strengths-based program)
Classic Threats to Internal Validity (3 of 3)
Interaction Effects: Threats to internal validity interact with each other.
Components of Design: What should be included in general
- Defining and describing the intervention or program elements to be evaluated
- Establishing the time order of the independent variable
- Manipulating the independent variable
- Establishing the relationship between the independent and dependent variables
- Controlling for rival hypotheses
- Using at least one control group
- Assigning the person who are subjects in a random manner
Pre-Test / Post-Test: Design Methods Activity
Work in small groups to discuss potential evaluation or aspect of your group you could test by pre-test / post-test (even if you aren’t going to do this or “wouldn’t be able to” and create a simple example pre/post test
Types of Group Designs
- Case study approach
- One group post-test design
- One-group pre-test and post-test
- Post-test only with nonequivalent groups
- Experimental design
- Matched comparison groups
Types of Group Designs: Planning in your groups
- Are you going to use a group design for your program evaluation or what method will you be using?
- What type of group design method are you going to use?
- What are the challenges that you think you will encounter
Case Study Approach
Description: The group in which an intervention has been introduced is the focus of the study that will chronicle the progress and process of the gorup describing the changes (or lack of change) after the introduction of the intervention
Strengths:
- Detailed exploration
- Ability to understand complexity
- Rich narrative
Limitations:
- No comparison group
- Case may not have same qualities as sample
- Difficult to weigh elements of narrative
One Group Post-Test Only Design
Description: This design invovles the implementation of an intervention with a group of people whom that intervention wth a group of people for whom that intervention was designed, and then the adminstration of a simple test or other measurement to ascertain the results of that intervention.
This can be described as an A-B design, with A being the pre-intervention status and B representing the post -intervention status
Strengths:
- Design is simple and practical
- Intervention is intended to increase positive outcome
- Intervention delivered and measured
Limitations: There are concerns about the validity of the findings, the validity of the measurement instrument, and consequently, the inability to present the effectiveness of the intervention with a high degree of confidence
One-Group Pre-Test & Post-Test Design
Description: A target group is assessed prior to the intervention and after the intervention they are assessed again using the same measurement tool. It is designed to measure the change that was presumably caused by the intervention.
Strengths:
- Can show comparison between before and after the intervention
- Progress is likely attributable in part to the intervention
Limitations:
- Threats to internal validity
- Historical considerations
- Maturation
- Testing and instrumentation
Post-Test Only Design with Nonequivalent Groups
Description: The post-test only aspect of this design means that the impact of the intervention is only delivered after the intervention. The experience annd success of othe clients also served by the agency, who have not recieved the intervention is also measured.
Strengths: Simplicity of the post-test only design combined with a simple, accessible method for comparison
Limitations: Concerns abut the ability to compare nonequivalent groups and the lac k of randomization mean that strong questions about the validity persist.
Experimental Design
Description: The persons to be studied are randomly assigned to two groups. one group is administered the intervention and the other group is not administered the intervention. The condition and status of both groups (e.g. experemental group and control) are measured.
Strengths:
- Allows ability to control threats to internal validity
- Presents a higher degree of confidence in the results of the evaluation and effectiveness of the intervention
Limitations:
- The cost and effort to create this type of experimental design is higher than others
- Ethical concerns association with withholding treatment
Experimental Design: Comparison Groups
Description: Control group not selected by randomly withholding the intervention
Strengths:
- May not present the dilemmas posed by an experimental design
- Is more compatible with ongoing service delivery
- Offers some degree of rigor as it attempts to answer the questions as to the effect of experiencing the benefits of the information
Limitations: Potentially challenging to identify comparison groups
Using different groups and impact of the intervention is a common technique for many program evaluations. This session looks at the list of group program evaluations (case study approach, one group post-test design, one-group pre-test and post-test, post-test only with nonequivalent groups, experimental design, matched comparison groups) along with their strengths and weaknesses. We also discuss validity and it’s relationship to methods. The agenda for class is as follows:
- Checking in for the group work plan
- Key components for evaluation methods
- Threats to validity
- Types of group designs
