Spring 2025 SOWK 460w Week 07 - Building the Foundation of a Program Evaluation

Spring 2025 SOWK 460w Week 07 - Building the Foundation of a Program Evaluation

title: Spring 2025 SOWK 460w Week 07 - Building the Foundation of a Program Evaluation date: 2025-03-03 15:06:51 location: Heritage University tags:

• Heritage University
• BASW Program
• SOWK 460w presentation_video: > “” description: >

Students will be submitting their logic models this week and should be considering their agency as a whole. The reading from Royse (2023) looks at formative and process evaluations, and we will start class by looking at some of the considerations we should have for process evaluations. During week seven, we will be considering your research question so that you can narrow down exactly what part of the program you are evaluating is being examined. Student will complete their logic models this week. During class, we will discuss how to use our logic models to develop a program description. There will also be a discussion about presentations, and students will be able to complete midterm feedback.

• Developing a research question
• Presentation planning
• Developing a program description
• Midterm feedback

Agenda

• Developing a research question
• Presentation planning
• Developing a program description
• Midterm feedback

Foci for Process Evaluations

I want to start our conversation off today talking about process evaluations and what the seven foci are for them.

• Context (political, economic & other influences on the program)
• Recruitment (methods used, successes and failures)
• Reach (% of target population participating, sufficient resources?)
• Dose Delivered (how much intervention typically delivered?)
• Dose Received (clients’ engagement & compliance with treatment)
• Fidelity (was the intervention applied as designed?)
• Implementation (a composite program implementation score can be created by averaging reach, dose delivered, dose received, and fidelity)

Developing a Research Question (1 of 7) Topic Introduction

I think that most of you already have a general idea what you want to do for your project. I want to make sure we are starting from the place of what do you want to know.

We are going to go through a process to help identify this.

Developing a Research Question (2 of 7) Frame In Steps of Evaluation

Before we start to look at what method will be used and how you will collect the data, analyze it, or use that information, please spend some time in your groups considering your research questions.

This diagram shows the first five steps in a program evaluation, which we discussed in week two of this class.

1. Identify Evaluation Question
2. Conceptualize Question
3. Operationalize Question

This should all happen before we can really pick a method. Today, my goal is for you to define your overarching research question. In your groups, you will have to come up with 3 to 5 subquestions that you are trying to determine with the components of your research project.

Developing a Research Question (3 of 7) Step 1

Step one is helping draw out your research question and understanding some potential questions you might be trying to answer. The following are some activities you could consider:

• What is your topic? i.e., burnout, how transitions happen, gaps in services, etc.
• What is the context or location of your research? i.e., DSHS, Domestic Violence, etc.
• What do you want to achieve? i.e., to discover, to describe, to change, to explore, to explain, etc.
• What is the nature of your question? i.e., a what, where, how, when, or why question?
• Are there potential relationships you want to explore? i.e., impacts, increases, decreases, relationships, correlations, causes, etc.

[Small Group Activity] Answer each of these questions using one or two word answers.

(O’Leary, 2018)

Developing a Research Question (4 of 7) Step 2 Examples

I’m drawing this activity from a neat little book, a part of little quick fix series and it is about research questions.

The second step is about starting with the nature of the question (e.g., who, what, why, where, how, when) put the pieces together.

The following are examples the author used. You will see that all three examples are about the same topic and context, but the other elements are all different.

Topic: pornography Context: high school Goal: to explore prevalence of watching porn Nature of your question: how much/ often Relationship: N/A

Question: How prevalent is watching pornography among high school students?

Topic: pornography Context: high school Goal: to understand how porn changes sexual expectations Nature of your question: how Relationship: watching porn and expectations

Question: How does watching pornography change sexual expectations among high school students?

Topic: pornography Context: high school Goal: to understand education programmes in high school that address porn viewing Nature of your question: what Relationship: N/A.

Question: What education programmes have been shown to have a positive impact on the pornography viewing habits of high school students?

(O’Leary, 2018, pp. 72-73)

Developing a Research Question (5 of 7) Step 2

You will write it out as follows:

Potential Question 1
Topic:
Context:
Goal:
Nature of Question:
Relationships:
Question:

Potential Question 2
Topic:
Context:
Goal:
Nature of Question:
Relationships:
Question:

etc.

[Small Group Activity] Work in your team to start to develop a few potential questions

(O’Leary, 2018)

Developing a Research Question (6 of 7) Step 3

Now we need to actually write out a question. Include the details identified information to make a well defined question.

(O’Leary, 2018)

Developing a Research Question (7 of 7) Step 4

1. Rewrite your question and circle terms that could be ambiguous.
2. Go through and clarify those terms.
3. Then, redraft your question, bringing more clarity and description

Presentation Planning

I wanted to spend some time tonight talking about our presentations that will take place on May 5th.

Discuss location

• Who do we invite
• What do we call it
• Do we have food

Show example of flyer from last year:

2024-04-29, Student Program Evaluation Symposium.pages

Evaluation Resources

To get us warmed up and thinking about logic models again, the following link (also in MyHeritage)

Select: Interactive logic model puzzles complete one as a group

• Seniors fall prevention
• Youth smoking cessation
• Teen parenting
• Community crime prevention

Evaluation Resources

Using Logic Models to Identify Key Information Needs

So, we talked about logic models last week and some of the format. Yours are due this week. How many of you have yours started or particially finish?

What do You See in a Logic Model

If you have your logic model or one of the examples shared or that you can find. Review it an ask “shat information do we need to evaluate the program’s processes and outcomes effectively?”

Process: A process is the implementation of a key program component. In the Kellogg logic model, processes are described as activities and quantified as outputs.

Outcomes: Outcomes are results that occur that are directly linked to program processes.

Residential Treatment Program

[Whole Class Activity] Discuss this as an example logical model. Review how it would be created, the details within it, and how it helps in determining the evaluative process

Resources/Inputs

• Funding
• Staff
• Clients
• Facilities

Activities

• Group therapy
• Family therapy
• Residential care

Outputs

• 4 weekly group therapy sessions
• Weekly family therapy sessions for 30 families
• 30 youth in residential care

Outcomes

• Youth maintain participation in employment or education program
• Youth maintain legal lifestyle

Impact

• Families functioning productively
• Families satisfied with functioning

Residential Treatment Program Components

Conceptualization for program and component and the indicators that we can track.

• Weekly group therapy sessions # of sessions
• Weekly family therapy sessions # of sessions

In our logic model these are conceptulizied as the outputs

Residential Treatment Program

Adding to the conceptualization

We could add the following two itmes. They are some of the impacts that… have defined them as more specific indicators.

• Youth maintain participation in employment or education program – % of youth employed or in education program 3 months, 6 months, and 12 months out
• Youth maintain legal lifestyle – % of youth with no encounters with the legal system with first 12 months

Using Logic Models to Identify Key Information Needs

so we come back to:

“What information do you need to effectively evaluate the processes and outcomes of the program?”

Writing About Your Description of the Program

In writing about your program description, you will include some of the following information.

• Need: What need or needs is this program designed to meet?
• Context: What is the program’s context? Be sure to discuss the contextual or cultural factors that may affect the program’s effectiveness?
• Population Addressed: Who is included in the population for whom activities are intended?
• Stage of Development: How long has the program been in place?
• Resources: What resources (e.g., staff, money, space, faculty time, partnerships, technology, etc.) are available to support the program?
• Activities: What specific activities are conducted to achieve the program’s outcomes? (Note: Here is where you can provide additional information that you may have been asked to eliminate from the logic model, such as information on hours and availability of services, length of tutoring sessions, etc.)
• Outputs: What do the activities produce? (Remember, these are goal numbers, in this case measuring services delivered.)
• Outcomes: What are the program’s intended outcomes? (Remember that outcomes are changes in attitudes, skills, or behaviors of participants. You may want to divide outcomes into short-term, intermediate, and long-term categories–it’s up to you.)
• Impact: What is the program’s intended impact on the larger community?

Complete you Midterm Feedback

Feedback can be completed at the course evaluation form.