Spring 2026 SOWK 460w Week 08 - Instruments and Validity

Slide 1
Pink square with a lightbulb icon and numbered list; text reads: 'Instruments, and Validity, Spring 2026, SOWK 460w, Week 08, Jacob Campbell, Ph.D. LICSW at Heritage University.'

Spring 2026 SOWK 460w Week 08 - Instruments and Validity

title: Spring 2026 SOWK 460w Week 08 - Instruments and Validity date: 2026-03-09 13:25:27 location: Heritage University tags:

  • Heritage University
  • BASW Program
  • SOWK 460w presentation_video: > “” description: >

During week eight, we will engage in a peer review of your logic models and consider the validity and instruments used to collect data for your program evaluation. We can also consider threats to validity to develop a more robust, more valid evaluation and to identify high-quality sources of information. We will have time in class to work on your projects. The agenda for the in-person class session includes:

  • Logic model peer review
  • Validity and threats to validity
  • Searching for sources
  • Time to work on projects

Learning Objectives

  • Conduct a peer review of a program logic model using established criteria to provide constructive feedback to peers.
  • Define validity and distinguish among types of validity used in program evaluation.
  • Identify and explain classic threats to internal validity and describe strategies for mitigating those threats in a program evaluation design.
  • Recognize and correct common item-writing problems in survey and instrument development.
Slide 2
**Object**: Slide**Action**: Displays agenda and objectives**Context**: Educational presentation**Text**:- **Title**: 'Week Eight Plan'- **Agenda**: - Logic model peer review - Validity and threats to validity - Searching for sources - Time to work on projects- **Learning Objectives**: - Conduct a peer review of a program logic model using established criteria to provide constructive feedback to peers. - Define validity and distinguish among types of validity used in program evaluation. - Identify and explain classic threats to internal validity and describe strategies for mitigating those threats in a program evaluation design. - Recognize and correct common item-writing problems in survey and instrument development.

Week Eight Plan

Agenda

  • Logic model peer review
  • Validity and threats to validity
  • Searching for sources
  • Time to work on projects

Learning Objectives

  • Conduct a peer review of a program logic model using established criteria to provide constructive feedback to peers.
  • Define validity and distinguish among types of validity used in program evaluation.
  • Identify and explain classic threats to internal validity and describe strategies for mitigating those threats in a program evaluation design.
  • Recognize and correct common item-writing problems in survey and instrument development.
Slide 3
The image shows a slide with a 'Logic Model' under 'Peer Review.' It instructs reviewing and submitting notes in the MyHeritage Forum. A rubric titled 'Logic Model Rubric for SOWK 460w' is included, categorizing descriptions from 'Initial' to 'Highly Developed.'

Peer Review of Logic Model

[Small Group Activity] Working in your learning team, you will review the logic models of other groups. Links to the assignment forum and the rubric we created are in My Heritage.

Slide 4
A typewriter icon appears next to a list detailing 'Methods for Evaluation: Basics for Every Method,' including sample selection, data collection, analysis, and reporting. (Kapp & Anderson, 2010) mentioned.

Methods for Evaluation

Any method for evaluation needs to include:

  • Sample selection
  • Data collection
  • Analysis
  • Reporting
Slide 5
Three pie charts show data distribution; the first two are divided with labeled percentages (75%-25%), while the third is a solid circle labeled 100%. Below, colorful text reads 'Sampling.'

Sampling

What kinds of sampling methods? [Whole Class Activity] Discussion regarding what types of sampling methods are planned for groups.

Slide 6
A compass lies on a wooden surface. Beside it, text titled 'Validity' lists types: content, face, concurrent, predictive, known groups, discriminant, and convergent validity with the question, 'What is it and how do we address it?'

Validity: How to Address Internal Validity

[Whole Class Activity] What is validity?

Your textbook describes the following types of validity

  • Content Validity: Whether the instrument captures the full range of the construct. Missing key components (like leaving out common hangover symptoms) means poor content validity.
  • Face Validity: A panel of colleagues or experts reviews the instrument and agrees it appears to measure what it’s supposed to. Along with content validity, this is a “minor” form that doesn’t provide empirical evidence.
  • Concurrent Validity: A new instrument is given alongside an established, validated instrument at the same time. If the two correlate well, the new one has concurrent validity.
  • Predictive Validity: The instrument can predict some future event or outcome. More difficult to establish because it requires following participants over time and tracking real-world outcomes.
  • Known Groups Validity: The instrument can accurately distinguish between groups expected to score high versus low on the construct (e.g., correctly classifying abusive vs. nonabusive parents).
  • Discriminant (Divergent) Validity: The instrument does not correlate with a construct it theoretically shouldn’t be related to, confirming it’s measuring something distinct.
  • Convergent Validity: Scores on the instrument align with an external criterion that should logically be related (e.g., high interest in nursing correlating with volunteer experience in health settings).
  • Factor Analysis: A statistical procedure examining the internal structure of an instrument to determine whether items form a single construct or multiple distinct dimensions (subscales).

There are a number of potential threats to validity. One of the ways that we address validity is by mitigating it’s threats

Next Slide ->

Slide 7
The image features icons representing 'Classic Threats to Internal Validity': a statue for History, hourglass for Maturation and Time, checklist for Testing, hammer for Instrumentation, and graph for Statistical Regression. Text: 'Classic Threats to internal validity' and source noted as '(Kapp & Anderson, 2010)'.

Classic Threats to Internal Validity (1 of 3)

  • History: Events that happen outside of evaluation or contextually during the evaluation that effect the event. (Corona Virus, people being laid off… etc)
  • Maturation and the passage of time: general growth that happens on it’s own. Especially true for children, but can be true for anybody.
  • Testing: Pre-test effects the outcome of the post-test.
  • Instrumentation: Change in the tools used to collect data during time of data collection (e.g. changing questions on pre-test/post-test)
  • Statistical regression: When there are significant changes (improvement / deterioration) that is based on their extreme behavior or position prior. (Think nowhere to go but up/down)

(Kapp & Anderson, 2010)

Slide 8
Diagram identifies 'Classic Threats to internal validity' including: 'Selection Bias,' 'Experimental Mortality and Attrition,' 'Ambiguity About Direction of Causal Influences,' 'Design Contamination,' 'Diffusion or Imitation of Treatments.' Features icons and arrows to illustrate concepts. (Kapp & Anderson, 2010).

Classic Threats to Internal Validity (2 of 3)

  • Selection bias: Problems related to selection of participants (more random and larger sample better)
  • Experimental mortality and attrition: Not completing the intervention or process.
  • Ambiguity about the direction of causal influences: Direction of impacts and influencing conditions not clear. (Depressed causes lack of sleep or lack of sleep causes depression)
  • Design contamination: change behaviors or actions because of being evaluated.
  • Diffusion or imitation of treatments: looking for unique qualities which might be used by other professions (many professionals use strengths-based practice… not only ones that work in a “strengths-based program)

(Kapp & Anderson, 2010)

Slide 9
Diagram depicting classic threats to internal validity, such as interaction effects. Icons include people, diagrams, charts, a magnifying glass, and other symbols. Text: 'Classic Threats to internal validity,' 'Interaction Effects,' 'A → B,' 'Kapp & Anderson, 2010.'

Classic Threats to Internal Validity (3 of 3)

Interaction Effects: Threats to internal validity interact with each other.

(Kapp & Anderson, 2010)

Slide 10
The image is a presentation slide listing 'Tips for Searching for Sources' with nine detailed tips on finding scholarly sources, including starting broad, using key words, and employing synonyms.

Tips for Searching for Sources: Finding Scholarly Sources

Your textbook provides this same tips, focused on finding instruments to use for your measurement. Works the same for literature.

TIP 1. Start with a Broad Social Science Database. TIP 2. Start by Searching Key Words in the Title Boxes. TIP 3. Use as Few Words as Possible. TIP 4. If the Title Search Is Unproductive, Broaden the Search by Changing the Secondary Search Term to an All-Text Search. TIP 5. If You Get Too Many Hits, Add Another (Third) Search Term. TIP 7. Use Synonyms and Then Repeat Your Searches. TIP 8. Use Quotation Marks for a Two-Term Phrase. TIP 9. Look for Systematic Literature Reviews, Systematic Reviews, or Reviews of the Literature.

(Royse, 2023)

Slide 11
The image shows the word 'DEMO' in a gradient blue-to-purple font. Below, the text reads 'Burnout and Child Welfare Workers' on a plain white background.

Demo: Burnout and Child Welfare Workers

Step 1: The topic we are considering.

Burnout

Step 2: Get some broad understanding of the topic if you don’t already have it.

Search for Wikipedia wikipedia burnout

or use ChatGPT etc. Give me an overview of the topic of burnout

Step 3: Start Searching for Articles

Google Scholar Heritage Search Sci-Hub

Narrowing Down Topics

  • adding more search terms
  • Using advanced search tools (advanced, filters, search codes)
  • Synonyms
  • Searching for specific types of articles systematic reviews

Step 4: Save Your Work

  • Save articles you read
  • Write notes about it
  • Highlight and copy notes from articles
Slide 12
Title slide lists item writing issues: double-barreled and leading questions, unavailable info, jargon, insensitive language, loaded and vague questions, non-exclusive choices, all-inclusive and negatively constructed terms.

Problems to Avoid in Item Writing

  • Double-Barreled Questions: Ask about two things in one question, making it impossible to know which part the respondent is answering. Stick to one concept per item.
  • Leading Questions: Guide the respondent toward a particular answer, often unintentionally. Review items from an opposing perspective to catch these.
  • Unavailable Information: Ask for details people can’t reasonably recall, producing unreliable data. Keep time frames short and realistic.
  • Jargon, Acronyms, and Technical Terms: Use language that some or many respondents won’t understand. Use plain, accessible wording unless your entire audience shares the specialized vocabulary.
  • Insensitive Language: Create an “us vs. them” dynamic or use phrasing that feels biased or othering toward respondents.
  • Loaded Questions: Include inflammatory or stigmatizing terms that push respondents toward socially desirable answers rather than honest ones.
  • Response Choices Not Mutually Exclusive: Offer overlapping or vague categories (like “several” or “a few”) that prevent accurate measurement or leave respondents unsure where they fit.
  • Vague and Ambiguous Terms: Use imprecise words like “regularly” or “often” that mean different things to different people. Aim for specific counts or timeframes.
  • All-Inclusive Terms: Use absolutes like “always” or “never,” which few people can honestly endorse and which tend to distort responses.
  • Negatively Constructed Items: Include “not” in statements, which can confuse meaning or slow comprehension. Phrase items positively instead.
  • Other Considerations: Sequence matters — start with approachable questions, save sensitive items for the end, watch for question order effects, keep instruments short to avoid fatigue, and favor close-ended response formats when possible.

(Royse, 2023)

Slide 13
Slide with text reads: 'Time to work on questionnaires or literature review,' using blue and purple gradient text on a white background.

Time to work on instruments

Where are people at in developing questionnaires and data collection? Time to work on instruments.