The objective of the YOS and COS Redesign Project is to develop a set of outcome measurement tools that meet the following three requirements:

• Easy for staff to administer
• Provide the data required to meet the needs of all BBBS stakeholders (match support, fundraising, marketing, recruitment, etc.)
• Be valid and reliable measures of BBBS outcomes

In this week’s post, we take a deeper look at the third requirement listed above and explain why reliability and validity are important elements of the YOS and COS.

Validity

Validity is a term used to signify confidence that what we think we are measuring with a set of survey questions is what we are actually measuring.  A valid set of survey questions actually measures what we intend it to measure. An invalid set of survey questions does not actually measure what we intend it to measure.

One of the best ways to establish the validity of a set of survey questions is to test the questions with diverse populations (e.g., race, ethnicity, age, gender) and test if they are indeed related to other measures we would expect them to be related to. Because certain measures have been tested repeatedly with diverse populations, we can be confident that they actually predict behaviors and outcomes that we believe they predict – they are valid.

Unfortunately, BBBS does not have the resources required to independently test question sets with the rigor required to develop new scales and establish validity. For this reason, we must rely on question sets developed and tested by outside groups.

The specific scales currently used in the YOS/COS have been widely adopted by the research community (including public health), and they have been used in longitudinal studies to determine if they relate to long-term outcome areas.  We are able to predict the long-term success of our own youth because of previously established research linking our desired outcomes to these measures.

Reliability

Reliability refers to the consistency or repeatability of measures. We would expect that a person would give the same answers to the same questions time after time, and that they would answer related questions in a similar manner (for example, if a person scores high on one question in the emotion regulation scale, they should score high on the other questions in the same scale).

One of the most important types of reliability for the YOS and COS is the internal reliability of the surveys’ question sets.

Internal Reliability

Internal reliability ensures that a measure is consistent within itself.  The only way to determine if this criterion has been met is to ask similar questions worded in slightly different ways, which sometimes makes questions seem repetitive.  Referring again to the emotion regulation example, we need to ask all four questions in the emotion regulation scale to get at the specific idea of “emotion regulation.”  If we only ask one of the questions, we lose reliability and can no longer speak to the overall concept of emotion regulation. Instead, we could only speak to specific question asked.

Here is an example of the difference in data reporting between using the emotion regulation scale (all four questions) and just one question from the scale:

• 96% of youth maintained or improved their ability to regulate their emotions.  High emotion regulation is associated with improved mental health, positive well-being, academic achievement, and the quality of their social interactions.
• 96% of youth maintained or improved their ability to relax when they feel tense. (By only asking one question, it can give the impression that we target that specific behavior instead of a more general, yet defined outcome area.)

While survey questions may sometimes seem repetitive, the reliability of survey measures often require similar questions to be asked in slightly different ways to capture the essence of the concept.

The Value of validity and Reliability

Using valid and reliable measures in the YOS and COS is valuable to BBBS for the following reasons:

1. It ensures we are in fact measuring what we believe we are measuring
2. It allows us to measure conceptual traits, such as emotional regulation, instead of merely specific behaviors such as the ability to relax
3. It allows us to compare our outcome data with external data, thus allowing us to understand how we are performing in comparison to non-BBBS organizations
4. It ensures that our measures are accurate across diverse demographic groups

Validity and reliability are highly important aspects of the YOS and COS. As such, the YOS and COS Redesign Project team is working hard to maintain these aspects of the surveys while also striving to increase ease of administration and ensure the surveys produce the data required by all BBBS stakeholders.

If you have any questions about the YOS and COS Redesign Project, you can click here to send the project team a message. We will get back to you as soon as possible.