Tải bản đầy đủ

Assessment in counseling chapter 4

Validity and Item
Chapter 4

∗ Concerns what instrument measures and how well
it does so
∗ Not something instrument “has” or “does not
∗ Informs counselor when it is appropriate to use
instrument and what can be inferred from results
∗ Reliability is a prerequisite for validity

Traditional Categories of Validity
∗ Content-Related
∗ Criterion-Related
∗ Construct

Validation Evidence: Standards for
Educational and Psychological Testing

Evidence Based on Test Content

Evidence Based on Response Processes

Evidence Based on Internal Structure

Evidence Based on Relations to Other Variables

Evidence Based on Consequences of Testing

Evidence Based on Test Content
∗ Degree to which the evidence indicates that items,
questions, or tasks adequately represent intended
behavior domain
∗ Central focus is typically on how the instrument’s
content was determined
∗ Content-related validation evidence should not be
confused with face validity

Evidence Based on Response Processes
∗ Concerns whether individuals respond in a manner
consistent with construct measured
∗ May also examine information processing differences
by subgroup
∗ Computer simulated performances

Evidence Based on Internal Structure
∗ Examining internal structure using factor analysis
∗ Can also examine internal structure of instrument for
different subgroups (differential item functioning)

Evidence Based on Relations to Other

Correlational method

∗ Convergent and discriminant evidence
∗ Multitrait-multimethod matrix

Prediction or instrument-criterion relationship

∗ Concurrent and predictive validity
∗ Regression
∗ Decision Theory

Validity Generalization

Correlational Method
∗ Correlational method 
examines relationship
between instrument and
criterion or other
pertinent variables
∗ Validity coefficient

∗ Multitrait-multimethod
matrix  examines
relationships between
instrument and traits that
are theoretically related
and unrelated to it

∗ Concurrent validity
∗ Predictive validity


a = y-intercept
b = slope
X = score

Standard Error of Estimate
∗ The margin of expected error in the individual’s
predicted criterion score as a result of imperfect


Sest = Sy 1− r xy

Decision Theory: Expectancy Table

Expectancy Table (cont.)

Expectancy Table (cont.)

Validity Generalization

Method of combining validation studies to determine if validity evidence can be generalized

∗ Must have a substantial number of studies
∗ Must use meta-analytic procedures

Evidence-Based Consequences
of Testing

Examples of social consequences:

∗ Group differences on tests used for employment selection
∗ Group differences in placement in special education

Counselors should consider both validation evidence and social implications

Conclusion on Validation Evidence
∗ Gradual accumulation of evidence
∗ Counselor must evaluate the information to
determine if it appropriate for which client under
what circumstance
∗ Validation evidence should also be considered in
informal assessments

Item Analysis
∗ Examining and evaluating each item in
∗ Item difficulty:
p = # who answered item correctly
total # of individuals

∗ Item discrimination:
∗ Extreme group method: d = upper % – lower %
∗ Correlational method

Item Response Theory
∗ Focus is on each item and establishing items that
measure ability or level of a latent trait
∗ Involves examining item characteristic curve and
calibration of each individual item
∗ Parameters:
∗ Difficulty
∗ Slope (discrimination)
∗ Probability of guessing

Item Characteristic Curve

Tài liệu bạn tìm kiếm đã sẵn sàng tải về

Tải bản đầy đủ ngay