Tải bản đầy đủ

Assessment in counseling chapter 4

Validity and Item
Analysis
Chapter 4


Validity
∗ Concerns what instrument measures and how well
it does so
∗ Not something instrument “has” or “does not
have”
∗ Informs counselor when it is appropriate to use
instrument and what can be inferred from results
∗ Reliability is a prerequisite for validity


Traditional Categories of Validity
∗ Content-Related
∗ Criterion-Related
∗ Construct



Validation Evidence: Standards for
Educational and Psychological Testing


Evidence Based on Test Content



Evidence Based on Response Processes



Evidence Based on Internal Structure



Evidence Based on Relations to Other Variables



Evidence Based on Consequences of Testing


Evidence Based on Test Content
∗ Degree to which the evidence indicates that items,
questions, or tasks adequately represent intended
behavior domain
∗ Central focus is typically on how the instrument’s
content was determined
∗ Content-related validation evidence should not be
confused with face validity


Evidence Based on Response Processes
∗ Concerns whether individuals respond in a manner
consistent with construct measured
∗ May also examine information processing differences
by subgroup
∗ Computer simulated performances



Evidence Based on Internal Structure
∗ Examining internal structure using factor analysis
∗ Can also examine internal structure of instrument for
different subgroups (differential item functioning)


Evidence Based on Relations to Other
Variables


Correlational method

∗ Convergent and discriminant evidence
∗ Multitrait-multimethod matrix


Prediction or instrument-criterion relationship

∗ Concurrent and predictive validity
∗ Regression
∗ Decision Theory


Validity Generalization


Correlational Method
∗ Correlational method 
examines relationship
between instrument and
criterion or other
pertinent variables
∗ Validity coefficient

∗ Multitrait-multimethod
matrix  examines
relationships between
instrument and traits that
are theoretically related
and unrelated to it


Prediction/Instrument-Criterion
Relationship
∗ Concurrent validity
∗ Predictive validity


Regression

a = y-intercept
b = slope
X = score


Standard Error of Estimate
∗ The margin of expected error in the individual’s
predicted criterion score as a result of imperfect
validity

2

Sest = Sy 1− r xy


Decision Theory: Expectancy Table


Expectancy Table (cont.)


Expectancy Table (cont.)


Validity Generalization


Method of combining validation studies to determine if validity evidence can be generalized

∗ Must have a substantial number of studies
∗ Must use meta-analytic procedures


Evidence-Based Consequences
of Testing


Examples of social consequences:

∗ Group differences on tests used for employment selection
∗ Group differences in placement in special education


Counselors should consider both validation evidence and social implications


Conclusion on Validation Evidence
∗ Gradual accumulation of evidence
∗ Counselor must evaluate the information to
determine if it appropriate for which client under
what circumstance
∗ Validation evidence should also be considered in
informal assessments


Item Analysis
∗ Examining and evaluating each item in
instrument
∗ Item difficulty:
p = # who answered item correctly
total # of individuals

∗ Item discrimination:
∗ Extreme group method: d = upper % – lower %
∗ Correlational method


Item Response Theory
∗ Focus is on each item and establishing items that
measure ability or level of a latent trait
∗ Involves examining item characteristic curve and
calibration of each individual item
∗ Parameters:
∗ Difficulty
∗ Slope (discrimination)
∗ Probability of guessing


Item Characteristic Curve



Tài liệu bạn tìm kiếm đã sẵn sàng tải về

Tải bản đầy đủ ngay

×