Reliability

Chapter 3

Classical Test Theory

∗ Every observed score is a combination of true

score and error

Obs. = T + E

∗ Reliability =

s

1−

s

2

E

2

O

s

=

s

2

T

2

O

Reliability

∗ Systematic versus unsystematic error

∗ Reliability only takes unsystematic error into

account

Reliability & Correlation

∗ Reliability often based on consistency between two

sets of scores

∗ Correlation: Statistical technique used to examine

consistency

Positive Correlation

Negative Correlation

Pearson-Product Moment

Correlation Coefficient

∗ Correlation coefficient: a numerical indicator of the

relationship between two sets of data

∗ Pearson-Product Moment correlation coefficient is

most common

r=

∑ z1 z 2

N

Coefficient of Determination

∗ The percentage of shared variance between two sets of

data

Types of Reliability

∗ Test-Retest

∗ Alternate/Parallel Forms

∗ Internal Consistency Measures

Test-Retest

∗ Correlating performance on first administration

with performance on the second

∗ Co-efficient of stability

Alternate/Parallel Forms

∗ Two forms of instrument, administered to same

individuals

Internal Consistency Measures

∗ Split-half reliability

∗ Spearman-Brown formula

∗ Kuder-Richardson formulas

∗ KR 20

∗ KR 21

∗ Coefficient Alpha

Nontypical Situations

∗ Typical methods for determining reliability may not

be suitable for:

∗ Speed tests

∗ Criterion-referenced tests

∗ Subjectively-scored instruments

∗ Interrater reliability

Evaluating Reliability Coefficients

∗ Examine purpose for using instrument

∗ Be knowledgeable about reliability coefficients of

other instruments in that area

∗ Examine characteristics of particular clients

against reliability coefficients

∗ Coefficients may vary based on SES, age,

culture/ethnicity, etc.

Standard Error of Measurement

SEM = s 1 − r

∗ Provides estimate of range of scores if someone were to

take instrument repeatedly

∗ Based on premise that when individuals take a test

multiple times, scores fall into normal distribution

SEM: Example

∗ Sam’s SAT Verbal = 550

∗ r = .91; s = 100

∗ SEM = 100 1 − .91

= 100 .09

= 100 × .3

= 30

∗ 68% of the time, Sam’s true score would fall between 520 and 580

∗ 95% of the time, Sam’s true score would fall between 490 and 610

∗ 99.5% of the time, Sam’s true score would fall between 460 and 640

Determining Range of Scores Using

SEM

Standard Error of Difference

∗ Method to determine if difference between two

scores is significant

∗ Takes into account SEM of both scores

Alternative Theoretical Model

∗ Generalizability or Domain Sampling Theory

∗ Focus is on estimating the extent to which specific

sources of variation under defined conditions are

contributing to the score on the instrument

Chapter 3

Classical Test Theory

∗ Every observed score is a combination of true

score and error

Obs. = T + E

∗ Reliability =

s

1−

s

2

E

2

O

s

=

s

2

T

2

O

Reliability

∗ Systematic versus unsystematic error

∗ Reliability only takes unsystematic error into

account

Reliability & Correlation

∗ Reliability often based on consistency between two

sets of scores

∗ Correlation: Statistical technique used to examine

consistency

Positive Correlation

Negative Correlation

Pearson-Product Moment

Correlation Coefficient

∗ Correlation coefficient: a numerical indicator of the

relationship between two sets of data

∗ Pearson-Product Moment correlation coefficient is

most common

r=

∑ z1 z 2

N

Coefficient of Determination

∗ The percentage of shared variance between two sets of

data

Types of Reliability

∗ Test-Retest

∗ Alternate/Parallel Forms

∗ Internal Consistency Measures

Test-Retest

∗ Correlating performance on first administration

with performance on the second

∗ Co-efficient of stability

Alternate/Parallel Forms

∗ Two forms of instrument, administered to same

individuals

Internal Consistency Measures

∗ Split-half reliability

∗ Spearman-Brown formula

∗ Kuder-Richardson formulas

∗ KR 20

∗ KR 21

∗ Coefficient Alpha

Nontypical Situations

∗ Typical methods for determining reliability may not

be suitable for:

∗ Speed tests

∗ Criterion-referenced tests

∗ Subjectively-scored instruments

∗ Interrater reliability

Evaluating Reliability Coefficients

∗ Examine purpose for using instrument

∗ Be knowledgeable about reliability coefficients of

other instruments in that area

∗ Examine characteristics of particular clients

against reliability coefficients

∗ Coefficients may vary based on SES, age,

culture/ethnicity, etc.

Standard Error of Measurement

SEM = s 1 − r

∗ Provides estimate of range of scores if someone were to

take instrument repeatedly

∗ Based on premise that when individuals take a test

multiple times, scores fall into normal distribution

SEM: Example

∗ Sam’s SAT Verbal = 550

∗ r = .91; s = 100

∗ SEM = 100 1 − .91

= 100 .09

= 100 × .3

= 30

∗ 68% of the time, Sam’s true score would fall between 520 and 580

∗ 95% of the time, Sam’s true score would fall between 490 and 610

∗ 99.5% of the time, Sam’s true score would fall between 460 and 640

Determining Range of Scores Using

SEM

Standard Error of Difference

∗ Method to determine if difference between two

scores is significant

∗ Takes into account SEM of both scores

Alternative Theoretical Model

∗ Generalizability or Domain Sampling Theory

∗ Focus is on estimating the extent to which specific

sources of variation under defined conditions are

contributing to the score on the instrument

## Fate of Pharmaceuticals in the Environment and in Water Treatment Systems - Chapter 3 pdf

## Wastewater Purification: Aerobic Granulation in Sequencing Batch Reactors - Chapter 3 potx

## WETLAND AND WATER RESOURCE MODELING AND ASSESSMENT: A Watershed Perspective - Chapter 3 doc

## AEROSOL CHEMICAL PROCESSES IN THE ENVIRONMENT - CHAPTER 3 pot

## Climate Change and Water Resources in South Asia - Chapter 3 ppt

## Heavy Metals in the Environment: Using Wetlands for Their Removal - Chapter 3 docx

## Integrated Assessment of Health and Sustainability of Agroecosystems - Chapter 3 potx

## Radionuclide Concentrations in Foor and the Environment - Chapter 3 pdf

## Sediment and Contaminant Transport in Surface Waters - Chapter 3 pdf

## Coastal and Estuarine Risk Assessment - Chapter 3 ppsx

Tài liệu liên quan