This Article  • References  • Full text PDF  • Reply to article  • Send to a friend  • Save in My Folder  • Save to citation manager  • Permissions  Citing Articles  • Citation map  • Citing articles on HighWire  • Contact me when this article is cited  Related Content  • Similar articles in this journal  Social Bookmarking   What's this?

Edward L. Spitznagel, PhD; John E. Helzer, MD

Arch Gen Psychiatry. 1985;42(7):725-728.

Abstract
• Because it corrects for chance agreement, kappa (K) is a useful statistic for calculating interrater concordance. However, K has been criticized because its computed value is a function not only of sensitivity and specificity, but also the prevalence, or base rate, of the illness of interest in the particular population under study. For example, it has been shown for a hypothetical case in which sensitivity and specificity remain constant at .95 each, that K falls from .81 to .14 when the prevalence drops from 50% to 1%. Thus, differing values of K may be entirely due to differences in prevalence. Calculation of agreement presents different problems depending on whether one is studying reliability or validity. We discuss quantification of agreement in the pure validity case, the pure reliability case, and those studies that fall somewhere between. As a way of minimizing the base rate problem, we propose a statistic for the quantification of agreement (the Y statistic), which can be related to K but which is completely independent of prevalence in the case of validity studies and relatively so in the case of reliability.

Author Affiliations
From the Department of Mathematics and Division of Biostatistics (Dr Spitznagel), and the Department of Psychiatry (Dr Helzer), Washington University, St Louis.

Footnotes
Accepted for publication Jan 10, 1985.

Reprint requests to Department of Psychiatry, Washington University School of Medicine, Medical School Box 8134, 4940 Audubon Ave, St Louis, MO 63110 (Dr Helzer).

CiteULike    Connotea    Del.icio.us    Digg    Reddit    Technorati    Twitter     What's this?

THIS ARTICLE HAS BEEN CITED BY OTHER ARTICLES

Including Omission Mistakes in the Calculation of Cohen's Kappa and an Analysis of the Coefficient's Paradox Features
Simon
Educational and Psychological Measurement 2006;66:765-777.
ABSTRACT  

Intimate Partner Aggression Reporting Concordance and Correlates of Agreement Among Men With Alcohol use Disorders and Their Female Partners
Panuzio et al.
Assessment 2006;13:266-279.
ABSTRACT  

Chance-corrected measures of reliability and validity in K K tables
Andres and Marzo
Stat Methods Med Res 2005;14:473-492.
ABSTRACT  

Assessing the reliability of ordered categorical scales using kappa-type statistics
Roberts and McNamee
Stat Methods Med Res 2005;14:493-514.
ABSTRACT  

Factors Related to the Inability of Individuals With Low Back Pain to Improve With a Spinal Manipulation
Fritz et al.
ptjournal 2004;84:173-190.
ABSTRACT | FULL TEXT  

Reliability of the PEDro Scale for Rating Quality of Randomized Controlled Trials
Maher et al.
ptjournal 2003;83:713-721.
ABSTRACT | FULL TEXT  

Validating a Dipstick Method for Detecting Recent Smoking
Gariti et al.
Cancer Epidemiol. Biomarkers Prev. 2002;11:1123-1125.
ABSTRACT | FULL TEXT  

On the relationship between the efficiency and the quality of the consultation. A validity study
Goedhuys and Rethans
Fam Pract 2001;18:592-596.
ABSTRACT | FULL TEXT  

Examining Diagnostic Tests: An Evidence-Based Perspective
Fritz and Wainner
ptjournal 2001;81:1546-1564.
ABSTRACT | FULL TEXT  

Kappa as a Parameter of a Symmetry Model for Rater Agreement
Schuster
JOURNAL OF EDUCATIONAL AND BEHAVIORAL STATISTICS 2001;26:331-342.
ABSTRACT  

A Method for Measuring Interrater Agreement on Checklists
Sinacore et al.
Eval Health Prof 1999;22:221-234.
ABSTRACT  

Monash Interview for Liaison Psychiatry (MILP): Development, Reliability, and Procedural Validity
Clarke et al.
Psychosomatics 1998;39:318-328.
ABSTRACT | FULL TEXT  

Measurement reliability and agreement in psychiatry
Shrout
Stat Methods Med Res 1998;7:301-317.
ABSTRACT  

Psychosocial Risk Characteristics of Children in Welfare Programmes in Holland: The Role of Risk Factor Analysis in the Planning of Welfare Services for Troubled Children
SCHOLTE
Childhood 1998;5:185-205.
ABSTRACT  

Severin Classification System for Evaluation of the Results of Operative Treatment of Congenital Dislocation of the Hip. A Study of Intraobserver and Interobserver Reliability
WARD et al.
JBJS 1997;79:656-63.
ABSTRACT | FULL TEXT  

Two-Unit Reliability Analysis of Questionnaires Used in a Regulatory System
Fleishman et al.
Eval Rev 1996;20:580-595.
ABSTRACT  

ADHD boys' behavior during structured classroom social activities: Effects of social demands, teacher proximity, and methylphenidate
Granger et al.
J Atten Disord 1996;1:16-30.
ABSTRACT  

The Evaluation of Behavioral Disturbances in Alzheimer's Disease: The Utility of Three Rating Scales
Mack and Patterson
J Geriatr Psychiatry Neurol 1994;7:99-115.
ABSTRACT  

The Reliability of Peer Assessments: A Meta-Analysis
Goldman
Eval Health Prof 1994;17:3-21.
ABSTRACT  

Diagnosing Personality Disorders: A Review of Issues and Research Methods
Zimmerman
Arch Gen Psychiatry 1994;51:225-245.
ABSTRACT  

Reliability of DSM-III-R Anxiety Disorder Categories: Using the Anxiety Disorders Interview Schedule--Revised (ADIS-R)
Di Nardo et al.
Arch Gen Psychiatry 1993;50:251-256.
ABSTRACT  

Progress Toward Achieving a Common Language in Psychiatry: Results From the Field Trial of the Clinical Guidelines Accompanying the WHO Classification of Mental and Behavioral Disorders in ICD-10
Sartorius et al.
Arch Gen Psychiatry 1993;50:115-124.
ABSTRACT  

Agraphia in Dementia of the Alzheimer Type
LaBarge et al.
Arch Neurol 1992;49:1151-1156.
ABSTRACT  

Modelling patterns of agreement and disagreement
Agresti
Stat Methods Med Res 1992;1:201-218.
ABSTRACT  

Attitudinal Predictors of Sexual Activity in Hispanic Adolescent Females
Gibson and Kempf
Journal of Adolescent Research 1990;5:414-430.
ABSTRACT  

Maximum Likelihood Estimates of the Accuracy of Four Diagnostic Techniques
Streiner and Miller
Educational and Psychological Measurement 1990;50:653-662.
ABSTRACT  

Assessment of Behavioral and Affective Symptoms in Alzheimer's Disease
Patterson et al.
J Geriatr Psychiatry Neurol 1990;3:21-30.
ABSTRACT  

Wilson's Disease: Psychiatric Symptoms in 195 Cases
Dening and Berrios
Arch Gen Psychiatry 1989;46:1126-1134.
ABSTRACT  

The Prevalence of Psychiatric Disorders in Patients With Alcohol and Other Drug Problems
Ross et al.
Arch Gen Psychiatry 1988;45:1023-1031.
ABSTRACT  

The Validity of a Self-Report Questionnaire for Diagnosing Major Depressive Disorder
Zimmerman and Coryell
Arch Gen Psychiatry 1988;45:738-740.
ABSTRACT  

A Partial Solution to the Base Rate Problem of the k Statistic
Stewart and Rey
Arch Gen Psychiatry 1988;45:504-505.
ABSTRACT  

Y's, k'S, p's and q's
Carey
Arch Gen Psychiatry 1987;44:1027-1027.
ABSTRACT  

Ontario Child Health Study: I. Methodology
Boyle et al.
Arch Gen Psychiatry 1987;44:826-831.
ABSTRACT  

The Spanish Diagnostic Interview Schedule: Reliability and Concordance With Clinical Diagnoses in Puerto Rico
Canino et al.
Arch Gen Psychiatry 1987;44:720-726.
ABSTRACT  

Quantification of Agreement in Psychiatric Diagnosis Revisited
Shrout et al.
Arch Gen Psychiatry 1987;44:172-177.
ABSTRACT  

Charlie Brown and Statistics: An Exchange
Kraemer
Arch Gen Psychiatry 1987;44:192-193.
ABSTRACT  

Epidemiology: Reflections on Testing the Validity of Psychiatric Interviews
Robins
Arch Gen Psychiatry 1985;42:918-924.
ABSTRACT  

A Comparison of Two Diagnostic Methods: Clinical ICD Diagnoses vs DSM-III and Research Diagnostic Criteria Using the Diagnostic Interview Schedule (Version 2)
Wittchen et al.
Arch Gen Psychiatry 1985;42:677-684.
ABSTRACT