How to use a statistical test (Krippendorff alpha) to check the reliability of a variable with ordinal data using a Windows PC and SPSS. Six observers have rated 30 students. The question was, "How would you rate this individual student?"
1 = Excellent
2 = Above Average
3 = Average
4 = Below Average

Here, I have six judges and no missing data. However, this statistical test can be used with any number of judges and with or without missing data.

In SPSS, click File, Open, Syntax, and open the macro "kalpha.sps".

If you do not have this special file, please see my previous video
"Nominal dichotomous yes/no data: Krippendorff alpha inter-rater reliability"
where I show you how to find and download it.

Execute this macro.

Open your data file.

Run the statistical test by clicking File, New, Syntax, and type:
kalpha judges = teacher1 teacher2 teacher3 teacher4 teacher5 teacher6/level = 2/detail = 0/boot = 10000

Then: Run, All

The Krippendorff's Alpha Reliability Estimate here is 0.6159.
An alpha below 0.67 indicates a really low inter-rater reliability.
Ideally, it should be over 0.8.
Below 0.8 but above 0.67 indicates low reliability.

Source: Krippendorff's book Content Analysis: An Introduction to Its Methodology. (Published by SAGE.)

The table shows an estimated 70.01 percent chance that the alpha would be below 0.67 if the whole population were tested.

A Krippendorff alpha of just 0.6159 is perhaps too low to be used in a report, but I still include an example here to show how these types of results are written out.

Method
Krippendorff's alpha test was used (Hayes & Krippendorff, 2007) to estimate the inter-coder reliability, and these alpha (α) values are reported in the results below.

Results
The results show that the inter-coder reliability was low (α = 0.6159), i.e., that the six observers did not agree.

Discussion
(Here, discuss possible reasons why the observers disagreed.)

References
Hayes, A. F., & Krippendorff, K. (2007). Answering the call for a standard reliability measure for coding data. Communication Methods and Measures 1(1), 77-89.

Let us do it again with another set of example data.
Here, we have four observers who are rating 12 individuals.

There is missing data, but that is no problem for the Krippendorff alpha test. It is one great advantage compared to other statistical tests.

File, New, Syntax.
kalpha judges = obsa obsb obsc obsd/level = 2/detail = 0/boot = 10000

The Krippendorff's Alpha Reliability Estimate here is 0.8095.
There is a 6.46 percent chance that the alpha would be below 0.67 if the whole population would be tested.

This is an example of how it can be reported in text:

Method
Krippendorff's alpha test was used (Hayes & Krippendorff, 2007) to estimate the inter-coder reliability, and these alpha (α) values are reported in the results below.

Results
The results show a relatively high inter-coder reliability (α = 0.8095), i.e., the four observers agreed with each other.

Discussion
(Discuss plausible reasons why the observers agreed and possible consequences.)

References
Hayes, A. F., & Krippendorff, K. (2007). Answering the call for a standard reliability measure for coding data. Communication Methods and Measures 1(1), 77-89.

Copyright
Text, video, and audio © Kent Löfgren, Sweden

свернуть