Tutorial on Sample Size Calculations for the precision and the testing of Cohen’s Kappa: a methodological review and new proposals

Sample Size for qualitative agreement

Abstract

Introduction

We have revised the steps of the sample sizes calculation for the biomedical research.

Then, focusing on the agreement studies on qualitative variables we have shown the derivation of the Cohen’s kappa together with the factors influencing its value, leading to have low kappa values even in presence of a relevant observed agreement.

We have revised the sample size calculation approaches for the Cohen’s kappa proposed in the statistical literature.

Methods

We have calculated the sample sizes for agreement studies based on the Cohen’s kappa statistics according to the main relevant literature proposals, such as those of Flack et al.[72] and of Donner et al. [90,92,96]. Then, we have proposed a partial extension of the common correlation model (PCCM) for 2x2 contingency table to cxc square contingency tables with a common correlation coefficient of the cells on the principal diagonal, considered pairwise. Finally, we have conceived a full generalization with a common correlation coefficient model (FCCM) of all cells of the square contingency table.

Results

From our PCCM, we have obtained very similar maximum and minimum values of the kappa variance, leading to have sample sizes slightly different. Otherwise, from FCCM, it is obtained a unique contingency table and, consequently, a unique value of the kappa variance to be used under the null and the alternative hypothesis leading to have only one value of the sample size. More relevantly, this latter sample size is within or equal to the sample sizes calculated by using the maximum and minimum value of the kappa variance under PCCM.

In the case of 2x2 contingency tables, all our sample sizes calculation proposed methods (SS-A&C-max, SS-A&C-min, and SS-A&C-full) gave equal sample sizes which are, in addition, equal to those calculated according to Flack et al.[72], being the sample sizes from Donner et al. [90,92] generally greater.

For 3x3 contingency tables, the sample sizes calculated according to our proposed models are lower than or equal to those calculated according to Flack et al.[72] and, also almost always, lower than those from Donner et al. [90,92]. The same occurs for the 4x4 tables.

Discussion / conclusions

Sample sizes from our proposed models can be generally recommended since they are lower than or at maximum equal to those obtained by Flack et al.’s approach[72]. In addition, our proposed procedures give very similar maximum and minimum sample size values under PCCM and only one sample size value under FCCM. Furthermore, sample size proposed by Donner et al [90,92,96] can be considered only in the case in which they are lower and the primary objective of the study is the “agreement or not” instead of a diversified level of agreement assessed through the weighted kappa and, finally, there are more than two raters.

Finally, the fact that there is a unique kappa variance value under the FCCM, leading to sample sizes always within those calculated from the maximum and minimum values of the kappa variance under PCCM, allows to argue that this model actually refers to the population probability table, and consequently, it has to be warmly recommended