Hello, I am trying to use your online calculator for my dissertation – I have a qualitative study with 17 categories, 6 subjects and 2 raters. But every time I try to enter the data and press calculate – I get different error messages. Any way you can help me figure out what I’m doing wrong? I’d greatly appreciate it! Thank you! Shivani
I tried out a hypothetical data set with those parameters and it
seemed to work. Be sure that all of the rows sum to 2, because that’s
how many raters you have. ‘m assuming that raters can only assign one
category to each case. Also be sure that all of the empty cells have
a zero in them. Don’t leave any cells blank. If that doesn’t work, let
me know what the error is and post your dataset and I’ll see if I can
figure out what’s happening.
THANKS SO MUCH FOR THE ON LINE CALCULATOR. BY THE WAY AM WRITING FROM NIGERIA WEST AFRICA.
EACH TIME I OPEN THE WEB PAGE OF THE CALCUTOR, I CAN ONLY SEE THE INSTRUCTIONS BUT THE TABLE NEVER APPEARS FOR ME TO LOAD DATA. I’VE SPENT SEVERAL HOUR TRYING BUT SAME RESULT.
I just checked and it is working from two of my computers. You need to have the latest version of java installed to get the calculator to work in your browser. If you already have Java, try to uninstall it and reinstall it. That has worked for others who have had the same problem. If worse comes to worse, you might have to use the calculator from a different computer.
i came to your online calculator as i have trouble doing Kappa in SPSS and need a calculator that can digest my dataset. Most online calculators only do 8 or 12 categories but not more.
I have a dataset with 2 coders, 34 categories, and 135 cases (which will grow up about 400 cases for each kappa i need to calculate); and as i have 6 2-way combinations of 4 raters i will have to compute 6 different kappas.
Of course, so many categories mean i get a lot of zero’s in the category-by-category square, which might cause my SPSS not to calculate it. If i enter all of these data into your calculator (format is different than the SPSS output i get, so i would have to do it all by hand, no C&P) do you think your calculator will be able to digest this? Or do you have any other recommendations how i could best do this?
I don’t think that I saw this before because it’s in the about section.
I think that the OKC will work for you. I’d put the data in a spreadsheet first and try to cut and paste. You don’t want to input all of that data by hand in case there’s an error!
HI!
I started to wonder if you or somebody else using this website could help me with the followning question.
I have conducted face to face interviews and from the transcribed text two independent researchers have identified meaning units that were linked to ICF (international classification of functioning disability and health).No I shoud calculate the kappa statistics on inter rater agreement. The problem is that the ICF core set that we have used has 130 categories. So everytime I choose a category I choose among 130 categories as well as the other rater.
Is it possible to calculate Kappa statistics of this?
That’s an interesting question. Theoretically, kappa would work in this case, but you’d have to be careful with the interpretation. I’ve never seen kappa with so many categories. You can inflate free-marginal kappa by adding superflous categories that aren’t theoretically defensible. With 130 categories the probability of change agreement is .008. However, the probability of chance agreement wouldn’t actually be .008 if many of the 130 categories are “throw away” categories that you or the other raters wouldn’t really pick. My suggestions is to only use as many categories as a theoretically justifiable . . . all of your 130 cats might be theoretically justifiable and you’d just have to make that argument. You might report the fixed marginal kappa in this case (or both).
With that said, I’ve never tried the OKC with 130 cats. Let me know what happens.
9 comments
Comments feed for this article
May 13, 2009 at 12:36 am
Shivani
Hello, I am trying to use your online calculator for my dissertation – I have a qualitative study with 17 categories, 6 subjects and 2 raters. But every time I try to enter the data and press calculate – I get different error messages. Any way you can help me figure out what I’m doing wrong? I’d greatly appreciate it! Thank you! Shivani
May 13, 2009 at 2:16 am
jrandolp
Hi Shivani,
I tried out a hypothetical data set with those parameters and it
seemed to work. Be sure that all of the rows sum to 2, because that’s
how many raters you have. ‘m assuming that raters can only assign one
category to each case. Also be sure that all of the empty cells have
a zero in them. Don’t leave any cells blank. If that doesn’t work, let
me know what the error is and post your dataset and I’ll see if I can
figure out what’s happening.
Take care,
Justus
May 21, 2009 at 7:32 pm
EMMANUEL
HI JUSTUS,
THANKS SO MUCH FOR THE ON LINE CALCULATOR. BY THE WAY AM WRITING FROM NIGERIA WEST AFRICA.
EACH TIME I OPEN THE WEB PAGE OF THE CALCUTOR, I CAN ONLY SEE THE INSTRUCTIONS BUT THE TABLE NEVER APPEARS FOR ME TO LOAD DATA. I’VE SPENT SEVERAL HOUR TRYING BUT SAME RESULT.
PLS TELL ME WHAT TO DO NEXT.
THANKS.
EMMMANUEL.
May 22, 2009 at 6:54 am
jrandolp
Hi Emmanuel,
I just checked and it is working from two of my computers. You need to have the latest version of java installed to get the calculator to work in your browser. If you already have Java, try to uninstall it and reinstall it. That has worked for others who have had the same problem. If worse comes to worse, you might have to use the calculator from a different computer.
Justus
June 25, 2010 at 6:16 pm
Gina
Hi Justus,
i came to your online calculator as i have trouble doing Kappa in SPSS and need a calculator that can digest my dataset. Most online calculators only do 8 or 12 categories but not more.
I have a dataset with 2 coders, 34 categories, and 135 cases (which will grow up about 400 cases for each kappa i need to calculate); and as i have 6 2-way combinations of 4 raters i will have to compute 6 different kappas.
Of course, so many categories mean i get a lot of zero’s in the category-by-category square, which might cause my SPSS not to calculate it. If i enter all of these data into your calculator (format is different than the SPSS output i get, so i would have to do it all by hand, no C&P) do you think your calculator will be able to digest this? Or do you have any other recommendations how i could best do this?
Thank you!
Gina
February 27, 2012 at 6:20 pm
Justus
Hi Gina,
I don’t think that I saw this before because it’s in the about section.
I think that the OKC will work for you. I’d put the data in a spreadsheet first and try to cut and paste. You don’t want to input all of that data by hand in case there’s an error!
February 27, 2012 at 1:07 am
Markku Paanalahti
HI!
I started to wonder if you or somebody else using this website could help me with the followning question.
I have conducted face to face interviews and from the transcribed text two independent researchers have identified meaning units that were linked to ICF (international classification of functioning disability and health).No I shoud calculate the kappa statistics on inter rater agreement. The problem is that the ICF core set that we have used has 130 categories. So everytime I choose a category I choose among 130 categories as well as the other rater.
Is it possible to calculate Kappa statistics of this?
February 27, 2012 at 6:32 pm
Justus
Markku,
That’s an interesting question. Theoretically, kappa would work in this case, but you’d have to be careful with the interpretation. I’ve never seen kappa with so many categories. You can inflate free-marginal kappa by adding superflous categories that aren’t theoretically defensible. With 130 categories the probability of change agreement is .008. However, the probability of chance agreement wouldn’t actually be .008 if many of the 130 categories are “throw away” categories that you or the other raters wouldn’t really pick. My suggestions is to only use as many categories as a theoretically justifiable . . . all of your 130 cats might be theoretically justifiable and you’d just have to make that argument. You might report the fixed marginal kappa in this case (or both).
With that said, I’ve never tried the OKC with 130 cats. Let me know what happens.
February 27, 2012 at 8:44 pm
Markku Paanalahti
Thank you Justus!
I continue to study the problem I have to confront in the near future,
Best Regards
Markku