Inter-rater reliability in labeling quality and pathological features of retinal OCT scans: A customized annotation software approach
2024

Reliability of Labeling Retinal OCT Scans

Sample size: 500 publication Evidence: moderate

Author Information

Author(s): Du Katherine, Shah Stavan, Bollepalli Sandeep Chandra, Ibrahim Mohammed Nasar, Gadari Adarsh, Sutharahan Shan, Sahel José-Alain, Chhablani Jay, Vupparaboina Kiran Kumar

Primary Institution: Department of Ophthalmology, University of Pittsburgh Medical Center

Hypothesis

This study aims to analyze the inter-rater reliability in labeling the quality and common imaging signatures of retinal OCT scans.

Conclusion

The study demonstrates significant inter-rater reliability in labeling the quality and retinal pathologies on OCT scans.

Supporting Evidence

  • The inter-rater reliability for overall scan quality was κ = 0.60, indicating substantial agreement.
  • The binary determination of presence and absence of retinal disease signatures showed almost complete agreement (κ = 0.85).
  • Specific retinal pathologies exhibited substantial concordance, with κ coefficients ranging from 0.52 to 0.87.

Takeaway

Two doctors looked at the same eye scans and mostly agreed on what they saw, which helps make sure doctors can trust the results.

Methodology

500 OCT scans were labeled by two masked reviewers using a customizable annotation software, and inter-rater reliability was evaluated using Cohen’s kappa coefficient.

Potential Biases

Differences in interpretation of scan quality and features could introduce variability in labeling.

Limitations

The study focused only on age-related macular degeneration eyes and used a specific OCT device, limiting the generalizability of the findings.

Participant Demographics

Patients diagnosed with age-related macular degeneration, screened between 2017 and 2019.

Digital Object Identifier (DOI)

10.1371/journal.pone.0314707

Want to read the original?

Access the complete publication on the publisher's website

View Original Publication