New Article: Automated Grading with AI? We Test Two Widely Used Tools

AI tools promise to relieve teachers by supporting the correction, feedback, and grading of student texts. In our new open-access article in SEMINAR – Lehrerbildung und Schule, Sean Quägwer and I test two widely used systems for automated feedback and AI-supported grading: FelloFish and Edaira.

The study follows up on our earlier study of Fobizz’s AI grading assistant and asks whether the problems observed there are limited to a single tool or point to structural limits of grading systems based on large language models. The result is clear: FelloFish and Edaira also show substantial variation for identical inputs, a lack of reproducibility in their assessments, and problematic effects when students revise texts on the basis of automated feedback. One particularly troubling finding is that verbatim adoption of automatically suggested formulations is sometimes rated more highly than independent revisions that are equivalent in content.

Our conclusion: such tools currently cannot be recommended as autonomous systems for grading student work. If they are used at all, they should be used only as supporting tools, transparently and under critical human supervision.

Abstract: The study examines the usability of AI-supported correction and feedback systems in schools using the tools FelloFish and Edaira. Based on controlled test series with simulated student texts, it analyzes in particular the consistency of automated assessments and the effects of iterative revisions based on automated feedback. The results show considerable grading volatility for identical inputs, a lack of reproducibility in judgments, and inconsistent effects when improvement suggestions are implemented. The study also shows that the systems tend to rate verbatim adoption of automatically suggested formulations more highly than independent revisions that are equivalent in content. Overall, the findings point to structural limits of grading systems based on large language models. The authors conclude that such tools are currently not suitable for autonomous performance assessment, but at most as supporting tools under critical human supervision.

The material appendix for the study is available here.

Download options and bibliographic data

1. 

   Mühlhoff, Rainer, und Sean Quägwer. 2026. „Automatisierte Korrektur mit KI? Wir testen zwei verbreitete Tools“. SEMINAR – Lehrerbildung und Schule 2/2026: 62–76. doi:10.3278/SEM2602W007.

   Full text bibtex

   ×

   @article{Mü-Quägwer2026,
     title = {Automatisierte Korrektur mit KI? Wir testen zwei verbreitete Tools},
     author = {Mühlhoff, Rainer and Quägwer, Sean},
     year = {2026},
     journaltitle = {SEMINAR – Lehrerbildung und Schule},
     volume = {2/2026},
     pages = {62–76},
     doi = {10.3278/SEM2602W007},
     url = {https://www.wbv.de/shop/openaccess-direct/SEM2602W007},
     urldate = {2026-07-08},
     keywords = {KI-gestützte Leistungsbewertung; Automatisiertes Feedback; Bewertungsvolatilität; Sprachmodelle im Bildungsbereich; Digitale Bildung; EdTech},
     web_thumbnail = {/assets/images/publications/Mü-Quägwer2026.jpg},
     web_group = {papers},
     web_fulltext = {https://www.wbv.de/shop/openaccess-direct/SEM2602W007}
   }