We tested Fobizz’s “AI Grading Assistant”: New Paper and 38C3 Presentation

Chatbots in the classroom? We tested a tool from the German market leader Fobizz and find serious shortcomings.

Our study examines the AI-powered grading tool “AI Grading Assistant” by the German company Fobizz, designed to support teachers in evaluating and providing feedback on student assignments. Against the societal backdrop of an overburdened education system and rising expectations for artificial intelligence as a solution to these challenges, the investigation evaluates the tool’s functional suitability through two test series. The results reveal significant shortcomings: The tool’s numerical grades and qualitative feedback are often random and do not improve even when its suggestions are incorporated. The highest ratings are achievable only with texts generated by ChatGPT. False claims and nonsensical submissions frequently go undetected, while the implementation of some grading criteria is unreliable and opaque. Since these deficiencies stem from the inherent limitations of large language models (LLMs), fundamental improvements to this or similar tools are not immediately foreseeable. The study critiques the broader trend of adopting AI as a quick fix for systemic problems in education, concluding that Fobizz’s marketing of the tool as an objective and time-saving solution is misleading and irresponsible. Finally, the study calls for systematic evaluation and subject-specific pedagogical scrutiny of the use of AI tools in educational contexts.

Read the paper:

1. 

   Mühlhoff, Rainer, und Marte Henningsen. 2024. „Chatbots im Schulunterricht: Wir testen das Fobizz-Tool zur automatischen Bewertung von Hausaufgaben“. doi:10.48550/arXiv.2412.06651.

   Full text pre-print bibtex

   ×

   @online{Mü-Henningsen2024,
     title = {Chatbots im Schulunterricht: Wir testen das Fobizz-Tool zur automatischen Bewertung von Hausaufgaben},
     shorttitle = {Chatbots im Schulunterricht},
     author = {Mühlhoff, Rainer and Henningsen, Marte},
     date = {2024-12-10},
     eprint = {2412.06651},
     eprinttype = {arXiv},
     eprintclass = {cs},
     doi = {10.48550/arXiv.2412.06651},
     url = {http://arxiv.org/abs/2412.06651},
     urldate = {2024-12-11},
     pubstate = {prepublished},
     keywords = {Computer Science - Computers and Society},
     web_thumbnail = {/assets/images/publications/Mü-Henningsen2024.jpg},
     web_group = {aktuell},
     web_fulltext = {http://arxiv.org/pdf/2412.06651},
     web_preprint = {http://arxiv.org/abs/2412.06651}
   }
   

Presentation at Chaos Computer Club – 38C3

At the annual meeting #38C3 of Chaos Computer Club, 29 Decemver 2024, in Hamburg, Germany, Marte Henningsen and Rainer Mühlhoff present the study. The presentation will also be available as a video (live and recording):

Event website