The use of generative artificial intelligence (AI) by students requires instructors to reconsider the materials and activities used to design and assess learning. This work presents Mind the Gap (MTG), an open-source hybrid AI platform that analyzes teaching materials against pedagogical requirements under educator control. The platform can be hosted and run on university infrastructure, allowing institutions to select the Large Language Models (LLMs) that best fit their needs and available resources. The design of MTG also draws on a systematic literature review conducted as part of the study, where 35,975 unique bibliographic records were examined and 78 primary studies were included. The findings indicate that few existing systems translate pedagogical and institutional guidance into executable and verifiable checks. MTG addresses this gap by translating requirements expressed in natural language into proposed rules that instructors can inspect, modify, and approve before activation. During analysis, a language model extracts the required data from documents according to an explicit contract. The code then combines these data and deterministically verifies whether the requirement is satisfied. Each reported issue remains linked to the rule that produced it, the version of the analyzed document, the values used in the check, and the text passages identified as evidence. The platform also proposes editable revision suggestions and supports comparison across successive analysis runs. The evaluation compared six open-weight language models on 200 requirement-document checks obtained by applying 40 requirements to 15 documents, covering 95 pages from a publicly available English-language course. Using manually written reference rules, agreement between the platform outputs and those of an external model used as a judge under an LLM-as-a-judge approach ranged from Cohen's κ = 0.551 to 0.723. Agreement decreased for every model when rule generation was also delegated to the models. Gemma-4-26B-A4B, selected as the platform’s default configuration, generated all 40 requirements and all expected report entries in every run. Across the eighteen reports produced, we found no references to rules outside the course and no summary counts unsupported by the values actually computed. The results also show that rule generation remains a major source of error. A rule can pass the platform’s validity checks and remain executable while testing a condition that differs from the one described in the original requirement. MTG preserves traceability for reported issues and allows users to edit proposed revisions, but the correctness of extracted data and identified evidence still requires further verification. The experiments were also conducted on materials from a single course, and the platform outputs were compared only with judgments produced by an LLM acting as evaluator. The next step is to evaluate MTG with instructors revising their own courses. This will allow us to study the relevance of reported issues, the effort required to modify and adapt the proposed suggestions, and the changes instructors actually make to their teaching materials.
L'uso dell'intelligenza artificiale (IA) generativa da parte degli studenti richiede ai docenti di ripensare i materiali e le attività utilizzate per progettare e valutare l'apprendimento. Questo lavoro presenta Mind the Gap (MTG), una piattaforma open source di IA ibrida che analizza i materiali didattici sulla base di requisiti pedagogici e sotto il controllo del docente. La piattaforma può essere ospitata ed eseguita sulle infrastrutture dell'università, permettendo alle istituzioni di scegliere i Large Language Models (LLM) più adatti alle proprie esigenze e alle risorse disponibili. La progettazione di MTG è stata effettuata anche sulla base di una revisione sistematica della letteratura, dove sono stati esaminati 35.975 record bibliografici unici e incluso 78 studi primari. I risultati mostrano che pochi sistemi esistenti trasformano le indicazioni pedagogiche e istituzionali in controlli eseguibili e verificabili. MTG affronta questa esigenza traducendo i requisiti espressi in linguaggio naturale in proposte di regole che i docenti possono esaminare, modificare e approvare prima dell'attivazione. Durante l'analisi, un modello linguistico estrae dai documenti i dati richiesti secondo un contratto esplicito. Il codice combina quindi questi dati e verifica in modo deterministico se il requisito sia soddisfatto. Ogni problema segnalato rimane collegato alla regola che lo ha prodotto, alla versione del documento analizzato, ai valori utilizzati per il controllo e ai passaggi del testo individuati come evidenze. La piattaforma propone inoltre suggerimenti di revisione modificabili e permette di confrontare i risultati di analisi successive. La valutazione ha confrontato sei modelli linguistici open-weight su 200 verifiche requisito-documento ottenute applicando 40 requisiti a 15 documenti, per un totale di 95 pagine, appartenenti a un corso in lingua inglese disponibile pubblicamente. Utilizzando regole di riferimento scritte a mano, l'accordo tra gli esiti prodotti dalla piattaforma e quelli di un modello esterno utilizzato come giudice secondo l'approccio LLM-as-a-judge varia tra Cohen's κ = 0,551 e 0,723. L'accordo diminuisce per tutti i modelli quando viene affidata loro anche la generazione delle regole. Gemma-4-26B-A4B, scelta come configurazione predefinita della piattaforma, ha generato tutti i 40 requisiti e tutte le voci di report previste in ciascuna esecuzione. Nei diciotto report prodotti non sono stati rilevati riferimenti a regole non appartenenti al corso né conteggi riepilogativi non supportati dai valori effettivamente calcolati. I risultati mostrano inoltre che la generazione delle regole rimane una fonte importante di errore. Una regola può superare i controlli di validità previsti dalla piattaforma ed essere quindi eseguibile, pur verificando una condizione diversa da quella descritta nel requisito originario. MTG mantiene tracciabili i problemi segnalati e permette di modificare i suggerimenti proposti, ma la correttezza dei dati estratti e delle evidenze individuate richiede ulteriori verifiche. Gli esperimenti sono stati inoltre condotti sui materiali di un solo corso e gli esiti della piattaforma sono stati confrontati soltanto con le valutazioni prodotte da un LLM utilizzato come giudice. Il passo successivo consiste nel valutare MTG con docenti impegnati nella revisione dei propri corsi. Questo permetterà di studiare la pertinenza dei problemi segnalati, il lavoro necessario per modificare e adattare i suggerimenti proposti e le modifiche che i docenti apportano effettivamente ai propri materiali didattici.
Intelligenza artificiale ibrida a supporto di un’interazione educativa strutturata tra persona e macchina
CROCETTI, VALERIO
2025/2026
Abstract
The use of generative artificial intelligence (AI) by students requires instructors to reconsider the materials and activities used to design and assess learning. This work presents Mind the Gap (MTG), an open-source hybrid AI platform that analyzes teaching materials against pedagogical requirements under educator control. The platform can be hosted and run on university infrastructure, allowing institutions to select the Large Language Models (LLMs) that best fit their needs and available resources. The design of MTG also draws on a systematic literature review conducted as part of the study, where 35,975 unique bibliographic records were examined and 78 primary studies were included. The findings indicate that few existing systems translate pedagogical and institutional guidance into executable and verifiable checks. MTG addresses this gap by translating requirements expressed in natural language into proposed rules that instructors can inspect, modify, and approve before activation. During analysis, a language model extracts the required data from documents according to an explicit contract. The code then combines these data and deterministically verifies whether the requirement is satisfied. Each reported issue remains linked to the rule that produced it, the version of the analyzed document, the values used in the check, and the text passages identified as evidence. The platform also proposes editable revision suggestions and supports comparison across successive analysis runs. The evaluation compared six open-weight language models on 200 requirement-document checks obtained by applying 40 requirements to 15 documents, covering 95 pages from a publicly available English-language course. Using manually written reference rules, agreement between the platform outputs and those of an external model used as a judge under an LLM-as-a-judge approach ranged from Cohen's κ = 0.551 to 0.723. Agreement decreased for every model when rule generation was also delegated to the models. Gemma-4-26B-A4B, selected as the platform’s default configuration, generated all 40 requirements and all expected report entries in every run. Across the eighteen reports produced, we found no references to rules outside the course and no summary counts unsupported by the values actually computed. The results also show that rule generation remains a major source of error. A rule can pass the platform’s validity checks and remain executable while testing a condition that differs from the one described in the original requirement. MTG preserves traceability for reported issues and allows users to edit proposed revisions, but the correctness of extracted data and identified evidence still requires further verification. The experiments were also conducted on materials from a single course, and the platform outputs were compared only with judgments produced by an LLM acting as evaluator. The next step is to evaluate MTG with instructors revising their own courses. This will allow us to study the relevance of reported issues, the effort required to modify and adapt the proposed suggestions, and the changes instructors actually make to their teaching materials.| File | Dimensione | Formato | |
|---|---|---|---|
|
Tesi___Valerio_Crocetti.pdf
embargo fino al 18/04/2028
Dimensione
4.45 MB
Formato
Adobe PDF
|
4.45 MB | Adobe PDF |
I documenti in UNITESI sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.
https://hdl.handle.net/20.500.12075/29206