paper
ACL-Verbatim: hallucination-free question answering for research
Abstract
Academic researchers need efficient and reliable methods for collecting high-quality information from trusted sources, but modern tools for AI-assisted research still suffer from the tendency of Large Language Models (LLMs) to produce factually inaccurate or nonsensical output, commonly referred to as hallucinations. We apply the extractive question answering system VerbatimRAG to research papers in the ACL Anthology, directly mapping user queries to verbatim text spans in retrieved documents. We contribute a novel ground truth dataset for the task of mapping user queries to relevant text spans in research papers, and use it to train and evaluate a variety of extractive models. Human annotation is performed by NLP researchers and is based on synthetic user queries generated using a custom pipeline based on the ScIRGen methodology, paired with chunks of research papers retrieved by VerbatimRAG. On this benchmark, a 150M-parameter ModernBERT token classifier trained on silver supervision from our pipeline achieves the best word-level F1 (53.6), ahead of the strongest evaluated LLM extractor (48.7).
TL;DR
- VerbatimRAG is applied to 110K+ papers of the ACL Anthology
- All papers are released in markdown format. A small sample of query-paper pairs is manually annotated for the extraction (highlighting) task
- Custom extractor models are trained on LLM-generated silver extraction data and outperform all LLMs on word-level F-score
- Open-source library at github.com/KRLabsOrg/acl-verbatim
Resources
For the full text of the paper, follow the paper link above. The open-source library is at https://github.com/KRLabsOrg/acl-verbatim and includes links to all models and datasets.
Cite
@misc{Recski:2026,
title={ACL-Verbatim: hallucination-free question answering for research},
author={Gábor Recski and Szilveszter Tóth and Nadia Verdha and István Boros and Ádám Kovács},
year={2026},
eprint={2605.21102},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2605.21102},
}