Retrieval-Augmented Question Answering using the EU AI Act

Generative Artificial Intelligence systems have significantly improved question answering capabilities; however, applying these systems to legal documents remains challenging due to the complexity of legal semantics and the need for relevant regulatory provisions retrieval accuracy. Retrieval-Augmented Generation (RAG) has emerged as a promising approach to address the challenge by combining language models with external knowledge sources. In this paper, we investigate the use of LightRAG, a graph-based Retrieval-Augmented Generation framework, for question answering over the European Union AI Act. Additionally, we examine the impact of incorporating a domain-specific ontology to guide entity extraction and retrieval, comparing the performance of the framework with and without ontology-based guidance. The evaluation follows two steps: 1) First, it uses ten expert-reviewed legal questions, including their answers and supporting legal references. The retrieved legal references from both configurations are compared against the expert validated legal references. 2) Second, it uses ten other legal queries without answers and regulatory references. The regulatory provisions identified by two LightRAG configurations are compared qualitatively with the outputs of GraphReader to assess the retrieval characteristics of the three approaches.