CALM: Foundations for causality-aware language models for causal question-answering

Causality is a crucial aspect of human rationality, and causal reasoning is a critical element in the development of our mental models of reality. Theories of causality drive scientific inquiry and pervade all dimensions of our daily lives, from the minutiae of deciding what to wear based on weather predictions to medical diagnoses for experienced ailments. Recent advances in language models and AI technologies have demonstrated remarkable performance across a wide range of reasoning and QA tasks, resulting in promises of human-level general intelligence. However, the causal reasoning capabilities of such models remain underexplored, as prior research has been disparate and limited to narrow investigations of causal question-answering (causal QA) tasks. In this thesis, we aim to unify causal QA research, measure the baseline causal reasoning capabilities of language models, and propose foundational resources for the development of Causality-Aware Language Models (CALM) which are effective across diverse causal QA tasks. Inspired by cognitive theories and philosophical accounts of reasoning, we posit a unified definition of causal QA aligned with human causality. Specifically, we introduce CALM-Bench, a multi-task benchmark that consists of diverse causal QA tasks, ranging from causal abduction to effect quantification, and conduct extensive transfer learning experiments. We then introduce CALM-Schema, a semantic schema for the organization of causal knowledge as causal systems, and CALM-KB, the first synthetically generated knowledge base consisting of approximately 5.4K causal systems. CALM-KB is extensively validated in the knowledge injection and RAG settings, and we further detail the benefits and limitations of causal knowledge across common causal reasoning categories. Finally, we investigate the causal explanation capabilities of LLMs and propose IBE-Eval, an interpretable framework for the automatic evaluation of LLM-generated causal explanations. Our research lays the groundwork for future studies on language model causal reasoning and introduces foundational resources for the advancement of CALM models.

Funder

Research Ireland

Publisher

University of Galway

Rights

Attribution-NonCommercial-NoDerivatives 4.0 International

cbn

Collections

University of Galway Theses (PhD Theses)