Do not trust the experts: How the lack of standard complicates NLP for historical Irish

Oksana Dereza, Theodorus Fransen, and John P. Mccrae. 2023. Do not Trust the Experts - How the Lack of Standard Complicates NLP for Historical Irish. In Proceedings of the Fourth Workshop on Insights from Negative Results in NLP, pages 82–87, Dubrovnik, Croatia. Association for Computational Linguistics.

Abstract

In this paper, we describe how we unearthed some fundamental problems while building an analogy dataset modelled on BATS (Gladkova et al., 2016) to evaluate historical Irish embeddings on their ability to detect orthographic, morphological and semantic similarity.performance of our models in the analogy task was extremely poor regardless of the architecture, hyperparameters and evaluation metrics, while the qualitative evaluation revealed positive tendencies. argue that low agreement between field experts on fundamental lexical and orthographic issues, and the lack of a unified editorial standard in available resources make it impossible to build reliable evaluation datasets for computational models and obtain interpretable results. We emphasise the need for such a standard, particularly for NLP applications, and prompt Celticists and historical linguists to engage in further discussion. We would also like to draw NLP scholars’ attention to the role of data and its (extra)linguistic properties in testing new models, technologies and evaluation scenarios.

Funder

Irish Research Council
Science Foundation Ireland

Publisher

Association for Computational Linguistics

Rights

Attribution-NonCommercial-NoDerivatives 4.0 International

cbn

Collections

Data Science Institute (Workshop Papers)