Entities, Dates, and Languages: Zero-Shot on Historical Texts with T0

Francesco De Toni; Christopher Akiki; Javier de la Rosa; Clémentine Fourrier; Enrique Manjavacas; Stefan Schweter; Daniel van Strien

doi:10.48550/arXiv.2204.05211

Entities, Dates, and Languages: Zero-Shot on Historical Texts with T0

Francesco De Toni^*, Christopher Akiki, Javier de la Rosa, Clémentine Fourrier, Enrique Manjavacas, Stefan Schweter, Daniel van Strien

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

7 Citations (Scopus)

Abstract

In this work, we explore whether the recently demonstrated zero-shot abilities of the T0 model extend to Named Entity Recognition for out-of-distribution languages and time periods. Using a historical newspaper corpus in 3 languages as test-bed, we use prompts to extract possible named entities. Our results show that a naive approach for prompt-based zero-shot multilingual Named Entity Recognition is error-prone, but highlights the potential of such an approach for historical languages lacking labeled datasets. Moreover, we also find that T0-like models can be probed to predict the publication date and language of a document, which could be very relevant for the study of historical texts.

Original language	English
Title of host publication	Challenges & Perspectives in Creating Large Language Models
Subtitle of host publication	Proceedings of BigScience Episode #5 Workshop
Editors	Angela Fan, Suzana Ilic, Thomas Wolf, Matthias Galle
Place of Publication	Stroudsburg PA, USA
Publisher	Association for Computational Linguistics (ACL)
Pages	75-83
Number of pages	9
ISBN (Electronic)	978-1-955917-26-1
DOIs	https://doi.org/10.48550/arXiv.2204.05211 https://doi.org/10.18653/v1/2022.bigscience-1.7
Publication status	Published - 2022
Externally published	Yes
Event	Challenges and Perspectives in Creating Large Language Models: BigScience Episode #5 – Workshop - Virtual, Dublin, Ireland Duration: 27 May 2022 → 27 May 2022 https://aclanthology.org/2022.bigscience-1.pdf

Workshop

Workshop	Challenges and Perspectives in Creating Large Language Models
Country/Territory	Ireland
City	Virtual, Dublin
Period	27/05/22 → 27/05/22
Other	This workshop is organized by the BigScience initiative and will also serve as the closing session of this one year-long initiative aimed at developing a multilingual large language model, which is gathering 1.000+ researchers from more than 60 countries and 250 institutions and research labs. Its goal is to investigate the creation of a large scale dataset and model from a very wide diversity of angles.
Internet address	https://aclanthology.org/2022.bigscience-1.pdf

Access to Document

Cite this

De Toni, F., Akiki, C., de la Rosa, J., Fourrier, C., Manjavacas, E., Schweter, S., & van Strien, D. (2022). Entities, Dates, and Languages: Zero-Shot on Historical Texts with T0. In A. Fan, S. Ilic, T. Wolf, & M. Galle (Eds.), Challenges & Perspectives in Creating Large Language Models: Proceedings of BigScience Episode #5 Workshop (pp. 75-83). Association for Computational Linguistics (ACL). https://doi.org/10.48550/arXiv.2204.05211, https://doi.org/10.18653/v1/2022.bigscience-1.7

De Toni, Francesco ; Akiki, Christopher ; de la Rosa, Javier et al. / Entities, Dates, and Languages : Zero-Shot on Historical Texts with T0. Challenges & Perspectives in Creating Large Language Models: Proceedings of BigScience Episode #5 Workshop. editor / Angela Fan ; Suzana Ilic ; Thomas Wolf ; Matthias Galle. Stroudsburg PA, USA : Association for Computational Linguistics (ACL), 2022. pp. 75-83

@inproceedings{95b8974648494c0195f4398965ce537f,

title = "Entities, Dates, and Languages: Zero-Shot on Historical Texts with T0",

abstract = "In this work, we explore whether the recently demonstrated zero-shot abilities of the T0 model extend to Named Entity Recognition for out-of-distribution languages and time periods. Using a historical newspaper corpus in 3 languages as test-bed, we use prompts to extract possible named entities. Our results show that a naive approach for prompt-based zero-shot multilingual Named Entity Recognition is error-prone, but highlights the potential of such an approach for historical languages lacking labeled datasets. Moreover, we also find that T0-like models can be probed to predict the publication date and language of a document, which could be very relevant for the study of historical texts.",

author = "{De Toni}, Francesco and Christopher Akiki and {de la Rosa}, Javier and Cl{\'e}mentine Fourrier and Enrique Manjavacas and Stefan Schweter and {van Strien}, Daniel",

note = "Publisher Copyright: {\textcopyright} 2022 Association for Computational Linguistics.; Challenges and Perspectives in Creating Large Language Models : BigScience Episode #5 – Workshop ; Conference date: 27-05-2022 Through 27-05-2022",

year = "2022",

doi = "10.48550/arXiv.2204.05211",

language = "English",

pages = "75--83",

editor = "Angela Fan and Suzana Ilic and Thomas Wolf and Matthias Galle",

booktitle = "Challenges & Perspectives in Creating Large Language Models",

publisher = "Association for Computational Linguistics (ACL)",

address = "United States",

url = "https://aclanthology.org/2022.bigscience-1.pdf",

}

De Toni, F, Akiki, C, de la Rosa, J, Fourrier, C, Manjavacas, E, Schweter, S & van Strien, D 2022, Entities, Dates, and Languages: Zero-Shot on Historical Texts with T0. in A Fan, S Ilic, T Wolf & M Galle (eds), Challenges & Perspectives in Creating Large Language Models: Proceedings of BigScience Episode #5 Workshop. Association for Computational Linguistics (ACL), Stroudsburg PA, USA, pp. 75-83, Challenges and Perspectives in Creating Large Language Models, Virtual, Dublin, Ireland, 27/05/22. https://doi.org/10.48550/arXiv.2204.05211, https://doi.org/10.18653/v1/2022.bigscience-1.7

Entities, Dates, and Languages: Zero-Shot on Historical Texts with T0. / De Toni, Francesco; Akiki, Christopher; de la Rosa, Javier et al.
Challenges & Perspectives in Creating Large Language Models: Proceedings of BigScience Episode #5 Workshop. ed. / Angela Fan; Suzana Ilic; Thomas Wolf; Matthias Galle. Stroudsburg PA, USA: Association for Computational Linguistics (ACL), 2022. p. 75-83.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Entities, Dates, and Languages

T2 - Challenges and Perspectives in Creating Large Language Models

AU - De Toni, Francesco

AU - Akiki, Christopher

AU - de la Rosa, Javier

AU - Fourrier, Clémentine

AU - Manjavacas, Enrique

AU - Schweter, Stefan

AU - van Strien, Daniel

PY - 2022

Y1 - 2022

N2 - In this work, we explore whether the recently demonstrated zero-shot abilities of the T0 model extend to Named Entity Recognition for out-of-distribution languages and time periods. Using a historical newspaper corpus in 3 languages as test-bed, we use prompts to extract possible named entities. Our results show that a naive approach for prompt-based zero-shot multilingual Named Entity Recognition is error-prone, but highlights the potential of such an approach for historical languages lacking labeled datasets. Moreover, we also find that T0-like models can be probed to predict the publication date and language of a document, which could be very relevant for the study of historical texts.

AB - In this work, we explore whether the recently demonstrated zero-shot abilities of the T0 model extend to Named Entity Recognition for out-of-distribution languages and time periods. Using a historical newspaper corpus in 3 languages as test-bed, we use prompts to extract possible named entities. Our results show that a naive approach for prompt-based zero-shot multilingual Named Entity Recognition is error-prone, but highlights the potential of such an approach for historical languages lacking labeled datasets. Moreover, we also find that T0-like models can be probed to predict the publication date and language of a document, which could be very relevant for the study of historical texts.

UR - http://www.scopus.com/inward/record.url?scp=85137734213&partnerID=8YFLogxK

U2 - 10.48550/arXiv.2204.05211

DO - 10.48550/arXiv.2204.05211

M3 - Conference contribution

AN - SCOPUS:85137734213

SP - 75

EP - 83

BT - Challenges & Perspectives in Creating Large Language Models

A2 - Fan, Angela

A2 - Ilic, Suzana

A2 - Wolf, Thomas

A2 - Galle, Matthias

PB - Association for Computational Linguistics (ACL)

CY - Stroudsburg PA, USA

Y2 - 27 May 2022 through 27 May 2022

ER -

De Toni F, Akiki C, de la Rosa J, Fourrier C, Manjavacas E, Schweter S et al. Entities, Dates, and Languages: Zero-Shot on Historical Texts with T0. In Fan A, Ilic S, Wolf T, Galle M, editors, Challenges & Perspectives in Creating Large Language Models: Proceedings of BigScience Episode #5 Workshop. Stroudsburg PA, USA: Association for Computational Linguistics (ACL). 2022. p. 75-83 doi: 10.48550/arXiv.2204.05211, 10.18653/v1/2022.bigscience-1.7

Entities, Dates, and Languages: Zero-Shot on Historical Texts with T0

Abstract

Workshop

Access to Document

Other files and links

Fingerprint

Cite this