[M]alaysian [E]nglish [N]ews Articles Dataset (MEN-Dataset) for Named Entity Recognition and Relation Extraction

Mohan Raj Chanthran*†, Soon Lay Ki†, Ong Huey Fang†† Bhawani Selvaretnam‡
†Monash University Malaysia
‡ Valiantlytix Sdn Bhd
LREC-COLING 2024
Paper Annotation Guideline Code arXiv

Abstract

Standard English and Malaysian English exhibit notable differences, posing challenges for natural language processing (NLP) tasks on Malaysian English. Unfortunately, most of the existing datasets are mainly based on standard English and therefore inadequate for improving NLP tasks in Malaysian English. An experiment using state-of-the-art Named Entity Recognition (NER) solutions on Malaysian English news articles highlights that they cannot handle morphosyntactic variations in Malaysian English. To the best of our knowledge, there is no annotated dataset available to improvise the model. To address these issues, we constructed a Malaysian English News (MEN) dataset, which contains 200 news articles that are manually annotated with entities and relations. We then fine-tuned the spaCy NER tool and validated that having a dataset tailor-made for Malaysian English could improve the performance of NER in Malaysian English significantly. This paper presents our effort in the data acquisition, annotation methodology, and thorough analysis of the annotated dataset. To validate the quality of the annotation, inter-annotator agreement was used, followed by adjudication of disagreements by a subject matter expert. Upon completion of these tasks, we managed to develop a dataset with 6,061 entities and 3,268 relation instances. Finally, we discuss on spaCy fine-tuning setup and analysis on the NER performance. This unique dataset will contribute significantly to the advancement of NLP research in Malaysian English, allowing researchers to accelerate their progress, particularly in NER and RE

Dataset Statistics

Statistics of total entities annotated
Statistics of total relations annotated

Model Performance

Model Performance after Fine-Tuning with MEN-Dataset

Evaluating ChatGPT on Malaysian English

*This work has been accepted in Generation, Evaluation & Metrics (GEM) Workshop at EMNLP 2023


ChatGPT has attracted a lot of interest from both researchers and the general public. In this study, we assess ChatGPT's capability in extracting entities and relations from the Malaysian English News (MEN) dataset. We propose a three-step methodology referred to as educate-predict-evaluate. The performance of ChatGPT is assessed using F1-Score across 18 unique prompt settings, which were carefully engineered for a comprehensive review.

Paper Code arXiv

BibTeX

@misc{chanthran2024malaysian,
        title={Malaysian English News Decoded: A Linguistic Resource for Named Entity and Relation Extraction}, 
        author={Mohan Raj Chanthran and Lay-Ki Soon and Huey Fang Ong and Bhawani Selvaretnam},
        year={2024},
        eprint={2402.14521},
        archivePrefix={arXiv},
        primaryClass={cs.CL}
      }
@misc{chanthran2023chatgpt,
        title={How well ChatGPT understand Malaysian English? An Evaluation on Named Entity Recognition and Relation Extraction},
        author={Mohan Raj Chanthran and Lay-Ki Soon and Huey Fang Ong and Bhawani Selvaretnam},
        year={2023},
        eprint={2311.11583},
        archivePrefix={arXiv},
        primaryClass={cs.CL}
    }