Alert button

GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLP

May 24, 2023
Md Tawkat Islam Khondaker, Abdul Waheed, El Moatez Billah Nagoudi, Muhammad Abdul-Mageed

Figure 1 for GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLP
Figure 2 for GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLP
Figure 3 for GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLP
Figure 4 for GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLP

Share this with someone who'll enjoy it:

The recent emergence of ChatGPT has brought a revolutionary change in the landscape of NLP. Although ChatGPT has consistently shown impressive performance on English benchmarks, its exact capabilities on most other languages remain largely unknown. To better understand ChatGPT's capabilities on Arabic, we present a large-scale evaluation of the model on a broad range of Arabic NLP tasks. Namely, we evaluate ChatGPT on 32 diverse natural language understanding and generation tasks on over 60 different datasets. To the best of our knowledge, our work offers the first performance analysis of ChatGPT on Arabic NLP at such a massive scale. Our results show that, despite its success on English benchmarks, ChatGPT trained in-context (few-shot) is consistently outperformed by much smaller dedicated models finetuned on Arabic. These results suggest that there is significant place for improvement for instruction-tuned LLMs such as ChatGPT.

* Work in progress  
View paper onarxiv icon

Share this with someone who'll enjoy it: