Alert button

A Statistical Model of Word Rank Evolution

Jul 21, 2021
Alex John Quijano, Rick Dale, Suzanne Sindi

Figure 1 for A Statistical Model of Word Rank Evolution
Figure 2 for A Statistical Model of Word Rank Evolution
Figure 3 for A Statistical Model of Word Rank Evolution
Figure 4 for A Statistical Model of Word Rank Evolution

Share this with someone who'll enjoy it:

The availability of large linguistic data sets enables data-driven approaches to study linguistic change. This work explores the word rank dynamics of eight languages by investigating the Google Books corpus unigram frequency data set. We observed the rank changes of the unigrams from 1900 to 2008 and compared it to a Wright-Fisher inspired model that we developed for our analysis. The model simulates a neutral evolutionary process with the restriction of having no disappearing words. This work explains the mathematical framework of the model - written as a Markov Chain with multinomial transition probabilities - to show how frequencies of words change in time. From our observations in the data and our model, word rank stability shows two types of characteristics: (1) the increase/decrease in ranks are monotonic, or (2) the average rank stays the same. Based on our model, high-ranked words tend to be more stable while low-ranked words tend to be more volatile. Some words change in ranks in two ways: (a) by an accumulation of small increasing/decreasing rank changes in time and (b) by shocks of increase/decrease in ranks. Most of the stopwords and Swadesh words are observed to be stable in ranks across eight languages. These signatures suggest unigram frequencies in all languages have changed in a manner inconsistent with a purely neutral evolutionary process.

* This manuscript - with 53 pages and 28 figures - is a draft for a journal research article submission  
View paper onarxiv icon

Share this with someone who'll enjoy it: