Code for Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP

Add code button
avatar
Github Iconmoses-smt/mosesdecoder/blob/master/scripts/tokenizer/tokenizer.perl

Human or AI?

Log in/sign up for free to see all code implementations