Urdu Tokenization using SpaCy

SpaCy is an NLP library which supports many languages. It’s fast and has DNNs build in for performing many NLP tasks such as POS and NER. It has extensive support and good documentation. It is fast and provides GPU support and can be integrated with Tensorflow, PyTorch, Scikit-Learn, etc.

SpaCy provides the easiest way to add any language support. A new language can be added by simply following Adding Languages article. I’ve added the Urdu language with dictionary-based lemmatization, lexical support and stop words(Urdu). Here is how you can use the tokenizer for the Urdu language.

First, install SpaCy.

$ pip install spacy

Now import spacy and create a blank object with support of Urdu language. I’m using blank because there is no proper model available for Urdu yet, but tokenization support available.

import spacy

nlp = spacy.blank('ur')

doc = nlp("کچھ ممالک ایسے بھی ہیں جہاں اس برس روزے کا دورانیہ 20 گھنٹے تک ہے۔")

print("Urdu Tokenization using SpaCy")

for word in doc:
    print(word)

Here is the output:

کچھ
ممالک
ایسے
بھی
ہیں
جہاں
اس
برس
روزے
کا
دورانیہ
20
گھنٹے
تک
ہے
۔

Note that Urdu has different punctuation symbols such as ۔ ، etc and it also uses English numbers 12 etc. Accuracy is 100% for Urdu language tokenization.

If you have any question feel free to ask in comments.