Python ISRIStemmer for Arabic text

Question

I am running the following code on IDLE(Python) and I want to enter Arabic string and get the stemming for it but actually it doesn't work

>>> from nltk.stem.isri import ISRIStemmer
>>> st = ISRIStemmer()
>>> w= 'حركات'
>>> join = w.decode('Windows-1256')
>>> print st.stem(join).encode('Windows-1256').decode('utf-8')

The result of running it is the same text in w which is 'حركات' which is not the stem

But when do the following:

>>> print st.stem(u'اعلاميون')

The result succeeded and returns the stem which is 'علم'

Why passing some words to stem() function doesn't return the stem?

Answer 1

Ok, I solved the problem by myself using the following:

w = 'حركات' 
st.stem(w.decode('utf-8'))

and it gives the root correctly which is "حرك"

Answer 2

This code above won't work in Python 3 because we are trying to decode an object that is already decoded. So, there is no need to decode from UTF-8 anymore.

Here is the new code that should work just fine in Python 3.

import nltk
from nltk.stem.isri import ISRIStemmer
st = ISRIStemmer()
w= 'حركات'
print(st.stem(w))

Answer 3

there is a new light arabicstemmer here developed with snowball framework

Answer 4

You can use this snippet to directly stem text:

from nltk import word_tokenize

from nltk.stem.isri import ISRIStemmer

st = ISRIStemmer()

w= " البحث العلمي أو البحث أو التجربة التنموية هو أسلوب منظم في جمع المعلومات الموثوقة وتدوين الملاحظات والتحليل الموضوعي لتلك المعلومات باتباع أساليب ومناهج علمية محددة بقصد التأكد من صحتها أو تعديلها أو إضافة الجديد لها، ومن ثم التوصل إلى بعض القوانين والنظريات والتنبؤ بحدوث مثل هذه الظواهر والتحكم في أسبابها"

for a in word_tokenize(w):

    print(st.stem(a))

Answer 5

Well, just notice that your two strings actually only differ by a mere "u" at the beginning of the second string :

w = 'حركات'
w2 = u'اعلاميون'

But that tiny "u" made all the difference : w is a UTF8 string (default character encoding in Python), while w2 is a Unicode string.

Hence all you really need to do is make sure your string is defined as a Unicode string, and then you can use the stem function normally without any extra decoding step :

w = u'حركات'
print st.stem(w)

Answer 6

Here is another example on how to use the stemmer (of course you can remove stop words first!)

from nltk import word_tokenize
from nltk.stem.isri import ISRIStemmer

st = ISRIStemmer()
word_list = "عرض يستخدم الى التفاعل مع المستخدمين في هاذا المجال !وعلمآ تكون الخدمه للستطلاع على الخدمات والعروض المقدمة"

# Define a function
def filter(word_list):
    wordsfilter=[]
    for a in word_tokenize(word_list):
        stem = st.stem(a)
        wordsfilter.append(stem)
    print(wordsfilter)

# Call the function
filter(word_list)

Here is the result:

['عرض', 'خدم', 'الى', 'فعل', 'مع', 'خدم', 'في', 'هذا', 'جال', '!', 'علمآ', 'تكون', 'خدم', 'طلع', 'على', 'خدم', 'عرض', 'قدم']