In this notebook, we explore whether it is possible to build a binary classifier for surahs based on the words used in their verses.
import numpy as np, pandas as pd
verses = pd.read_csv("data/verses.txt", header=0, delimiter="|", quoting=3, encoding='utf-8')
labels = np.genfromtxt("data/surah-labels.csv", delimiter=",")This step can take up to a minute or so to run. It can be optimized but its not necessary for a fixed corpus, like the text of the Quran
# -*- coding: utf-8 -*-
n_verses = verses.shape[0]
word_bag = list()
valid_verse_counter = 0
for verse in range(n_verses):
if type(verses["Text"][verse]) is str:
valid_verse_counter += 1
for word in verses["Text"][verse].split(" "):
word_bag.append(word)
print("The Quran has",len(word_bag)," words.")
word_bag = list(set(word_bag))
print("The Quran has",len(word_bag),"unique words, allowing for vowel variations.")
word_vectors = np.zeros(shape=[valid_verse_counter,len(word_bag)])
verse_labels = np.zeros(shape=[valid_verse_counter])-1
verse_chapter_mapping = np.zeros(shape=[valid_verse_counter])
valid_verse_counter = 0
for verse in range(n_verses):
if type(verses["Text"][verse]) is str:
for word in verses["Text"][verse].split(" "):
index = word_bag.index(word)
word_vectors[valid_verse_counter,index] += 1
#apply labels to individual verses
ch_number = int(verses["Chapter"][verse])
verse_labels[valid_verse_counter] = labels[ch_number-1,1] #-1 for 0-indexed np array
#create a mapping between verses and chapters
verse_chapter_mapping[valid_verse_counter] = ch_number
valid_verse_counter += 1
print("The Quran has approximately",np.count_nonzero(verse_labels)," Meccan verses.")
word_bag = list(set(word_bag))
print("The Quran has approximately",len(verse_labels)-np.count_nonzero(verse_labels)," Medinan verses.")
print("This is approximate because some Meccan surahs include Medinan verses and vice versa")The Quran has 78245 words.
The Quran has 14870 unique words, allowing for vowel variations.
The Quran has approximately 4613 Meccan verses.
The Quran has approximately 1623 Medinan verses.
This is approximate because some Meccan surahs include Medinan verses and vice versa
#split based on surah level (to preserve independence between training and validation set)
def partition(features, labels, verse_chapter_mapping, train_fraction=0.4):
n_chapters = int(np.max(verse_chapter_mapping))
n_train = int(train_fraction*n_chapters)
n_valid = n_chapters - n_train
chapters = np.random.permutation(n_chapters) + 1 #zero-indexed np array
train_chapters, valid_chapters = chapters[:n_train], chapters[n_train:]
train_indices = np.where(np.in1d(verse_chapter_mapping,train_chapters))[0]
valid_indices = np.where(np.in1d(verse_chapter_mapping,valid_chapters))[0]
features_train = features[train_indices]
labels_train = labels[train_indices]
features_valid = features[valid_indices]
labels_valid = labels[valid_indices]
return (features_train, labels_train, features_valid, labels_valid)
features_train, labels_train, features_valid, labels_valid = partition(word_vectors, verse_labels, verse_chapter_mapping)
print("The training set has:",len(labels_train),"verses")
print("The validation set has:",len(labels_valid),"verses")The training set has: 2357 verses
The validation set has: 3879 verses
from sklearn.linear_model import LogisticRegression
model = LogisticRegression()
model = model.fit(features_train, labels_train)# check the accuracy on the training set
print('We achieve', str(round(100*model.score(features_train, labels_train), 2))+'% accuracy on the training set')
print('We achieve', str(round(100*model.score(features_valid, labels_valid), 2))+'% accuracy on the validation set')We achieve 98.68% accuracy on the training set
We achieve 86.26% accuracy on the validation set
Given the large number of features, its not surprising that there is some overfitting happening. However, our results on the validation set do suggest that the classifier is learning something. How much better is this than chance? Well, we started with an unbalanced dataset, so if our classifier was just classifying every verse as Meccan, it would get an accuracy of about 74%. Depending on the specific partitioning of the training and validation, this is around 10 percentage points higher than that.
We can also see how well the classifier can predict an entire surah is Meccan or Medinan. This is also more fair since many Meccan surahs include Medinan verses and vice versa, which we don't take into accout when training or measuring the performance of our results.
We will simply use majority voting to decide if a surah is Meccan or Medinan. If a majority of a surahs' verses are Meccan, the entire surah will be Meccan and same for Medinan.
chapter_predictions = np.zeros(shape=[int(np.max(verse_chapter_mapping)), 2])-1
for chapter in range(1,int(np.max(verse_chapter_mapping))+1):
#get all the verse of that chapter
verses = np.where(verse_chapter_mapping==chapter)[0]
verse_predictions = model.predict(word_vectors[verses])
#see what majority of voters say
chapter_pred = np.round(np.mean(verse_predictions))
chapter_conf = np.mean(verse_predictions)
chapter_predictions[chapter-1, 0] = chapter_pred
chapter_predictions[chapter-1, 1] = round(100*(0.5+abs(0.5-chapter_conf))
#compare to actual chapter labels
pred_errors = chapter_predictions[:,0] - labels[:,1]
print("Surahs It Misclassified as Meccan:")
print("---------------")
for i in np.where(pred_errors>0)[0]:
print(i+1, " -- % Verses:", str(chapter_predictions[i,1]))
print("\nSurahs It Misclassified as Medinan:")
print("---------------")
for i in np.where(pred_errors<0)[0]:
print(i+1)Surahs It Misclassified as Meccan:
---------------
13 -- % Verses: 81.0
22 -- % Verses: 63.0
47 -- % Verses: 53.0
55 -- % Verses: 100.0
76 -- % Verses: 97.0
99 -- % Verses: 100.0
Surahs It Misclassified as Medinan:
---------------
Further insight can be obtained by examining the weights in the logistic regression model. For example, we can see what words are most associated with a a Meccan surah, and what words are associated with a Medinan surah.
TOP_N = 10
weights = model.coef_.flatten()
idx = np.argpartition(weights, -TOP_N)[-TOP_N:]
print("Top Meccan Words\n------------------")
for i in idx:
print(word_bag[i])
idx = np.argpartition(weights, TOP_N)[:TOP_N]
print("Top Medinan Words\n------------------")
for i in idx:
print(word_bag[i])Top Meccan Words
------------------
بعهدكم
وتقسطوا
العالمون
أولاهما
لتحصنكم
العمى
نور
لمستقر
لنفد
تنزيل
Top Medinan Words
------------------
تقتلني
والله
أكلها
وإذ
وبكفرهم
مبصرة
زلزلة
بآية
سخرناها
وجهرا
def weight_of_word(word):
i = word_bag.index(word)
w = model.coef_.flatten()[i]
mx = max(abs(max(model.coef_.flatten())), abs(min(model.coef_.flatten())))
prob = (w + mx) / (2*mx)
print("The weight of this word is:", w)
print("A very rough probability that the word is Meccan:", prob)
weight_of_word("الشمس")The weight of this word is: 0.164471590085
A very rough probability that the word is Meccan: 0.536413190447