Analyzing the Biblia Hebraica Stuttgartensia with Text Fabric in Python

I am very curious about Hebrew, specially the way the language is hierarchically designed. Put very simply, three letter words are commonly known as trilateral roots and the rest of the words are variation of these three letters having a common root word and having related meaning somehow, thus creating a “family” if you will. I am not going to get into these details since it is not the scope of this post.

Long story short, the journey of trying to learn Hebrew started probably between 2006 to 2008 and I haven’t had much success at it. Or lets say that the expectations don’t match reality due to lack of consistent effort on my side.

I thought that I could try to somehow start with the most common words in the scriptures and that is what motivated to parse the Aleppo codex and analyze its content in python. I quickly learned that my approach was rather naive given the prefixes that causes variations to the words, even thought the meaning of the word itself is the same.

This blog post is a second attempt to tackle the same problem but with a more sophisticated and accurate approach using Text Fabric.

A corpus of ancient texts and (linguistic) annotations represents a large body of knowledge.

Text-Fabric is a Python package for processing and access a corpus of ancient text and linguistic annotations. In this specific case I am using the Hebrew Bible Database, containing the text of the Hebrew Bible augmented with linguistic annotations compiled by the Eep Talstra Centre for Bible and Computer from the VU University Amsterdam.

The text is based on the Biblia Hebraica Stuttgartensia edited by Karl Elliger and Wilhelm Rudolph, Fifth Revised Edition, edited by Adrian Schenker, © 1977 and 1997 Deutsche Bibelgesellschaft, Stuttgart.

The text-fabric version has been prepared by Dirk Roorda Data Archiving and Networked Services, with thanks to Martijn Naaijer, Cody Kingham, and Constantijn Sikkel.

It is amazing to see the work these researchers did compiling all this data and making it public for free. I am very thankful for it.

I am using a literate programming approach with Doom Emacs and org-mode. This blog post is a bit more technical oriented. So it is okay if you pass through the code or if you don’t understand some of it.

Create the virtual environment #

This snippet will create a virtual environment and install some libs. Due to some incompatibilities with the word cloud lib, I had to downgrate do Python 3.6.

#virtualenv ~/.workon-home/venv-textfabric
virtualenv --python=/usr/bin/python3.6 ~/.workon-home/venv-textfabric
cd ~/.workon-home/venv-textfabric
source ./bin/activate.fish
pip install text-fabric pandas requests

Activate the virtual environment #

Use C-c here on Doom Emacs

(pyvenv-activate "~/.workon-home/venv-textfabric")

Load the ETCBC/bhsa dataset #

Use run-python and ober-eval-block-in-repl to evaluate each block (C-c r at the time being). For the sake of a more practical approach to writing using literate programming the output of the blocks will follow the code. The documentation of the BHSA dataset can be found here.

from tf.app import use
import os
import collections
from itertools import chain

A = use("ETCBC/bhsa", hoist=globals())

A.indent(reset=True)
A.info("counting objects ...")

for otype in F.otype.all:
    i = 0

    A.indent(level=1, reset=True)

    for n in F.otype.s(otype):
        i += 1

    A.info("{:>7} {}s".format(i, otype))

A.indent(level=0)
A.info("Done")

<IPython.core.display.HTML object>
<IPython.core.display.HTML object>
<IPython.core.display.HTML object>
<IPython.core.display.HTML object>
This is Text-Fabric 9.5.2
Api reference : https://annotation.github.io/text-fabric/tf/cheatsheet.html

122 features found and 0 ignored
<IPython.core.display.HTML object>
<IPython.core.display.HTML object>
<IPython.core.display.HTML object>
<IPython.core.display.HTML object>
  0.00s counting objects ...
   |     0.00s      39 books
   |     0.00s     929 chapters
   |     0.00s    9230 lexs
   |     0.00s   23213 verses
   |     0.00s   45179 half_verses
   |     0.00s   63717 sentences
   |     0.00s   64514 sentence_atoms
   |     0.00s   88131 clauses
   |     0.00s   90704 clause_atoms
   |     0.01s  253203 phrases
   |     0.01s  267532 phrase_atoms
   |     0.01s  113850 subphrases
   |     0.02s  426590 words
  0.07s Done

Analyzing the type of word structures available #

Text-Fabric allows to query the “types” of words. We can do this by using:

print(F.sp.freqList())

(('subs', 125583), ('verb', 75451), ('prep', 73298), ('conj', 62737), ('nmpr', 35607), ('art', 30387), ('adjv', 10141), ('nega', 6059), ('prps', 5035), ('advb', 4603), ('prde', 2678), ('intj', 1912), ('inrg', 1303), ('prin', 1026))

Defining some help functions #

These functions will facilitate the processing of the data. The first one will get the occurrences of a specific type of lexeme. I am not a linguist so I am learning on the go and I will try to explain it easily. The most common lexeme verb is אמר (say). It appears 5307 times. We can see that Yah indeed wants to communicate and instruct us.

A specific instance of that verb has other properties (node features) like morphology which contains the verbal stem (qal, piel, nif, hif), the verbal tense (perf, impf, wayq) and gender (m, f) among other information. A lexeme is the representation of that word (verb in this case) in the broad aspect, i.e regardless of the verbal stem, tense, gender, etc.

import pandas as pd

def get_lexeme_by_type(lex_type):
    rows = []
    A.indent(reset=True)
    for w in F.otype.s("lex"):
        if F.sp.v(w) != lex_type:
            continue
        row = {'w': w,
               'freq_lex': F.freq_lex.v(w),
               #'sp': F.sp.v(w),
               'lex_utf8': F.lex_utf8.v(w),
               'gloss': F.gloss.v(w),
               #'phono': F.phono.v(w),
               #'g_word_utf8': F.g_word_utf8.v(w),
               #'g_lex_utf8': F.g_lex_utf8.v(w),
               #'g_cons_utf8': F.g_cons_utf8.v(w),
               #'gn': F.gn.v(w),
               #'nu': F.nu.v(w),
               #'ps': F.ps.v(w),
               #'st': F.st.v(w),
               #'vs': F.vs.v(w),
               #'vt': F.vt.v(w),
               #'book': F.book.v(w),
               }
        rows.append(row)
    df = pd.DataFrame(rows)
    return(df)

def export_df_to_org_table(input_df, rows_qty):
    input_df["lex_utf8"] = input_df.apply(lambda x: remove_diacritics(x["lex_utf8"]), axis=1)
    output_df = input_df.head(rows_qty)
    return([list(output_df)] + [None] + output_df.values.tolist())

Most frequent prepositions #

df_preps = get_lexeme_by_type('prep')
df_cloud = df_preps.sort_values(by=['freq_lex'], ascending=False)
print_df = export_df_to_org_table(df_cloud, 100)

Fair enough, this is a good starting point to learn some words.

Most frequent names (people and places) #

Now lets see the 30 most cited names in scriptures

df_nmpr = get_lexeme_by_type('nmpr')
df_cloud = df_nmpr.sort_values(by=['freq_lex'], ascending=False)
print_df = export_df_to_org_table(df_cloud, 30)

And now lets generate a cloud image for in the Hebrew

And also a cloud image for the English translations of those names

To me this is very exciting, with just some lines of code (and of course standing in the shoulders of giants that made TF possible and the BHSA dataset), we are able to perform simple tasks like this.

Most frequent verbs #

I want more, so lets repeat the same procedure but with verbs now. And lets increase the amount of verbs to 100 verbs sorted by frequency.

df_verbs = get_lexeme_by_type('verb')
df_cloud = df_verbs.sort_values(by=['freq_lex'], ascending=False)
print_df = export_df_to_org_table(df_cloud, 100)

Here we can see the id of the lexeme (w), the frequency (freq_lex), the utf8 representation of the word without vowel pointing and the english meaning used for the translation. All automatically, pretty cool isn’t it?

Now lets create word clouds out of those verbs.

First, the top 30 verbs in Hebrew:

Now lets try the 50 most common verbs in Hebrew:

And now lets increase the amount to 100 verbs in Hebrew:

Now lets generate a cloud words image for the top 30 verbs in English:

Similarly, for the 50 top verbs in English:

And finally, the top 100 top verbs in English:

I have noticed something “unexpected” here with vowel pointing¹. The word “Turn” is the bigger one in english, yet is not the most frequent word in the table. Why is that? Although I am not completely sure about it, I believe it is due that it appears twice on the list since there are two Hebrew words translated as turn with different amount of frequencies. I have not taken that into account in the code. This doesn’t happen with the Hebrew words since the frequency would already be increased in the freq_lex column.

To solve this, I would probably have to parse the pandas dataframe to manually sum the frequencies of the same english word. But I won’t do that right now since I am more curious about exploring other words.

Furthermore, the “mistake” is rather interesting and I believe it speaks volumes in itself, since I truly believe that the will of the heart of our Father goes towards teshuva indeed, turn of our own ways. Seeing this unexpected event was rather interesting to me.

Most frequent substantives #

df = get_lexeme_by_type('subs')
df_cloud = df.sort_values(by=['freq_lex'], ascending=False)
print_df = export_df_to_org_table(df_cloud, 100)

Most frequent conjunctions #

df = get_lexeme_by_type('conj')
df_cloud = df.sort_values(by=['freq_lex'], ascending=False)
print_df = export_df_to_org_table(df_cloud, 100)

Most frequent adjectives #

df = get_lexeme_by_type('adjv')
df_cloud = df.sort_values(by=['freq_lex'], ascending=False)
print_df = export_df_to_org_table(df_cloud, 100)

Most frequent pronouns #

df = get_lexeme_by_type('prps')
df_cloud = df.sort_values(by=['freq_lex'], ascending=False)
print_df = export_df_to_org_table(df_cloud, 100)

Most frequent adverbs #

df = get_lexeme_by_type('advb')
df_cloud = df.sort_values(by=['freq_lex'], ascending=False)
print_df = export_df_to_org_table(df_cloud, 100)

Most frequent demonstrative pronouns #

df = get_lexeme_by_type('prde')
df_cloud = df.sort_values(by=['freq_lex'], ascending=False)
print_df = export_df_to_org_table(df_cloud, 100)

Most frequent interrogative particles #

df = get_lexeme_by_type('inrg')
df_cloud = df.sort_values(by=['freq_lex'], ascending=False)
print_df = export_df_to_org_table(df_cloud, 100)