nlp6. Treebank tokenizer in Python NLTK

The program below works the same as last, since Treebank Tokenizer is the default word tokenizer.

Parts of the DocString is printed. We import nltk.tokenize.TreebankWordTokenizer as the alias TWT.

Instead of using the regular expressions in Penn Treebank, we may also create new rules.

# nlp6.py
from __future__ import print_function, division
from nltk.tokenize import TreebankWordTokenizer as TWT
lines = """Dr. Brown gave a speech. I'd wish I knew then
what I know now. Finally, he praised Python! At 8 o'clock,
he went home.""" 

t = '\t'
A = TWT()
B = A.tokenize(lines)
print("DocString")
for i in TWT.__doc__.split('\n')[1:4]:
    print(i)
for i,b in enumerate(B):
    print(t,i,b)

#DocString
#    The Treebank tokenizer uses regular expressions to tokenize
# text as in Penn Treebank. This is the method that is invoked by
# ``word_tokenize()``.  It assumes that the text has already been
# segmented into sentences, e.g. using ``sent_tokenize()``.
#         0 Dr.
#         1 Brown
#         2 gave
#         3 a
#         4 speech.
#         5 I
#         6 'd
#         7 wish
#         8 I
#         9 knew
#         10 then
#         11 what
#         12 I
#         13 know
#         14 now.
#         15 Finally
#         16 ,
#         17 he
#         18 praised
#         19 Python
#         20 !
#         21 At
#         22 8
#         23 o'clock
#         24 ,
#         25 he
#         26 went
#         27 home
#         28 .

nlp5. Word Tokenization in Python NLTK

Usually, we want a text to be broken into words, which is done by nltk.tokenize.word_tokenize.

As we can see from the DocString, it uses sentence tokenizing as well.

Instead of using enumerate, we can always iterate over the indices.

# nlp5.py
from __future__ import print_function, division
from nltk.tokenize import word_tokenize
lines = """This is the first sentence. Dr. Brown gave a speech.
Finally, he praised Python! At 8 o'clock, he went home.""" 

A = word_tokenize(lines)
print("DocString for %s:\n%s" % ("word_tokenize",
                                 word_tokenize.__doc__.strip()))
for i in range(len(A)):
    print(i,A[i])

#    DocString for word_tokenize:
#    Return a tokenized copy of *text*,
#        using NLTK's recommended word tokenizer
#        (currently :class:`.TreebankWordTokenizer`
#        along with :class:`.PunktSentenceTokenizer`).
#    0 This
#    1 is
#    2 the
#    3 first
#    4 sentence
#    5 .
#    6 Dr.
#    7 Brown
#    8 gave
#    9 a
#    10 speech
#    11 .
#    12 Finally
#    13 ,
#    14 he
#    15 praised
#    16 Python
#    17 !
#    18 At
#    19 8
#    20 o'clock
#    21 ,
#    22 he
#    23 went
#    24 home
#    25 .

nlp4. Directly loading a tokenizer in Python NLTK

This program does the same thing as the last.

Now we explicity load our tokenizer. It has to be found in ntlk_data folder. This load was implicit in the last program.

We also print the DocString, after removing whitespace characters.

# nlp4.py
from __future__ import print_function, division
from nltk.data import load
lines = """This is the first sentence. Dr. Brown gave a speech.
Finally, he praised Python! At 8 o'clock, he went home.""" 

tok = load("tokenizers/punkt/english.pickle")
print("DocString:\n",tok.tokenize.__doc__.strip())
A = tok.tokenize(lines)

print('type(A)=',type(A))
for i,j in enumerate(A):
    print(i,': ',j)

# DocString:
#  Given a text, returns a list of the sentences in that text.
# type(A)= <type 'list'>
# 0 :  This is the first sentence.
# 1 :  Dr. Brown gave a speech.
# 2 :  Finally, he praised Python!
# 3 :  At 8 o'clock, he went home.

nlp3. Sentence tokenization in Python NLTK

A text has to be broken into sentences for further processing.

We can always write a bunch of rules, or we can use nltk.tokenize.sent_tokenize.

# nlp3.py
from __future__ import print_function, division
from nltk.tokenize import sent_tokenize
lines = """This is the first sentence. Dr. Brown gave a speech.
Finally, he praised Python! At 8 o'clock, he went home.""" 

A = sent_tokenize(lines)
print('type(A)=',type(A))
for i,j in enumerate(A):
    print(i,': ',j)

#    type(A)= <type 'list'>
#    0 :  This is the first sentence.
#    1 :  Dr. Brown gave a speech.
#    2 :  Finally, he praised Python!
#    3 :  At 8 o'clock, he went home.

nlp2. Concordance in Python NLTK

Concordance gives the context of some text inside a corpus.

Here, we iterate over three strings in a Python list and see what is contained in Wall Street Journal for those entries.

Unlike the count method, which returns the integer, the concordance method returns None, but just prints its results.

# nlp2.py
from __future__ import print_function, division
from nltk.book import text7
print('text7 =',text7)
print('text 7 length =',len(text7))
St = ["Indonesia","Singapore","Malaysia"]
for st in St:
    n = text7.count(st)
    print("The string %s ocurrs %d times" % (st,n))
    print("The occurences:")
    text7.concordance(st,50)
    
#    text7 = <Text: Wall Street Journal>
#    text 7 length = 100676
#    The string Indonesia ocurrs 2 times
#    The occurences:
#    Displaying 2 of 2 matches:
#     and export them to Indonesia . `` The effect wil
#    aysia , Singapore , Indonesia , the Philippines a
#    The string Singapore ocurrs 4 times
#    The occurences:
#    Displaying 4 of 4 matches:
#     tobacco smoke . In Singapore , a new law require
#    cial said 0 *T*-1 . Singapore already bans smokin
#    ailand , Malaysia , Singapore , Indonesia , the P
#    es closed higher in Singapore , Taipei and Wellin
#    The string Malaysia ocurrs 6 times
#    The occurences:
#    Displaying 6 of 6 matches:
#    ing slow progress in Malaysia . '' She did n't ela
#    eocassette piracy in Malaysia and disregard for U.
#    ood restaurants . In Malaysia , Siti Zaharah Sulai
#    such as Thailand and Malaysia , the investment wil
#    assemble the sets in Malaysia and export them to I
#    ations -- Thailand , Malaysia , Singapore , Indone

nlp1. Reading a text in Python NLTK

The NLTK module in Python can be used to load a text, or corpus. In nltk_data folder, you can find the included texts. This assumes all the data files have been downloaded to the computer using nltk.download().

Here Shakespeare’s Julius Caesar is read as a raw string. We may also use the xml loader which will allow parsing the tree, for example the <LINE> elements.

The <LINE> elements are extracted using regular expressions. Only a subset of the lines are printed; those with the word 'Pompey'.

# nlp1.py
from __future__ import print_function, division
from nltk.corpus import shakespeare
import re
sp = " " * 2
jc = shakespeare.raw("j_caesar.xml")
jc_lines = re.findall(r"<LINE>.+</LINE>", jc)
for line in jc_lines:
    lin = line[6:-7]
    if lin.count("Pompey"):
        print(sp+lin)
        
#  Knew you not Pompey? Many a time and oft
#  To see great Pompey pass the streets of Rome:
#  That comes in triumph over Pompey's blood? Be gone!
#  In Pompey's porch: for now, this fearful night,
#  Repair to Pompey's porch, where you shall find us.
#  That done, repair to Pompey's theatre.
#  Who rated him for speaking well of Pompey:
#  That now on Pompey's basis lies along
#  Even at the base of Pompey's statua,
#  As Pompey was, am I compell'd to set

ML1. K-Nearest Neighbor in Python

K-Nearest Neighbor is a supervised lazy learning technique.

The Iris dataset is used, with 150 instances, 4 features and 3 classes. The first 50 observations (rows) correspond to class 0, next 50 rows to class 1 and last 50 rows to class 2. The program prints the class names.

10-fold cross validation is used. Thus 150/10 = 15 instances are used for testing, and the rest for training. This is done 10 times, each time with a new set of indices. The KFold function has the shuffle parameter set to True so each test/training will have samples from all 3 classes.

The accuracy_score function is used to find the fraction of correctly labelled test values. Since there are 135 training labels, we thus find 135 distances in 4D space during each train-test iteration.

# ML1.py
from __future__ import print_function, division
from sklearn.neighbors import KNeighborsClassifier
from sklearn.cross_validation import KFold
from sklearn.metrics import accuracy_score
from sklearn.datasets import load_iris

# Loading data (150,4)
data = load_iris()
x = data.data
y = data.target
print('The three classes are',data.target_names)

# Use 5 nearest neighbors
classifier = KNeighborsClassifier(n_neighbors=5)

# Running 10 tests using 10-fold cross validataon
test = set()
acc = []
kf = KFold(len(x), n_folds=10, shuffle=True)
for trn,tst in kf:
    x_train = x[trn]
    y_train = y[trn]
    print('length of x_train:',len(x_train))
    classifier.fit(x_train, y_train)
    x_test = x[tst]
    y_test = y[tst]
    test = test.intersection(tst)
    print('length of x_test:',len(x_test))
    print('tst:',tst)
    pred = classifier.predict(x_test)
    acc.append(accuracy_score(y_test,pred))

# Accuracy
print('Result: {}'.format(sum(acc)/len(acc)))
print('length of test: {}'.format(len(test)))

#The three classes are ['setosa' 'versicolor' 'virginica']
#length of x_train: 135
#length of x_test: 15
#tst: [ 11  20  37  42  58  88  94  95  99 101 117 121 132 136 146]
#length of x_train: 135
#length of x_test: 15
#tst: [  0  13  19  26  47  64  76  86  97  98 104 105 120 133 143]
#length of x_train: 135
#length of x_test: 15
#tst: [ 12  18  24  27  30  33  35  38  48  51  55  60 106 122 144]
#length of x_train: 135
#length of x_test: 15
#tst: [  5  32  45  52  65  66  81  83  90 102 116 131 137 139 148]
#length of x_train: 135
#length of x_test: 15
#tst: [  3   4  17  23  29  31  40  41  49  79  85  87 109 114 145]
#length of x_train: 135
#length of x_test: 15
#tst: [  1   2  54  57  61  80  89  96 113 115 118 127 128 134 141]
#length of x_train: 135
#length of x_test: 15
#tst: [  7   8  10  15  16  71  74  82 125 129 130 135 140 142 149]
#length of x_train: 135
#length of x_test: 15
#tst: [  9  14  53  56  68  69  73  75  77  91 100 103 107 110 111]
#length of x_train: 135
#length of x_test: 15
#tst: [  6  21  25  28  34  44  46  62  63  70  92 119 126 138 147]
#length of x_train: 135
#length of x_test: 15
#tst: [ 22  36  39  43  50  59  67  72  78  84  93 108 112 123 124]
#Result: 0.973333333333
#length of test: 0