nlp13. WordNet in Python NLTK

WordNet provides a dictionary-like structure of Synset objects.

We can give it a string and optionally part of speech. If we don't give part of speech, it will return all the matching synsets of different parts of speech.

We use the definitions method, as we iterate over the objects returned.

# nlp13.py
from __future__ import print_function, division
from nltk.corpus import wordnet
arr = "\t-->"
A = wordnet.synsets('love')
for s in A:
    print(s)
    print(arr+s.definition())

# Synset('love.n.01')
#        -->a strong positive emotion of regard and affection
# Synset('love.n.02')
#        -->any object of warm affection or devotion; 
# Synset('beloved.n.01')
#        -->a beloved person; used as terms of endearment
# Synset('love.n.04')
#        -->a deep feeling of sexual desire and attraction
# Synset('love.n.05')
#        -->a score of zero in tennis or squash
# Synset('sexual_love.n.02')
#        -->sexual activities (often including sexual intercourse)
#           between two people
# Synset('love.v.01')
#        -->have a great affection or liking for
# Synset('love.v.02')
#        -->get pleasure from
# Synset('love.v.03')
#        -->be enamored or in love with
# Synset('sleep_together.v.01')
#        -->have sexual intercourse with

nlp12. Fileids in Python NLTK

We can access a specific text within a corpus by using a fileid.

The length of inaugural, that is, len(inaugural.words()) is 145735. However, by putting a fileid, in the call to the words method, we can select only a particular text.

The particular text we selected has a world length of, that is, len(inaugural.words('1789-Washington.txt')) is equal to 1538. We can use the fileids attribute of inaugural, or whatever the corpus happens to be, to get a list with the text names.

The first few words of the first inaugural is printed.

# nlp12.py
from __future__ import print_function, division
from nltk.corpus import inaugural
A = inaugural.fileids()
s = 2*' '
for a in A[:5]:
    print(s+a)
B = inaugural.words(A[0])
for b in B[:20]:
    print(b, end = s)

#  1789-Washington.txt
#  1793-Washington.txt
#  1797-Adams.txt
#  1801-Jefferson.txt
#  1805-Jefferson.txt
# Fellow  -  Citizens  of  the  Senate  and  of
# the  House  of  Representatives  :  Among  the
# vicissitudes  incident  to  life  no  

nlp11. RegexpTokenizer in Python NLTK

We can use RegexpTokenizer to write our own tokenizers.

Our sentences here are alternating numbers and words. The regular expression splits the numbers and words. It will consider a period (.) to be a number.

This only tokens selected have a period, digits, and letters. Thus ? or ! will not be selected.

# nlp11.py
from __future__ import print_function, division
from nltk.tokenize import RegexpTokenizer
A = "I'll3finish45my987project2.2today!3a"
tok = RegexpTokenizer("([a-zA-z']+|[0-9.]+)")
B = tok.tokenize(A)
for b in B: print('\t'+b)
#        I'll
#        3
#        finish
#        45
#        my
#        987
#        project
#        2.2
#        today
#        3
#        a

nlp10. PunktWordTokenizer and WordPunctTokenizer in Python NLTK

PunktWordTokenizer and WordPunctTokenizer will give different tokens for words such as I'll.

The same line is tokenized with different word tokenizers, and the resulting list is either B1,B2,B3, and of different lengths. To show until the last index of the max(length of B1,B2, B3), there are try-clauses to print only if an index exists. Since each clause is only 1-statement, we may put in the sole statement after the colon.

# nlp10.py
from __future__ import print_function, division
from nltk.tokenize import (PunktWordTokenizer,
                           WordPunctTokenizer, word_tokenize)
A = "I'll finish my project today."
PWT = PunktWordTokenizer()
WPT = WordPunctTokenizer()
w = word_tokenize
B1 = PWT.tokenize(A)
B2 = WPT.tokenize(A)
B3 = w(A)
L1,L2,L3 = len(B1),len(B2),len(B3)
print('B1\tB2\tB3')
for i in range(max(L1,L2,L3)):
    try: print(B1[i],end='\t')
    except: print(end='\t')
    try: print(B2[i],end='\t')
    except: print(end='\t')
    try: print(B3[i],end='\t')
    except: print(end='\t')
    print()

#    B1      B2      B3
#    I       I       I       
#    'll     '       'll     
#    finish  ll      finish  
#    my      finish  my      
#    project my      project 
#    today.  project today   
#            today   .       

nlp9. Lexical Dispersion Plot in Python NLTK

A lexical dispersion plot will plot occurences of words in a text.

Here, we select a subset of stopwords that occur more than 90 times and less than 100 times. There are 4 such words and they form the to_plot list, which is sent to the dispersion_plot function.

#nlp9.py
from __future__ import print_function, division
from nltk.corpus import stopwords
from nltk.book import text4
print("%s has a vocabulary of %d" % (text4,len(set(text4))))
words = stopwords.words('english')
to_plot = []
tot = 0
for word in words:
    count = text4.count(word)
    tot += count
    if 90<count<100:
        to_plot.append(word)
        print("count of %s is %d" % (word,count))
print("A total of %d stop words were used." % tot)
print("Total text length is",len(text4))
text4.dispersion_plot(to_plot)

# <Text: Inaugural Address Corpus> has a vocabulary of 9754
# count of between is 93
# count of through is 99
# count of out is 91
# count of some is 91
# A total of 64854 stop words were used.
# Total text length is 145735

Output:

nlp8. Shortening stop word list in Python NLTK

In the previous program, the stop list S contained 127 words.

We remove the words 'then' and 'now' using set operations so its new size is 125. Even though S changed from the list to set, the rest of the program did not have to change.

# nlp8.py
from __future__ import print_function, division
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
lines = """Dr. Brown gave a speech. I'd wish I knew then
what I know now. Finally, he praised Python! At 8 o'clock,
he went home.""" 
S = stopwords.words("english")
nS = ['then','now']
S = set(S)-set(nS)
t = '\t'
A = word_tokenize(lines.lower())
for a in A:
    if a not in S:
        print(t,a)

#         dr.
#         brown
#         gave
#         speech
#         .
#         'd
#         wish
#         knew
#         then
#         know
#         now
#         .
#         finally
#         ,
#         praised
#         python
#         !
#         8
#         o'clock
#         ,
#         went
#         home
#         .

nlp7. Stop word removal in Python NLTK

The function nltk.corpus.stopwords.words gets a list of 127 stop words which usually do not add much to the meaning of sentences. However, it is always possible to find exceptions.

The list is put in S. If you are getting too much filtering, you should try to shorten the stoplist.

# nlp7.py
from __future__ import print_function, division
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
lines = """Dr. Brown gave a speech. I'd wish I knew then
what I know now. Finally, he praised Python! At 8 o'clock,
he went home.""" 
S = stopwords.words("english")
t = '\t'
A = word_tokenize(lines.lower())
for a in A:
    if a not in S:
        print(t,a)

#    dr.
#    brown
#    gave
#    speech
#    .
#    'd
#    wish
#    knew
#    know
#    .
#    finally
#    ,
#    praised
#    python
#    !
#    8
#    o'clock
#    ,
#    went
#    home
#    .