Combining filters in nltk collocations

Question

I want to be able to apply a combination of multiple filters to bigrams using nltk collocations. The problem is as follows :

I need to keep bigrams that either occur more than once or is present on a word list. Note that, I want to KEEP the bigrams that are in the word list -- if I wanted to remove them I could simply apply one filter after the other.

I know that there is a frequency filter, and also I can separately check for words in a list using the following :

lambda *w: w not in [w1,w2,w3]

But I do not know how to check the frequency of bigram in this function.

How do we get the frequency of the bigrams passed to lambda ?

Answer 1

Okay, you want to filter both consulting the frequencies AND a word list. So, create a function that construct a filter that consults the scored dictionary and a defined list of words, like this:

def create_filter_minfreq_inwords(scored, words, minfreq):
    def bigram_filter(w1, w2):
        return (w1 not in words and w2 not in words) and (
                (w1, w2) in scored and scored[w1, w2] <= minfreq)
    return bigram_filter

Then, you need to create a dictionary with the frequencies first using finder.score_ngrams with BigramAssocMeasures.raw_freq, and use the above function to create the bigram filter. Here is an example, step by step:

import nltk

# create some text and tokenize it
text = "This is a text. Written for test purposes, this is a text."
tokens = nltk.wordpunct_tokenize(text)

# initialize finder object with the tokens
finder = nltk.collocations.BigramCollocationFinder.from_words(tokens)

# build a dictionary with bigrams and their frequencies
bigram_measures = nltk.collocations.BigramAssocMeasures()
scored = dict(finder.score_ngrams(bigram_measures.raw_freq))

# build a word list
words = ['test', 'This']

# create the filter...
myfilter = create_filter_minfreq_inwords(scored, words, 0.1)

print 'Before filter:\n', list(finder.score_ngrams(bigram_measures.raw_freq))

# apply filter
finder.apply_ngram_filter(myfilter)

print '\nAfter filter:\n', list(finder.score_ngrams(bigram_measures.raw_freq))