a Regex for extracting sentence from a paragraph in python

Question

I'm trying to extract a sentence from a paragraph using regular expressions in python.
Usually the code that I'm testing extracts the sentence correctly, but in the following paragraph the sentence does not get extracted correctly.

The paragraph:

"But in the case of malaria infections and sepsis, dendritic cells throughout the body are concentrated on alerting the immune system, which prevents them from detecting and responding to any new infections." A new type of vaccine?

The code:

def splitParagraphIntoSentences(paragraph):

import re

sentenceEnders = re.compile('[.!?][\s]{1,2}(?=[A-Z])')
sentenceList = sentenceEnders.split(paragraph)
return sentenceList
if __name__ == '__main__':
    f = open("bs.txt", 'r')
    text = f.read()
    mylist = []
    sentences = splitParagraphIntoSentences(text)
    for s in sentences:
        mylist.append(s.strip())
        for i in mylist:
            print i

When tested with the above paragraph it gives output exactly as the input paragraph but the output should look like-

But in the case of malaria infections and sepsis, dendritic cells throughout the body are concentrated on alerting the immune system, which prevents them from detecting and responding to any new infections

A new type of vaccine

Is there anything wrong with the regular expression?

Answer 1

The paragraph you've posted as an example has its first sentence enclosed in double quotes ", and the closing quote comes immediately after the full stop: infections."

Your regexp [.!?]\s{1,2} is looking for a period followed by one or two spaces as sentence terminator, so it won't catch it.

It can be adjusted to cope with this case by allowing for optional closing quotes:

sentenceEnders = re.compile(r'''[.!?]['"]?\s{1,2}(?=[A-Z])''')

However, with the above regexp you would be removing the end quote from the sentence. Keeping it is slightly more tricky and can be done using a look-behind assertion:

sentenceEnders = re.compile(r'''(?<=[.!?]['"\s])\s*(?=[A-Z])''')

Note, however, that there are a lot of cases where a regexp-based splitter fails, e.g.:

• Abbreviations: "In the works of Dr. A. B. Givental ..." -- according to your regexp, this will be incorrectly split after "Dr.", "A." and "B." (You can adjust the single-letter case, but you cannot detect an abbreviation unless you hard-code it.)

• Use of exclamation marks in the middle of the sentence: "... when, lo and behold! M. Deshayes himself appeared..."

• Use of multiple quote marks and nested quotes, etc.

Answer 2

Riccardo Murri's answer is correct, but I thought I'd throw a bit more light on the subject.

There was a similar question asked with regard to PHP: php sentence boundaries detection. My answer to that question includes handling the exceptions such as "Mr.", "Mrs." and "Jr.". I've adapted that regex to work with Python, (which places more restrictions on lookbehinds). Here is a modified and tested version of your script which uses this new regex:

def splitParagraphIntoSentences(paragraph):
    import re
    sentenceEnders = re.compile(r"""
        # Split sentences on whitespace between them.
        (?:               # Group for two positive lookbehinds.
          (?<=[.!?])      # Either an end of sentence punct,
        | (?<=[.!?]['"])  # or end of sentence punct and quote.
        )                 # End group of two positive lookbehinds.
        (?<!  Mr\.   )    # Don't end sentence on "Mr."
        (?<!  Mrs\.  )    # Don't end sentence on "Mrs."
        (?<!  Jr\.   )    # Don't end sentence on "Jr."
        (?<!  Dr\.   )    # Don't end sentence on "Dr."
        (?<!  Prof\. )    # Don't end sentence on "Prof."
        (?<!  Sr\.   )    # Don't end sentence on "Sr."
        \s+               # Split on whitespace between sentences.
        """, 
        re.IGNORECASE | re.VERBOSE)
    sentenceList = sentenceEnders.split(paragraph)
    return sentenceList

if __name__ == '__main__':
    f = open("bs.txt", 'r')
    text = f.read()
    mylist = []
    sentences = splitParagraphIntoSentences(text)
    for s in sentences:
        mylist.append(s.strip())
    for i in mylist:
        print i

You can see how it handles the special cases and it is easy to add or remove them as required. It correctly parses your example paragraph. It also correctly parses the following test paragraph (which includes more special cases):

This is sentence one. Sentence two! Sentence three? Sentence "four". Sentence "five"! Sentence "six"? Sentence "seven." Sentence 'eight!' Dr. Jones said: "Mrs. Smith you have a lovely daughter!"

But note that there are other exceptions that can fail which Riccardo Murri has correctly pointed out.

Answer 3

Yes, there is something wrong. You take the separator into account only if it is followed by one or two spaces and then a capital letter, so the end of "A new type of vaccine?" sentence won't get matched for example.

I would not be too restrictive about the spaces either, unless it is an intent (the text might not be well formated), because e.g. "Hello Lucky Boy!How are you today?" would not get splitted.

I also do not understand your example, why is only the first sentence is in enclosed in " ?

Anyway:

>>> Text="""But in the case of malaria infections, dendritic cells and stuff.
            A new type of vaccine? My uncle!
         """
>>> Sentences = re.split('[?!.][\s]*',Text)
>>> Sentences
    ['But in the case of malaria infections, dendritic cells and stuff',
     'A new type of vaccine',
     'My uncle',
     '']

You might also filter the empty sentences:

>>> NonemptyS = [ s for s in Senteces if s ]