Parsing colon delimited data

Question

I have the following text chunk:

string = """
    apples: 20
    oranges: 30
    ripe: yes
    farmers:
            elmer fudd
                   lives in tv
            farmer ted
                   lives close
            farmer bill
                   lives far
    selling: yes
    veggies:
            carrots
            potatoes
    """

I am trying to find a good regex that will allow me to parse out the key values. I can grab the single line key values with something like:

'(.+?):\s(.+?)\n'

However, the problem comes when I hit farmers, or veggies.

Using the re flags, I need to do something like:

re.findall( '(.+?):\s(.+?)\n', string, re.S), 

However, I am having a heck of a time grabbing all of the values associated with farmers.

There is a newline after each value, and a tab, or series of tabs before the values when they are multiline.

and goal is to have something like:

{ 'apples': 20, 'farmers': ['elmer fudd', 'farmer ted'] }

etc.

Thank you in advance for your help.

Answer 1

Here's a really dumb parser that takes into account your (apparent) indentation rules:

def parse(s):
    d = {}
    lastkey = None
    for fullline in s:
        line = fullline.strip()
        if not line:
            pass
        elif ':' not in line:
            indent = len(fullline) - len(fullline.lstrip())
            if lastindent is None:
                lastindent = indent
            if lastindent == indent:
                lastval.append(line)
        else:
            if lastkey:
                d[lastkey] = lastval
                lastkey = None
            if line.endswith(':'):
                lastkey, lastval, lastindent = key, [], None
            else:
                key, _, value = line.partition(':')
                d[key] = value.strip()
    if lastkey:
        d[lastkey] = lastval
        lastkey = None
    return d

import pprint
pprint(parse(string.splitlines()))

The output is:

{'apples': '20',
 'oranges': '30',
 'ripe': ['elmer fudd', 'farmer ted', 'farmer bill'],
 'selling': ['carrots', 'potatoes']}

I think this is already complicated enough that it would look cleaner as an explicit state machine, but I wanted to write this in terms that any novice could understand.

Answer 2

You might look at PyYAML, this text is very close to, if not actually valid YAML.

Answer 3

Here's a totally silly way to do it:

import collections


string = """
    apples: 20
    oranges: 30
    ripe: yes
    farmers:
            elmer fudd
                   lives in tv
            farmer ted
                   lives close
            farmer bill
                   lives far
    selling: yes
    veggies:
            carrots
            potatoes
    """


def funky_parse(inval):
    lines = inval.split("\n")
    items = collections.defaultdict(list)
    at_val = False
    key = ''
    val = ''
    last_indent = 0
    for j, line in enumerate(lines):
        indent = len(line) - len(line.lstrip())
        if j != 0 and at_val and indent > last_indent > 4:
            continue
        if j != 0 and ":" in line:
            if val:
                items[key].append(val.strip())
            at_val = False
            key = ''
        line = line.lstrip()
        for i, c in enumerate(line, 1):
            if at_val:
                val += c
            else:
                key += c
            if c == ':':
                at_val = True
            if i == len(line) and at_val and val:
                items[key].append(val.strip())
                val = ''
        last_indent = indent

    return items

print dict(funky_parse(string))

OUTPUT

{'farmers:': ['elmer fudd', 'farmer ted', 'farmer bill'], 'apples:': ['20'], 'veggies:': ['carrots', 'potatoes'], 'ripe:': ['yes'], 'oranges:': ['30'], 'selling:': ['yes']}