Getting the error “TypeError: 'NoneType' object is not iterable” when trying to use whisper-merge

Question

I'm trying to use whisper-merge to merge 2 wsp files. They have identical retention strategies, one just has older data than the other.

When I run whisper-merge oldfile.wsp newfile.wsp I get this error

Traceback (most recent call last):
  File "/usr/local/src/whisper-0.9.12/bin/whisper-merge.py", line 32, in <module>
    whisper.merge(path_from, path_to)
  File "/usr/local/lib/python2.7/dist-packages/whisper.py", line 821, in merge
    (timeInfo, values) = fetch(path_from, fromTime, untilTime)
TypeError: 'NoneType' object is not iterable

Any ideas?

Here's the meta data output for the 2 files:

• "from_file" http://sprunge.us/dBHC
• "to_file" http://sprunge.us/eIVG

Answer 1

Snippet from whisper.py

def fetch(path,fromTime,untilTime=None):
    """fetch(path,fromTime,untilTime=None)

    path is a string
    fromTime is an epoch time
    untilTime is also an epoch time, but defaults to now.

    Returns a tuple of (timeInfo, valueList)
    where timeInfo is itself a tuple of (fromTime, untilTime, step)

    Returns None if no data can be returned
    """
    fh = open(path,'rb')
    return file_fetch(fh, fromTime, untilTime)

Suggests that whisper.fetch() is returning None, which in turn, (along with the final line in the traceback) suggests that there is a problem with your path_from file.
Looking a little deeper, whisper.file_fetch() appears to have two places where it can return None (explicitly, at least):

def file_fetch(fh, fromTime, untilTime):
    header = __readHeader(fh)
    now = int( time.time() )
    if untilTime is None:
        untilTime = now
    fromTime = int(fromTime)
    untilTime = int(untilTime)

    # Here we try and be flexible and return as much data as we can.
    # If the range of data is from too far in the past or fully in the future, we
    # return nothing
    if (fromTime > untilTime):
        raise InvalidTimeInterval("Invalid time interval: from time '%s' is after until time '%s'" % (fromTime, untilTime))

    oldestTime = now - header['maxRetention']
    # Range is in the future
    if fromTime > now:
        return None               # <== Here
    # Range is beyond retention
    if untilTime < oldestTime:
        return None               # <== ...and here
    ...

Answer 2

Line 812 in whisper.py is broken for files that contain multiple archives. https://github.com/graphite-project/whisper/blob/0.9.12/whisper.py#L812

fromTime = int(time.time()) - headerFrom['maxRetention']

To fix, immediately following line 813, assign fromTime based on the archive retention. https://github.com/graphite-project/whisper/blob/0.9.12/whisper.py#L813

for archive in archives: # this line already exists
  fromTime = int(time.time()) - archive['retention'] # add this line