How can I moderate how much data I get from a file stream in python?

Question

I've got an embedded system I'm writing a user app against. The user app needs to take a firmware image and split it into chunks suitable for sending to the embedded system for programming. I'm starting with S-record files, and using Xmodem for file transfer (meaning each major 'file' transfer would need to be ended with an EOF), so the easiest thing for me to do would be to split the image file into a set files of full s-records no greater than the size of the receive buffer of the (single threaded) embedded system. My user app is written in python, and I have a C program that will split the firmware image into properly sized files, but I thought there may be a more 'pythonic' way of going about this, perhaps by using a custom stream handler.

Any thoughts?

Edit : to add to the discussion, I can feed my input file into a buffer. How could I use range to set a hard limit going into the buffer of either the file size, or a full S-record line ('S' delimited ASCII text)?

Answer 1

I thought this was an interesting question and the S-record format isn't too complicated, so I wrote an S-record encoder that appears to work from my limited testing.

import struct

def s_record_encode(fileobj, recordtype, address, buflen):
    """S-Record encode bytes from file.

    fileobj      file-like object to read data (if any)
    recordtype   'S0' to 'S9'
    address      integer address
    buflen       maximum output buffer size
    """
    # S-type to (address_len, has_data)
    record_address_bytes = {
        'S0':(2, True), 'S1':(2, True), 'S2':(3, True), 'S3':(4, True),
        'S5':(2, False), 'S7':(4, False), 'S8':(3, False), 'S9':(2, False)
    }

    # params for this record type
    address_len, has_data = record_address_bytes[recordtype]

    # big-endian address as string, trimmed to length
    address = struct.pack('>L', address)[-address_len:]

    # read data up to 255 bytes minus address and checksum len
    if has_data:
        data = fileobj.read(0xff - len(address) - 1)
        if not data:
            return '', 0
    else:
        data = ''

    # byte count is address + data + checksum
    count = len(address) + len(data) + 1
    count = struct.pack('B', count)

    # checksum count + address + data
    checksummed_record = count + address + data
    checksum = struct.pack('B', sum(ord(d) for d in checksummed_record) & 0xff ^ 0xff)

    # glue record type to hex encoded buffer
    record = recordtype + (checksummed_record + checksum).encode('hex').upper()

    # return buffer and how much data we read from the file
    return record, len(data)



def s_record_test():
    from cStringIO import StringIO

    # from an example, this should encode to given string
    fake_file = StringIO("\x0A\x0A\x0D\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00")
    encode_to = "S1137AF00A0A0D0000000000000000000000000061"
    fake_file.seek(0)
    record, buflen = s_record_encode(fake_file, 'S1', 0x7af0, 80)
    print 'record', record
    print 'encode_to', encode_to
    assert record == encode_to

    fake_file = StringIO()
    for i in xrange(1000):
        fake_file.write(struct.pack('>L', i))
    fake_file.seek(0)

    address = 0

    while True:
        buf, datalen = s_record_encode(fake_file, 'S2', address, 100)
        if not buf:
            break
        print address, datalen, buf
        address += datalen

Answer 2

If you already have a C program then you're in luck. Python is like a scripting language over C with most of the same functions. See Core tools for working with streams for all the familiar C I/O functions. Then you can make your program more Pythonic by rolling methods into Classes and using things like The Python Slice Notation.