Sending data chunks over named pipe in linux

Question

I want to send data blocks over named pipe and want receiver to know where data blocks end. How should I do it with named pipes? Should I use some format for joining and splitting blocks (treat pipe always as stream of bytes) or are there any other method?

I've tried opening and closing pipe at sender for every data block but data becomes concatenated at receiver side (EOF not send):

for _ in range(2):
     with open('myfifo', 'bw') as f:
         f.write(b'+')

Result:

rsk@fe temp $ cat myfifo 
++rsk@fe temp $

Answer 1

You can either use some sort of delimiter or a frame structure over your pipes, or (preferably) use multiprocessing.Pipe like objects and run Pickled Python objects through them.

The first option is simply defining a simple protocol you will be running through your pipe. Add a header to each chunk of data you send so that you know what to do with it. For instance, use a length-value system:

import struct

def send_data(file_descriptor, data):
    length = struct.pack('>L', len(data))
    packet = "%s%s" % (length, data)
    file_descriptor.write(packet)

def read_data(file_descriptor):
    binary_length = file_descriptor.read(4)
    length = struct.unpack('>L', binary_length)[0]

    data = ''
    while len(data) < length:
        data += file_descriptor.read(length - len(data))

As for the other option - You can try reading the code of the multiprocessing module, but essentially, you just run through the pipe the result of cPickle.dumps and then you read it into cPickle.loads to get Python objects.

Answer 2

I would just use lines of JSON ecoded data. These are easy to debug and the performance is reasonable.

For an example on reading and writing lines: http://www.tutorialspoint.com/python/file_writelines.htm

For an example of using ujson (UltraJSON): https://pypi.python.org/pypi/ujson

Answer 3

In addition to other solutions, you don't need to stick on named pipes. Named sockets aren't worse and provide more handy features. With AF_LOCAL and SOCK_SEQPACKET, message boundaries transport is maintained by the kernel, so what is written by a single send() will be got on the opposite side with a single recv().