Working around NSFileHandle NSTask blocking when passing large amounts of data

Question

I've got a bit of code that handles exporting data from my application. It takes in an NSString full of XML and runs it through a PHP script to generate HTMl, RTF, etc. It works well unless a user has a large list. This is apparently due to it overrunning the 8k or so buffer of NSPipe.

I worked around it (I think) in the readPipe and readHandle, but I'm not sure how to handle it in the writeHandle/writePipe. The application will beachball at [writeHandle writeData:[in... unless I break on it in gdb, wait a few seconds and and then continue.

Any help on how I can workaround this in my code?

- (NSString *)outputFromExporter:(COExporter *)exporter input:(NSString *)input {
  NSString *exportedString = nil;
  NSString *path = [exporter path];
  NSTask *task = [[NSTask alloc] init];

  NSPipe *writePipe = [NSPipe pipe];
  NSFileHandle *writeHandle = [writePipe fileHandleForWriting];
  NSPipe *readPipe = [NSPipe pipe];
  NSFileHandle *readHandle = [readPipe fileHandleForReading];

  NSMutableData *outputData = [[NSMutableData alloc] init];
  NSData *readData = nil;

  // Set the launch path and I/O for the task
  [task setLaunchPath:path];
  [task setStandardInput:writePipe];
  [task setStandardOutput:readPipe];

  // Launch the exporter, it will convert the raw OPML into HTML, Plaintext, etc
  [task launch];

  // Write the raw OPML representation to the exporter's input stream
  [writeHandle writeData:[input dataUsingEncoding:NSUTF8StringEncoding]];
  [writeHandle closeFile];

  while ((readData = [readHandle availableData]) && [readData length]) {
    [outputData appendData:readData];
  }

  exportedString = [[NSString alloc] initWithData:outputData encoding:NSUTF8StringEncoding];
  return exportedString;
}

Answer 1

The simple, painful truth is that writing a lot of data to a subprocess and then reading a lot of data back from it is not something you can do in a single function or method without blocking the UI.

The solution is just as simple, and is certainly a painful-looking prospect: Make the export asynchronous. Write data as you can, and read data as you can. Not only are you then not blocking the UI, you also gain the ability to update a progress indicator for a really long export, and to do multiple exports in parallel (e.g., from separate documents).

It's work, but the UI payoffs are big, and the result is a cleaner design both internally and externally.

Answer 2

There's a new API since 10.7, so you can avoid using NSNotifications.

task.standardOutput = [NSPipe pipe];
[[task.standardOutput fileHandleForReading] setReadabilityHandler:^(NSFileHandle *file) {
    NSData *data = [file availableData]; // this will read to EOF, so call only once
    NSLog(@"Task output! %@", [[NSString alloc] initWithData:data encoding:NSUTF8StringEncoding]);

    // if you're collecting the whole output of a task, you may store it on a property
    [self.taskOutput appendData:data];
}];

Probably you want to repeat the same above for task.standardError.

IMPORTANT:

When your task terminates, you have to set readabilityHandler block to nil; otherwise, you'll encounter high CPU usage, as the reading will never stop.

[task setTerminationHandler:^(NSTask *task) {

    // do your stuff on completion

    [task.standardOutput fileHandleForReading].readabilityHandler = nil;
    [task.standardError fileHandleForReading].readabilityHandler = nil;
}];

This is all asynchronous (and you should do it async), so your method should have a ^completion block.

Answer 3

NSFileHandle availableData seems to block infinitely:

I made a test program that I call with NSTask from another program. I assign the NSFileHandle to stdin and read the data from pipe. The test program floods the stdout with lots of text using NSLog function. There seems to be no way around it, no matter what API in the NSFileHandle I use, sooner or later the availableData blocks and then the app will hang infinitely and will do nothing. It actually stops to the data read statement, no matter if it is placed in the while or inside it. I tried also reading bytes, one by one, does not help either:

data = [file readDataOfLength: 1];  // blocks infinitely

data = [file availableData]; // blocks infinitely

This works for a while until it also freezes. Seems that I have noticed that the NSFileHandle API does not really work with shell commands that output lots of data, so I have to work around this by using Posix API instead.

Every single example of how to read the data in parts with this API found from Stack Overflow or other sites in the Internet, either synchronously, or asynchronously, seems to block to last fileAvailableData read.

Answer 4

To be clear, this isn't just slow but it's actually freezing until you break in with a debugger? It's not a problem with your sub-process?

One would expect NSFileHandle to handle any data you throw at it, but perhaps you can split your data into smaller chunks using -subdataWithRange: to see what effect that has. You could also grab the fileDescriptor and use the POSIX APIs (fdopen, fwrite, etc.) to write to the stream. The POSIX APIs would provide more flexibility, if indeed that's what you need.

Answer 5

I believe what was happening was I was running into a deadlock caused by the [writeHandle writeData:[input dataUsingEncoding:NSUTF8StringEncoding]]; line filling overfilling its buffer and causing the application to hang until it is (never) emptied.

I worked around it by dispatching the write operation off to a separate thread.

Answer 6

Have a look at asynctask.m from this gist.

asynctask.m allows to process more than 8k of input data.