Alter a file (without fseek or + modes) or concatenate two files with minimal copying

Question

I am writing an audio file to an SD/MMC storage card in real time, in WAVE format, working on an ARM board. Said card is (and must remain) in FAT32 format. I can write a valid WAVE file just fine, provided I know how much I'm going to write beforehand.

I want to be able to put placeholder data in the Chunk Data Size field of the RIFF and data chunks, write my audio data, and then go back and update the Chunk Data Size field in those two chunks so that they have correct values, but...

I have a working filesystem and some stdio functions, with some caveats:

• fwrite() supports r, w, and a, but not any + modes.
• fseek() does not work in write mode.

I did not write the implementations of the above functions (I am using ARM's RL-FLashFS), and I am not certain what the justification for the restrictions/partial implementations is. Adding in the missing functionality personally is probably an option, but I would like to avoid it if possible (I have no other need of those features, do not forsee any, and can't really afford to spend too much time on it.) Switching to a different implementation is also not an option here.

I have very limited memory available, and I do not know how much audio data will be received, except that it will almost certainly be more than I can keep in memory at any one time.

I could write a file with the raw interleaved audio data in it while keeping track of how many bytes I write, close it, then open it again for reading, open a second file for writing, write the header into the second file, and copy the audio data over. That is, I could post-process it into a properly formatted valid WAVE file. I have done this and it works fine. But I want to avoid post-processing large amounts of data if at all possible.

Perhaps I could somehow concatenate two files in place? (I.e. write the data, then write the chunks to a separate file, then join them in the filesystem, avoiding much of the time spent copying potentially vast amounts of data.) My understanding of that is that, if possible, it would still involve some copying due to block orientation of the storage.

Suggestions?

EDIT: I really should have mentioned this, but there is no OS running here. I have some stdio functions running on top of a hardware abstraction layer, and that's about it.

Answer 1

This should be possible, but it involves writing a set of FAT table manipulation routines.

The concept of FAT is simple: A file is stored in a chain of "clusters" - fixed size blocks. The clusters do not have to be contiguous on the disk. The Directory entry for a file includes the ID of the first cluster. The FAT contains one value for each cluster, which is either the ID of the next cluster in the chain, or an "End-Of-Chain" (EOC) marker.

So you can concatenate files together by altering the first file's EOC marker to point to the head cluster of the second file.

For your application you could write all the data, rewrite the first cluster (with the correct header) into a new file, then do FAT surgery to graft the new head onto the old tail:

• Determine the FAT cluster size (S)
• Determine the size of the WAV header up to the first data byte (F)
• Write the incoming data to a temp file. Close when stream ends.
• Create a new file with the desired name.
• Open the temp file for reading, and copy the header to the new file while filling in the size field(s) correctly (as you have done before).
• Write min(S-F, bytes_remaining) to the new file.
• Close the new file.
• If there are no bytes remaining, you are done,
• else,
  • Read the FAT and Directory into memory.
  • Read the Directory to get
    • the first cluster of the temp file (T1) (with all the data),
    • the first cluster of the wav file (W1). (with the correct header)
  • Read the FAT entry for T1 to find the second temp cluster (T2).
  • Change the FAT entry for W1 from "EOC" to T2.
  • Change the FAT entry for T1 from T2 to "EOC".
  • Swap the FileSize entries for the two files in the Directory.
  • Write the FAT and Directory back to disk.
  • Delete the Temp file.

Of course, by the time you do this, you will probably understand the file system well enough to implement fseek(fp,0,SEEK_SET), which should give you enough functionality to do the header fixup through standard library calls.

Answer 2

We are working with exactly the same scenario as you in our project - recorder application. Since the length of file is unknown - we write a RIFF header with 0 length in the beginning (to reserve space) and on closing - go back to the 0 position (with fseek) and write correct header. Thus, I think you have to debug why fseek doesn't work in write mode, otherwise you will not be able to perform this task efficiently.

By the way, you'd better off from file system internal specific workarounds like concatenating blocks, etc - this is hardly possible, will not be portable and can bring you new problems. Let's use standard and proven methods instead.

Update

(After finding out that your FS is ARM's RL-FlashFS) why not using rewind http://www.keil.com/support/man/docs/rlarm/rlarm_rewind.htm instead of fseek?