splitting files for P2P application

Question

I have to implement a middleware system for file sharing, and it has to split the files not unlike what happens on bittorrent, where it sends and receives separate pieces simultaneously from varios sources. How do i do that? Is it a library or i have to implement the file splitting myself?

Answer 1

Split the files into blocks let's say of 100KB each. Then calculate a SHA hash (or some other hashing algorithm) on each of the blocks. so if the file is 905KB, you would have 10 such hashes calculated.

The server would contain a hash definition file for each file that it serves. This hash definition file would contain a list of all of the blocks of the file, along with the hash. So if the server is serving our 905KB file called test.exe. Then we would have another file called test.exe.hashes which contains a listing of the 10 hashes of the file.

The client would download the hash definition file, and ensure that it has all of the blocks. The client can request each block individually and after it's downloaded, it can calculate the hash on its end again to ensure there is no corruption.

You don't need to physically split the file, splitting a file is just reading the part of it that you are interested in. The first block of the file is from byte range 0 to 102399, the next block is from 102400 to 204800, and so on. So just open the file, seek to that position, read the data, and close the file.

Answer 2

Look at the implementation of Split & Concat (GNU software).

Answer 3

You might want to consider using Solomon-Reed encoding. It will make getting the final blocks much easier. This is the route Microsoft took in Avalaunch.

Answer 4

Out of interest: Why not just implement BitTorrent or something like that? There are many OpenSource Clients (i.e. Azureus) and the protocol is really simple. There is also an article with a little more detail, but this contains some extensions - in doubt, the official spec is always right.