Domanda

In my test Android application, I need to process a large file such as compression, encryption, erasure encoding, and etc...
In order to speed-up the process, I spawn multiple threads, each thread read and process different part of a file, and finally merge/append the result together. (Using Java NIO)

I already tried it and there is really some speed-up, 50% or more depending on which storage technique was involved.
There are many similar SO question on this BUT they mainly discussed how it will not improve the I/O speed due to the limitation on single spinning hard disk.
But in my case, it is on multi-cores Android device, which uses flash memory.
Therefore I am not really sure if the speed-up is due to parallel processing or due to caching in RAM.

and my main question is:
Am I doing the right thing? (since I am on multi-cores Android device)
or is this method bad? Bad in terms of what?
Given that interoperability (compression, encryption) on other system is not a concern here.

More details:
I also somehow using the concept of pipelining.
For example:
i) [Sequential] Compress and then encrypt a file will take 10 + 20 = 30 seconds,
ii) [Pipelining] Compress the first half of the file, immediately start encryption after compression is done. At the same time, start compressing the second half of the file, and finally encrypt the second half after the compression is done. It might only take 20 seconds.
(I know it is a bad example, but just to give the idea of applying storage techniques into pipeline)
I am not sure about this, but since each chunk of a file does not depends on the previous chunk (no data dependencies problem), pipelining a file should be ok right?

È stato utile?

Soluzione

Whether it will actually speed up your program or not depends on a lot of factors. This includes- is the file in RAM or on disk? If its on disk is the program IO bound or CPU bound (if IO bound then it won't help)? How is the scheduler of the OS actually going to assign the threads- is it going to assign them to the same core or multiple cores? Do the different threads ever need to interact (will they be waiting locked for so long that its not a speedup or is very buggy)?

Your technique is a fairly standard one for parallel processing. Whether its good for your app or not requires pretty much implementing and checking.

Autorizzato sotto: CC-BY-SA insieme a attribuzione
Non affiliato a StackOverflow
scroll top