What's the right way to seek in amzon s3 from php?

Question

I am using amazon S3 to save files which can be as large as 1GB. I need sections of these files in php to stream to the browser. I am using StreamWrapper which allows a file system approach to files on S3. I have two ways of getting the right parts.

HTTP Range header : The http url for the S3 accept Range headers. I can send an authenticated curl request to S3 with appropriate range headers to get required part.

curl --header "Range=$startbit-$endbit" $url

fread() to the required point: I can also use followng to get the required part.

$f=fopen('s3://bucket.key');
fread($f, $startbit);
echo fread($f, $length);

Which of these is more economical? Is there any better ways to do this?

Answer 1

The fread method is insane.

• It wastes network and CPU.
• It will certainly introduce delay (proportional to how far into the file you want to seek)
• It can easily kill your server the way you've implemented: Doing fread(..$length) means the program will use $length of RAM. If you do a bunch of these at once, your server will probably go out to lunch.

It is perfectly reasonable to do streaming via range headers. Your current library may not support it, but it should be easy to add (i.e. pass the range header to fopen somehow)