Fetching blob from Google Blobstore without knowing size before hand

Question

I need to fetch a blob from the Blobstore programmatically, without knowing the size before hand. Does anyone know how to do that?

I have tried using

BlobstoreService blobStoreService = BlobstoreServiceFactory.getBlobstoreService();
byte[] picture = blobStoreService.fetchData(blobKey, 0, Integer.MAX_VALUE);

but I get an error since (at least seemingly) Integer.MAX_VALUE is too big.

java.lang.IllegalArgumentException: Blob fetch size 2147483648 it larger than maximum size 1015808 bytes.
at com.google.appengine.api.blobstore.BlobstoreServiceImpl.fetchData(BlobstoreServiceImpl.java:250)

So does anyone know how to do this correctly? Also if you could tell me in passing, is it better to same images into the blobstore as "jpeg" or as "png"?

Answer 1

Hope this helps, this is the way I have been doing it for a while:

        BlobstoreService blobstoreService = BlobstoreServiceFactory.getBlobstoreService();
        BlobKey blobKey = new BlobKey(KEY);

        // Start reading
        ByteArrayOutputStream out = new ByteArrayOutputStream();

        long inxStart = 0;
        long inxEnd = 1024;
        boolean flag = false;

        do {
            try {
                byte[] b = blobstoreService.fetchData(blobKey,inxStart,inxEnd);
                out.write(b);

                if (b.length < 1024)
                    flag = true;

                inxStart = inxEnd + 1;
                inxEnd += 1025;

            } catch (Exception e) {
                flag = true;
            }

        } while (!flag);

        byte[] filebytes = out.toByteArray();

I used to use:

BlobInfo blobInfo = blobInfoFactory.loadBlobInfo(blobKey);
filesize = blobInfo.getSize();

to obtaint the size, but for some reason, sometimes this info was null.

Maybe all this can give you an idea.

Answer 2

def blob_fetch(blob_key):
  blob_info = blobstore.get(blob_key)
  total_size = blob_info.size
  unit_size = blobstore.MAX_BLOB_FETCH_SIZE
  pos = 0
  buf = cStringIO.StringIO()
  try:
    while pos < total_size:
      buf.write(blobstore.fetch_data(blob_key, pos, min(pos + unit_size - 1, total_size)))
      pos += unit_size
    return buf.getvalue()
  finally:
    buf.close()

The MAX_BLOB_FETCH_SIZE is not obvious in the document.

Answer 3

In Python:

from google.appengine.ext.blobstore import BlobInfo
from google.appengine.api import blobstore
import cStringIO as StringIO

blobinfo = BlobInfo.get(KEY)

offset = 0
accumulated_content = StringIO.StringIO()
while True:
  fetched_content = blobstore.fetch_data(
      blobinfo.key(),
      offset,
      offset + blobstore.MAX_BLOB_FETCH_SIZE - 1)
  accumulated_content.write(fetched_content)
  if len(fetched_content) < blobstore.MAX_BLOB_FETCH_SIZE:
    break
  offset += blobstore.MAX_BLOB_FETCH_SIZE