GAE "The API call urlfetch.Fetch() required more quota than is available" when resumable upload Video files

Question

my GAE application reads files from Drive by Drive API into a FileStream, and then the FileStream is uploaded into Youtube by Youtube API v3 with "resumable upload". When the file size gets larger (e.g. 60M ), the Youtube API returns this error "The API call urlfetch.Fetch() required more quota than is available"

I also have tried with "direct upload" for uploading 60M size video file, then error message would be "java.lang.OutOfMemory: Java heap space at com.google.protobuf.ByteString.copyFrom (ByteString.java:178)".

Here is the brief version of my code:

GoogleCredential credential = new GoogleCredential.Builder()
            .setTransport(HTTP_TRANSPORT)
            .setJsonFactory(JSON_FACTORY)
            .setServiceAccountId(SERVICE_ACCOUNT_EMAIL)
            .setServiceAccountScopes(YouTubeScopes.YOUTUBE)
            .setServiceAccountPrivateKeyFromP12File(new File(P12))
            .setServiceAccountUser(account).build();
YouTube service = new YouTube.Builder(HTTP_TRANSPORT, JSON_FACTORY, credential).setApplicationName("VSP").build();
Video videoObjectDefiningMetadata = new Video();
VideoSnippet snippet = new VideoSnippet();
snippet.setTitle(title);
videoObjectDefiningMetadata.setSnippet(snippet);
InputStreamContent mediaContent = new InputStreamContent(VIDEO_FILE_FORMAT, new BufferedInputStream(filestream));
mediaContent.setLength(filesize);
YouTube.Videos.Insert videoInsert = service.videos().insert("snippet,statistics,status", videoObjectDefiningMetadata, mediaContent);

MediaHttpUploader uploader = videoInsert.getMediaHttpUploader();
uploader.setDirectUploadEnabled(false);
uploader.setChunkSize(7864320);

Video returnedVideo = videoInsert.execute();

error message "The API call urlfetch.Fetch() required more quota than is available" comes at last line of the code. sometimes the uploading is done successfully with the error message, sometimes not, by setting the ChunkSize differently.

I couldn't find any useful information about this error message. But my guess is that GAE application can only send certain mount of requests during certain mount of time. Since "resumable upload" is breaking the filestream into chunks, and send them in a sequence of requests, it reaches the limit easily. if my guess is right, what is the limit? and how do i solve this problem? if my guess is wrong, where do you think the problem is?

Thanks

Answer 1

Thanks guys!

Here is the limit for incoming & outgoing bandwidth for URL Fetch in GAE: https://developers.google.com/appengine/docs/quotas

By default, the limit is 22M/min, with bill enabled, the limit becomes 740M/min. so with 22M/min limit, a GAE task queue can upload about 220M video files to Youtube (22M * 10min)

But this leads to the problem of using the upper code

Video returnedVideo = videoInsert.execute();

, becoz we cannot control the how many chunks are sent every minute in that code. The solution which i did, is to follow the description in the following link to handle each of the requests by myself. https://developers.google.com/youtube/v3/guides/using_resumable_upload_protocol in this way, we can control the size of stream which could be sent each minute.