IO Exception when writing files to Google Cloud Storage

Question

I am trying to write files from Google App Engine to Google Cloud, and I use backend to process a request which writes a bunch of files to google cloud storage. But it seems that I get IOException in approximately about 5 minutes (I tried different file size, sleep the thread for 30s for each file etc, and only the time seems to be consistent).

Then I tested the following code:

         String testString = "1,1,1,1,1,1,1,1"; 
         for (int i = 0; i < 200; i++) {
            log.info("Dumping file " + i);
            String fileName = "test/test" + i + ".csv";
            FileService fs = FileServiceFactory.getFileService();
            GSFileOptionsBuilder optionsBuilder = new GSFileOptionsBuilder()
                .setBucket(BUCKET_NAME)
                .setKey(fileName)
                .setAcl("public_read");
            AppEngineFile writableFile = fs.createNewGSFile(optionsBuilder.build());
            FileWriteChannel writeChannel = fs.openWriteChannel(writableFile, true);
            PrintWriter out = new PrintWriter(Channels.newWriter(writeChannel, "UTF8"));
            for (int j = 0; j < 10; j++) { 
                out.println(testString);
            }
            out.close();
            writeChannel.closeFinally();
            Thread.sleep(30000);
        }

And I get IOException as below:

java.io.IOException
    at com.google.appengine.api.files.FileServiceImpl.translateException(FileServiceImpl.java:617)
    at com.google.appengine.api.files.FileServiceImpl.makeSyncCall(FileServiceImpl.java:590)
    at com.google.appengine.api.files.FileServiceImpl.create(FileServiceImpl.java:498)
    at com.google.appengine.api.files.FileServiceImpl.createNewGSFile(FileServiceImpl.java:153)

In the google cloud storage bucket, 11 files are written and are good (each has 160 bytes), and the request fails on creation of the twelfth file. For my real application, it will fail on writeChannel.closeFinally().

java.io.IOException
    at com.google.appengine.api.files.FileServiceImpl.translateException(FileServiceImpl.java:617)
    at com.google.appengine.api.files.FileServiceImpl.makeSyncCall(FileServiceImpl.java:590)
    at com.google.appengine.api.files.FileServiceImpl.close(FileServiceImpl.java:551)
    at com.google.appengine.api.files.FileServiceImpl.close(FileServiceImpl.java:441)
    at com.google.appengine.api.files.FileWriteChannelImpl.closeFinally(FileWriteChannelImpl.java:81)

btw. I am sure I give full control access to the google service account of my GAE application in order to write to the corresponding bucket.

Thanks very much!

Answer 1

The files API is still experimental - I think you're now just running into general issues with the service because it's still experimental.

I know that there is some very active work happening in improving the reliability of this service so for now you should probably code defensively and implement retries when you see transiently errors like this.