Should I move my GAE static files to Google Cloud Service?

Question

I have a lot of static json files in my GAE project that are loaded dynamically with ajax on each page. There is going to be about 3GB of them, so I was thinking of moving them to Google cloud storage, which has 5 GB free maximum. I am not sure about the differences this will make. And also:

• How to bulk upload them to GCS. Is there any user-friendly tool for that? I've tried uploading them via GCS console interface (https://console.developers.google.com/storage), but it adds wrong type (binary/octet-stream) instead of json, and makes files private by default, which can't be bulk changed.

• Is this a good idea? Is there any cost difference for bandwidth between GAE static files and GCS? These files won't be uploaded or changed dynamically, but somehow it makes more sense to me not to store them with project code.

Answer 1

There is no general-purpose bulk-uploading tool, as far as I know. Depending on the language that you use, you can write a relatively simple code to move these files into a bucket, and set the correct options (mime type, access rights) on them.

Network traffic pricing is the same for the first terabyte. After that Google Cloud Storage gets cheaper, while I don't see a similar discounted schedule on App Engine pricing page.

Answer 2

You can add a -m option to gsutil cp to do a "multi" upload, nowadays, since year 2017 at least, see blog post link below. From the gsutil docs:

$ gsutil help cp
...
If you have a large number of files to transfer you might want to use the
gsutil -m option, to perform a parallel (multi-threaded/multi-processing)
copy:

  gsutil -m cp -r dir gs://my-bucket
...

As per the graphs in this blog post from year 2017, this can be many times faster, and the author writes:

Gsutil provides the -m option which performs a parallel (multi-threaded/multi-processing) copy which can significantly increase the performance of an upload