How to do gsutil cp -R while ignoring files like .git, .gitignore?

Question

I am trying to automate the process of syncing my web assets with Google Cloud Storage. I basically need to copy everything in my development directory up to the cloud. However, I need to ignore the .git directory and some other irrelevant files.

I can't just do a gsutil cp -R . <dest> because that takes absolutely everything, including .git. I tried find . | fgrep git | gsutil cp -I <dest> but that flattens all directories and puts them in root!

Is there a way I can solve this with gsutil or do I have to do a loop in script which uploads all directories (except .git) with -R and then uploads individual files in current directory?

Answer 1

You have two options:

A) Remove the git files after they are uploaded:

gsutil rm gs://bucket/\*.git\*

B) Use find to exclude git files:

find . -not -path '*/.git' -type f -printf '%P\n' | xargs -I '{}' gsutil cp '{}' gs://bucket/'{}'

Source: https://groups.google.com/forum/#!topic/gsutil-discuss/zoHhkTPhiNc

It would've been much easier if gsutil implemented rsync, this would've been easier with their --exclude flag.

Answer 2

You could use a command like:

gsutil rsync -x '\.git.*' dev_dir gs://your-bucket

See Google Storage - rsync - Synchronize content of two buckets/directories

Answer 3

Your web assets are text files - JS, CSS, etc. - which you want to serve compressed, don't you?

Then you need to be aware that GCS requires you to upload such files compressed, to serve them compressed!

Therefore you want to use the -z parameter of gsutil to compress such files.

Expanding on Aziz Saleh's answer then, you probably want to do this:

path '*/.git' -type f -printf '%P\n' | xargs -I '{}' gsutil cp -z js,css,json,html,htm,xml '{}' gs://bucket/'{}'

(You cannot use -z parameter with the Mike Schwartz's gsutil rsync solution.)

Read more about gsutil cp here and about "Transcoding of gzip-compressed files" here.