Git local mirror and repository

Question

Here is what I would like to do:

• Have a local git repository that mirrors an upstream one
• Be able to push "local" branches / changes to that repository and keep those locally
• Keep this repository in sync with the upstream one, including:
  • Fetch any new branch
  • Delete any reference of branches that are deleted upstream

I setup my cron job to fetch all the changes from upstream and prune any branch that have been deleted like this:

*/5 * * * * cd /home/git/myrepo.git && git fetch origin && git remote prune origin > /dev/null

So far what I have tried (and why it failed):

1- Setup the git repository as a mirror (as described here)

git clone --bare --mirror URL

The problem with that is when it does the git remote prune, it is also deleting references to the "local" changes that have been pushed there (and not to the upstream server).

I also tried to have this local repository be the mirror for two separate repositories (with the same master but some different branches) and I hit a similar problem when doing git remote prune, it will delete the branches coming from the other repository.

2- Setup git only as a bare repository:

git clone --bare URL

But then git fetch origin is not updating properly, it seems to be downloading the objects, but does not create the refs and then only prints

 * branch            HEAD       -> FETCH_HEAD

and the "location" of the current branches is not being updated with what's in the upstream server.

I also tried git remote update as described here, with the same result.

I can convert that repository as a mirror with:

git config remote.origin.fetch 'refs/heads/*:refs/heads/*'

But that only brings me back to the problem in (1)

Answer 1

Assuming you can drop the "mirror" requirement, and have "local (bare) repo $X also copies upstream repo $UX using refs/heads/upstream/$branch to name upstream branches known there as refs/heads/$X", use your second approach, but do this instead:

$ cd /tmp; mkdir tt; cd tt; git clone --bare ssh://$upstream_host/tmp/t
$ cd t.git
$ git config remote.origin.fetch '+refs/heads/*:refs/heads/upstream/*'
$ git fetch -p # accidentally omitted this step from cut/paste earlier

This assumes you won't use branch names like upstream/master for anything yourself. (You could also/instead do something like:

git config remote.origin.fetch '+refs/*:refs/upstream/*'

but refs/upstream/* references are not copied over by normal git clone, git fetch, etc., so this is more of a pain for "normal" git users.)

Let's make a clone of the --bare repo too to see what happens as we go on. (For reference, on $upstream_host, I have /tmp/t, a regular git repo. On $local_host, the not-quite-mirror machine, I have /tmp/tt/t.git, a --bare repo that does this upstream tracking thing. I am actually using the same host for both but the principle applies...)

$ cd /tmp; mkdir xt; cd xt; git clone ssh://$local_host/tmp/tt/t.git
Cloning into 't'...
remote: Counting objects: 96, done.
remote: Compressing objects: 100% (54/54), done.
remote: Total 96 (delta 33), reused 96 (delta 33)
Receiving objects: 100% (96/96), 17.11 KiB | 0 bytes/s, done.
Resolving deltas: 100% (33/33), done.
Checking connectivity... done

Now I made a change on $upstream_host in /tmp/t, and commited it. Back on $local_host:

$ cd /tmp/tt/t.git; git fetch -p origin # -p will prune deleted upstream/foo's
remote: Counting objects: 5, done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 4 (delta 1), reused 0 (delta 0)
Unpacking objects: 100% (4/4), done.
From ssh://$host/tmp/t
 + c10e54c...5e01371 master     -> upstream/master  (forced update)

Thus, changes made upstream will appear in your "sort of a mirror but not exactly" bare git repo as a change to upstream/master rather than master, or more generally, upstream/$branch for any $branch. If you want to merge them you'll have to do that manually. My example below is a bit messy because the change I made on $upstream_host was a history rewrite (hence all the forced update stuff), which winds up getting exposed here via the clones. If you don't want it exposed you'll have to note which updates were history rewrites and (in effect) manually copy them to your own not-quite-mirror, and then on to any clones of that. I'll just go ahead and make a real merge.

So, now we go to the non-bare repo on $local_host, in /tmp/xt/t:

$ cd /tmp/xt/t
$ git fetch
remote: Counting objects: 5, done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 4 (delta 1), reused 1 (delta 0)
Unpacking objects: 100% (4/4), done.
From ssh://$local_host/tmp/tt/t
 + c10e54c...5e01371 upstream/master -> origin/upstream/master  (forced update)
$ git status
# On branch master
nothing to commit, working directory clean
$ git log --oneline --decorate --graph
* 5e01371 (origin/upstream/master) add ast example
| * c10e54c (HEAD, origin/master, origin/HEAD, master) add ast example
|/  
* 309b36c add like_min.py
... [snipped]
$ git merge origin/upstream/master

Merge remote-tracking branch 'origin/upstream/master'

# Please enter a commit message to explain why this merge is necessary,
# especially if it merges an updated upstream into a topic branch.
#
# Lines starting with '#' will be ignored, and an empty message aborts
# the commit.
...
$ git push
warning: push.default is unset; its implicit value is changing in
Git 2.0 from 'matching' to 'simple'. To squelch this message
...
Counting objects: 1, done.
Writing objects: 100% (1/1), 244 bytes | 0 bytes/s, done.
Total 1 (delta 0), reused 0 (delta 0)
To ssh://$local_host/tmp/tt/t.git
   c10e54c..e571182  master -> master

I've now updated the --bare clone ($local_host, /tmp/tt/t.git) via the non-bare clone to merge the upstream work into my local not-exactly-a-mirror. The HEAD revision is my merge, HEAD^1 is the original (broken) update that used to be origin/upstream/master (before all the "forced update"ing), and HEAD^2 is the corrected update that is now origin/upstream/master (afterward):

$ git rev-parse HEAD^2 origin/upstream/master
5e013711f5d6eb3f643ef562d49a131852aa4aa1
5e013711f5d6eb3f643ef562d49a131852aa4aa1

(The name is just upstream/master in the --bare clone, so the git rev-parse above is from /tmp/xt/t not /tmp/tt/t.git.)