Recover A Corrupt Git Bare Repository

Recently we had lot of fun fixing our git repository. Due to a manual power outage our machines were restarted and caused many dangling objects in git. The Internet discussions handle corrupt blob object well. But our system had also corrupt tree objects which are discussed less! This entry describes our way to get the git bare repository up and running again.

First, we ran into problems at checkout:

$ git checkout rewoo-1937
fatal: unable to read tree 4836d984c7f7910b5930dd9e343a0ccd3483f605

Other commands failed as well: git clone failed. git pull failed. git push failed. So our repository was indeed broken.

Searching the Internet lead us to a mail from Linus. He described in [1] how to find your corrupt objects with “git fsck --full” but our system reported a fatal corruption error:

root$ cd /srv/gitosis/repositories/rewoo.git # switch to our bare repository
root$ git fsck --full
fatal: object 00f74052e7dc6c2483d6d43e3d0b6a092ad81d3e is corrupted

Mm. Bad start. So we deleted the object 00f74052e7dc6c2483d6d43e3d0b6a092ad81d3e

root$ rm -f ./objects/00/f74052e7dc6c2483d6d43e3d0b6a092ad81d3e
root$ git fsck --full
fatal: object 03569db3014d231aa306bb1ada9d597502ce1a83 is corrupted

Another object? Well. We just deleted all:

root$ while [ true ]; do \
f=`git fsck --full 2>&1|awk '{print $3}'|sed -r 's/(^..)(.*)/objects\/\1\/\2/'`; \
if [ ! -f "$f" ]; then break; fi; \
echo delete $f; rm -f "$f"; done
delete objects/00/f74052e7dc6c2483d6d43e3d0b6a092ad81d3e
delete objects/03/569db3014d231aa306bb1ada9d597502ce1a83
delete objects/0a/51475b3a6394cbaaed35464a6a2a030c655226
delete objects/12/c186ceb03e31c2a91247e0a9a8725d20d6ce4b
delete objects/1b/2099131b444b3342b15f3bd4ba63ef96d8b1cd
delete objects/21/07e3566773f8f2069e1673d527232f28b04754

Now git fsck --all ran fine.

root$ git fsck --full
dangling commit 8d1954001d105ccd4d4ea9603b73820c5d7f9129
dangling commit 30f5168dd24a6e244ae41ff314127ecd6c9757db

Now we cleaned up our repository with the garbage collector:

root$ git gc --prune=today

No luck. git checkout rewoo-1937 still produced errors. But we noticed that branch ‘rewoo-1937’ was described in packed-refs and in refs/heads/rewoo-1937. We deleted the line in packed-refs and pointed the ref of rewoo-1937 to an earlier commit of that day.

root$ echo "595fb43daff52a65ec49a3c91c5bde4fcc79873d" > refs/heads/rewoo-1937

Again we did a garbage collection on the repository and finally we could checkout again!

$ git fetch --all
Fetching origin
From git.rewoo.lan:/rewoo
+ 30f5168...595fb43 rewoo-1937 -> origin/rewoo-1937  (forced update)

Finally. We were able to push our old commits to the repository again.

PS: Another helpful discussion was found at [2] but git log --all --raw ended with a fatal error. With git log --branches and git log --branches=rewoo-* we found the corrupt branch of our repository (eg. git log --branches=master did not cause problems, git log --branches=rewoo-* did).

[1] Recover Corrupted Blob Object
[2] StackOverflow git-which-commit-has-this-blob