Looks like I found a solution, but it's kind of sucky.
I changed
XSendFilePath /opt/gitorious-2.4.12-1/apps/gitorious/tarballs
to
XSendFilePath /opt/gitorious-2.4.12-1/apps/gitorious/repositories
Which caused the following error:
Wed Jul 03 12:18:51 2013] [error] [client 143.122.172.123] (13)Permission denied: xsendfile: cannot open file: /opt/gitorious-2.4.12-1/apps/gitorious/repositories/test/test.git/
I solved this by running
sudo chmod 775 /opt/gitorious-2.4.12-1/apps/gitorious/repositories
Nothing else I tried worked. I'm not sure if those permissions are safe, but they let me clone over HTTP. I cannot push but Gitorious doesn't support that over HTTP anyway.