Ok, the comments were a great start to understand what the root cause of the problem probably was. I don't really understand the git filter-branch
command though, so I was a little wary of just using that.
I came across this tool: https://rtyley.github.io/bfg-repo-cleaner/
It worked wonders. My repo is now under 10MB.