The culprit was found. Like I was suspecting, a dependent assembly was referencing log4net 1.2.12.0. The tool used to find out was dotPeek. Luckily we do have the source code for the dependent assembly and we can change it to reference the latest log4net. This solves this issue.
So if anyone has a similar problem, use dotPeek or a similar tool to go through the dependent assemblies to find out what versions of assemblies it is referencing.