The Load methods in the Assembly class suffer from a naming problem. Nobody ever gets Load() wrong. But LoadFile() has a much too attractive name. LoadFrom should have been LoadFile and LoadFile should have been LoadButUseThisOnlyWhenYouReallyKnowWhatItDoes().
The loading context for an assembly is difficult to understand. Suzanne Cooke has blogged about it a lot, not otherwise putting much of a dent into explaining it :) Roughly, when you use LoadFile, the CLR does not keep track of where the assembly came from and won't stop you from loading the assembly more than once. The types in that assembly are unique and the same type loaded from two LoadFile calls are not identical. You only ever use LoadFile when you intentionally don't want a match. The kind of thing that programs that read metadata would be interested in, a decompiler for example.
The "does not keep track of where the assembly came from" clause is the problem here. You need to use LoadFrom().