I modified NHibernate; how can I get Castle to use a different-keyed NH assembly?

Question

I made a change to NHibernate 2.1.2, and as such have been doing battle with .NET and strong assembly names. I've set up a ResolveEventHandler to load my version of NHibernate, which I've signed, which I register before building my session factory:

Assembly ModifiedNHibernateAssemblyResolver(object sender, ResolveEventArgs args) {
    var name = new AssemblyName(args.Name);
    switch (name.Name) {
        case "NHibernate":
            return typeof(NHibernate.NHibernateUtil).Assembly;
    }
    return null;
}

I get single-references to "NHibernate.XmlSerializers, Version=2.1.2.4000, Culture=neutral, PublicKeyToken=3557c07e7d6e1f91" now that it's signed (rather than one with that token followed by one with the token = null), but it's failing to load when it encounters Castle:

Could not load type NHibernate.ByteCode.Castle.ProxyFactoryFactory, NHibernate.ByteCode.Castle.
System.IO.FileLoadException: The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)

When un-signed, it was failing on the SysCache load, now it's here, and I have no idea why. I don't see any logging references to loading SysCache, so I think the loading order has changed, but my problem is the error itself. The only differences between the two are the public key tokens, versions are the same, I only have one NHibernate assembly which I reference in the project by browsing (none in GAC), and I'm stuck.

Is there a way to load an assembly with a different public key? If not, how should I go about making modifications to NHibernate so I can still use other libraries, some of which I may not be able to compile under my key?

---

Edit: to make this a more generic question, as it's likely a generic problem: how can I load an assembly with a different public key token than another project expects?
It seems like this would mean death to making your own modifications to open source projects if there's no way, as then you couldn't use it with anything. If you sign it, anything which uses it is stuck to your code. If you don't, anything which is signed can't call your code...