This sounds like a plugin architecture to me. The dll, what the plugin implementor should implement should ideally contain a clean interface. Of course that interface may contain some types what belongs to your code too. You have to refactor our architecture in a way that those types would be interfaces too, and ideally would be in one or few assemblies. Then you only have to provide the plugin-in interface assembly and the other slim interface assembly(s) to the client.
BTW, have you looked into MEF? Some years ago it was a CodePlex project but for a while now it's part of the .NET base now. It provides convenient way to decouple such architecture.
link
You just have to use [Export(...)]
and [Import(...)]
attributes and configure it's discovery. MEF also can serve as an IoC/Dependency Injection framework.