EDIT
There is a library (a set of libraries actually) called NRefactory. Have a look here for an indepth look at its features.
Basically what you can do is analyse all of the DLL's that you load at runtime using NRefactory and create wrappers with a simple C# program that you will write yourself.
The idea is that you create some boilerplate invoke wrappers that will be part of your gateway class and then process all of the libraries you are interested in and populate the gateway class.
Now you have a gateway that is able to call every method in question. Then you just cut the invoke wrappers that you don't need.
It sounds complicated but in reality it's not. I've used this approach to produce something similar on another project. NRefactory is a very powerful tool that gives you amazing possibilities when it comes to analysing and emitting C# code.
ORIGINAL
From what I know (which is not a lot :) ) you can't do this. The simple reason being that your domain will get the dll and all of it's publicly accessible members.
Blocking invokes on those members is not possible without some sort of a gateway.
Implementation of such a gateway is a different story altogether. But as you said it will take time and effort to do this.
I don't think that limiting use of UnityEngine is a good idea. You are basically cutting all relevant Unity functionality.
Perhaps you shouldn't restrict your modders in such a 'strict' way.