There is only one way a loaded type can be unloaded: if it is a part of a collectible assembly.
But collectible assemblies can be only created through Reflection.Emit, not CSharpCodeProvider. Because of that, I'm not sure if using a collectible assembly is an option for you. If not, you will need some other option (like unloading AppDomains).