First, the application configuration settings will always trump the plugin settings. You can't avoid that (as far as I know).
However, you can always remove the log4j configuration from the application's config.groovy and make your plugin wire in the log4j configuration within the doWithSpring event of the plugin's lifecycle. This will ensure that the plugin is responsible for configuring log4j.
I will assume you don't need any help wiring up the log4j.xml portion since you didn't ask about that. Just in case, that might look something like this:
MyLoggingPlugin.groovy
...
def doWithSpring = {
...
log4jConfigurer(org.springframework.beans.factory.config.MethodInvokingFactoryBean){
targetClass = "org.springframework.util.Log4jConfigurer"
targetMethod = "initLogging"
arguments = "path/to/log4j.xml"
}
...
}
...