As indicated from the answers, you just can't do it, especially with attributes, it will always be two dependencies (which at most you could mush into a single DLL). So i've changed the structure to use:
[AttributeUsage(AttributeTargets.Interface, AllowMultiple = false)]
public class AppSettingsFromConfigAttribute : Attribute {}
as a simple marker. Looked at the Windsor source and all the interfaces implemented were internally being used to add a PropertyDescriptor to the type when making the dictionary adapter. Windsor being windsor provides an overload to specify a PropertyDescriptor externally, so i now create that myself at runtime using the vanilla attribute, in the bootstrapper - the one place that needs Castle.