When bootstrapping a different container, use the IWantCustomInitialization interface together with the IConfigureThisEndpoint like @JohnSimons mentioned.
Also, when you are implementing the IWantCustomInitialization in the IConfigureThisEndpoint, there is no bus yet, so an instance has not been created at this point, so you'd need to use Configure.With() instead of Configure.Instance.
NOTE: You don't need to specify UsingTransport as Msmq is the default transport. You also don't need to specify Configure.Serialization.Xml() as Xml is the default serializer.
So, if you change your code to something like below, it should work:
public class EndpointConfig : IConfigureThisEndpoint, AsA_Server, IWantCustomInitialization
{
public void Init()
{
Configure.Features.Enable<Sagas>();
Configure.With()
.CastleWindsorBuilder()
.UseInMemoryTimeoutPersister()
.MsmqSubscriptionStorage();
}
}