The restart manager isn't needed to remove a service. Just use the ServiceInstall and ServiceControl elements and as long your component is removed the service will be removed.
Restart Manager is more about hibernating a process and restoring it's state so that file locks are avoided and availability in maintained.