You need to call connector.destroy(). The port binding is done in init by default, so correspondingly, unbinding is done in destroy.
If you want to re-use the connector, then set bindOnInit to false. Then the port will be bound/unbound on start/stop.
connector.setProperty("bindOnInit", "false")
A connector is stopped before removing, so you need not explicitly stop it when calling removeConnector.