Closing the ServerSocket and creating a new one doesn't accomplish anything useful. The only effect of that would be to cause connection losses on pending, un-accepted connections. Remove that.
You have to close the accepted sockets, and have the clients react to that by attempting to reconnect.