The easiest way is mark clients which should be removed and then remove them in "safe" place.
public void unsubscribe(Object client)
{
clientsToRemove.add(client);
}
And in code something like this :
for (Object client : clients) {
sendTo(client, message);
}
client.removeAll(clientsToRemove);