The people from the eclipse emf forums has answered me, this is the answer:
How are the adapters attached? You should consider attaching them in response to the Resource.RESOURCE__IS_LOADED notification much like org.eclipse.emf.ecore.change.util.ChangeRecorder.notifyChanged(Notification) does. You can also guard with ((Resource.Internal)eObject.eResource()).isLoading().
I have made the second option, the guard with isLoading and it worked like a charm.