Listeners are a reference from the element to whatever is listening- this means that the element will keep whatever is listening around until the element is GC'd.
Listening does not add a reference from the listener to the element, so the element can be GC'd before the listener.
The way to think about GC is references- when object A has a reference to object B, object A keeps object B around. But unless object B has a reference back to object A, B will not prevent A from being GC'd. Events are references from the target to the listener, but not the other way.
In addition, if object A has a reference to object B and B has a reference to A, then they are both keeping each other around, but if nothing has a reference to either of them then they can both be GC'd.
What this all means is that if the code listening to the element is expected to have a lifecycle approximating the element, or longer than the element then it does not need to unlisten. But if the element is expected to be long-lived and the listener short-lived, then the listener should be detached (which makes sense- why would a short-lived listener still be listening to something?). (Answer courtesy of Google engineer Pete Blois).