The weak reference is an implementation detail and just adds complication for the API users for no reason. Unless they are every going to do with the weak reference just have them pass the object in and do the weak reference yourself.
Be aware though that there are limitations with this pattern, I tried it myself before and discovered it was actually more hindrance than help. Because the weak references get dropped I couldn't actually just add a listener and then forget about it - I had to also keep a reference to that listener elsewhere just to stop it getting GC.
This may not apply to your use-case but in many cases listeners are implemented as anonymous inner classes and they are attached to listen but no other reference is then kept to them. That is not possible if you store them using weak references.