What is happening is that your mouse events are completely halting your events. I suggest filtering them out using list comprehension beforehand:
try:
events = [event for event in events if event.key >= 273 and event.key <= 276]
except AttributeError:
continue
This makes sure that the keys are one of the arrow keys. Otherwise, remove it from the list of events.
Hope this helps!