remove_filter( 'comment_author', 'floated_admin_avatar' ); doesn’t work

Question

In a small plugin I’ve written to add an IP address column to the comments list I want to remove the avatars.
In line 156 I tried to use remove_filter:

remove_filter( 'comment_author', 'floated_admin_avatar', 50 );

Well … I know this function fails sometimes, but why does it fail here?

My workaround is:

$_GET['comment_status'] == 'spam' 
    and add_filter( 'pre_option_show_avatars', '__return_zero' );

I guess I stared too long at this code to see what went wrong. I need a hint into the right direction.