Can't pass table to $wpdb->prepare

Question

I'm having trouble passing a table variable to $wpdb->prepare(); Here is functioning code:

$table = $wpdb->get_blog_prefix( $blog_id ) . 'my_table';

$voters = $wpdb->get_row($wpdb->prepare("SELECT * FROM $table WHERE user_id= %d AND post_id = %d;", $user_ID, $post->ID  ));

This works great. However I think I should also be including my table in the prepare statement. But, it breaks when I change it to this:

$table = $wpdb->get_blog_prefix( $blog_id ) . 'my_table';

$voters = $wpdb->get_row($wpdb->prepare("SELECT * FROM %s WHERE user_id= %d AND post_id = %d;", $table, $user_ID, $post->ID  ));

Any ideas on why it might be breaking?