The circumstances were slightly different, but both my co-workers and I have seen evidence that if you have something like this:
select something
from LinkedServer.DataBase.Owner.Table
where whatever
then sql server will select the entire table from the other server first, and apply the where clause afterwards. That might be happening to you.
We solve the problem by using openquery instead of the fully qualified method shown above, and by putting the openquery results into a temp table. Then we join to the temp table.