You are switching the detail table in and out of filtered mode. This switching is what's taking up the time. Because the switch causes a refresh from the database.
As per greymatter's suggestion you'd be better of with to TTable's one permanently linked to the master table and one free.
Another way to speed things up more is to have 2 TDbGrids. One visible the other one hidden. If you want to show all records hide the grid linked to the detail table and show the one linked to the free table and visa versa.
That way you'll not incurr the cost of the switch. You may want to remember syncing the active row on the displayed and hidden grid (if possible) before the swap. Otherwise it might be jarring for the user to see the rows jump around too much.
Obviously you'd need to disable (grey out) the master table's Grid, so as to let the user know he's now looking at the detail table on its own without filtering.
Finally another option is to dispense with the master-detail and show a joined view of both tables. This means that rows from the master table would repeat. Something like.
select d.d1, d.d2, d.id, m.m1 from detail d
left join master m on (m.id = d.masterid)
Whether this makes sense in your application you'll have to decide.