If I'm understanding your question correctly, you're trying to maintain the restriction of the limit and start ordering by the offset? If so, something like this could perhaps work:
select id
from mytable
order by !(id>3), id
limit 5
I've altered the input to include a 6th record to show the limit functionality.