The sorting is happening within each group, aka, "working as intended." You're creating a somewhat arbitrary and random set of record groups based on row position in an un-ordered set.
You need to find a way to sort properly (in the SQL?) prior to applying any sort of grouping, or use a less random means of paging.