You should do following to make it faster.
Paging : Do custom paging instead of standard paging so only page-size amount of data is stored for the current page. It will hugely improve the performance. Please refer Custom Paging.
Also in the page, disable the view-state of controls if not required.
You can also use caching if grid-view data is read-only. Caching