Read http://www.salesforce.com/us/developer/docs/pages/Content/pages_controller_get_request.htm
especially that part:
- The constructor methods on the associated custom controller or controller extension classes are called, instantiating the controller objects.
- If the page contains any custom components, they are created and the constructor methods on any associated custom controllers or controller extensions are executed.
So in the first initial request the constructors (and getters) of main page and all embedded pages/components will fire to render the initial state. That means it's all one transaction, one context and the 50K rows limit is shared across them.
Maybe you can annotate your pages with readonly="true"
or try to make the constructors slim (not wasting too many queries). Put a "fetch data" button & action method - user will have to make 1 more click but this will count as separate action = fresh 50K rows limit.