I'd avoid averages are they are prone to outliers influencing the results - what you want is a histogram of page load times so you can tell how long the page takes to load for say 50% or 95% of visitors.
Google Analytics offers some Real User Measurement (uses Navigation Timing API which isn't supported by Safari yet)
You can find the numbers via Content > Site Speed > Page Timings and then the performance tab.
By default it uses a very low sample rate and you can increase this by adding
_gaq.push(['_setSiteSpeedSampleRate', 100]);
(Google Limits you to 10K samples a day though)
Armed with the load times, compare them against your business goals - conversions etc. - to see what effect performance is having on visitor behaviour.
EDIT
Other thing to do is use webpagetest.org and create a video of the page loading