Suppose your larger image size is 800x600 pixels (which is already very small by today's standards), and that a typical JPG of this size is 120 KB.
Now suppose you want to display a page full of thumbnails (let's say there are 100 thumbnails in your page).
The browser will have to download 100 * 120KB = 12MBs. Where I live, with a decent ADSL line, that takes 30 seconds.
Now let's say you reduce the images to the target thumbnail size, which makes each thumbnail 25KB. That makes a total of 2500 KBs to download, which reduce the time to load the whole page to 6 seconds, to diaplay exactly the same information.
And if you pay for your server bandwidth, you'll see a similar reduction in cost as well.
So yes, it's a good idea to reduce the image sizes. Especially if your largest image size is big. If you take the same example and start with a 1280x960 image, you'll start at something like 500 KB per image, and make it more than 2 minutes to see your thumbnails.
And I don't even talk about rural or less-developed regions, where the bandwidth is much less than mine.