I've seen that behavior quiet often, first of all let me tell you that finishing an activity doesn't necessarily mean that the OS will release the resources, OS is smart enough to leave some resources around in case that you might decide to open again that activity even when you previously destroyed it, now as you open more activities the OS could decide to completely let go some resources of previous "destroyed" activities since it might need those resources for the newest activity.
Regarding to your memory issue I'm almost sure it has to do with the Image Resizing Mechanism, even when your application has images with just a few KBs, those images might become several MBs if not stored in the proper drawable-folder, because the OS will try to resize those images to fit properly in your activity by using the formula: ImageWidth * ImageHeight * 4Bytes, 1Byte for Alpha/R/G/B, and in the DDMS by tracking the Heap you can notice if when you go to one activity the Heap increases stupidly fast.
In order to make sure the issue is not related to the Resizing Mechanism, put all your drawables in a folder named: drawable-nodpi (This way you tell the OS NOT TO RESIZE), if you see a huge difference in performance then it means that the problem is that your drawables are not properly designed in size per each screen density...
Hope this helps.
Regards!