No, your app's size won't directly increase, because they are all built on top of the same base SDK.
Some things (among many others) that may increase the size are:
- additional libraries that you may need to add if you want an old version of iOS to mimic a new feature;
- or any additional code that you may need to grant compatibility with older versions;
- different image sizes for retina/4" compatibility;
- different nib files for different screen sizes;
- multiple architectures
Edit:
As Neal remembered, different architectures will increase the bundle size, and older devices have different architectures than newer devices. So in a certain point of view, yes, your app will probably be larger if you support from iOS 5 to iOS 7, for instance. But that really depends on which hardware to you want to support, which can indirectly increase the app size.