The biggest disadvantage would be that there could be slight implementation changes when using the actual android jar vs the compatibility library. This is why the compatibility library does not switch implementation when running on 3.0 and above, even though it could.
To keep everything consistent and avoid headaches, keep the support library implementation globally unless you are implementing something specific to one api version.