Patch or Core Hack

Question

When I'm on a system upgrade project, one of the things I do is diff a client's system against a fresh Magento installation. I'm looking for core hacks or additional files that aren't part of standard Magento to make sure I catch any shody-but-business-critical-work done by a previous freelancer, contractor, consultant, or agency.

One thing that always stumps me though is patches. Over the years Magento has issued "between-version" patches — usually to address a security fix, or a change in a shipping/payment vendor's API.

The problem is, from the point of view of a diff, patches are indistinguishable from core hacks — especially when you don't know which patches (if any) have been applied to the system.

Which leads to my question.

How do you differentiate between a core hack and a patch?