Calling a p/invoke from managed code requires that your code be executed with full trust (I think prior to v4 it needed the UnmanagedCode permission which was a full trust equivalent) since calling native code could be dangerous, so it's not possible for code that's being run in a mode that the CLR doesn't trust to p/invoke.
Code marked as unsafe also will only run if the code is trusted. It means something very specific, though, and is different from calling native code via a p/invoke. Being marked 'unsafe' means that the code itself is unverifiable. That is to say, the code itself may be doing dangerous, non-type-safe things.
So, they're just two different ways that .Net programs can be potentially dangerous. The unsafe keyword isn't meant to be an overall category for any potentially unsafe operations, only those specifically related to whether or not the managed code is verifiable or not (it says nothing about whether that 'verifiable' code is potentially launching new processes, or modifying important system files or ACLs - also dangerous things requiring full trust).
More 'unsafe' details here: http://msdn.microsoft.com/en-us/library/t2yzs44b.aspx