I would say it is not a good solution.
It will work, but from a client implementer's point of view it looks like you will have these operators defined, but then they depend on the streams being in binary mode (which is not explicit or suggested by your interface - don't do that) and it requires the use of reinterpret_cast (don't do that either). Your restriction to POD types will be hidden as well, unless you drop comments somewhere ("// DO NOT declare non-POD types here
").
If you add non-POD types (or use non-binary streams), the errors generated will be silent data corruptions.
You are better off with a separate implementation for each of them; This will not impose any constraints with working with PODs either and the implementation will respect the principle of least surprise.