You can (try) and with sufficient effort, you can probably make things work; even w/o resorting to unsafe
code. But I think you'll find the C++/CLI syntax for interop situations sufficiently similar to C#.
Perhaps the biggest problem with this approach however is the long-term maintenance of your interop layer. Every time something changes on the C++ side, you could have to manually make changes to C#. And you won't get much help from the C# compiler alerting you to such things. It's only at runtime when the mangled C++ name can't be located in the DLL that you'll know.
It's can also be error-prone to re-implement C++ .H files in C#, especially if you need the same code to work for both 32-bit and 64-bit.