Why not use unmanaged safe code in C#

Question

There is an option in C# to execute code unchecked. It's generally not advised to do so, as managed code is much safer and it overcomes a lot of problems.

However I am wondering, if you're sure your code won't cause errors, and you know how to handle memory then why (if you like fast code) follow the general advice?

I am wondering this since I wrote a program for a video camera, which required some extremely fast bitmap manipulation. I made some fast graphical algorithms myself, and they work excellent on the bitmaps using unmanaged code.

Now I wonder in general, if you're sure you don't have memory leaks, or risks of crashes, why not use unmanaged code more often?

PS my background: I kinda rolled into this programming world and I work alone (I do so for a few years) and so I hope this software design question isn't that strange. I don't really have other people out there like a teacher to ask such things.