Is this a legitimate way to get around having an "unsafe" assembly, or is accessing the byte[] 1 byte at a time the only legitimate way to do this in a safe manner?
This is legitimate, in that it 1) works, and 2) can produce the correct results.
Granted, it's not necessarily "safe" in that you can do things that are "nasty" by doing this, but it will still function. You are definitely violating the spirit of safe code in this case, as you can do things that shouldn't be allowed in "safe" code, such as effectively bypass array length checking and cause buffer overruns.
Accessing the array one element at a time is more in the spirit of "safe" code, but there are times when using tricks like this is valuable, as it provides a mechanism for performance when required. If performance is required for your specific scenario, and the normal mechanisms will not perform adequately, this may be a way to make code that "works" in that it meets your specifications (the perf. requirements) without violating the safe constraints of marking an assembly as unsafe.