I'm not an expert in this subject, but I believe you have some false assumptions here:
Value typed structs are stack based variables that reside in processor cache rather than RAM. So by avoiding trip via system bus from processor to RAM using value-typed variables should be faster than using reference type variables.
Just because something is a reference type doesn't mean that the CPU cache won't be used. The stack isn't the only area of memory that can use the cache. In addition, the CPU is pretty smart at things like pre-fetching to cache, so you don't typically have to micro-manage your memory like this.
Also, keep in mind that as you access your struct
within your loop, it's not just the instructions within the getter and setter you have to worry about; there's also overhead anytime you do a method call, which includes indexers. A method call involves pushing the parameters to the stack, doing the jump to the method instructions, pushing the return value to the stack, doing the return jump, etc. So it's no surprise that this is more costly than the simple instructions for setting an array value.