Indeed, endianness is your issue here. While not difficult to work around, it can be annoying at times on Intel-based systems due to them being in Little Endian whereas Network Order is Big Endian. In short, your bytes are in the reverse order.
Here is a little convenience method you may use to solve this issue (and even on various platforms):
static uint MakeIPAddressInt32(byte[] array)
{
// Make a defensive copy.
var ipBytes = new byte[array.Length];
array.CopyTo(ipBytes, 0);
// Reverse if we are on a little endian architecture.
if(BitConverter.IsLittleEndian)
Array.Reverse(ipBytes);
// Convert these bytes to an unsigned 32-bit integer (IPv4 address).
return BitConverter.ToUInt32(ipBytes, 0);
}