The P/Invoke marshaler can't handle the last parameter as an array. In .Net arrays have to have a size associated with them, while in C they are just memory blocks. Since the marhsaler doesn't know the size of the array it can't handle it. However you can handle it manually using an IntPtr
[DllImport("myLib.dll", CallingConvention = CallingConvention.Cdecl)]
private static extern void f2(float[] a, float[] b, uint n, out IntPtr rPtr);
Then you can pull the array values out of the pointer using Marshal.Copy:
float[] r = new float[size];
Marshal.Copy(rPtr, r, 0, size);
Also you'll need a function call in your DLL to free the memory allocated for the array by f2
since there isn't any way to free memory that was allocated in a C DLL.