If the coordinates you supply to polyarea
are measured in pixels, then the result will be measured in square pixels.
If you know the width of a pixel (in mm), then your area (in mm2) is pixelarea * pixelwidth2, i.e.:
area = polyarea(X,Y) * power(pixelwidth,2)