Operators are typically defined as static members:
type shape =
...
static member (-) (x, y) =
match x, y with
| Point (x0,y0), Point (x1,y1) -> Vector (x0-x1,y0-y1)
| Point (x0,y0), Vector (x1,y1) -> Point (x0+x1,y0+y1)
| _ -> failwith "invalid arguments"
A few notes about your attempt:
- union cases are not types, so they can't be used to define method overloads
- functions can't be overloaded