You can implement any type of coordinate transform you want in shaders. However, there is an important restriction:
If you draw two connected vertices (i.e. a straight line, or the edge of a triangle), the result will always be a straight line on the screen — it is not possible to do otherwise in OpenGL.
The Cartesian-polar transform turns straight lines into curved lines. This means that if you want to transform a straight-sided shape and get the “right” curved result, you must draw it using a sequence of closely-spaced vertices — as many as you need to produce the “resolution” of smooth curvature you want. This is generally not hard to program, but it is something to be aware of.