Turns out, you've got to use switchTextAppearance and define textAppeareance attribute in style resources to change (or do equivalent in your java code).
The textColor attribute in Switch class is pretty misleading in this case but necessary as the class inherits (not directly) from TextView class