I think that people mean that it breaks ABI (binary) compatibility, not the API (source) compatibility.
Although the syntax is identical, behind the scenes, access to properties and access to member variables are compiled differently.
That said, if your variable/property is not to be used from an assembly that you yourself do not compile, then there is no harm in changing it. But if it is part of a public interface, then it is better to make it a property, so that you will not regret it in the future.