What you're looking for is discretization. Behind this obscene name stands a simple principle : you can't deal with an infinite amount of data.
You need then to perform a transformation on your world : instead of allowing your character/unit to go at any location (x and y being real numbers), you can divide your world into some sort of a grid (this is what the navigation mesh and waypoints are doing), and only allow your char to go on these cells (or points, you can see it as you want). This is discretising : you are going from continuous values (real coordinates) to discrete values (integer coordinates / point). The more precise you go, the nicer it'll look.
After doing this, assigning moving costs between cells/points is rather simple, and performing A* on it as well.