This is hazy by design, and it's nothing you can easily fix. IP-based geolocation services can only use the position information they guess from the various points from their server to the target IP. (You can see those points when you do a traceroute <targetip>
on your console). How exact IP geolocation is depends on your user's ISP's architecture, and results can be way off in some areas, but pretty exact in others (especially cities).
If you seriously need exact geolocation, use client-side HTML 5 geolocation. This requires the user's consent, though, and can be imprecise as well if the client device doesn't have a GPS.