Jim Lyon's answer pointed me to the exact solution using basic triangle geometry:
to-report link-distance [ x y ]
let a [ distancexy x y ] of end1
let b [ distancexy x y ] of end2
let c link-length
let d (0 - a ^ 2 + b ^ 2 + c ^ 2) / (2 * c)
if d > c [
report a
]
if d < 0 [
report b
]
report sqrt (b ^ 2 - d ^ 2)
end
This works for both wrapping and non-wrapping worlds.