Normally you would not use field values as the second and third argument in the geodist call. The first parameter is the location field to use, the second and third are the lat / long to compare this to. This is usually based on search input.
If you use field values for this, you are basically comparing the distance between values within document itself. I don't even know if Solr supports that.
Can you manually create a Solr query that returns the results you need? I can probably 'translate' that to Solarium calls.