The estimation of a suboptimal radius is described in the OPTICS paper
Looking for Natural Patterns in Analytical Data. 2. Tracing Local Density with OPTICS
As outline in the paper there are assumptions to make this formulation useful.
To sum up, citing the article, the density of the objects of a dataset can be compared with the density of the same number of objects uniformly distributed in the same volume as the data set. If the data set has a uniform distribution, then the neighborhood radius eps , containing k points can be estimated.