DBSCAN doesn't claim the radius is the maximum cluster size.
Have you read the article? It's looking for arbitrarily shaped clusters; eps
is just the core size of a point; roughly the size used for density estimation; any point within this radius of a core point will be part of a cluster.
This makes it essentially the maximum step size to connect dense points. But they may still form a chain of density connected points, of arbitary shape or size.
I don't know what cluster 0 is in your R implementation. I've experimented with the R implementation, but it was waaaay slower than all the others. I don't recommend using R, there are much better tools for cluster analysis available, such as ELKI. Try running DBSCAN with your settings on ELKI, with LatLngDistanceFunction and and sort-tile-recursive loaded R-tree index. You'll be surprised how fast it can be, compared to R.
OPTICS is looking for the same density connected type of clusters. Are you sure this arbitrarily-shaped type of clusters is what you are looking for?
IMHO, you are using the wrong method for your goals (and you aren't really explaining what you are trying to achieve)
If you want a hard limit on the cluster diameter, use complete-linkage hierarchical clustering.