Found a solution:
# makes everything nan to start with
bestepsilons1 = numpy.zeros(R.shape[0])+numpy.nan
# finds the indices where the entire column would be nan, so the nanargmin would raise an error
d0 = numpy.nanmin(R, axis=1)
# on the indices where we do not have a nan-column, get the right index with nanargmin, and than put the right value in those points
bestepsilons1[~numpy.isnan(d0)] = epsilon[numpy.nanargmin(R[~numpy.isnan(d0),:], axis=1)]
This basically is a workaround, by only taking the nanargmin on the places where it will not give an error, since at those places we want the resulting index to be a nan anyways