There are two approaches to avoiding overlap in a d3 force layout.
The first is to adjust the parameters of the force object, the most relevant of which is the "charge" parameter. Nodes with negative "charge" values push other nodes away (versus nodes with positive values pull other nodes closer), and you can increase the amount of charge to cause a greater push.
The default value for the "charge" is -30, so from there you can adjust it until you get an effect you want. There's no simple formula for determining the value you want, since it also depends on the other parameters and the amount of links in your graph.
If for any reason adjusting the charge doesn't work for you (for example, if you want the nodes to cluster close to each other -- not repel each other -- but still not overlap), then you can manually check for overlapping nodes, as in this Mike Bostock example suggested by Josh in the comments.