It's nontrivial. We are open to volunteers to help with layout algorithms. Emden Gansner and I made a big start on a complete overhaul of dot (with cleaner data structures for subgraphs and long edges, and use sifting for mincross) but we didn't finish. We wrote new level assignment cpde that I think Emden back-hacked into the old dot layout engine. it is probably a few months of effort if someone wants to pick it up and take it further. Contact us at graphviz.org
In general constraint machines (like dot) can be a little brittle. Either they get the right solution on their own, or maybe you can add a few tweaks but as soon as the input changes even a little, the tweaks break everything and it's no good. Because of the serial nature of the layout passes, it is hard for example to favor a different level assignment in order to reduce crossings later. There might be some work in the Graph Drawing conference if someone wants to look it up and implement it.
Stephen North