The solution to your problem is Be sure to set the right minimum-stability. To get started or be sure this is no issue, set minimum-stability to "dev".
Since my/project requires my/otherproject dev-master, your root package must allow my/otherproject to be installed in dev versions.
Either you set "minmum-stability":"dev"
, which will allow all packages to be installed in dev, or you can add a require like "my/otherproject":"@dev"
which just allows this one to be dev, but not others.
Obviously a better solution to this is to start tagging releases. That way my/project can require 1.* or something of my/otherproject, and you won't have problems anymore.