What you are seeing is expected, because you are (probably) comparing two different things:
When forcing a rebuild in the Eclipse IDE, you are simply compiling all java-files. (For our 270-bundle project, this takes about 1-2 minutes.)
When building with PDE-build, you are packaging the product as well (generating bundles and features, resolving plugin dependencies, JAR-ing everything, etc), and this takes much more time. (For our project this takes about 6-7 minutes.) The equvivalent of this in the IDE would be to open your .product file, and on the "overview"-tab in the editor, you select "eclipse product export wizard".
As for tips on how to speed up the build: There is no silver bullet that I know of. But: Search for "tycho build eclipse", and/or see http://www.vogella.com/tutorials/EclipseTycho/article.html and https://eclipse.org/tycho/. In other words: You can change your build to use Tycho, which enabeles you to build each plugin separately (and incrementally if you want).