PDE build with Ant is extremly slow compared to Eclipse

Question

I have a very large OSGi project with about 300 bundles. When I import the project in Eclipse and do a Project -> Clean... and Project -> Build All it takes about 10 minutes until everything is build and I can start my application.

I think that 10 minutes are already pretty long, but when I execute a headless PDE build with Ant scripts it takes about an hour! I think something goes pretty wrong...

Do you have any hint for me how to speed-up the Ant-based build? Or how to track the problem down? I use the normal PDE build with a command like below. I have no idea why it is so much slower compared to a complete build in Eclipse.

java -jar eclipse/plugins/org.eclipse.equinox.launcher_<version>.jar -application org.eclipse.ant.core.antRunner -buildfile  eclipse/plugins/org.eclipse.pde.build_<version>/scripts/build.xml -Dbuilder=<path to the build configuration folder>

Thanks, Thomas

PS.: I know the question is very general, but even a hint how to find the performance lack would be helpful.

Answer 1

What you are seeing is expected, because you are (probably) comparing two different things:

When forcing a rebuild in the Eclipse IDE, you are simply compiling all java-files. (For our 270-bundle project, this takes about 1-2 minutes.)

When building with PDE-build, you are packaging the product as well (generating bundles and features, resolving plugin dependencies, JAR-ing everything, etc), and this takes much more time. (For our project this takes about 6-7 minutes.) The equvivalent of this in the IDE would be to open your .product file, and on the "overview"-tab in the editor, you select "eclipse product export wizard".

As for tips on how to speed up the build: There is no silver bullet that I know of. But: Search for "tycho build eclipse", and/or see http://www.vogella.com/tutorials/EclipseTycho/article.html and https://eclipse.org/tycho/. In other words: You can change your build to use Tycho, which enabeles you to build each plugin separately (and incrementally if you want).

Answer 2

I work on a large RCP project with around 50k classes spread over 850 projects/bundles in the workspace (with a similar number of bundle sin the target platform). We have SSDs in both our desktops and in the Jenkins servers to speed up the build. Builds are far quicker on Linux than on Windows. Windows workspace refresh and clean all takes around 2-3 minutes whereas a full Linux headless build takes around 25 minutes (on Windows it is over an hour).

The PDE headless build process is not very efficient, and it does a lot of crazy I/O. At the start of the build it is common to see the scripts pause for minutes on the "generateScript" and "generateFeature" stages. If you profile these you will see that there are millions of reads to build.properties files when it calculates the dependency tree (I got to 46 million reads for ~1000 build.properties and then decided to stop) so something isn't right there. The PDE Headless build system is now effectively abandoned at Eclipse in favour of Tycho, so I doubt it will get looked at.

If you have Eclipse features, make sure the inclusions are sane - by including the same feature in multiple places we found that the above generateScript/generateFeature stages could take much longer than needed.

If you can build your project in phases such that you can use the product of one phase as a target platform for the next do so - it really helps.

Finally, try and move your builds to Tycho/Maven. We haven't been able to do this except on some of our supporting projects yet (partly because of how our target platform is built, partly due to the use of customBuildCallbacks in several places, and generally due to the lack of time) but it does appear to be a much better build system.