Build server: Best practices managing third party components?

StackOverflow https://stackoverflow.com/questions/1017023

Question

I'm maintaining a quite large legacy application. The source tree is a real mess. I'm trying to set up a build server.

On the source tree, I've third party component with sources (also in the project's include path). These components are also installed within the IDE.

My question is : How to manage those components ?

I thought to manage this way:

  • Install the IDE on the build server
  • Install all the third party component
  • Remove the component sources from the project sources tree (and keep them on the project root on a dedicated folder each zipped)
  • Each time we need to customize (or debug) a third party component we re-build the package and re-install it in the IDE of the build server (and on each developers workstation)

What's the difference between having the components installed in the IDE and having the sources in the include path ? How the linker handle that case ?

Was it helpful?

Solution

We have set up our daily builds using simple command files.

  1. Each project (.dpr) has an associated Build.cmd file.
  2. All Build.cmd files are called from our main BuildServerRun.cmd file.

The BuildServerRun.cmd file takes care of

  1. Deleting the entire source tree on the buildserver.
  2. Getting a latest version from our source control repository.
  3. Call each Build.cmd and pipe the output to a file.
  4. Mail the results to all developers.

All paths to external components are configured in the dcc32.cfg file

..    
-u"c:\Program files\Developer Express Inc\ExpressInplaceEditors\Delphi 5\Lib"
-u"c:\Program files\Developer Express Inc\ExpressQuantumGrid\Delphi 5\Lib"
..
-r"c:\Program Files\Borland\Delphi5\Lib"
-r"C:\Program Files\jvcl\jvcl\resources"
..
-i"C:\Program Files\jvcl\jvcl\run"
-i"C:\Program Files\jvcl\jcl\source"

Example of a Build.cmd.

Note: we have a policy to compile to bin\dcu, exe to bin, hence the -N, -E directives.

@echo on
dcc32speed -B -Q -W -H -Nbin\dcu -Ebin BpABA.dpr
@echo off

Example of snipped BuildServerRun.cmd

SET %Drive%=E:

:BuildServer
REM *************************************************
REM     Clear files
REM *************************************************
ECHO. > "%Temp%\BuildLieven.txt"
ECHO. > "%Temp%\TestRunLieven.txt"

REM *************************************************
REM     Set start time
REM *************************************************
echo | TIME | FIND "Huidige tijd" > "%Temp%\ResultLieven.txt"

REM *************************************************
REM     Get latest versions
REM *************************************************
IF %LatestVersion%==1 CALL %Drive%\buildserver\latestversion.cmd
ECHO "Latest versions opgehaald" >> "%Temp%\ResultLieven.txt"

REM *************************************************
REM     Build projects
REM *************************************************
CD %Drive%\Projects\

ECHO ***************************************************************** >> "%Temp%\BuildLieven.txt"
ECHO BpABA >> "%Temp%\BuildLieven.txt"
ECHO ***************************************************************** >> "%Temp%\BuildLieven.txt"
CD %Drive%\Projects\BPABA\production
ECHO Building BPABA\production
CALL Build.cmd >> "%Temp%\BuildLieven.txt"
CD %Drive%\Projects\BPABA\test
ECHO Building BPABA\test
CALL Build.cmd >> "%Temp%\BuildLieven.txt"
CD %Drive%\Projects\BPABA\test\dunit
ECHO Building BPABA\test\dunit
CALL Build.cmd >> "%Temp%\BuildLieven.txt"
ECHO BPABATests >> "%Temp%\TestRunLieven.txt"
ECHO Running BPABATests
CALL bin\BPABATests >> "%Temp%\TestRunLieven.txt"
CD %Drive%\Projects
ECHO. >> "%Temp%\BuildLieven.txt"
ECHO. >> "%Temp%\BuildLieven.txt"
ECHO. >> "%Temp%\BuildLieven.txt"

REM *****************************************************************
REM     Gather (Fatal)Errors/Hints/Warnings & Failures
REM *****************************************************************
ECHO ***************************************************************** >> "%Temp%\ResultLieven.txt"
ECHO (Fatal)Errors/Hints/Warnings en Failures >> "%Temp%\ResultLieven.txt"
ECHO ***************************************************************** >> "%Temp%\ResultLieven.txt"
ECHO Fatal errors during build >> "%Temp%\ResultLieven.txt"
TYPE "%Temp%\BuildLieven.txt" | FIND /c "Fatal:" >> "%Temp%\ResultLieven.txt"


ECHO Errors during build >> "%Temp%\ResultLieven.txt"
TYPE "%Temp%\BuildLieven.txt" | FIND /c "Error:" >> "%Temp%\ResultLieven.txt"

ECHO Warnings during build >> "%Temp%\ResultLieven.txt"
TYPE "%Temp%\BuildLieven.txt" | FIND /c "Warning:" >> "%Temp%\ResultLieven.txt"

ECHO Hints during build >> "%Temp%\ResultLieven.txt"
TYPE "%Temp%\BuildLieven.txt" | FIND /c "Hint:" >> "%Temp%\ResultLieven.txt"

ECHO Failures during test >> "%Temp%\ResultLieven.txt"
TYPE "%Temp%\TestRunLieven.txt" | FIND /c "Failures:" >> "%Temp%\ResultLieven.txt"
ECHO. >> "%Temp%\ResultLieven.txt"

ECHO ***************************************************************** >> "%Temp%\ResultLieven.txt"
ECHO Controle #Projecten = #Compiles >> "%Temp%\ResultLieven.txt"
ECHO ***************************************************************** >> "%Temp%\ResultLieven.txt"
ECHO #Projecten >> "%Temp%\ResultLieven.txt"
TYPE "%Drive%\buildserver\buildserverrun.cmd" | FIND /i "cmd >> " | FIND /i "Lieven" | FIND /i /v /c "FIND /i /v /c" >> "%Temp%\ResultLieven.txt"
ECHO #Compiles >> "%Temp%\ResultLieven.txt"
TYPE "%Temp%\buildLieven.txt" | FIND /i /c "dcc32" >> "%Temp%\ResultLieven.txt"
ECHO #Tests expected to run >> "%Temp%\ResultLieven.txt"
TYPE "%Drive%\buildserver\buildserverrun.cmd" | FIND /i "TestRunLieven" | FIND /i "CALL" | FIND /i /v /c "FIND /i /v /c" >> "%Temp%\ResultLieven.txt"
ECHO #Tests actually run >> "%Temp%\ResultLieven.txt"
TYPE "%Temp%\TestRunLieven.txt" | FIND /i /c "DUnit / Testing" >> "%Temp%\ResultLieven.txt"
ECHO. >> "%Temp%\ResultLieven.txt"
ECHO. >> "%Temp%\ResultLieven.txt"

ECHO ***************************************************************** >> "%Temp%\ResultLieven.txt"
ECHO Detail (Fatal)Errors/Hints/Warnings en Failures >> "%Temp%\ResultLieven.txt"
ECHO ***************************************************************** >> "%Temp%\ResultLieven.txt"
TYPE "%Temp%\BuildLieven.txt" | FIND "Fatal:" >> "%Temp%\ResultLieven.txt"
TYPE "%Temp%\BuildLieven.txt" | FIND "Error:" >> "%Temp%\ResultLieven.txt"
TYPE "%Temp%\BuildLieven.txt" | FIND "Warning:" >> "%Temp%\ResultLieven.txt"
TYPE "%Temp%\BuildLieven.txt" | FIND "Hint:" >> "%Temp%\ResultLieven.txt"
TYPE "%Temp%\TestRunLieven.txt" | FIND "Failures:" >> "%Temp%\ResultLieven.txt"

REM *************************************************
REM     Set stop time
REM *************************************************
ECHO | TIME | FIND "Huidige tijd" >> "%Temp%\ResultLieven.txt"

REM *************************************************
REM     Send results
REM *************************************************
CALL %drive%\buildserver\Blat.cmd

OTHER TIPS

My answer is more general than Lieven's answer, which is Delphi-specific. I wrote this one shortly after the question, but went to a co-worker before submitting ;)

I refuse to install any IDE on our main Windows build agent. Sounds like a nightmare to me. The MSBuild engine handles all build scenarios well, and other than .NET, you just need the Windows SDK installed. Or you could use NAnt and even CMake, whatever. Just don't install IDEs. It's not fun on build servers.

Now you have tagged this as Delphi. I don't know how good it works there, but as Lieven wrote, Delphi comes with a command line compiler. I just don't have any experience how it works with third-party compnents, but I think Delphi supports MSBuild in the latest version.

I'm also unsure whether including thiry-party components into version control is a good thing, because of the space it takes - though you can also put them somewhere else and include them as external, which makes it much smaller, but also imposes the problem that upgrading the components for one app will upgrade them for all - so you better have good integration tests. But that is the point of a build server anyway.

Apart from that, it is always a good thing to check-out and have all components required to build the application available. You don't need to install components into an IDE if they were made well. Depending on what components they are, in many cases you don't even need to install them on developer machines. Many .NET components, for example, are available in the designer when you add a reference to them. And licensing is typically no more than "put the license file into the same directory". Well, that's how it should be, at least. If that's not how it works in Delphi today, that's likely one of the reasons Delphi is on its way out. Other than the Borland/Inprise/DevCo/Codegear/Embarcadero hassle.

Similar situation here, fortunately it did not start with a big mess. It is true that the real problem lies in the IDE configuration, which needs to point to the correct versions of the third party components if you check out an older version. The only solution I have heard of is the use of different registry branches for the different product release configurations.

The components are kept in a separate directory structure, and use version numbers in the directory names when possible. This makes it easier to check out old versions and have the build scripts point to the correct version.

We use Apache Ant as the main build tool for years now and it really does everything we need, including unit test invocation and Innosetup script generation.

Compilation of packages is only necessary you ship the executable with BPLs, otherwise it is not necessary on a build server.

Installing the components in the IDE is also not necessary on the build server.

You can use Owly CI tool.

It allow build projects in easy way by defining manifest file. It also allows handle dependencies - you can wrap 3rd party components in owlyci packages and mark them as dependencies to the main project.

There is an example, how to use it with Jenkins CI system.

Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top