why does Delphi decide to recompile certain files needlessly

Question

When I build my projects, I expect to see all the units recompile in the compilation progress. When I "make" the project, I expect only those units where the source has changes to be recompiled. When I make immediately after a full build I expect to see the link happen and nothing else.

For some reason Delphi has got it into it's head to recompile certain units regardless. The main one I notice go past is idIOHandler.pas - part of Indy. It sometimes compiles others - always some unit in Indy. I have searched the Indy source folders in vain for files with screwy datestamps.

(Occasionally I see the opposite problem - where source units I know I have changed are not recompiled. I put this down to time differences between my PC and the server where the sources are held.)

It's not a biggie, but I'd be keen to hear an explanation.

Answer 1

Unfortunately the source code for the Delphi compiler and its dependencies is closed source, so this question calls for speculation. However, if you would be interested in avoiding having Indy units recompile, you could of course place only the DCU files in the library path, and remove the Indy source files from your search path (project level) and library path (global IDE level) configuration. If you are indeed compiling source code across mapped network drives, then I think you're crazy. I recommend you check out mercurial and this cool feature where you can clone repos and keep copies of all your source code completely on the same computer you are building from. It's cool.

Anyways, the compiler... What I have observed so far is that:

1. What looks like a recompilation to you (and to me) is in fact, actually possibly only a scan of the Interface section, and is in fact the way that a tree of dependencies is built. Unlike a C (makefile) or Java (ant or maven) style build environment, setting the compiler loose at least on the interface section of your code is the only way to find out all the dependencies of a particular module. I think that even if you remove the Indy units from your search path, you may still see compiler progress showing a "compile" of IdIOHandler.pas, but in this case, what it would be doing is loading up a compiled representation of the interface section of your pascal unit from the DCU file, and then deciding what other files to read. Sometimes the IDE progress will show "compiling" on units but not modify the DCU file at the end.

2. When fighting a large battle to refactor or fix a large project after some breaking change I have observed the following: a. That it is very difficult if not impossible to guess where the next syntax error will break compilation. b. That the compiler breaking on fatal error in unit A, then halfway through unit B, then on unit A again, then on unit C, then on unit B again, suggests that the compiler is nothing like what your mental model of a C compiler based on makefiles would be like: First we compile unit A, then we compile unit B, then we compile unit D. All the evidence I have suggests that the real process is far more convoluted than this, and that inside what someone at the compiler team would call a "single pass" over file A, there are "scans" of other files contents or scans of other files' internal representations in memory (rather than a parse of the physical file content) that leads to "compilation" not being an all or nothing or all at once thing, and rather a big in-memory free for all, and that the compiler's order of unit evaluation is driven by some in-memory tree built by the interface, and then implementation clauses of each unit. Predicting the behaviour of this large cyclic graph in a real world application is beyond my skill.

Answer 2

It depends entirely on how Delphi decides that something needs recompilation.

I have seen one thing cause this (in a totally non-Delphi environment). Where, for some reason, the files are given dates in the future, files that depend on them will always recompile.

For example, let's say for some reason myprog.c has been given a date in 2027. When you first compile it (to myprog.o ), the result is given today's date in 2013.

The next time you go to make, the date on myprog.c is after the one of myprog.o and it recompiles because of that.

Not sure if that's what's causing your problem but it may be worth looking into.

The other thing that you need to watch out for. If your unit depends on another unit where the interface has changed, it will be recompiled. See also this answer to a different question but it provides relevant information to this one as well.

I tend to do full builds most of the time because I don't trust computers - I program them so I know how dodgy they can be :-)

Answer 3

As said by Warren the unit compilation order is quite unpredictable (from an human viewpoint, I'm sure the compiler is deterministic) and interfaces and implementation are separate compilation stages (since parsing the implementation uses clause can force other units to be compiled before continuing with the implementation), but I would expect each of those two parts to be compiled in one go.

The only reason for deviation from that (which I never observed, but never searched for either) would then be multiple "compile" worker threads inside the delphi compiler.

Additional complicating factors:

1. circular/mutual dependencies, also of the allowed (implementation A uses B while B interface uses A) type. The direct A <->B deps are often properly detected, but larger cycles can cause recompilations.
2. Multiple copies of sources and/or (almost) the same include files in multiple directories (indy is notorious for this). Different includefiles can enable slightly different compiler options that might (?) trigger recompilation in some cases.
3. inline, generics. (that cross interface <-> implementation separation, possibly requiring implementation information before processing using code)
4. The fact that file paths in the .dpr have some influence. E.g. selecting a specific copy of a file that exists in multiple forms. (you checked out in a new path, but the absolute path in the dpr points to the old checkout). This is specially noticable in commandline building when doing partial builds. (building units, not program/library projects)
5. (not entirely sure. I sometimes have the feeling that Delphi can find files open in the IDE, even while they are not in the searchpath. This could induce selective behaviour in the case of multiple copies too)

The most important two are #1 and #2, and the effects are stronger in FPC than in Delphi. (better cycle logic in Delphi?). Excessive recompilations are quite normal when recompiling Lazarus.

The #3 comes from theory, #4 is seen sometimes in larger trees compiled by batchfiles. Since inlining is formally optional it shouldn't be a problem, but it is not documented if it at least attempts to go beyond what is required.