Why is the startup of an App on linux slower when using shared libs?

Question

On the embedded device I'm working on, the startup time is an important issue. The whole application consists of several executables that use a set of libraries. Because space in FLASH memory is limited we'd like to use shared libraries.

The application workes as usual when compiled and linked with shared libraries and the amount of FLASH memory is reduced as expected. The difference to the version that is linked to static libs is that the startup time of the application is about 20s longer and I have no idea why.

The application runs on an ARM9 CPU at 180 MHz with Linux 2.6.17 OS, 16 MB FLASH (JFFS File System) and 32 MB RAM.

Answer 1

Bacause shared libraries have to be linked to at runtime, usually by dlopen() or something similar. There's no such step for static libraries.

Edit: some more detail. dlopen has to perform the following tasks.

• Find the shared library
• Load it into memory
• Recursively load all dependencies (and their dependencies....)
• Resolve all symbols

This requires quite a lot of IO operations to accomplish.

In a statically linked program all of the above is done at compile time, not runtime. Therefore it's much faster to load a statically linked program.

In your case, the difference is exaggerated by the relatively slow hardware your code has to run on.

Answer 2

This is a fine example of the classic tradeoff of speed and space.

You can statically link all your executables so that they are faster but then they will take more space

OR

You can have shared libraries that take less space but also more time to load.

So decide what you want to sacrifice.

There are many factors for this difference (OS, compiler e.t.c) but a good list of reasons can be found here. Basically shared libraries were created for space reasons and much of the "magic" involved to make them work takes a performance hit.

(As a historical note the original Netscape navigator on Linux/Unix was a statically linked big fat executable).

Answer 3

This may help others with similar problems:

The reason why startup took so long in my case was, that the default setting of the GCC is to export all symbols inside of a library. A big improvement is to set a compiler setting "-fvisibility=hidden".

All symbols that the lib has to export have to be augmented with the statement

__attribute__ ((visibility("default")))

see gcc wiki
and the very fine article how to write shared libraries

Answer 4

Ok, I have learned now that the usage of shared libraries has it's disadvatages concerning speed. I found this article about dynamic linking and loading enlighting. The loading process seems to be much lengthier than I have expected.

Answer 5

Interesting.. typically loading time for a shared library is unnoticeable from a fat app that is statically linked. So I can only surmise that the system is either very slow to load a library from flash memory, or the library that is loaded is being checked in some way (eg .NET apps run a checksum for all loaded dlls, reducing startup time considerably in some cases). It could be that the shared libraries are being loaded as-needed, and unloaded afterwards which could indicate a configuration problem.

So, sorry I can't help say why, but I think its an issue with your ARM device/OS. Have you tried instrumenting the startup code, or statically linking with 1 of the most commonly-used libraries to see if that makes a large difference. Also put the shared libs in the same directory as the app to reduce the time it takes to search the FS for the lib.

Answer 6

One option which seems obvious to me, is to statically link the several programs all into a single binary. That way you continue to share as much code as possible (probably more than before), but you will also avoid the overhead of the dynamic linker AND save the space of having the dynamic linker on the system at all.

It's pretty easy to combine several executables into the same one, you normally just examine argv and decide which routine to call based on that.