read many time config file or use gen_server status?

Question

I implemented an otp system using gen_server and gen_fsm. There is a configuration file to be read for some values that the software needs in order to run, an example may be:

{values, [value1, value2, value3]}.

I used a macro to extract one of these values

define(VALUES, my_utility:get_conf_value(values)). 

The question is the following: since ?VALUES may be called very often, and therefore the configuration file is parsed many times, should I embed ?VALUES inside the state of my gen_server of gen_fsm and extract it with a call any time I need it?

In fact I really appreciated the previous implementation because one could change the behaviour of the software just by changing the values inside the configuration file, without any #state{} change or call.

Which solution do you prefer?

Answer 1

The solution will depend upon your requirements. Performance vs "correctness".

A possible solution would be to keep the configuration in a process state and re-read it regularly (checking if the file modification time has changed). This might be a good compromise between the two worlds.

Summary:

• Re-read from file everytime: always up to date, but slow with unnecessary IO
• Re-read on interval, store in process state: fast to read, but lagging behind
• Manual re-read: fast to read, but lagging behind and needs manual trigger
• Read once and store in process state: fast to read, needs restart to update

Requirements not considered: security, stability (corrupt config file?)

Answer 2

This is actually a topic that is much harder than what one would first think. For example :

• How is the configuration changed?
• Are there any process that should be notified if it changes?
• Do you store the configuration in some intermediate process/state? Would this have to be cleared?
• How do you ensure everybody is seeing the same configuration in all different processes on all different nodes?
• Or is your system able to function even with incorrect configuration?
• How often is the configuration accessed?
• What data do you store there?

So, to analyse your current solutions:

• Parsing config files every time you need a value will be too slow and create too much garbage, if you care about these matters
• Asking the gen_server for the value might also be too slow, depending on your use case.
• If you need to do many lookups, the gen_server might also be overloaded
• If the data is big, copying between the gen_server and your process might
• If your application is running on multiple machines and you cannot tolerate inconsistency in configuration, you have to orchestrate the update to happen as close as posssible to the same time (this you would have with any solution I can think of, if you care about consistency)

Another solution would be to store the configuration as code. You could for example have a special configuration module, which you create using a parse transform or by generating the actual source file. With this solution, the configuration values resides in the constant pool of the VM and fetching them creates as little garbage as possible (depending on what do you with the data later).

Answer 3

Personally I think best solution is make configuration as module. Benefits are:

• Really very fast
• Can by changed by hot code change
• Can be machine generated from any form of config file formats
• Compiling and hot code swapping gives you atomic update by only "correct" configuration for free

P.S.: See network module load (aka nl/1 in shell) for cluster wide config change.