Best way to declare N number of workers in a supervisor, without knowing N beforehand

Question

I'm developing an application with 1 supervisor and several workers. Each one of those workers will just open a tcp socket, execute a listen, and then accept connections, spawning a process for each client as they arrive (I dont mind supervising these).

I'd like to make the listening addresses configurable in the application configuration, so I can have as many addresses to listen as needed (at least 1 address is needed, but any number can be specified). Each address is an ip (or hostname) and a port address. So far, nothing new.

My question is about how to declare, start, and supervise an unknown number of workers. My solution was to dynamically generate (at runtime) the result of the init/1 function in the supervisor code, like this:

-define(
   CHILD(Name, Args),
   {Name, {
       ?MODULE, start_listen, Args
       }, permanent, 5000, worker, [?MODULE]
   }
).

init([]) ->
   {ok, Addresses} = application:get_env(listen),
   Children = lists:map(
       fun(Address) ->
           {X1,X2,X3} = os:timestamp(),
           ChildName = string:join([
               "Listener-",
               integer_to_list(X1),
               integer_to_list(X2),
               integer_to_list(X3)
           ], ""),
           ?CHILD(ChildName, [Address])
       end,
       Addresses
   ),
   {ok, { {one_for_one, 5, 10}, Children }}.

This allows me to dynamically generate a name for each worker and also generate the worker definitions. So:

1. Is this solution acceptable? It seems to me that it's not that elegant. Is there any standard solution (or best practice, etc) for this kind of use cases?

2. I know about the "simple_one_for_one" strategy, that allows to dynamically add workers to a supervisor. But can it also be used to dynamically generate the worker's names? Is it better (in any way) to use "simple_one_for_one" instead of my own solution that uses "one_for_one"? (again, for this particular situation).

Thanks in advance, and sorry for the long post! :)

Answer 1

With simple_one_for_one :

in the init function of your workers, you can register them in a table to associate a name with their PID, so you will be able to get the pid from the name.

You can use the global module (or gproc !) to associate a name with a pid. When a process dies, the name is automatically deleted by global or gproc, so when the supervisor restarts the child, the name is available.

You would pass the names in the arguments list (2nd param) of supervisor:start_child/2

Using simple_one_for_one will allow you to dynamically add more listeners after your supervisor initialization.

If you need this feature, simple_one_for_one is the good solution.

If you stick with your solution of a dynamic content inside the init function of your sup, you could clean the code like this, it may, or may not, seem more elegant :

-define(
   CHILD(Name, Args),
   {Name, {
       ?MODULE, start_listen, Args
       }, permanent, 5000, worker, [?MODULE]
   }
).

generate_names(Adresses) -> generate_names(Adresses, 1, []).

generate_names([], _, Acc) -> Acc;
generate_names([Addr|Addresses], ID, Acc) -> generate_names(Addresses, ID+1, [{id_name(ID), Addr}|Acc]).

id_name(ID) -> "listener-" ++ integer_to_list(ID).

init([]]) ->
   {ok, Addresses} = application:get_env(listen),
   Children = [?CHILD(Name, Address) || {Name, Address} <- generate_names(Addresses)],
   {ok, { {one_for_one, 5, 10}, Children }}.

Or use a lists:foldl instead of all theese little functions to keep the code short.

But anyway i would pass the Adresses in the Args list of init and not call get_env inside init to keep it pure.

Answer 2

From your code, I know that you want to get the number of children from environment. In the source code of worker_pool_sup.erl of rabbitmq open source project, I have read the almost similar requirement code, and the code is much elegant, I think it is helpful to you. 3 files are related, worker_pool_sup.erl, worker_pool_worker.erl, work_pool.erl.

The following code is from worker_pool_sup.erl.

init([WCount]) ->
    {ok, {{one_for_one, 10, 10},
          [{worker_pool, {worker_pool, start_link, []}, transient,
            16#ffffffff, worker, [worker_pool]} |
           [{N, {worker_pool_worker, start_link, [N]}, transient, 16#ffffffff,
             worker, [worker_pool_worker]} || N <- lists:seq(1, WCount)]]}}.

For how to use it, the following code is from worker_pool.erl

get_worker_pid(WId) ->
    [{WId, Pid, _Type, _Modules} | _] =
        lists:dropwhile(fun ({Id, _Pid, _Type, _Modules})
                              when Id =:= WId -> false;
                            (_)               -> true
                        end,
                        supervisor:which_children(worker_pool_sup)),
    Pid.