Erlang: how to handle a long init callback?

Question

I have gen_server , which at startup tries to start a certain number of child processes (usually 10-20) under the supervisor in the observation tree. The init gen_server callback calls supervisor:start_child/2 for each child process. The supervisor:start_child/2 call is synchronous, so it does not return until the start of the child process. All child processes are also gen_servers, so the start_link call is not returned until the init callback is returned. In the init callback, a call is made to a third-party system, which may take some time to answer (I found this problem when calls to a third-party system failed after 60 seconds). Meanwhile, the init call is blocked, which means that supervisor:start_child/2 also blocked. Thus, the gen_server process calling supervisor:start_child/2 all the time does not respond. Calls the gen_server timeout while it waits for the start_child function to return. Since it can last 60 seconds or more. I would like to change this since my application is suspended in some kind of half-running state while it is waiting.

What is the best way to solve this problem?

The only solution I can think of is to move the code that interacts with the third-party system from the init callback and handle_cast callback. This will speed up the init callback. The downside is that I will need to call gen_server:cast/2 after the child processes are started.

Is there a better way to do this?

Answer 1

One approach I've seen is to use the init/1 timeout and handle_info/2 .

 init(Args) -> {ok, {timeout_init, Args} = _State, 0 = _Timeout}. ... handle_info( timeout, {timeout_init, Args}) -> %% do your inicialization {noreply, ActualServerState}; % this time no need for timeout handle_info( .... 

Almost all results can be returned with an additional timeout parameter, which is basically the timeout for another message. It sets the time that handle_info/2 is called, with the timeout atom and the server. In our case, with a timeout of 0, the timeout should occur before gen_server:start . This means that handle_info should be called before we can return the pid of our server to anyone else. So this timeout_init should be the first call made on our server, and give us some confidence that we are completing the initialization before handling anything else.

If you do not like this approach (it is not readable actually), you can try to send a message to yourself in init/1

 init(Args) -> self() ! {finish_init, Args}, {ok, no_state_yet}. ... handle_info({finish_init, Args} = _Message, no_state_yet) -> %% finish whateva {noreply, ActualServerState}; handle_info( ... % other clauses 

Again, you will make sure that the message to complete the initialization is sent as soon as possible to this server, which is very important in the case of gen_servers, which are registered under some atom.

---

EDIT After a more thorough study of the OTP source code.

This approach is good enough when you communicate with your server through it pid. Mostly because pid is returned after the init/1 functions return. But it is slightly different in the case of gen_.. , starting with start/4 or start_link/4 , where we automatically register the process under the same name. There is one race condition that you might encounter, which I would like to explain in more detail.

If the process is a register, usually all calls are simplified and executed on the server, for example:

 count() -> gen_server:cast(?SERVER, count). 

Where ?SERVER usually the name of the module (atom) and will work very well until (and live) the process is registered under this name. And of course, under the hood, this cast is the standard Erlang message sent with ! . Nothing magical about this, much like your init with self() ! {finish ... self() ! {finish ...

But in our case, we accept one more thing. Not only the registration part, but also the fact that our server has completed its initialization. Of course, since we are dealing with a message field, it does not matter how long something happens, but it is important what message we receive. More precisely, we would like to receive the finish_init message before receiving the count message.

Unfortunately, such a scenario may occur. This is because the gen in OTP is registered before init/1 calls the callback. Therefore, theoretically, while one process calls the start function, which will go to the registration part than the other, you can find our server and send a count message, and immediately after that the init/1 function will be called with the finish_init message, the chances are small ( very, very few ), but still it can happen.

There are three solutions.

At first it would do nothing. In the event of such a race condition, handle_cast will fail (due to a function suggestion, since our state is not_state_yet atom), and the supervisor will simply restart everything.

The second case will ignore this bad message / state incident. This is easily achieved with

  ... ; handle_cast( _, State) -> {noreply, State}. 

as your last sentence. And, unfortunately, most people using templates use such an unsuccessful (IMHO) template.

In both cases, you may lose one count message. If this is really a problem, you can still fix it by changing the last sentence to

  ... ; handle_cast(Message, no_state_yet) -> gen_server:cast( ?SERVER, Message), {noreply, no_state_yet}. 

but it has other obvious advantages, I would prefer "let it not fit."

The third option is the registration process a little later. Instead of using start/4 and asking for automatic registration, use start/3 , get the pid and register it yourself.

 start(Args) -> {ok, Pid} = gen_server:start(?MODULE, Args, []), register(?SERVER, Pid), {ok, Pid}. 

Thus, we send the finish_init message before registration and before anyone else can send the message and count .

But this approach has its drawbacks, mainly registration itself, which can fail in several ways. You could always check how OTP handles this and duplicates this code. But this is another story.

Thus, in the end, it all depends on what you need, or even on the problems that you will encounter in production. It is important to have some idea that the bad can happen, but I personally do not try to fix any of them until I really suffered from this state of the race.