Phoenix Workflow Queue

Question

I need something like PubSub, but instead of broadcasting to all subscribers, the message is sent to only one subscriber (preferably, the subscriber is selected automatically based on the number of messages in it receive buffer, better below).

I am trying to send several hundred thousand HTTP requests using a controlled number of distributed workers.

Answer 1

To solve this problem, the first thing I will try is to get workers to pull out requests, not push them.

, Agent, HTTP-, API . N , worker(Task, ...), Supervisor one_for_one, . Agent HTTP- , , URL-.

, HTTP- , , , , , .

, . , URL-, , .

Answer 2

, , Poolboy , N (N - URL-), -. , , , , url . , , .

Poolboy :poolboy.checkout/2, , . , :full, pid.

:

def crawl_parallel(urls) do
  urls
  |> Enum.map(&crawl_task/1)
  |> Enum.map(&Task.await/1)
end

defp crawl_task(url) do
  case :poolboy.checkout Crawler, false do
    :full ->
      # No free workers, wait a bit and retry
      :timer.sleep 100
      crawl_task url
    worker_pid -> 
      # We have a worker, asynchronously crawl the url
      Task.async fn ->
        Crawler.Worker.crawl worker_pid, url
        :poolboy.checkin Crawler, worker_pid
      end
  end
end