Distributed Systems NSQ Topology Structure on Docker Containers

Is it possible to replicate the "NSQ real-time distributed messaging platform" described in the last example, "Topology Templates" with Docker ? Does anyone have a docker file or image?

+6
source share
1 answer

I will crack it while I wait for a few background tasks to finish.

The distributed messaging platform mentioned by @Luca G. Soave can be seen here: source: www.nsq.io/deployment/topology_patterns.html

I believe this question has a fundamental problem: a misunderstanding of which Docker containers exist.

For the purposes of our discussion, let's assume that the Docker container is just another name for the virtual machine. The question "Can a distributed system be built using virtual machines?" not quite suitable, because in reality it is just a matter of configuration, abstraction and coordination.

The above diagram can be recreated with each contact / responsibility point (node) that is offline in a Docker container or virtual machine. i.e:.

  • Each API / nsqd node is in its own container
  • Each nsq_to_file node is in its own container
  • Each nsqlookupd node is in its own container

Depending on how you customize your Docker images, you can implement a distributed (multi-way) version of this in several ways. Some ideas:

  • Map the internal ports of the container to the same host port and configure your nodes to transfer themselves as the ip host, so that when other nodes connect to them, they are locked to the host’s external IP address on the port associated with the container; thus connecting directly to the container.

  • Using a service discovery package such as consul , replace nsqlookupd to add additional metadata. This would be useful if you run many containers, each of which is internally bound to the same port (for example, port 9090), but allows the docker process on the host to manage random external port mappings.

Since this applies to Docker, there have been some recent developments in the field of translating information between nodes into appropriate containers; which would be one way to seed your api / nsqd containers with nsqlookupd container information.

I have had success using MaestroNG for small deployments, but this is definitely not a great solution for large-scale docker deployments.

+1
source

Source: https://habr.com/ru/post/959159/


All Articles