Dynamic scalable and adaptive architecture

Question

I am a PhD student in Cloud Computing, I plan to use the microservices architecture with consul and zeromq for my research project. I had a few questions that are hard for me to understand. Can someone help me share my experience.

• We have docker-based microservices, we have zeromq, and we have a consul. Can you mention how we could combine all three together to have a dynamic adaptive environment?

Although I understand what zeromq, docker and consul individually are, I still can’t get a clear idea of ​​how they all function as a whole. We have docker containers in which microservices work inside them on the host. We use zeromq to transport (Pub-sub / pipe) messages between docker containers. Containers can operate on the same host / data center or on different hosts / data centers. Then we use the consul to discover the services. I understand what is right here?

2. How does the up / down architecture dynamically scale to fit the workload?

Let's say I have a situation where I need more work nodes for a certain calculation for a while. Who is spinning more work nodes. Which component determines / makes this decision?

Is there a planning component? If so, can someone briefly explain how this happens or which component performs this function?

3. So what is the main role of the consul? Is it used only for service discovery? Can it be used for configurations? If so, what is its limitation?

I see that even zeromq has service discovery mechanisms, so why do we need a consul?

4. How does a failure occur in node information in architecture? Which component is responsible? Is this just a consul? Or zeroMq as well?

Please advice.

Answer 1

I participate in a large project using microservices based on Docker and Consul. (We use another queue service - RabbitMQ, so I can’t talk about this in detail, but in general, a queue is a queue.)

Neither Docker, Consul, nor any queue technology that I know of provides autoscaling features. Docker provides an easy way to deploy multiple instances of a service, and Consul provides a service discovery (as you said) and a key / value store. A queue is simply a way to pass messages between service instances. You did not mention anything that handles autoscaling.

To add autoscaling, you need to look at something like Kubernetes.

You can watch something like CloudFoundry or Mesos. However, both of them require a level of virtualization such as OpenStack or VMWare vSphere. There is considerable value with these products, but also price and complexity.

Instead of taking this route, I recommend that you familiarize yourself with Amazon Web Services. Using AWS, you can easily launch docker containers and configure autoscaling based on processor load, queued messages, time of day (or day of week), etc. I know that using AWS carries a price tag, but when well designed and managed, it will cost you far less than trying to design, implement, and support yourself. You can also use the smallest (i.e., free) machines and or point instances to minimize costs.

Answer 2

This is an interesting question. All the components you are talking about are independent, clearly differing strengths and roles in the microservice architecture. The unusual part is the use of messages, not HTTP. I believe that this is a valuable departure, because it allows you to create more flexible computing templates (a working producer should not be a consumer of a product or even receive notifications). The special beauty of HTTP skipping is to avoid costs (both OPEX and maintenance delays), complexity, and additional failure modes of load balancers.

• Docker: manage the packaging of individual services and delivery to the ZeroMQ infrastructure: manage effective peer-to-peer or intermediary communications between services Consul: discover a service (for example, find out where a user's service is located)

• Auto scan is not performed by Docker. You can do this using your own microservice, which checks the load indicators (for example, average load, physical / swap memory usage, etc.) and spins replicas and updates of Consul.

  Alternatively, you can rely on auto-scalable solutions detailed with @drhender: Kubernetes, Mososphere DCOS or AWS Autoscaling Groups. Please note, however, that using AWS Autoscaling Groups greatly limits the portability of your solution.

  Regardless of the autoscaling mechanism selected, make sure your ZeroMQ message templates support or add or remove services. The ZeroMQ Guide has good tips on this topic.

• A consul can provide both discovery services and service configuration settings. Be aware of storage security issues if you store sensitive data, such as PHI or PII. They are better stored with overfill protection, such as storage inventory.

  Agent Consul collects telemetry , which you can track on issues. There is also a fairly simple Consul KV test suite that you can use to test your configuration repository.

  ZeroMQ is not intended for direct service discovery. You can choose a centrally brokerage architecture that makes a separate service discovery unnecessary, but has different effects of scalability and resiliency. It has the overhead of a message for a message, assuming you are using multicast messages and signing subscriptions when binding SUB sockets. There are many ways to discover services using ZeroMQ individually, but that would not be trivial, especially considering the fact of tolerance and consensus.

• Node Failure is an interesting task. The consul can find out through health checks, but ZeroMQ creates some problems depending on the messaging patterns. For example, using a REQ-REP pair, if the request is sent and the responder dies after delivery, the REQ socket will be blocked forever in my experience. There are ways to get around this with timeouts.

  I would rely on the Consul for this and was ready to interrupt or recreate the REQ connectors in the event of a failure. Avoid RPC-style interactions completely by using Staged Event Driven Architecture (SEDA), where input manufacturers are not consumers of the outputs of assemblies of this almost completely. You always have the task of losing the queued inputs or outputs in the event of a failure, so you will need system level monitoring and restart mechanisms if the loss of work is fatal.

ContainerBuddy allows you to place any running application in a Docker container and register it with Consul. It can simplify things for you.