How to use message queue in real scenario?

Question

Ok, so I'm interested in the message queue. In fact, I have already researched this topic a lot. I read the "azure programming windows" (about Azure Queues), I read a lot of tutorials and Azure service bus information, I watched 9 channel videos on messaging templates, etc.

But what I just do not understand: how can I use this in a real scenario? All examples simply put a single line or object with some data in the queue, and they read it from the queue on the "other side". But how do you know what to do with this data? For example: I could try to save the Client, Order and address in the database, so I put these 3 objects in the queue to read on the other side, to put them in my database. How to find out what to do with these objects?

Just some questions that I have:

• What data will be queued. Perhaps the command that is executed when it is read, so everything in the queue has the same interface, and the object itself calculates what to do?
• Meanwhile, which layers will the queues use? I was thinking about putting things in the queue at the service level (after checking) and reading it at the data access level so that I can put it in the database.
• How many queues need to be done? Only one queue between the layers you want to connect, or several queues between them (maybe for different purposes, although I can’t think of one)?
• This form of free communication allows us to order requests so that they can be processed later (for example: when we would like to restart the database). That's cool, but what if I want to READ data instead of writing? Then the database must be online. And should I read the data through the queue, or did my service level just pull the data from the data access layer and pass it to the presentation layer?

I think these were the biggest questions that arose in my head. I hope someone can clean this up.

Answer 1

• Well, you can put where you want. This is just a message, so no matter what your layers agree with, it will work. We put messages indicating when to send emails, when to process videos, etc.
• I would not think about the order between the layers, just like in the queue between real-time user processes (for example, web servers) and background processing. Use the queue to do what you don't want your web roles to take while the user is waiting.
• In fact, you can do as much as you want. If there is a field in the queue messages that says what they mean, then you could hypothetically put everything in one queue, although I would not recommend this. We have about a dozen queues: one for sending notifications, one for video processing, one for reporting requests, etc.
• I would definitely not use a queue to store and retrieve data. This is simply the wrong tool for this job, for example, trying to use TCP to store data.

Note that a single instance of a work object (i.e. a virtual machine) can read as many queues as you want. You just need to write the code that way. If you have a lot of queues that don’t see much traffic, this is a smart way to cut costs. Another alternative would be to combine several types of messages in one queue. Thus, you have only one place to search for messages.

In addition, you can have many workers reading the same line to evenly distribute work on them.

Answer 2

In the simplest example, imagine that you serialize to the queue the action you want to perform, and then the serialized version of the arguments:

Q: Delete: User 5

which is great, as your Worker Role can read it and act accordingly. This example does not seem to be a big bonus, since the queue can take as much time as the deletion, but what if you want to do BI for all your users? Your web page or application will never be updated, waiting for a synchronous request for processing. But if you sent this request to the queue:

Q: RunSome2HourProcess: User *

Now you can return that you have queued the operation, and continue your fun journey, while your server, which pulls out of the queue, will be able to process at your leisure.

The larger it is, the better the scaling scenarios. What if the last message was split? Therefore, instead of doing all the users, you did something like:

Q RunSome2HourProcess: User A*-M*

Q RunSome2HourProcess: User N*-Z*

The two working roles of the azure tree will share the load and process your information in parallel.

There are also a number of scenarios related to retries, workflows, asynchronous multi-role communication, etc., but this should give you some reasons why the queue might be good in the right scenarios.

So, in these examples: 1. We placed the actions and identifiers / requests in the queue. The role of the working role from the queue will know how to deal with it.

• You can use the queue between the service level and processing levels. You can use the queue between processing levels. You can use the queue inside one layer. Sky is the limit

• I usually did 1 turn per operation. So in the example above, I will have a Run2HourProcess queue and just write the parameters. And create a new queue for any other type of process. But they are super flexible, so it depends on the developer.

• If this is a long read (i.e. an example of generating thumbnails in samples or the result of slow data processing), you should be as fast as in the database.

Answer 3

In addition to what Rob mentions, a message queue is also used to process traffic packets that can wait (in a queue) until a free worker becomes available.

I believe that Azure Service Bus, queues, and topics use message queuing under covers to accomplish the same thing. I also see tips that AppFabric, hosted by the Workflow Foundation, can do the same.