Search for tutorials and load balancing information between threads

Question

I know that the term “load balancing" can be very broad, but the topic I'm trying to explain is more specific, and I do not know the correct terminology. What I am creating is a set of Server / Client applications. The server should be able to handle huge amounts of data transfer, as well as client connections, so I began to study multithreading.

In essence, there are 3 ways that I can implement the implementation of any type of thread for a server ...

• One thread processing all requests (hits the target of the thread if 500 clients are registered)
• One thread for the user (which is dangerous for creating 1 thread for each of the 500 clients)
• A pool of threads that evenly distributes work for any number of clients (what I'm looking for)

The third is what I would like to know. This consists of the following setting:

• Maximum 250 threads at a time.
• 500 clients will not create 500 threads, but share 250
• The request queue will be sent to the stream.
• The thread is not tied to the client and vice versa
• The server decides which thread to send the request based on activity (load balance)

I'm currently not looking for any code yet, but information on how this setting works, and preferably a tutorial for this in Delphi (XE2). Even the right word or name to put on this topic would be enough for me to do the search myself.

EDIT

I found it necessary to explain a little why it will be used. I will broadcast both commands and images, there will be an installation with two sockets, where there is one "Main Command Socket" and another "Additional routing for streaming images." So really one connection is 2 socket connections.

Each connection to the main server socket creates (or reuses) an object that represents all the data necessary for this connection, including streams, images, settings, etc. For each connection to the main socket, the streaming socket is also associated. This is not always a streaming image, but the command socket is always ready.

The fact is that I already have a streaming mechanism in my current installation (1 stream per session object), and I would like to transfer this to a multi-threaded environment with a pool. These two connections require a higher level of control over these streams, and I cannot rely on something like Indy to maintain synchronization, I would better know how everything works than learn to trust something else to do the job for me .

Answer 1

IOCP server. This is the only high-performance solution. It is essentially asynchronous in user mode ("I / O overlaps in the M $ -peak), the thread pool calls WSARecv, WSASend, AcceptEx, and then everyone waits in the IOCP queue for completion records. When something useful happens, the kernel Threadpool does the actual I / O and then writes the completion records.

You need at least a buffer class and a socket class (and possibly others for the high-performance classes objectPool and pooledObject so that you can create socket and buffer pools).

Answer 2

500 threads cannot be a problem on a server class computer. A blocking TCP stream does not do much while it waits for a server response.

There is nothing stopping you from creating some kind of server-side work queue serviced by a pool of threads of a limited size. A simple thread-safe TList works fine as a queue, and you can easily place a message handler on each server thread for notifications.

However, at some point you may have too much work or too many threads to process the server. This is usually due to the addition of another application server.

To provide scalability, code is for the idea of ​​multiple servers, and you can continue to scale by adding hardware.

There may be some reason to limit the number of actual workflows, such as restricting lock contention in the database or something similar, however, in general, you distribute the work by adding threads and allow the hardware (CPU, redirector, switch, NAS, etc.) d.) plan to download.

Answer 3

Your implementation is completely tied to the communication components used. If you are using Indy or anything based on Indy, this is one thread per connection - period! Unable to change this. Indy will scale to 100 connections, but not 1000. Your best hope is to use thread pools with your communication components - IOCP, but here your choice is limited by the absence of third-party components. I did all the investigation before, and you can see my question at fooobar.com/questions/244772 / ....

I have a fully working distributed development environment (flows and communications) that has been used in the production of more than half a dozen separate systems for more than 3 years and basically covers everything that you have requested so far. Code can also be found on the Internet.