Patterns and technologies for a system capable of processing 40,000 messages per second

Question

We need to create a system capable of processing 40,000 messages per second. No messages can be lost in the event of a software or hardware failure.

The size of each message is about 2-4 KB.

Processing a message consists of checking the message, performing simple arithmetic calculations, storing the result in a database, and (sometimes) sending notifications to other systems.

Preferred software technology is .Net.

What software and hardware templates are most suitable for such a task?

How much hardware will be required?

Answer 1

• Message queue Your process flow sounds like its main goal.
• Clustering / load balancing.
• Optimize Your Code

The first thing I would do is queue for notifications. Then I would queue all database records that should not return a value. Then I will look at scaling.

Other considerations: * Avoid a large clumsy structure that does more work behind than the scene than you probably need. * Use cache and static variables whenever possible.

40 000 , IO , . . , , ( ) .

Edit:

. . , . . , .

Answer 2

4k * 40.000/s = 160MB/s - .

, , , .

, , .

db , 40 000 ( 3,5 ), .

, .Net .

Answer 3

, , , . " " . , 5000 5000 . , , .

, -, . , , .

, , . , .

Answer 4

Microsoft, MSMQ (Microsoft Message Queuing). , . MSMQ.

, -. , .

MSMQ , , . , , .

Answer 5

MSMQ , , . , , , - , , . . , . ( , . .)

, MSMQ, , C:, . , , .

, . , MSMQ , ..

IBM MQ is probably better suited to the task.

Answer 6

My advice is to hire someone who has already created a similar system. Let them choose the architecture and development tools. To work with such high transaction rates, you will need special knowledge in the field of hardware and software, and the cheapest way to get such knowledge is to pay money for it.