Is there a quick and scalable solution for saving data?

Question

I am developing a service that needs to be scalable on a Windows platform.

Initially, it will receive approximately 50 connections in seconds (each connection will send data of about 5 KB), but it must be scalable to receive more than 500 future ones.

It is impractical (I think) to save the received data into a common database such as Microsoft SQL Server.

Is there any other solution for saving data? Given that he will receive more than 6 million "records" per day.

There are 5 steps:

• Get data through the http handler (C #);
• Save the received data; <- HERE
• Request stored data for processing;
• Process the requested data;
• Save the processed data. <- HERE

My preliminary decision:

• Get data through the http handler (C #);
• Saving the received data to the Message Queue ;
• A request from MSQ for the stored data to be processed using Windows Services;
• Process the requested data;
• Save processed data to Microsoft SQL Server (bottleneck here);

Answer 1

6 million records per day do not seem particularly huge. In particular, this is not 500 per second for 24 hours a day - do you expect traffic to be "bloated"?

- . , , . / . .

, .

(, , "" ) . , , , , ( , ), .

, , ( ). , , .

Answer 2

, . , , , , .

, , , .

Answer 3

:

1.)
2.)
3.)

This will save you from having to request it if you already have one. I would be more worried about your table structure and your database machine, and then about the actual thread. I am sure your inserts will be as cheap as possible. If this is not possible, then the order of work makes sense. I would not use the message queue myself. Assuming you have a decent SQL Server machine, 6 million records per day should be good if you don't write a ton of data in each record.