I have a program that receives real-time data on 1000 topics. It receives - on average - 5,000 messages per second. Each message consists of two lines, a subject and a message value. I would like to save these lines along with a timestamp indicating the arrival time of the message.
I use 32-bit Windows XP for hardware and programming the Core 2 core in C #.
I would like to save this data in 1000 files - one for each topic. I know that many people will want to tell me to save the data in the database, but I do not want to go this way.
I reviewed several approaches:
1) Open 1000 files and write them to each when data arrives. I have two problems. I don’t know if 1000 files can be opened at the same time, and I don’t know what effect this will have on disk fragmentation.
2) Write to one file and - somehow - process it later to produce 1000 files.
3) Keep everything in RAM until the end of the day, and then write one file at a time. I think this will work well if I have enough bar, although I may need to switch to 64 bits to overcome the 2 GB limit.
How do you approach this problem?
Joe h source
share