High Speed ​​C # Data Processing

Question

I wrote an application that logs trace data from the embedded system via UDP. I am currently receiving datagrams and parsing variable length records and storing them in a list. The front end can access the list and present data (graphs and text lists, etc.).

The problem I am facing is that sometimes I need to register an extraordinary amount of data. So much so that my list implementation raises an exception from memory.

My requirements:

• Allow multithreaded read and write data (not just send the process)
• Processing large amounts of data (worst case ~ 2 MB / s ... 7.2 GB / hour of logging)
• Allow Dataset Storage
• Random read, index, access

Does anyone have any suggestions on how to attack this? Here are a few thoughts that I had:

• I need a great disk with memory, a cached list. It is similar to what will exist, but I did not find it.
• Local database? I don’t know too much about databases, but it seems like this is unnecessary.
• Store the data in a file right away. Store a list in memory that contains a byte offset for each record index. Can my reader use this thread at the same time?

Answer 1

- , . , UDP , ( , : , - ). .

( ) " ", OOM.

Answer 2

.NET 4 Lock. , UDP- , .

Answer 3

, , Log (string contents). , , . , , .

, . , , .

, 1 2 , DISK I/O Sucks. , , , .

private static StreamWriter sw;
private static Queue<string> logQueue = new Queue<string>();
public static string logLock = "";
public static void LogLoop()
{
    sw = new StreamWriter("logFilePath.log"), true);
    sw.AutoFlush = true;
    while (true)
    {
        while (logQueue.Count > 0)
        {
            string s = "";
            lock (logLock) // get a lock on the queue
            {
                s = logQueue.Dequeue();
            }
            sw.WriteLine(s);                
        }
        Thread.Sleep(10);
    }
}
public static void Log(string contents)
{
    contents = DateTime.Now.ToString("MM-dd-yy - HH:mm:ss ffff") + " - " + contents; // add a timestamp

    lock (logLock) // get a lock on the queue
    {
        logQueue.Enqueue(contents);
    }
}

.

Thread logThread = new Thread(LogLoop);
logThread.IsBackground = true;
logThread.Name = "Logging Thread";
logThread.Start();

Answer 4

Josiah Logger. , while (true), false.

while (logging)  // instead of while(true)
{
    while (logQueue.Count > 0)
    {
        string s = "";
        lock (logLock)
        {
           s = logQueue.Dequeue();
        }
        write(s);
    }
    Thread.Sleep(timer);
}

, , , logQueue.Count .

for (int i = 0; i <5000; i++)
{
     lock (logLock)
     {
       logQueue.Enqueue(i.toString());
     }
}
logging = false;

LogLoop , . false , , logQueue.Count , .