How to read from a text file faster / smarter?

Question

I want to know if it is possible to read with text file faster and smarter.

This is a typical format of my data in a text file :

Call this "part":

 ID:1; FIELD1 :someText; FIELD2 :someText; FIELD3 :someText; FIELD4 :someText; FIELD5 :someText; FIELD6 :someText; FIELD7 :someText; FIELD8 :someText; END_ID : 01: someData; 02: someData; ... ... 48: someData; ENDCARD: 

I have thousands of them in a text file.

Is it possible to use LINQ to read the "part" using the "part"? I do not want to iterate over each line.

Would it be possible for LINQ start with ID:1; and ended with ENDCARD: :?

The reason for this is that I want to create an object for each "part" ...

I had something like this:

 string[] lines = System.IO.File.ReadAllLines(SomeFilePath); //Cleaning up the text file of unwanted text var cleanedUpLines = from line in lines where !line.StartsWith("FIELD1") && !line.StartsWith("FIELD5") && !line.StartsWith("FIELD8") select line.Split(':'); //Here i want to LINQtoText "part" by "part" //This i do not want to do!!! foreach (string[] line in cleanedUpLines) { } 

Answer 1

Here you go:

 static void Main() { foreach(var part in ReadParts("Raw.txt")) { // all the fields for the part are available; I'm just showing // one of them for illustration Console.WriteLine(part["ID"]); } } static IEnumerable<IDictionary<string,string>> ReadParts(string path) { using(var reader = File.OpenText(path)) { var current = new Dictionary<string, string>(); string line; while((line = reader.ReadLine()) != null) { if(string.IsNullOrWhiteSpace(line)) continue; if(line.StartsWith("ENDCARD:")) { yield return current; current = new Dictionary<string, string>(); } else { var parts = line.Split(':'); current[parts[0].Trim()] = parts[1].Trim().TrimEnd(';'); } } if (current.Count > 0) yield return current; } } 

What is it: create an iterator block (a finite state machine that reads and “displays” data as it repeats, does not read the entire file at a time), which scans the lines; if this is the end of the card, the card has “lost”; otherwise, it adds data to the dictionary for storage.

Note. If you have your own class that represents data, you can use something like reflection or FastMember to set values ​​by name.

It does not use LINQ directly; however, it is implemented as an enumerated sequence, which is the building block of LINQ-to-Objects, so you can use this with LINQ, i.e.

 var data = ReadParts("some.file").Skip(2).First(x => x["ID"] == "123");