How to split an array of file paths into several smaller arrays?

Question

I read several different questions from SO about this already, but still have not been able to get it to work. I use public static string[] files = Directory.GetFiles(CurrentDirectory, "*.wav", SearchOption.AllDirectories); to get an array of file paths that will then be passed to the stream. The operations performed by the file stream took too much time, and only one stream processed all the files. So I decided that I split the array and passed these smaller arrays to different threads.

I got the code I used for this from another SO question and used it to pass the split array, but it only worked with one file in the first array, but I know what the problem is:

 var thing = from index in Enumerable.Range(0, files.Length) group files[index] by index/600; foreach(var set in thing) string.Join(";", set.ToArray()); 

(This is not quite the way I used it, I mixed up with it so much that I can’t remember.) The problem is that everything was considered as one massive file path, I have a foreach loop that gets each file from a smaller array , but processed each file in it as one, throwing a filepathtoolong exception when several files were returned from the search. My function takes an array and then uses foreach (string file in smallerArray) to write to each. What I need to do is split the file array into 4 smaller arrays and start new threads, such as new Thread(() => { DoWork(newArray); }).Start(); but nothing i tried worked.

Answer 1

So, I decided that I split the array and passed these smaller arrays to different threads.

It sounds like you are doing it hard :) Let the framework handle it for you with Parallel.ForEach :

 Parallel.ForEach(files, file => { // Do stuff with one file }); 

Answer 2

Here is an example

  public static IEnumerable<IEnumerable<T>> Split<T>(this IEnumerable<T> source, int blockSize) { if (source == null) throw new ArgumentNullException("source"); if (blockSize <= 0) throw new ArgumentException("blockSize = {0}".FormatWith(blockSize), "blockSize"); var result = new List<IEnumerable<T>>(); for (int blockStartIndex = 0; blockStartIndex < source.Count(); blockStartIndex += blockSize) { int blockStart = blockStartIndex; int blockEnd = blockStartIndex + blockSize - 1; IEnumerable<T> block = source.Where((x, i) => i >= blockStart && i <= blockEnd); result.Add(block); } return result; } 

here is a test

  [Test] public void TestSplit() { var list = new[] {0, 1, 2, 3, 4, 5, 6, 7, 8, 9}; IEnumerable<IEnumerable<int>> splitted = list.Split(10); Assert.That(splitted.Count(), Is.EqualTo(1)); Assert.That(splitted.First().Count(), Is.EqualTo(10)); splitted = list.Split(11); Assert.That(splitted.Count(), Is.EqualTo(1)); Assert.That(splitted.First().Count(), Is.EqualTo(10)); splitted = list.Split(9); Assert.That(splitted.Count(), Is.EqualTo(2)); Assert.That(splitted.First().Count(), Is.EqualTo(9)); Assert.That(splitted.ElementAt(1).Count(), Is.EqualTo(1)); splitted = list.Split(3); Assert.That(splitted.Count(), Is.EqualTo(4)); Assert.That(splitted.First().Count(), Is.EqualTo(3)); Assert.That(splitted.ElementAt(1).Count(), Is.EqualTo(3)); Assert.That(splitted.ElementAt(2).Count(), Is.EqualTo(3)); Assert.That(splitted.ElementAt(3).Count(), Is.EqualTo(1)); }