Is there a way to train the Encog neural network without loading the entire training set into memory?

Question

I use the .NET framework to train a neural network, and my dataset is really large, with lots of inputs.

I decided to switch to Encog because of some problems with the old base that I used, and also because Encog looks much richer and offers additional features.

My problem is that Encog loads all the training data at once into a 2D array, and this does not suit me, since I have a very large data set. I tried to get around this and also checked other questions (like this one , but could not find a good answer).

I tried working with SQLNeuralDataSet and other implementations that support stream operation, but they all seem to end up loading data into memory (using MemoryDataLoader), and this does not solve my problem.

Is there a way I can load (and train) my network one element at a time? Or is this option not available in Encog?

thanks

EDIT

I ended up copying the source code and modifying it according to my needs. Basically, it all boils down to the Process(IMLDataPair pair) in GradientWorker , which, according to the documentation:

Process one element of a set of exercises.

Answer 1

Looking at the source code for the data sources for Encog, there are a number of options. See Buffer/BufferedMLDataSet.cs for example:

 /// This class is not memory based, so very long files can be used, without /// running out of memory. This dataset uses a Encog binary training file as a /// buffer.