Java / MongoDB message length error on Windows but not Linux

Question

We are currently working on importing huge JSON files (~ 100 MB) into MongoDB using the java driver. We are currently splitting the files into smaller chunks since we first encountered problems importing the entire file. Of course, we know about the MongoDB restriction that the maximum document size is 16 MB, however, our pieces that we are importing now are much smaller.

Oddly enough, the import procedure works at startup on Linux (eclipse), but the same program throws an exception saying that "I can not say something" on Windows (eclipse). When observing a log from the database, the error message says

> "Thu Sep 13 11:38:48 [conn1] recv(): message len 1835627538 is too > large1835627538" 

Restarting the import in the same dataset always results in the same error message regarding the length of the message. We examined the size of our documents for import (using .toString (). Length ()) - the piece that caused the error was only some kB large.

It doesn't matter which OS the mongo database is running on, but it depends on where the import code is running (using the same java-mongo-driver

Answer 1

"we are currently working on importing huge JSON files (~ 100 MB) into MongoDB using the java driver"

Are we talking about a JSON file containing 1000 JSON objects OR 1 JSON object ~ 100 MB in size? Because, if I remember correctly, the 16 MB limit is not per object per JSON file containing 1000 units of JSON objects.

AND

 "Thu Sep 13 11:38:48 [conn1] recv(): message len 1835627538 is too large1835627538" 

the piece causing the error was only some kB big.

If 1835627538 really is in kb, this is quite a big value, because there are ~ 1750 GigaBytes !!

To get around a JSON file containing 1000 JSON objects, why don't you iterate over the data files line by line and do your inserts this way? With my method, it doesn't matter how big your data file is, an iterator is just a pointer to a specific line. It does not load the FULL FILE into memory and inserts.

NOTE It is assumed that your data file contains 1 JSON object per line .

Using Apache Commons IO FileUtils (click here ), you can use their String Iterator to iterate through your file, for example (not fully working code, you need to import the correct libraries):

 LineIterator line_iter; try { line_iter = FileUtils.lineIterator(data_file); while (line_iter.hasNext()) { line = line_iter.next(); try { if (line.charAt(0) == '{') this.mongodb.insert(line); } catch (IndexOutOfBoundsException e) {} } } line_iter.close(); // close the iterator } catch (IOException e) { e.printStackTrace(); }