Mongoimport speed when using -jsonArray is very slow

Question

I have a 15 GB file with over 25 lines of milion which is in this json format (which is accepted by mongodb for import:

[
    {"_id": 1, "value": "\u041c\..."}
    {"_id": 2, "value": "\u041d\..."}
    ...
]

When I try to import it into mongodb with the following command, I get a speed of only 50 lines per second, which is very slow for me.

mongoimport --db wordbase --collection sentences --type json --file C:\Users\Aleksandar\PycharmProjects\NLPSeminarska\my_file.json -jsonArray

When I tried to insert data into the collection using python with pymongo, the speed was even worse. I also tried to increase the priority of the process, but it did not matter.

The next thing I tried was the same thing, but without use -jsonArray, and although I received a large increase in speed (~ 4000 / sec), he said that the BSON representation of the supplied JSON is too large.

5 , 20 /.

, , 8 . , , .

, json bson , , , ?

.

Answer 1

160 . , 3% -jsonArray 15 .

[ trailing ]:

sed 's/^\[//; s/\]$/' -i filename.json

-jsonArray:

mongoimport --db "dbname" --collection "collectionname" --file filename.json

, sed , , , . ( , @guillermobox):

int main(int argc, char *argv[])
{
    FILE * f;
    const size_t buffersize = 2048;
    size_t length, filesize, position;
    char buffer[buffersize + 1];

    if (argc < 2) {
        fprintf(stderr, "Please provide file to mongofix!\n");
        exit(EXIT_FAILURE);
    };

    f = fopen(argv[1], "r+");

    /* get the full filesize */
    fseek(f, 0, SEEK_END);
    filesize = ftell(f);

    /* Ignore the first character */
    fseek(f, 1, SEEK_SET);

    while (1) {
        /* read chunks of buffersize size */
        length = fread(buffer, 1, buffersize, f);
        position = ftell(f);

        /* write the same chunk, one character before */
        fseek(f, position - length - 1, SEEK_SET);
        fwrite(buffer, 1, length, f);

        /* return to the reading position */
        fseek(f, position, SEEK_SET);

        /* we have finished when not all the buffer is read */
        if (length != buffersize)
            break;
    }

    /* truncate the file, with two less characters */
    ftruncate(fileno(f), filesize - 2);

    fclose(f);

    return 0;
};

P.S.: , , .