JSON as a database export format

Problem. We register things in the database. To limit the use of disk space, we export from the database to files that you can copy, or simply delete the plane. Some authority above me wants to see this as JSON.

I see one JSON file as a separate object. Therefore, in this case, we will create an object with a list of log messages. The problem is that this file may contain several million log entries, which, I think, would strangle most parsers. Therefore, the only way to do this, I think, for each element of the log has its own JSON object.

This means that JSON parsers cannot process the file. But we could write a line analyzer to read in the file and pass each line through the JSON parser.

Does this sound right?

I believe that XML will have the same problem, but at least we have SAX .. Or we could do it like a bunch of minidocks, all prefixed by their length.

Thanks.

+3
source share
4 answers

The whole idea of ​​JSON is incompatible with saving several million entries in a file ...

The whole point of JSON was to remove the overhead caused by XML. If you write each record as a JSON object, you return to storing service bits that do not matter. The next logical step is to write out a regular CSV file with a header, which everyone on the planet understands how to import.

If for some reason you have child records, you should see how EDI works.

+5
source

, json-, , , , ( json-, ... ).

, , - , csv

:

{"name":"bob","position":"ceo","start_date":"2007-08-10"}
{"name":"tom","position":"cfo","start_date":"2007-08-11"}

.

csv:

["name","position","start_date"]
["bob","ceo","2007-08-10"]
["tom","cfo","2007-08-11"]

, .

csv, , :

["bill","cto","2007-08-12",{"projects":["foo","bar","baz"]}]

CSV.

+4

: JSON / JSON JSON.

, . , ( , , ), , , ( "0-10000", "10001-20000" ..) - . , /. , - - , . , / , JSON, .

+3
source

This means that JSON parsers cannot process the file like it. But we could write a line parser to read into a file and push each line through a JSON parser.

Does this sound right?

That sounds reasonable ... so you end up with a large array of lines separated by line breaks, each line consisting of a single JSON object.

+1
source

Source: https://habr.com/ru/post/1702306/


All Articles