Elasticearch Bulk JSON Data

Question

This question arises from this SO stream .

It seems to me that I have a similar, but not the same request, it would be better to have a separate question for others, as @Val suggested.

So, similar to the above, I need to insert a huge amount of data into the index (my initial testing is about 10,000 documents, but this is only for POC, there are much more). The data I would like to insert is in the .json document and looks something like this (snippet):

[ { "fileName": "filename", "data":"massive string text data here" }, { "fileName": "filename2", "data":"massive string text data here" } ] 

By my own admission, I am new to ElasticSearch, however, after reading the documentation, my assumptions are that I can take the .json file and create an index from the data inside. Now I realized that every element in json should have a “header”, something like:

 {"index":{}} { "fileName": "filename", "data":"massive string text data here" } 

Meaning that this is not the actual json format (as such), but rather a managed string.

I would like to know if there is a way to import my json data as is (in json format), without having to manually manually manipulate the text (since my test data contains 10,000 entries, I'm sure you can understand why I would prefer not to do this manually).

Any suggestions or suggested automated tools to help with this?

PS - I use the Windows installer and Postman for calls.

Answer 1

You can easily convert your file with a single shell command like this. If your file is called input.json , you can do this:

 jq -c -r ".[]" input.json | while read line; do echo '{"index":{}}'; echo $line; done > bulk.json 

After that, you will have a file called bulk.json , which is correctly formatted to be sent to the primary endpoint.

You can then invoke your primary endpoint as follows:

 curl -XPOST localhost:9200/your_index/your_type/_bulk -H "Content-Type: application/x-ndjson" --data-binary @bulk.json 

Note: you need to install jq if you do not already have it.