How to move elasticsearch data from one server to another

Question

How to transfer Elasticsearch data from one server to another?

I have a server running Elasticsearch 1.1.1 on one local node with multiple indexes. I would like to copy this data to server B running Elasticsearch 1.3.4

Procedure so far

• Shut down ES on both servers and
• copy all the data to the correct data directory on the new server. (the data is apparently located in / var / lib / elasticsearch / in my debian blocks)
• change permissions and ownership of elasticsearch: elasticsearch
• start a new ES server

When I look at a cluster with the ES head plugin, no indexes appear.

It seems that the data is not loaded. Did I miss something?

Answer 1

The selected answer makes it sound a little more complicated than it is, the next one is what you need (install npm on your system first).

npm install -g elasticdump elasticdump --input=http://mysrc.com:9200/my_index --output=http://mydest.com:9200/my_index --type=mapping elasticdump --input=http://mysrc.com:9200/my_index --output=http://mydest.com:9200/my_index --type=data 

You can skip the first elastic dump command for subsequent copies if the mappings remain constant.

I just made the switch from AWS to Qbox.io with the above without any problems.

More on:

https://www.npmjs.com/package/elasticdump

Help page (as of February 2016) for completeness:

 elasticdump: Import and export tools for elasticsearch Usage: elasticdump --input SOURCE --output DESTINATION [OPTIONS] --input Source location (required) --input-index Source index and type (default: all, example: index/type) --output Destination location (required) --output-index Destination index and type (default: all, example: index/type) --limit How many objects to move in bulk per operation limit is approximate for file streams (default: 100) --debug Display the elasticsearch commands being used (default: false) --type What are we exporting? (default: data, options: [data, mapping]) --delete Delete documents one-by-one from the input as they are moved. Will not delete the source index (default: false) --searchBody Preform a partial extract based on search results (when ES is the input, (default: '{"query": { "match_all": {} } }')) --sourceOnly Output only the json contained within the document _source Normal: {"_index":"","_type":"","_id":"", "_source":{SOURCE}} sourceOnly: {SOURCE} (default: false) --all Load/store documents from ALL indexes (default: false) --bulk Leverage elasticsearch Bulk API when writing documents (default: false) --ignore-errors Will continue the read/write loop on write error (default: false) --scrollTime Time the nodes will hold the requested search in order. (default: 10m) --maxSockets How many simultaneous HTTP requests can we process make? (default: 5 [node <= v0.10.x] / Infinity [node >= v0.11.x] ) --bulk-mode The mode can be index, delete or update. 'index': Add or replace documents on the destination index. 'delete': Delete documents on destination index. 'update': Use 'doc_as_upsert' option with bulk update API to do partial update. (default: index) --bulk-use-output-index-name Force use of destination index name (the actual output URL) as destination while bulk writing to ES. Allows leveraging Bulk API copying data inside the same elasticsearch instance. (default: false) --timeout Integer containing the number of milliseconds to wait for a request to respond before aborting the request. Passed directly to the request library. If used in bulk writing, it will result in the entire batch not being written. Mostly used when you don't care too much if you lose some data when importing but rather have speed. --skip Integer containing the number of rows you wish to skip ahead from the input transport. When importing a large index, things can go wrong, be it connectivity, crashes, someone forgetting to `screen`, etc. This allows you to start the dump again from the last known line written (as logged by the `offset` in the output). Please be advised that since no sorting is specified when the dump is initially created, there no real way to guarantee that the skipped rows have already been written/parsed. This is more of an option for when you want to get most data as possible in the index without concern for losing some rows in the process, similar to the `timeout` option. --inputTransport Provide a custom js file to us as the input transport --outputTransport Provide a custom js file to us as the output transport --toLog When using a custom outputTransport, should log lines be appended to the output stream? (default: true, except for `$`) --help This page Examples: # Copy an index from production to staging with mappings: elasticdump \ --input=http://production.es.com:9200/my_index \ --output=http://staging.es.com:9200/my_index \ --type=mapping elasticdump \ --input=http://production.es.com:9200/my_index \ --output=http://staging.es.com:9200/my_index \ --type=data # Backup index data to a file: elasticdump \ --input=http://production.es.com:9200/my_index \ --output=/data/my_index_mapping.json \ --type=mapping elasticdump \ --input=http://production.es.com:9200/my_index \ --output=/data/my_index.json \ --type=data # Backup and index to a gzip using stdout: elasticdump \ --input=http://production.es.com:9200/my_index \ --output=$ \ | gzip > /data/my_index.json.gz # Backup ALL indices, then use Bulk API to populate another ES cluster: elasticdump \ --all=true \ --input=http://production-a.es.com:9200/ \ --output=/data/production.json elasticdump \ --bulk=true \ --input=/data/production.json \ --output=http://production-b.es.com:9200/ # Backup the results of a query to a file elasticdump \ --input=http://production.es.com:9200/my_index \ --output=query.json \ --searchBody '{"query":{"term":{"username": "admin"}}}' ------------------------------------------------------------------------------ Learn more @ https://github.com/taskrabbit/elasticsearch-dump`enter code here` 

Answer 2

Use ElasticDump

1) yum install epel-release

2) yum install nodejs

3) yum install npm

4) npm installation of elastic disk

5) cd node_modules / elasticdump / bin

6)

 ./elasticdump \ --input=http://192.168.1.1:9200/original \ --output=http://192.168.1.2:9200/newCopy \ --type=data 

Answer 3

You can use the snapshot / restore function available in Elasticsearch. Once you set up file system-based snapshot storage, you can move it between clusters and restore it in another cluster

Answer 4

I tried on ubuntu to move data from ELK 2.4.3 to ELK 5.1.1

Following are the steps

$ sudo apt-get update

$ sudo apt-get install -y python-software-properties python g++ make

$ sudo add-apt-repository ppa:chris-lea/node.js

$ sudo apt-get update

$ sudo apt-get install npm

$ sudo apt-get install nodejs

$ npm install colors

$ npm install nomnom

$ npm install elasticdump

in the goto $ cd node_modules/elasticdump/ home directory

run the command

If you need basic http auth, you can use it like this:

--input=http://name:password@localhost:9200/my_index

Copy index from production:

$ ./bin/elasticdump --input="http://Source:9200/Sourceindex" --output="http://username:password@Destination:9200/Destination_index" --type=data

Answer 5

If you can add a second server to the cluster, you can do this:

• Add server B to the cluster with server A
• Increasing Replicas for Indexes
• ES automatically copies indexes to server B
• Close Server A
• Reducing the number of replicas for indexes

This will only work if the number of substitutions is equal to the number of nodes.

Answer 6

If someone is facing the same problem, when trying to reset from elasticsearch <2.0 to> 2.0 you need to do:

 elasticdump --input=http://localhost:9200/$SRC_IND --output=http://$TARGET_IP:9200/$TGT_IND --type=analyzer elasticdump --input=http://localhost:9200/$SRC_IND --output=http://$TARGET_IP:9200/$TGT_IND --type=mapping elasticdump --input=http://localhost:9200/$SRC_IND --output=http://$TARGET_IP:9200/$TGT_IND --type=data --transform "delete doc.__source['_id']" 

Answer 7

I always managed to just copy the index directory / folder to the new server and restart it. You can find the index identifier by running GET /_cat/indices , and the folder corresponding to that identifier is located in data\nodes\0\indices (usually in your elasticsearch folder if you haven't moved it).

Answer 8

If you just need to transfer data from one Flexiblesearch server to another, you can also use asticsearch-document-Transfer .

Stages:

1. Open the directory in your terminal and run
   $ npm install elasticsearch-document-transfer.
2. Create config.js file
3. Add connection information for both config.js servers in config.js
4. Set appropriate values ​​in options.js
5. Run in terminal
   $ node index.js