Solr DataImport: configuration management for different environments (development / production / production)

Question

The configuration files for the SolR DataImportHandler contain SQL queries for working with the database, how to match the received columns with the fields of the SolR field, and, of course, the parameters necessary for connecting to the database.

In our project, the parameters for connecting to the database (in particular, database passwords) vary from environment to environment, which forces us to maintain one slightly different copy of the entire XML configuration file for each environment.

Is there a way to configure database connection parameters (especially passwords) separately from SQL statements and declarations, so each configuration is supported once and only once?

Answer 1

This is a known issue in Solr.

If you look at the Solr document or the Solr Entreprise server book, they say that you can use core1.properties with the key = value and use the key in your xml configuration files ... but in my experience it doesn’t Work. I tried several ways, there are open questions on the solr mailing list about this.

Thus, you need to resort to ugly workarounds (for example, use the xml template file and replace # password # with a real password, etc.).

Answer 2

In fact, this can be done using the standard Solr configuration.

First you need to define your data sources in the solrconfig.xml file [See Adding a data source to Solrconfig]

Secondly, you can export DIH configurations to a separate file using XInclude

I use this approach both to use local configuration files and to centralize connections across different kernels.

---

Example: In the solrconfig.xml file add:

<xi:include href="../../common-config/local.dih.xml" xmlns:xi="http://www.w3.org/2001/XInclude" /> 

local.dih.xml will look something like this:

 <requestHandler name="/dataimport" class="org.apache.solr.handler.dataimport.DataImportHandler"> <lst name="defaults"> <str name="config">data-config.xml</str> <lst name="datasource"> <str name="name">mongo</str> <str name="type">MongoDataSource</str> <str name="database">myMongoDb</str> </lst> <lst name="datasource"> <str name="name">psql</str> <str name="driver">org.postgresql.Driver</str> <str name="type">JdbcDataSource</str> <str name="url">jdbc:postgresql://localhost:5432/myPsqlDb</str> <str name="user">dbUser</str> <str name="password">dbPassword</str> </lst> </lst> </requestHandler>