What are some guidelines for handling large logstash configuration files?

Question

I am going to deploy a logstash instance that will handle various inputs and perform several filter actions. The configuration file will most likely end up with a lot of if-then instructions, given the complexity and number of inputs.

My questions:

• Is there a way to make the configuration file more “modular”? In terms of programming, I would create functions / routines so that I can test myself. I was thinking about dynamically creating mini-configuration files that I can use for testing. These mini files can then be combined into a single production configuration.

• Are there "best practices" for testing, deploying, and managing more complex Logstash configurations?

Thanks!

Answer 1

There is no support for functions / routines as such. I split various filters into separate files in order to maintain logical separation and avoid creating giant files. I also have inputs and outputs in different files. This way I can combine all filters with debug I / O, e.g.

input { stdin {} } output { stdout { codec => rubydebug } } 

and manually call Logstash to check the results of this input. Since I use filter ordering issues, I use the fact that Logstash reads configuration files in alphabetical order, so the files are called NN-some-descriptive-name.conf, where NN is an integer.

I also wrote a script that automates this process, allowing you to write a specification with test inputs and expected result messages, and if there is a mismatch, this will lead to an error and display diff. I can open the source code.

For deployment, use any configuration management system, such as Puppet, Chef, SaltStack, Ansible, CFEngine, or similar, that you are familiar with. I am very pleased with Ansible.

Answer 2

As @Magnus Bäck said, the answer to question 1. is not. There is currently no feature support.

But as for your second question, there is a way to make logstash configuration more modular. you can split the configuration file into several files and point logstash to the file directory.

check directory parameter in logstash man:

 -f, --config CONFIG_PATH Load the logstash config from a specific file or directory. If a direcory is given, all files in that directory will be concatonated in lexicographical order and then parsed as a single config file. You can also specify wildcards (globs) and any matched files will be loaded in the order described above.