Testing Logstash Custom Filters

Question

We use Ansible and have Logstash.

How to write some tests to cover our custom static statistics filters? I want to do the following:

• with Logstash configuration with filters,
• pass it a log line (or a multi-line log entry),
• to see how he successfully disassembled into pieces.

I know this is https://github.com/elastic/logstash/wiki/Tips:Testing-your-filters , but I don’t understand how useful this is - it is deprecated.

Answer 1

gmile 1.5 logstash. grok logstash-core. logstash 2.2:

# encoding: utf-8

require 'spec_helper'
require "logstash/patterns/core"

# solution based on https://github.com/logstash-plugins/logstash-filter-grok/blob/master/spec/filters/grok_spec.rb
module LogStash::Environment
  # running the grok code outside a logstash package means
  # LOGSTASH_HOME will not be defined, so let set it here
  # before requiring the grok filter

  # the path that is set is the plugin root path
  unless self.const_defined?(:LOGSTASH_HOME)
    LOGSTASH_HOME = File.expand_path("../../../", __FILE__)
  end

  # also :pattern_path method must exist so we define it too

  # method is called by logstash-filter-grok to create patterns_path array
  #
  #   logstash-filter-grok/lib/logstash/filters/grok.rb(line ~230):
  #
  #   @@patterns_path += [
  #     LogStash::Patterns::Core.path,
  #     LogStash::Environment.pattern_path("*")
  #
  # patterns defined in spec/patterns/ will be joined to the array by grok 

  unless self.method_defined?(:pattern_path)
    def pattern_path(path)
      ::File.join(LOGSTASH_HOME, "spec", "patterns", path)
    end
  end
end

require "logstash/filters/grok"
require "logstash/filters/<tested-plugin>"

spec .

, Gemfile . gemspec :

# Gem dependencies
s.add_runtime_dependency "logstash-core-plugin-api", "~> 1.0"
s.add_runtime_dependency "<tested-plugin>"
s.add_development_dependency 'logstash-devutils', '~> 0'
s.add_development_dependency 'logstash-filter-grok', '~> 3.2'
s.add_development_dependency 'logstash-patterns-core', '~> 4.0'

exapmle

Answer 2

:

# simple_filter_spec.rb
#
# run using:
#   bundle exec rspec simple_filter_spec.rb

require "logstash/devutils/rspec/spec_helper"

LogStash::Environment::LOGSTASH_HOME = `gem which logstash-core`
module LogStash::Environment
  unless self.method_defined?(:pattern_path)
    def pattern_path(path)
      ::File.join(LOGSTASH_HOME, "patterns", path)
    end
  end
end


require "logstash/filters/grok"

describe LogStash::Filters::Grok do
  config <<-CONFIG
  filter {
    grok {
      match => { "message" => "%{SYSLOGLINE}" }
      singles => true
      overwrite => [ "message" ]
    }
  }
  CONFIG

  sample "Mar 16 00:01:25 evita postfix/smtpd[1713]: connect from camomile.cloud9.net[168.100.1.3]" do
    insist { subject["tags"] }.nil?
    insist { subject["logsource"] } == "evita"
    insist { subject["timestamp"] } == "Mar 16 00:01:25"
    insist { subject["message"] } == "connect from camomile.cloud9.net[168.100.1.3]"
    insist { subject["program"] } == "postfix/smtpd"
    insist { subject["pid"] } == "1713"
  end
end

Gemfile :

source 'https://www.rubygems.org'

platform :jruby do
  gem 'pry'
  gem 'rspec'
  gem 'logstash-core'
  gem 'logstash-devutils'
  gem 'logstash-filter-grok'
end

Answer 3

Logstash-Tester - unit test . json, logstash-tester logstash . ( : )

Answer 4

From this blog :

$ git clone https://github.com/elastic/logstash
$ cd logstash
$ git checkout 2.1
$ rake bootstrap
$ rake test:install-core

Instead of checking branch 2.1, you should probably check the marked version of logstash that you are actually running, for example. v2.3.2 (note the "v").

After executing the above commands, you can run bin/rspec /some/path/your_filter_spec.rblogstash in the repository.

It is important . I found that an encoding string is required # encoding: utf-8, otherwise the match will fail.

Sample test file:

# encoding: utf-8

require "spec_helper"

describe "simple request log" do
  config (<<-CONFIG)
  filter {
    grok {
      match => { "message" => "%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}" }
    }
  }
  CONFIG

  sample '55.3.244.1 GET /index.html 15824 0.043' do
    insist { subject['client']   } == '55.3.244.1'
    insist { subject['method']   } == 'GET'
    insist { subject['request']  } == '/index.html'
    insist { subject['bytes']    } == '15824'
    insist { subject['duration'] } == '0.043'
  end
end