Introduce the GrokProcessor
Also moved all processor classes into a subdirectory and introduced a ConfigException class to be a catch-all for things that can go wrong when constructing new processors with configurations that possibly throw exceptions. The GrokProcessor loads patterns from the resources directory. fix resource path issue, and add rest-api-spec test for grok fix rest-spec tests changes: license, remove configexception, throw IOException add more tests and fix iso8601-hour pattern move grok patterns from resources to config fix tests with pom changes, updated IngestClientIT with grok processor update gradle build script for grok deps and test configuration move config files to src/main/packaging move Env out of Processor, fix test for src/main/packaging change add docs clean up test resources task update Grok to be immutable - Updated the Grok class to be immutable. This means that all the pattern bank loading is handled by an external utility class called PatternUtils. - fixed tabs in the nagios patterns file's comments
This commit is contained in:
Tal Levy
parent
a8d76f0117
commit
464b46437f
|
|
@ -3,6 +3,145 @@
|
|||
|
||||
TODO
|
||||
|
||||
=== Processors
|
||||
|
||||
==== Grok Processor
|
||||
|
||||
The Grok Processor extracts structured fields out of a single text field within a document. You choose which field to
|
||||
extract matched fields from, as well as the Grok Pattern you expect will match. A Grok Pattern is like a regular
|
||||
expression that supports aliased expressions that can be reused.
|
||||
|
||||
This tool is perfect for syslog logs, apache and other webserver logs, mysql logs, and in general, any log format
|
||||
that is generally written for humans and not computer consumption.
|
||||
|
||||
The processor comes packaged with over 120 reusable patterns that are located at `$ES_HOME/config/ingest/grok/patterns`.
|
||||
Here, you can add your own custom grok pattern files with custom grok expressions to be used by the processor.
|
||||
|
||||
If you need help building patterns to match your logs, you will find the <http://grokdebug.herokuapp.com> and
|
||||
<http://grokconstructor.appspot.com/> applications quite useful!
|
||||
|
||||
===== Grok Basics
|
||||
|
||||
Grok sits on top of regular expressions, so any regular expressions are valid in grok as well.
|
||||
The regular expression library is Oniguruma, and you can see the full supported regexp syntax
|
||||
https://github.com/kkos/oniguruma/blob/master/doc/RE[on the Onigiruma site].
|
||||
|
||||
Grok works by leveraging this regular expression language to allow naming existing patterns and combining them into more
|
||||
complex patterns that match your fields.
|
||||
|
||||
The syntax for re-using a grok pattern comes in three forms: `%{SYNTAX:SEMANTIC}`, `%{SYNTAX}`, `%{SYNTAX:SEMANTIC:TYPE}`.
|
||||
|
||||
The `SYNTAX` is the name of the pattern that will match your text. For example, `3.44` will be matched by the `NUMBER`
|
||||
pattern and `55.3.244.1` will be matched by the `IP` pattern. The syntax is how you match. `NUMBER` and `IP` are both
|
||||
patterns that are provided within the default patterns set.
|
||||
|
||||
The `SEMANTIC` is the identifier you give to the piece of text being matched. For example, `3.44` could be the
|
||||
duration of an event, so you could call it simply `duration`. Further, a string `55.3.244.1` might identify
|
||||
the `client` making a request.
|
||||
|
||||
The `TYPE` is the type you wish to cast your named field. `int` and `float` are currently the only types supported for coercion.
|
||||
|
||||
For example, here is a grok pattern that would match the above example given. We would like to match a text with the following
|
||||
contents:
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
3.44 55.3.244.1
|
||||
--------------------------------------------------
|
||||
|
||||
We may know that the above message is a number followed by an IP-address. We can match this text with the following
|
||||
Grok expression.
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
%{NUMBER:duration} %{IP:client}
|
||||
--------------------------------------------------
|
||||
|
||||
===== Custom Patterns and Pattern Files
|
||||
|
||||
The Grok Processor comes pre-packaged with a base set of pattern files. These patterns may not always have
|
||||
what you are looking for. These pattern files have a very basic format. Each line describes a named pattern with
|
||||
the following format:
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
NAME ' '+ PATTERN '\n'
|
||||
--------------------------------------------------
|
||||
|
||||
You can add this pattern to an existing file, or add your own file in the patterns directory here: `$ES_HOME/config/ingest/grok/patterns`.
|
||||
The Ingest Plugin will pick up files in this directory to be loaded into the grok processor's known patterns. These patterns are loaded
|
||||
at startup, so you will need to do a restart your ingest node if you wish to update these files while running.
|
||||
|
||||
Example snippet of pattern definitions found in the `grok-patterns` patterns file:
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
YEAR (?>\d\d){1,2}
|
||||
HOUR (?:2[0123]|[01]?[0-9])
|
||||
MINUTE (?:[0-5][0-9])
|
||||
SECOND (?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)
|
||||
TIME (?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9])
|
||||
--------------------------------------------------
|
||||
|
||||
===== Using Grok Processor in a Pipeline
|
||||
|
||||
[[grok-options]]
|
||||
.Grok Options
|
||||
[options="header"]
|
||||
|======
|
||||
| Name | Required | Default | Description
|
||||
| `match_field` | yes | - | The field to use for grok expression parsing
|
||||
| `match_pattern` | yes | - | The grok expression to match and extract named captures with
|
||||
|======
|
||||
|
||||
Here is an example of using the provided patterns to extract out and name structured fields from a string field in
|
||||
a document.
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
{
|
||||
"message": "55.3.244.1 GET /index.html 15824 0.043"
|
||||
}
|
||||
--------------------------------------------------
|
||||
|
||||
The pattern for this could be
|
||||
|
||||
[source]
|
||||
--------------------------------------------------
|
||||
%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}
|
||||
--------------------------------------------------
|
||||
|
||||
An example pipeline for processing the above document using Grok:
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
{
|
||||
"description" : "...",
|
||||
"processors": [
|
||||
{
|
||||
"grok": {
|
||||
"match_field": "message",
|
||||
"match_pattern": "%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
--------------------------------------------------
|
||||
|
||||
This pipeline will insert these named captures as new fields within the document, like so:
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
{
|
||||
"message": "55.3.244.1 GET /index.html 15824 0.043",
|
||||
"client": "55.3.244.1",
|
||||
"method": "GET",
|
||||
"request": "/index.html",
|
||||
"bytes": 15824,
|
||||
"duration": "0.043"
|
||||
}
|
||||
--------------------------------------------------
|
||||
|
||||
=== Put pipeline API
|
||||
|
||||
The put pipeline api adds pipelines and updates existing pipelines in the cluster.
|
||||
|
|
|
|||
|
|
@ -23,8 +23,17 @@ esplugin {
|
|||
}
|
||||
|
||||
dependencies {
|
||||
compile 'org.jruby.joni:joni:2.1.6'
|
||||
testCompile 'org.elasticsearch:securemock:1.1'
|
||||
}
|
||||
|
||||
sourceSets {
|
||||
test {
|
||||
resources {
|
||||
srcDir "src/main/packaging/config"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
compileJava.options.compilerArgs << "-Xlint:-rawtypes,-unchecked"
|
||||
compileTestJava.options.compilerArgs << "-Xlint:-rawtypes,-unchecked"
|
||||
|
|
|
|||
|
|
@ -0,0 +1 @@
|
|||
6bc17079fcaa8823ea8cd0d4c66516335b558db8
|
||||
|
|
@ -0,0 +1,17 @@
|
|||
Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
this software and associated documentation files (the "Software"), to deal in
|
||||
the Software without restriction, including without limitation the rights to
|
||||
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
|
||||
of the Software, and to permit persons to whom the Software is furnished to do
|
||||
so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
|
|
@ -0,0 +1 @@
|
|||
JCodings is released under the MIT License.
|
||||
|
|
@ -0,0 +1 @@
|
|||
0f23c95a06eaecbc8c74c7458a8bfd13e4fd2d3a
|
||||
|
|
@ -0,0 +1,17 @@
|
|||
Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
this software and associated documentation files (the "Software"), to deal in
|
||||
the Software without restriction, including without limitation the rights to
|
||||
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
|
||||
of the Software, and to permit persons to whom the Software is furnished to do
|
||||
so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Joni is released under the MIT License.
|
||||
|
|
@ -1 +0,0 @@
|
|||
This plugin has no third party dependencies
|
||||
|
|
@ -22,6 +22,7 @@ package org.elasticsearch.ingest;
|
|||
|
||||
import org.elasticsearch.ingest.processor.Processor;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Collections;
|
||||
import java.util.List;
|
||||
|
|
@ -82,7 +83,7 @@ public final class Pipeline {
|
|||
this.id = id;
|
||||
}
|
||||
|
||||
public void fromMap(Map<String, Object> config, Map<String, Processor.Builder.Factory> processorRegistry) {
|
||||
public void fromMap(Map<String, Object> config, Map<String, Processor.Builder.Factory> processorRegistry) throws IOException {
|
||||
description = (String) config.get("description");
|
||||
@SuppressWarnings("unchecked")
|
||||
List<Map<String, Map<String, Object>>> processors = (List<Map<String, Map<String, Object>>>) config.get("processors");
|
||||
|
|
@ -105,7 +106,7 @@ public final class Pipeline {
|
|||
this.description = description;
|
||||
}
|
||||
|
||||
public void addProcessors(Processor.Builder... processors) {
|
||||
public void addProcessors(Processor.Builder... processors) throws IOException {
|
||||
for (Processor.Builder processor : processors) {
|
||||
this.processors.add(processor.build());
|
||||
}
|
||||
|
|
|
|||
|
|
@ -22,6 +22,8 @@ package org.elasticsearch.ingest.processor;
|
|||
|
||||
import org.elasticsearch.ingest.Data;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.nio.file.Path;
|
||||
import java.util.Map;
|
||||
|
||||
/**
|
||||
|
|
@ -48,7 +50,7 @@ public interface Processor {
|
|||
/**
|
||||
* Builds the processor based on previous set settings.
|
||||
*/
|
||||
Processor build();
|
||||
Processor build() throws IOException;
|
||||
|
||||
/**
|
||||
* A factory that creates a processor builder when processor instances for pipelines are being created.
|
||||
|
|
@ -60,6 +62,11 @@ public interface Processor {
|
|||
*/
|
||||
Builder create();
|
||||
|
||||
/**
|
||||
*/
|
||||
default void setConfigDirectory(Path configDirectory) {
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
}
|
||||
|
|
|
|||
|
|
@ -0,0 +1,160 @@
|
|||
/*
|
||||
* Licensed to Elasticsearch under one or more contributor
|
||||
* license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright
|
||||
* ownership. Elasticsearch licenses this file to you under
|
||||
* the Apache License, Version 2.0 (the "License"); you may
|
||||
* not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing,
|
||||
* software distributed under the License is distributed on an
|
||||
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
* KIND, either express or implied. See the License for the
|
||||
* specific language governing permissions and limitations
|
||||
* under the License.
|
||||
*/
|
||||
|
||||
package org.elasticsearch.ingest.processor.grok;
|
||||
|
||||
import java.io.BufferedReader;
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.io.InputStreamReader;
|
||||
import java.lang.Object;
|
||||
import java.lang.String;
|
||||
import java.lang.StringIndexOutOfBoundsException;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.util.HashMap;
|
||||
import java.util.Iterator;
|
||||
import java.util.Locale;
|
||||
import java.util.Map;
|
||||
|
||||
import org.jcodings.specific.UTF8Encoding;
|
||||
import org.joni.*;
|
||||
import org.joni.exception.ValueException;
|
||||
|
||||
public class Grok {
|
||||
|
||||
private static final String NAME_GROUP = "name";
|
||||
private static final String SUBNAME_GROUP = "subname";
|
||||
private static final String PATTERN_GROUP = "pattern";
|
||||
private static final String DEFINITION_GROUP = "definition";
|
||||
private static final String GROK_PATTERN =
|
||||
"%\\{" +
|
||||
"(?<name>" +
|
||||
"(?<pattern>[A-z0-9]+)" +
|
||||
"(?::(?<subname>[A-z0-9_:.-]+))?" +
|
||||
")" +
|
||||
"(?:=(?<definition>" +
|
||||
"(?:" +
|
||||
"(?:[^{}]+|\\.+)+" +
|
||||
")+" +
|
||||
")" +
|
||||
")?" + "\\}";
|
||||
private static final Regex GROK_PATTERN_REGEX = new Regex(GROK_PATTERN.getBytes(StandardCharsets.UTF_8), 0, GROK_PATTERN.getBytes(StandardCharsets.UTF_8).length, Option.NONE, UTF8Encoding.INSTANCE, Syntax.DEFAULT);
|
||||
private final Map<String, String> patternBank;
|
||||
private final boolean namedCaptures;
|
||||
private final Regex compiledExpression;
|
||||
private final String expression;
|
||||
|
||||
|
||||
public Grok(Map<String, String> patternBank, String grokPattern) {
|
||||
this(patternBank, grokPattern, true);
|
||||
}
|
||||
|
||||
@SuppressWarnings("unchecked")
|
||||
public Grok(Map<String, String> patternBank, String grokPattern, boolean namedCaptures) {
|
||||
this.patternBank = patternBank;
|
||||
this.namedCaptures = namedCaptures;
|
||||
|
||||
this.expression = toRegex(grokPattern);
|
||||
byte[] expressionBytes = expression.getBytes(StandardCharsets.UTF_8);
|
||||
this.compiledExpression = new Regex(expressionBytes, 0, expressionBytes.length, Option.DEFAULT, UTF8Encoding.INSTANCE);
|
||||
}
|
||||
|
||||
|
||||
public String groupMatch(String name, Region region, String pattern) {
|
||||
try {
|
||||
int number = GROK_PATTERN_REGEX.nameToBackrefNumber(name.getBytes(StandardCharsets.UTF_8), 0, name.getBytes(StandardCharsets.UTF_8).length, region);
|
||||
int begin = region.beg[number];
|
||||
int end = region.end[number];
|
||||
return new String(pattern.getBytes(StandardCharsets.UTF_8), begin, end - begin, StandardCharsets.UTF_8);
|
||||
} catch (StringIndexOutOfBoundsException e) {
|
||||
return null;
|
||||
} catch (ValueException e) {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* converts a grok expression into a named regex expression
|
||||
*
|
||||
* @return named regex expression
|
||||
*/
|
||||
public String toRegex(String grokPattern) {
|
||||
byte[] grokPatternBytes = grokPattern.getBytes(StandardCharsets.UTF_8);
|
||||
Matcher matcher = GROK_PATTERN_REGEX.matcher(grokPatternBytes);
|
||||
|
||||
int result = matcher.search(0, grokPatternBytes.length, Option.NONE);
|
||||
if (result != -1) {
|
||||
Region region = matcher.getEagerRegion();
|
||||
String namedPatternRef = groupMatch(NAME_GROUP, region, grokPattern);
|
||||
String subName = groupMatch(SUBNAME_GROUP, region, grokPattern);
|
||||
// TODO(tal): Support definitions
|
||||
String definition = groupMatch(DEFINITION_GROUP, region, grokPattern);
|
||||
String patternName = groupMatch(PATTERN_GROUP, region, grokPattern);
|
||||
String pattern = patternBank.get(patternName);
|
||||
|
||||
String grokPart;
|
||||
if (namedCaptures && subName != null) {
|
||||
grokPart = String.format(Locale.US, "(?<%s>%s)", namedPatternRef, pattern);
|
||||
} else if (!namedCaptures) {
|
||||
grokPart = String.format(Locale.US, "(?<%s>%s)", patternName + "_" + String.valueOf(result), pattern);
|
||||
} else {
|
||||
grokPart = String.format(Locale.US, "(?:%s)", pattern);
|
||||
}
|
||||
|
||||
String start = new String(grokPatternBytes, 0, result, StandardCharsets.UTF_8);
|
||||
String rest = new String(grokPatternBytes, region.end[0], grokPatternBytes.length - region.end[0], StandardCharsets.UTF_8);
|
||||
return start + toRegex(grokPart + rest);
|
||||
}
|
||||
|
||||
return grokPattern;
|
||||
}
|
||||
|
||||
public boolean match(String text) {
|
||||
Matcher matcher = compiledExpression.matcher(text.getBytes(StandardCharsets.UTF_8));
|
||||
int result = matcher.search(0, text.length(), Option.DEFAULT);
|
||||
return (result != -1);
|
||||
}
|
||||
|
||||
public Map<String, Object> captures(String text) {
|
||||
byte[] textAsBytes = text.getBytes(StandardCharsets.UTF_8);
|
||||
Map<String, Object> fields = new HashMap<>();
|
||||
Matcher matcher = compiledExpression.matcher(textAsBytes);
|
||||
int result = matcher.search(0, textAsBytes.length, Option.DEFAULT);
|
||||
if (result != -1) {
|
||||
Region region = matcher.getEagerRegion();
|
||||
for (Iterator<NameEntry> entry = compiledExpression.namedBackrefIterator(); entry.hasNext();) {
|
||||
NameEntry e = entry.next();
|
||||
int number = e.getBackRefs()[0];
|
||||
|
||||
String groupName = new String(e.name, e.nameP, e.nameEnd - e.nameP, StandardCharsets.UTF_8);
|
||||
String matchValue = null;
|
||||
if (region.beg[number] >= 0) {
|
||||
matchValue = new String(textAsBytes, region.beg[number], region.end[number] - region.beg[number], StandardCharsets.UTF_8);
|
||||
}
|
||||
GrokMatchGroup match = new GrokMatchGroup(groupName, matchValue);
|
||||
fields.put(match.getName(), match.getValue());
|
||||
}
|
||||
} else {
|
||||
return null;
|
||||
}
|
||||
|
||||
return fields;
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -0,0 +1,67 @@
|
|||
/*
|
||||
* Licensed to Elasticsearch under one or more contributor
|
||||
* license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright
|
||||
* ownership. Elasticsearch licenses this file to you under
|
||||
* the Apache License, Version 2.0 (the "License"); you may
|
||||
* not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing,
|
||||
* software distributed under the License is distributed on an
|
||||
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
* KIND, either express or implied. See the License for the
|
||||
* specific language governing permissions and limitations
|
||||
* under the License.
|
||||
*/
|
||||
|
||||
package org.elasticsearch.ingest.processor.grok;
|
||||
|
||||
import java.lang.Float;
|
||||
import java.lang.Integer;
|
||||
import java.lang.Object;
|
||||
import java.lang.String;
|
||||
|
||||
public class GrokMatchGroup {
|
||||
private static final String DEFAULT_TYPE = "string";
|
||||
private final String patternName;
|
||||
private final String fieldName;
|
||||
private final String type;
|
||||
private final String groupValue;
|
||||
|
||||
public GrokMatchGroup(String groupName, String groupValue) {
|
||||
String[] parts = groupName.split(":");
|
||||
patternName = parts[0];
|
||||
if (parts.length >= 2) {
|
||||
fieldName = parts[1];
|
||||
} else {
|
||||
fieldName = null;
|
||||
}
|
||||
|
||||
if (parts.length == 3) {
|
||||
type = parts[2];
|
||||
} else {
|
||||
type = DEFAULT_TYPE;
|
||||
}
|
||||
this.groupValue = groupValue;
|
||||
}
|
||||
|
||||
public String getName() {
|
||||
return (fieldName == null) ? patternName : fieldName;
|
||||
}
|
||||
|
||||
public Object getValue() {
|
||||
if (groupValue == null) { return null; }
|
||||
|
||||
switch(type) {
|
||||
case "int":
|
||||
return Integer.parseInt(groupValue);
|
||||
case "float":
|
||||
return Float.parseFloat(groupValue);
|
||||
default:
|
||||
return groupValue;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,119 @@
|
|||
/*
|
||||
* Licensed to Elasticsearch under one or more contributor
|
||||
* license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright
|
||||
* ownership. Elasticsearch licenses this file to you under
|
||||
* the Apache License, Version 2.0 (the "License"); you may
|
||||
* not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing,
|
||||
* software distributed under the License is distributed on an
|
||||
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
* KIND, either express or implied. See the License for the
|
||||
* specific language governing permissions and limitations
|
||||
* under the License.
|
||||
*/
|
||||
|
||||
package org.elasticsearch.ingest.processor.grok;
|
||||
|
||||
import org.elasticsearch.common.inject.Inject;
|
||||
import org.elasticsearch.env.Environment;
|
||||
import org.elasticsearch.ingest.Data;
|
||||
import org.elasticsearch.ingest.processor.Processor;
|
||||
|
||||
import java.io.BufferedReader;
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.io.InputStreamReader;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.nio.file.DirectoryStream;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.nio.file.StandardOpenOption;
|
||||
import java.util.HashMap;
|
||||
import java.util.Map;
|
||||
|
||||
public final class GrokProcessor implements Processor {
|
||||
|
||||
public static final String TYPE = "grok";
|
||||
|
||||
private final String matchField;
|
||||
private final String matchPattern;
|
||||
private final Grok grok;
|
||||
|
||||
public GrokProcessor(Grok grok, String matchField, String matchPattern) throws IOException {
|
||||
this.matchField = matchField;
|
||||
this.matchPattern = matchPattern;
|
||||
this.grok = grok;
|
||||
}
|
||||
|
||||
@Override
|
||||
public void execute(Data data) {
|
||||
Object field = data.getProperty(matchField);
|
||||
// TODO(talevy): handle invalid field types
|
||||
if (field instanceof String) {
|
||||
Map<String, Object> matches = grok.captures((String) field);
|
||||
if (matches != null) {
|
||||
matches.forEach((k, v) -> data.addField(k, v));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
public static class Builder implements Processor.Builder {
|
||||
|
||||
private Path grokConfigDirectory;
|
||||
private String matchField;
|
||||
private String matchPattern;
|
||||
|
||||
public Builder(Path grokConfigDirectory) {
|
||||
this.grokConfigDirectory = grokConfigDirectory;
|
||||
}
|
||||
|
||||
public void setMatchField(String matchField) {
|
||||
this.matchField = matchField;
|
||||
}
|
||||
|
||||
public void setMatchPattern(String matchPattern) {
|
||||
this.matchPattern = matchPattern;
|
||||
}
|
||||
|
||||
public void fromMap(Map<String, Object> config) {
|
||||
this.matchField = (String) config.get("field");
|
||||
this.matchPattern = (String) config.get("pattern");
|
||||
}
|
||||
|
||||
@Override
|
||||
public Processor build() throws IOException {
|
||||
Map<String, String> patternBank = new HashMap<>();
|
||||
Path patternsDirectory = grokConfigDirectory.resolve("patterns");
|
||||
try (DirectoryStream<Path> stream = Files.newDirectoryStream(patternsDirectory)) {
|
||||
for (Path patternFilePath : stream) {
|
||||
try(InputStream is = Files.newInputStream(patternFilePath, StandardOpenOption.READ)) {
|
||||
PatternUtils.loadBankFromStream(patternBank, is);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Grok grok = new Grok(patternBank, matchPattern);
|
||||
return new GrokProcessor(grok, matchField, matchPattern);
|
||||
}
|
||||
|
||||
public static class Factory implements Processor.Builder.Factory {
|
||||
private Path grokConfigDirectory;
|
||||
|
||||
@Override
|
||||
public Processor.Builder create() {
|
||||
return new Builder(grokConfigDirectory);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void setConfigDirectory(Path configDirectory) {
|
||||
this.grokConfigDirectory = configDirectory.resolve("ingest").resolve("grok");
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,47 @@
|
|||
/*
|
||||
* Licensed to Elasticsearch under one or more contributor
|
||||
* license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright
|
||||
* ownership. Elasticsearch licenses this file to you under
|
||||
* the Apache License, Version 2.0 (the "License"); you may
|
||||
* not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing,
|
||||
* software distributed under the License is distributed on an
|
||||
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
* KIND, either express or implied. See the License for the
|
||||
* specific language governing permissions and limitations
|
||||
* under the License.
|
||||
*/
|
||||
package org.elasticsearch.ingest.processor.grok;
|
||||
|
||||
import java.io.BufferedReader;
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.io.InputStreamReader;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.util.Map;
|
||||
|
||||
public final class PatternUtils {
|
||||
private PatternUtils() {}
|
||||
|
||||
public static void loadBankFromStream(Map<String, String> patternBank, InputStream inputStream) throws IOException {
|
||||
String line;
|
||||
BufferedReader br = new BufferedReader(new InputStreamReader(inputStream, StandardCharsets.UTF_8));
|
||||
while ((line = br.readLine()) != null) {
|
||||
String trimmedLine = line.replaceAll("^\\s+", "");
|
||||
if (trimmedLine.startsWith("#") || trimmedLine.length() == 0) {
|
||||
continue;
|
||||
}
|
||||
|
||||
String[] parts = trimmedLine.split("\\s+", 2);
|
||||
if (parts.length == 2) {
|
||||
patternBank.put(parts[0], parts[1]);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
|
|
@ -22,6 +22,7 @@ package org.elasticsearch.plugin.ingest;
|
|||
import org.elasticsearch.common.inject.AbstractModule;
|
||||
import org.elasticsearch.common.inject.multibindings.MapBinder;
|
||||
import org.elasticsearch.ingest.processor.Processor;
|
||||
import org.elasticsearch.ingest.processor.grok.GrokProcessor;
|
||||
import org.elasticsearch.ingest.processor.simple.SimpleProcessor;
|
||||
import org.elasticsearch.plugin.ingest.rest.IngestRestFilter;
|
||||
|
||||
|
|
@ -40,6 +41,7 @@ public class IngestModule extends AbstractModule {
|
|||
binder().bind(PipelineStoreClient.class).asEagerSingleton();
|
||||
|
||||
registerProcessor(SimpleProcessor.TYPE, SimpleProcessor.Builder.Factory.class);
|
||||
registerProcessor(GrokProcessor.TYPE, GrokProcessor.Builder.Factory.class);
|
||||
|
||||
MapBinder<String, Processor.Builder.Factory> mapBinder = MapBinder.newMapBinder(binder(), String.class, Processor.Builder.Factory.class);
|
||||
for (Map.Entry<String, Class<? extends Processor.Builder.Factory>> entry : processors.entrySet()) {
|
||||
|
|
|
|||
|
|
@ -29,12 +29,14 @@ import org.elasticsearch.common.inject.Inject;
|
|||
import org.elasticsearch.common.regex.Regex;
|
||||
import org.elasticsearch.common.settings.Settings;
|
||||
import org.elasticsearch.common.unit.TimeValue;
|
||||
import org.elasticsearch.env.Environment;
|
||||
import org.elasticsearch.gateway.GatewayService;
|
||||
import org.elasticsearch.ingest.Pipeline;
|
||||
import org.elasticsearch.ingest.processor.Processor;
|
||||
import org.elasticsearch.search.SearchHit;
|
||||
import org.elasticsearch.threadpool.ThreadPool;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.util.*;
|
||||
|
||||
public class PipelineStore extends AbstractLifecycleComponent {
|
||||
|
|
@ -51,12 +53,15 @@ public class PipelineStore extends AbstractLifecycleComponent {
|
|||
private volatile Map<String, PipelineReference> pipelines = new HashMap<>();
|
||||
|
||||
@Inject
|
||||
public PipelineStore(Settings settings, ThreadPool threadPool, ClusterService clusterService, PipelineStoreClient client, Map<String, Processor.Builder.Factory> processors) {
|
||||
public PipelineStore(Settings settings, ThreadPool threadPool, Environment environment, ClusterService clusterService, PipelineStoreClient client, Map<String, Processor.Builder.Factory> processors) {
|
||||
super(settings);
|
||||
this.threadPool = threadPool;
|
||||
this.clusterService = clusterService;
|
||||
this.pipelineUpdateInterval = settings.getAsTime("ingest.pipeline.store.update.interval", TimeValue.timeValueSeconds(1));
|
||||
this.client = client;
|
||||
for (Processor.Builder.Factory factory : processors.values()) {
|
||||
factory.setConfigDirectory(environment.configFile());
|
||||
}
|
||||
this.processorFactoryRegistry = Collections.unmodifiableMap(processors);
|
||||
clusterService.add(new PipelineStoreListener());
|
||||
}
|
||||
|
|
@ -101,7 +106,7 @@ public class PipelineStore extends AbstractLifecycleComponent {
|
|||
return result;
|
||||
}
|
||||
|
||||
void updatePipelines() {
|
||||
void updatePipelines() throws IOException {
|
||||
// note: this process isn't fast or smart, but the idea is that there will not be many pipelines,
|
||||
// so for that reason the goal is to keep the update logic simple.
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,11 @@
|
|||
S3_REQUEST_LINE (?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})
|
||||
|
||||
S3_ACCESS_LOG %{WORD:owner} %{NOTSPACE:bucket} \[%{HTTPDATE:timestamp}\] %{IP:clientip} %{NOTSPACE:requester} %{NOTSPACE:request_id} %{NOTSPACE:operation} %{NOTSPACE:key} (?:"%{S3_REQUEST_LINE}"|-) (?:%{INT:response:int}|-) (?:-|%{NOTSPACE:error_code}) (?:%{INT:bytes:int}|-) (?:%{INT:object_size:int}|-) (?:%{INT:request_time_ms:int}|-) (?:%{INT:turnaround_time_ms:int}|-) (?:%{QS:referrer}|-) (?:"?%{QS:agent}"?|-) (?:-|%{NOTSPACE:version_id})
|
||||
|
||||
ELB_URIPATHPARAM %{URIPATH:path}(?:%{URIPARAM:params})?
|
||||
|
||||
ELB_URI %{URIPROTO:proto}://(?:%{USER}(?::[^@]*)?@)?(?:%{URIHOST:urihost})?(?:%{ELB_URIPATHPARAM})?
|
||||
|
||||
ELB_REQUEST_LINE (?:%{WORD:verb} %{ELB_URI:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})
|
||||
|
||||
ELB_ACCESS_LOG %{TIMESTAMP_ISO8601:timestamp} %{NOTSPACE:elb} %{IP:clientip}:%{INT:clientport:int} (?:(%{IP:backendip}:?:%{INT:backendport:int})|-) %{NUMBER:request_processing_time:float} %{NUMBER:backend_processing_time:float} %{NUMBER:response_processing_time:float} %{INT:response:int} %{INT:backend_response:int} %{INT:received_bytes:int} %{INT:bytes:int} "%{ELB_REQUEST_LINE}"
|
||||
|
|
@ -0,0 +1,50 @@
|
|||
BACULA_TIMESTAMP %{MONTHDAY}-%{MONTH} %{HOUR}:%{MINUTE}
|
||||
BACULA_HOST [a-zA-Z0-9-]+
|
||||
BACULA_VOLUME %{USER}
|
||||
BACULA_DEVICE %{USER}
|
||||
BACULA_DEVICEPATH %{UNIXPATH}
|
||||
BACULA_CAPACITY %{INT}{1,3}(,%{INT}{3})*
|
||||
BACULA_VERSION %{USER}
|
||||
BACULA_JOB %{USER}
|
||||
|
||||
BACULA_LOG_MAX_CAPACITY User defined maximum volume capacity %{BACULA_CAPACITY} exceeded on device \"%{BACULA_DEVICE:device}\" \(%{BACULA_DEVICEPATH}\)
|
||||
BACULA_LOG_END_VOLUME End of medium on Volume \"%{BACULA_VOLUME:volume}\" Bytes=%{BACULA_CAPACITY} Blocks=%{BACULA_CAPACITY} at %{MONTHDAY}-%{MONTH}-%{YEAR} %{HOUR}:%{MINUTE}.
|
||||
BACULA_LOG_NEW_VOLUME Created new Volume \"%{BACULA_VOLUME:volume}\" in catalog.
|
||||
BACULA_LOG_NEW_LABEL Labeled new Volume \"%{BACULA_VOLUME:volume}\" on device \"%{BACULA_DEVICE:device}\" \(%{BACULA_DEVICEPATH}\).
|
||||
BACULA_LOG_WROTE_LABEL Wrote label to prelabeled Volume \"%{BACULA_VOLUME:volume}\" on device \"%{BACULA_DEVICE}\" \(%{BACULA_DEVICEPATH}\)
|
||||
BACULA_LOG_NEW_MOUNT New volume \"%{BACULA_VOLUME:volume}\" mounted on device \"%{BACULA_DEVICE:device}\" \(%{BACULA_DEVICEPATH}\) at %{MONTHDAY}-%{MONTH}-%{YEAR} %{HOUR}:%{MINUTE}.
|
||||
BACULA_LOG_NOOPEN \s+Cannot open %{DATA}: ERR=%{GREEDYDATA:berror}
|
||||
BACULA_LOG_NOOPENDIR \s+Could not open directory %{DATA}: ERR=%{GREEDYDATA:berror}
|
||||
BACULA_LOG_NOSTAT \s+Could not stat %{DATA}: ERR=%{GREEDYDATA:berror}
|
||||
BACULA_LOG_NOJOBS There are no more Jobs associated with Volume \"%{BACULA_VOLUME:volume}\". Marking it purged.
|
||||
BACULA_LOG_ALL_RECORDS_PRUNED All records pruned from Volume \"%{BACULA_VOLUME:volume}\"; marking it \"Purged\"
|
||||
BACULA_LOG_BEGIN_PRUNE_JOBS Begin pruning Jobs older than %{INT} month %{INT} days .
|
||||
BACULA_LOG_BEGIN_PRUNE_FILES Begin pruning Files.
|
||||
BACULA_LOG_PRUNED_JOBS Pruned %{INT} Jobs* for client %{BACULA_HOST:client} from catalog.
|
||||
BACULA_LOG_PRUNED_FILES Pruned Files from %{INT} Jobs* for client %{BACULA_HOST:client} from catalog.
|
||||
BACULA_LOG_ENDPRUNE End auto prune.
|
||||
BACULA_LOG_STARTJOB Start Backup JobId %{INT}, Job=%{BACULA_JOB:job}
|
||||
BACULA_LOG_STARTRESTORE Start Restore Job %{BACULA_JOB:job}
|
||||
BACULA_LOG_USEDEVICE Using Device \"%{BACULA_DEVICE:device}\"
|
||||
BACULA_LOG_DIFF_FS \s+%{UNIXPATH} is a different filesystem. Will not descend from %{UNIXPATH} into it.
|
||||
BACULA_LOG_JOBEND Job write elapsed time = %{DATA:elapsed}, Transfer rate = %{NUMBER} (K|M|G)? Bytes/second
|
||||
BACULA_LOG_NOPRUNE_JOBS No Jobs found to prune.
|
||||
BACULA_LOG_NOPRUNE_FILES No Files found to prune.
|
||||
BACULA_LOG_VOLUME_PREVWRITTEN Volume \"%{BACULA_VOLUME:volume}\" previously written, moving to end of data.
|
||||
BACULA_LOG_READYAPPEND Ready to append to end of Volume \"%{BACULA_VOLUME:volume}\" size=%{INT}
|
||||
BACULA_LOG_CANCELLING Cancelling duplicate JobId=%{INT}.
|
||||
BACULA_LOG_MARKCANCEL JobId %{INT}, Job %{BACULA_JOB:job} marked to be canceled.
|
||||
BACULA_LOG_CLIENT_RBJ shell command: run ClientRunBeforeJob \"%{GREEDYDATA:runjob}\"
|
||||
BACULA_LOG_VSS (Generate )?VSS (Writer)?
|
||||
BACULA_LOG_MAXSTART Fatal error: Job canceled because max start delay time exceeded.
|
||||
BACULA_LOG_DUPLICATE Fatal error: JobId %{INT:duplicate} already running. Duplicate job not allowed.
|
||||
BACULA_LOG_NOJOBSTAT Fatal error: No Job status returned from FD.
|
||||
BACULA_LOG_FATAL_CONN Fatal error: bsock.c:133 Unable to connect to (Client: %{BACULA_HOST:client}|Storage daemon) on %{HOSTNAME}:%{POSINT}. ERR=(?<berror>%{GREEDYDATA})
|
||||
BACULA_LOG_NO_CONNECT Warning: bsock.c:127 Could not connect to (Client: %{BACULA_HOST:client}|Storage daemon) on %{HOSTNAME}:%{POSINT}. ERR=(?<berror>%{GREEDYDATA})
|
||||
BACULA_LOG_NO_AUTH Fatal error: Unable to authenticate with File daemon at %{HOSTNAME}. Possible causes:
|
||||
BACULA_LOG_NOSUIT No prior or suitable Full backup found in catalog. Doing FULL backup.
|
||||
BACULA_LOG_NOPRIOR No prior Full backup Job record found.
|
||||
|
||||
BACULA_LOG_JOB (Error: )?Bacula %{BACULA_HOST} %{BACULA_VERSION} \(%{BACULA_VERSION}\):
|
||||
|
||||
BACULA_LOGLINE %{BACULA_TIMESTAMP:bts} %{BACULA_HOST:hostname} JobId %{INT:jobid}: (%{BACULA_LOG_MAX_CAPACITY}|%{BACULA_LOG_END_VOLUME}|%{BACULA_LOG_NEW_VOLUME}|%{BACULA_LOG_NEW_LABEL}|%{BACULA_LOG_WROTE_LABEL}|%{BACULA_LOG_NEW_MOUNT}|%{BACULA_LOG_NOOPEN}|%{BACULA_LOG_NOOPENDIR}|%{BACULA_LOG_NOSTAT}|%{BACULA_LOG_NOJOBS}|%{BACULA_LOG_ALL_RECORDS_PRUNED}|%{BACULA_LOG_BEGIN_PRUNE_JOBS}|%{BACULA_LOG_BEGIN_PRUNE_FILES}|%{BACULA_LOG_PRUNED_JOBS}|%{BACULA_LOG_PRUNED_FILES}|%{BACULA_LOG_ENDPRUNE}|%{BACULA_LOG_STARTJOB}|%{BACULA_LOG_STARTRESTORE}|%{BACULA_LOG_USEDEVICE}|%{BACULA_LOG_DIFF_FS}|%{BACULA_LOG_JOBEND}|%{BACULA_LOG_NOPRUNE_JOBS}|%{BACULA_LOG_NOPRUNE_FILES}|%{BACULA_LOG_VOLUME_PREVWRITTEN}|%{BACULA_LOG_READYAPPEND}|%{BACULA_LOG_CANCELLING}|%{BACULA_LOG_MARKCANCEL}|%{BACULA_LOG_CLIENT_RBJ}|%{BACULA_LOG_VSS}|%{BACULA_LOG_MAXSTART}|%{BACULA_LOG_DUPLICATE}|%{BACULA_LOG_NOJOBSTAT}|%{BACULA_LOG_FATAL_CONN}|%{BACULA_LOG_NO_CONNECT}|%{BACULA_LOG_NO_AUTH}|%{BACULA_LOG_NOSUIT}|%{BACULA_LOG_JOB}|%{BACULA_LOG_NOPRIOR})
|
||||
|
|
@ -0,0 +1,13 @@
|
|||
# https://www.bro.org/sphinx/script-reference/log-files.html
|
||||
|
||||
# http.log
|
||||
BRO_HTTP %{NUMBER:ts}\t%{NOTSPACE:uid}\t%{IP:orig_h}\t%{INT:orig_p}\t%{IP:resp_h}\t%{INT:resp_p}\t%{INT:trans_depth}\t%{GREEDYDATA:method}\t%{GREEDYDATA:domain}\t%{GREEDYDATA:uri}\t%{GREEDYDATA:referrer}\t%{GREEDYDATA:user_agent}\t%{NUMBER:request_body_len}\t%{NUMBER:response_body_len}\t%{GREEDYDATA:status_code}\t%{GREEDYDATA:status_msg}\t%{GREEDYDATA:info_code}\t%{GREEDYDATA:info_msg}\t%{GREEDYDATA:filename}\t%{GREEDYDATA:bro_tags}\t%{GREEDYDATA:username}\t%{GREEDYDATA:password}\t%{GREEDYDATA:proxied}\t%{GREEDYDATA:orig_fuids}\t%{GREEDYDATA:orig_mime_types}\t%{GREEDYDATA:resp_fuids}\t%{GREEDYDATA:resp_mime_types}
|
||||
|
||||
# dns.log
|
||||
BRO_DNS %{NUMBER:ts}\t%{NOTSPACE:uid}\t%{IP:orig_h}\t%{INT:orig_p}\t%{IP:resp_h}\t%{INT:resp_p}\t%{WORD:proto}\t%{INT:trans_id}\t%{GREEDYDATA:query}\t%{GREEDYDATA:qclass}\t%{GREEDYDATA:qclass_name}\t%{GREEDYDATA:qtype}\t%{GREEDYDATA:qtype_name}\t%{GREEDYDATA:rcode}\t%{GREEDYDATA:rcode_name}\t%{GREEDYDATA:AA}\t%{GREEDYDATA:TC}\t%{GREEDYDATA:RD}\t%{GREEDYDATA:RA}\t%{GREEDYDATA:Z}\t%{GREEDYDATA:answers}\t%{GREEDYDATA:TTLs}\t%{GREEDYDATA:rejected}
|
||||
|
||||
# conn.log
|
||||
BRO_CONN %{NUMBER:ts}\t%{NOTSPACE:uid}\t%{IP:orig_h}\t%{INT:orig_p}\t%{IP:resp_h}\t%{INT:resp_p}\t%{WORD:proto}\t%{GREEDYDATA:service}\t%{NUMBER:duration}\t%{NUMBER:orig_bytes}\t%{NUMBER:resp_bytes}\t%{GREEDYDATA:conn_state}\t%{GREEDYDATA:local_orig}\t%{GREEDYDATA:missed_bytes}\t%{GREEDYDATA:history}\t%{GREEDYDATA:orig_pkts}\t%{GREEDYDATA:orig_ip_bytes}\t%{GREEDYDATA:resp_pkts}\t%{GREEDYDATA:resp_ip_bytes}\t%{GREEDYDATA:tunnel_parents}
|
||||
|
||||
# files.log
|
||||
BRO_FILES %{NUMBER:ts}\t%{NOTSPACE:fuid}\t%{IP:tx_hosts}\t%{IP:rx_hosts}\t%{NOTSPACE:conn_uids}\t%{GREEDYDATA:source}\t%{GREEDYDATA:depth}\t%{GREEDYDATA:analyzers}\t%{GREEDYDATA:mime_type}\t%{GREEDYDATA:filename}\t%{GREEDYDATA:duration}\t%{GREEDYDATA:local_orig}\t%{GREEDYDATA:is_orig}\t%{GREEDYDATA:seen_bytes}\t%{GREEDYDATA:total_bytes}\t%{GREEDYDATA:missing_bytes}\t%{GREEDYDATA:overflow_bytes}\t%{GREEDYDATA:timedout}\t%{GREEDYDATA:parent_fuid}\t%{GREEDYDATA:md5}\t%{GREEDYDATA:sha1}\t%{GREEDYDATA:sha256}\t%{GREEDYDATA:extracted}
|
||||
|
|
@ -0,0 +1,13 @@
|
|||
EXIM_MSGID [0-9A-Za-z]{6}-[0-9A-Za-z]{6}-[0-9A-Za-z]{2}
|
||||
EXIM_FLAGS (<=|[-=>*]>|[*]{2}|==)
|
||||
EXIM_DATE %{YEAR:exim_year}-%{MONTHNUM:exim_month}-%{MONTHDAY:exim_day} %{TIME:exim_time}
|
||||
EXIM_PID \[%{POSINT}\]
|
||||
EXIM_QT ((\d+y)?(\d+w)?(\d+d)?(\d+h)?(\d+m)?(\d+s)?)
|
||||
EXIM_EXCLUDE_TERMS (Message is frozen|(Start|End) queue run| Warning: | retry time not reached | no (IP address|host name) found for (IP address|host) | unexpected disconnection while reading SMTP command | no immediate delivery: |another process is handling this message)
|
||||
EXIM_REMOTE_HOST (H=(%{NOTSPACE:remote_hostname} )?(\(%{NOTSPACE:remote_heloname}\) )?\[%{IP:remote_host}\])
|
||||
EXIM_INTERFACE (I=\[%{IP:exim_interface}\](:%{NUMBER:exim_interface_port}))
|
||||
EXIM_PROTOCOL (P=%{NOTSPACE:protocol})
|
||||
EXIM_MSG_SIZE (S=%{NUMBER:exim_msg_size})
|
||||
EXIM_HEADER_ID (id=%{NOTSPACE:exim_header_id})
|
||||
EXIM_SUBJECT (T=%{QS:exim_subject})
|
||||
|
||||
|
|
@ -0,0 +1,86 @@
|
|||
# NetScreen firewall logs
|
||||
NETSCREENSESSIONLOG %{SYSLOGTIMESTAMP:date} %{IPORHOST:device} %{IPORHOST}: NetScreen device_id=%{WORD:device_id}%{DATA}: start_time=%{QUOTEDSTRING:start_time} duration=%{INT:duration} policy_id=%{INT:policy_id} service=%{DATA:service} proto=%{INT:proto} src zone=%{WORD:src_zone} dst zone=%{WORD:dst_zone} action=%{WORD:action} sent=%{INT:sent} rcvd=%{INT:rcvd} src=%{IPORHOST:src_ip} dst=%{IPORHOST:dst_ip} src_port=%{INT:src_port} dst_port=%{INT:dst_port} src-xlated ip=%{IPORHOST:src_xlated_ip} port=%{INT:src_xlated_port} dst-xlated ip=%{IPORHOST:dst_xlated_ip} port=%{INT:dst_xlated_port} session_id=%{INT:session_id} reason=%{GREEDYDATA:reason}
|
||||
|
||||
#== Cisco ASA ==
|
||||
CISCO_TAGGED_SYSLOG ^<%{POSINT:syslog_pri}>%{CISCOTIMESTAMP:timestamp}( %{SYSLOGHOST:sysloghost})? ?: %%{CISCOTAG:ciscotag}:
|
||||
CISCOTIMESTAMP %{MONTH} +%{MONTHDAY}(?: %{YEAR})? %{TIME}
|
||||
CISCOTAG [A-Z0-9]+-%{INT}-(?:[A-Z0-9_]+)
|
||||
# Common Particles
|
||||
CISCO_ACTION Built|Teardown|Deny|Denied|denied|requested|permitted|denied by ACL|discarded|est-allowed|Dropping|created|deleted
|
||||
CISCO_REASON Duplicate TCP SYN|Failed to locate egress interface|Invalid transport field|No matching connection|DNS Response|DNS Query|(?:%{WORD}\s*)*
|
||||
CISCO_DIRECTION Inbound|inbound|Outbound|outbound
|
||||
CISCO_INTERVAL first hit|%{INT}-second interval
|
||||
CISCO_XLATE_TYPE static|dynamic
|
||||
# ASA-1-104001
|
||||
CISCOFW104001 \((?:Primary|Secondary)\) Switching to ACTIVE - %{GREEDYDATA:switch_reason}
|
||||
# ASA-1-104002
|
||||
CISCOFW104002 \((?:Primary|Secondary)\) Switching to STANDBY - %{GREEDYDATA:switch_reason}
|
||||
# ASA-1-104003
|
||||
CISCOFW104003 \((?:Primary|Secondary)\) Switching to FAILED\.
|
||||
# ASA-1-104004
|
||||
CISCOFW104004 \((?:Primary|Secondary)\) Switching to OK\.
|
||||
# ASA-1-105003
|
||||
CISCOFW105003 \((?:Primary|Secondary)\) Monitoring on [Ii]nterface %{GREEDYDATA:interface_name} waiting
|
||||
# ASA-1-105004
|
||||
CISCOFW105004 \((?:Primary|Secondary)\) Monitoring on [Ii]nterface %{GREEDYDATA:interface_name} normal
|
||||
# ASA-1-105005
|
||||
CISCOFW105005 \((?:Primary|Secondary)\) Lost Failover communications with mate on [Ii]nterface %{GREEDYDATA:interface_name}
|
||||
# ASA-1-105008
|
||||
CISCOFW105008 \((?:Primary|Secondary)\) Testing [Ii]nterface %{GREEDYDATA:interface_name}
|
||||
# ASA-1-105009
|
||||
CISCOFW105009 \((?:Primary|Secondary)\) Testing on [Ii]nterface %{GREEDYDATA:interface_name} (?:Passed|Failed)
|
||||
# ASA-2-106001
|
||||
CISCOFW106001 %{CISCO_DIRECTION:direction} %{WORD:protocol} connection %{CISCO_ACTION:action} from %{IP:src_ip}/%{INT:src_port} to %{IP:dst_ip}/%{INT:dst_port} flags %{GREEDYDATA:tcp_flags} on interface %{GREEDYDATA:interface}
|
||||
# ASA-2-106006, ASA-2-106007, ASA-2-106010
|
||||
CISCOFW106006_106007_106010 %{CISCO_ACTION:action} %{CISCO_DIRECTION:direction} %{WORD:protocol} (?:from|src) %{IP:src_ip}/%{INT:src_port}(\(%{DATA:src_fwuser}\))? (?:to|dst) %{IP:dst_ip}/%{INT:dst_port}(\(%{DATA:dst_fwuser}\))? (?:on interface %{DATA:interface}|due to %{CISCO_REASON:reason})
|
||||
# ASA-3-106014
|
||||
CISCOFW106014 %{CISCO_ACTION:action} %{CISCO_DIRECTION:direction} %{WORD:protocol} src %{DATA:src_interface}:%{IP:src_ip}(\(%{DATA:src_fwuser}\))? dst %{DATA:dst_interface}:%{IP:dst_ip}(\(%{DATA:dst_fwuser}\))? \(type %{INT:icmp_type}, code %{INT:icmp_code}\)
|
||||
# ASA-6-106015
|
||||
CISCOFW106015 %{CISCO_ACTION:action} %{WORD:protocol} \(%{DATA:policy_id}\) from %{IP:src_ip}/%{INT:src_port} to %{IP:dst_ip}/%{INT:dst_port} flags %{DATA:tcp_flags} on interface %{GREEDYDATA:interface}
|
||||
# ASA-1-106021
|
||||
CISCOFW106021 %{CISCO_ACTION:action} %{WORD:protocol} reverse path check from %{IP:src_ip} to %{IP:dst_ip} on interface %{GREEDYDATA:interface}
|
||||
# ASA-4-106023
|
||||
CISCOFW106023 %{CISCO_ACTION:action}( protocol)? %{WORD:protocol} src %{DATA:src_interface}:%{DATA:src_ip}(/%{INT:src_port})?(\(%{DATA:src_fwuser}\))? dst %{DATA:dst_interface}:%{DATA:dst_ip}(/%{INT:dst_port})?(\(%{DATA:dst_fwuser}\))?( \(type %{INT:icmp_type}, code %{INT:icmp_code}\))? by access-group "?%{DATA:policy_id}"? \[%{DATA:hashcode1}, %{DATA:hashcode2}\]
|
||||
# ASA-4-106100, ASA-4-106102, ASA-4-106103
|
||||
CISCOFW106100_2_3 access-list %{NOTSPACE:policy_id} %{CISCO_ACTION:action} %{WORD:protocol} for user '%{DATA:src_fwuser}' %{DATA:src_interface}/%{IP:src_ip}\(%{INT:src_port}\) -> %{DATA:dst_interface}/%{IP:dst_ip}\(%{INT:dst_port}\) hit-cnt %{INT:hit_count} %{CISCO_INTERVAL:interval} \[%{DATA:hashcode1}, %{DATA:hashcode2}\]
|
||||
# ASA-5-106100
|
||||
CISCOFW106100 access-list %{NOTSPACE:policy_id} %{CISCO_ACTION:action} %{WORD:protocol} %{DATA:src_interface}/%{IP:src_ip}\(%{INT:src_port}\)(\(%{DATA:src_fwuser}\))? -> %{DATA:dst_interface}/%{IP:dst_ip}\(%{INT:dst_port}\)(\(%{DATA:src_fwuser}\))? hit-cnt %{INT:hit_count} %{CISCO_INTERVAL:interval} \[%{DATA:hashcode1}, %{DATA:hashcode2}\]
|
||||
# ASA-6-110002
|
||||
CISCOFW110002 %{CISCO_REASON:reason} for %{WORD:protocol} from %{DATA:src_interface}:%{IP:src_ip}/%{INT:src_port} to %{IP:dst_ip}/%{INT:dst_port}
|
||||
# ASA-6-302010
|
||||
CISCOFW302010 %{INT:connection_count} in use, %{INT:connection_count_max} most used
|
||||
# ASA-6-302013, ASA-6-302014, ASA-6-302015, ASA-6-302016
|
||||
CISCOFW302013_302014_302015_302016 %{CISCO_ACTION:action}(?: %{CISCO_DIRECTION:direction})? %{WORD:protocol} connection %{INT:connection_id} for %{DATA:src_interface}:%{IP:src_ip}/%{INT:src_port}( \(%{IP:src_mapped_ip}/%{INT:src_mapped_port}\))?(\(%{DATA:src_fwuser}\))? to %{DATA:dst_interface}:%{IP:dst_ip}/%{INT:dst_port}( \(%{IP:dst_mapped_ip}/%{INT:dst_mapped_port}\))?(\(%{DATA:dst_fwuser}\))?( duration %{TIME:duration} bytes %{INT:bytes})?(?: %{CISCO_REASON:reason})?( \(%{DATA:user}\))?
|
||||
# ASA-6-302020, ASA-6-302021
|
||||
CISCOFW302020_302021 %{CISCO_ACTION:action}(?: %{CISCO_DIRECTION:direction})? %{WORD:protocol} connection for faddr %{IP:dst_ip}/%{INT:icmp_seq_num}(?:\(%{DATA:fwuser}\))? gaddr %{IP:src_xlated_ip}/%{INT:icmp_code_xlated} laddr %{IP:src_ip}/%{INT:icmp_code}( \(%{DATA:user}\))?
|
||||
# ASA-6-305011
|
||||
CISCOFW305011 %{CISCO_ACTION:action} %{CISCO_XLATE_TYPE:xlate_type} %{WORD:protocol} translation from %{DATA:src_interface}:%{IP:src_ip}(/%{INT:src_port})?(\(%{DATA:src_fwuser}\))? to %{DATA:src_xlated_interface}:%{IP:src_xlated_ip}/%{DATA:src_xlated_port}
|
||||
# ASA-3-313001, ASA-3-313004, ASA-3-313008
|
||||
CISCOFW313001_313004_313008 %{CISCO_ACTION:action} %{WORD:protocol} type=%{INT:icmp_type}, code=%{INT:icmp_code} from %{IP:src_ip} on interface %{DATA:interface}( to %{IP:dst_ip})?
|
||||
# ASA-4-313005
|
||||
CISCOFW313005 %{CISCO_REASON:reason} for %{WORD:protocol} error message: %{WORD:err_protocol} src %{DATA:err_src_interface}:%{IP:err_src_ip}(\(%{DATA:err_src_fwuser}\))? dst %{DATA:err_dst_interface}:%{IP:err_dst_ip}(\(%{DATA:err_dst_fwuser}\))? \(type %{INT:err_icmp_type}, code %{INT:err_icmp_code}\) on %{DATA:interface} interface\. Original IP payload: %{WORD:protocol} src %{IP:orig_src_ip}/%{INT:orig_src_port}(\(%{DATA:orig_src_fwuser}\))? dst %{IP:orig_dst_ip}/%{INT:orig_dst_port}(\(%{DATA:orig_dst_fwuser}\))?
|
||||
# ASA-5-321001
|
||||
CISCOFW321001 Resource '%{WORD:resource_name}' limit of %{POSINT:resource_limit} reached for system
|
||||
# ASA-4-402117
|
||||
CISCOFW402117 %{WORD:protocol}: Received a non-IPSec packet \(protocol= %{WORD:orig_protocol}\) from %{IP:src_ip} to %{IP:dst_ip}
|
||||
# ASA-4-402119
|
||||
CISCOFW402119 %{WORD:protocol}: Received an %{WORD:orig_protocol} packet \(SPI= %{DATA:spi}, sequence number= %{DATA:seq_num}\) from %{IP:src_ip} \(user= %{DATA:user}\) to %{IP:dst_ip} that failed anti-replay checking
|
||||
# ASA-4-419001
|
||||
CISCOFW419001 %{CISCO_ACTION:action} %{WORD:protocol} packet from %{DATA:src_interface}:%{IP:src_ip}/%{INT:src_port} to %{DATA:dst_interface}:%{IP:dst_ip}/%{INT:dst_port}, reason: %{GREEDYDATA:reason}
|
||||
# ASA-4-419002
|
||||
CISCOFW419002 %{CISCO_REASON:reason} from %{DATA:src_interface}:%{IP:src_ip}/%{INT:src_port} to %{DATA:dst_interface}:%{IP:dst_ip}/%{INT:dst_port} with different initial sequence number
|
||||
# ASA-4-500004
|
||||
CISCOFW500004 %{CISCO_REASON:reason} for protocol=%{WORD:protocol}, from %{IP:src_ip}/%{INT:src_port} to %{IP:dst_ip}/%{INT:dst_port}
|
||||
# ASA-6-602303, ASA-6-602304
|
||||
CISCOFW602303_602304 %{WORD:protocol}: An %{CISCO_DIRECTION:direction} %{GREEDYDATA:tunnel_type} SA \(SPI= %{DATA:spi}\) between %{IP:src_ip} and %{IP:dst_ip} \(user= %{DATA:user}\) has been %{CISCO_ACTION:action}
|
||||
# ASA-7-710001, ASA-7-710002, ASA-7-710003, ASA-7-710005, ASA-7-710006
|
||||
CISCOFW710001_710002_710003_710005_710006 %{WORD:protocol} (?:request|access) %{CISCO_ACTION:action} from %{IP:src_ip}/%{INT:src_port} to %{DATA:dst_interface}:%{IP:dst_ip}/%{INT:dst_port}
|
||||
# ASA-6-713172
|
||||
CISCOFW713172 Group = %{GREEDYDATA:group}, IP = %{IP:src_ip}, Automatic NAT Detection Status:\s+Remote end\s*%{DATA:is_remote_natted}\s*behind a NAT device\s+This\s+end\s*%{DATA:is_local_natted}\s*behind a NAT device
|
||||
# ASA-4-733100
|
||||
CISCOFW733100 \[\s*%{DATA:drop_type}\s*\] drop %{DATA:drop_rate_id} exceeded. Current burst rate is %{INT:drop_rate_current_burst} per second, max configured rate is %{INT:drop_rate_max_burst}; Current average rate is %{INT:drop_rate_current_avg} per second, max configured rate is %{INT:drop_rate_max_avg}; Cumulative total count is %{INT:drop_total_count}
|
||||
#== End Cisco ASA ==
|
||||
|
||||
# Shorewall firewall logs
|
||||
SHOREWALL (%{SYSLOGTIMESTAMP:timestamp}) (%{WORD:nf_host}) kernel:.*Shorewall:(%{WORD:nf_action1})?:(%{WORD:nf_action2})?.*IN=(%{USERNAME:nf_in_interface})?.*(OUT= *MAC=(%{COMMONMAC:nf_dst_mac}):(%{COMMONMAC:nf_src_mac})?|OUT=%{USERNAME:nf_out_interface}).*SRC=(%{IPV4:nf_src_ip}).*DST=(%{IPV4:nf_dst_ip}).*LEN=(%{WORD:nf_len}).?*TOS=(%{WORD:nf_tos}).?*PREC=(%{WORD:nf_prec}).?*TTL=(%{INT:nf_ttl}).?*ID=(%{INT:nf_id}).?*PROTO=(%{WORD:nf_protocol}).?*SPT=(%{INT:nf_src_port}?.*DPT=%{INT:nf_dst_port}?.*)
|
||||
#== End Shorewall
|
||||
|
|
@ -0,0 +1,102 @@
|
|||
USERNAME [a-zA-Z0-9._-]+
|
||||
USER %{USERNAME}
|
||||
EMAILLOCALPART [a-zA-Z][a-zA-Z0-9_.+-=:]+
|
||||
EMAILADDRESS %{EMAILLOCALPART}@%{HOSTNAME}
|
||||
HTTPDUSER %{EMAILADDRESS}|%{USER}
|
||||
INT (?:[+-]?(?:[0-9]+))
|
||||
BASE10NUM (?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+)))
|
||||
NUMBER (?:%{BASE10NUM})
|
||||
BASE16NUM (?<![0-9A-Fa-f])(?:[+-]?(?:0x)?(?:[0-9A-Fa-f]+))
|
||||
BASE16FLOAT \b(?<![0-9A-Fa-f.])(?:[+-]?(?:0x)?(?:(?:[0-9A-Fa-f]+(?:\.[0-9A-Fa-f]*)?)|(?:\.[0-9A-Fa-f]+)))\b
|
||||
|
||||
POSINT \b(?:[1-9][0-9]*)\b
|
||||
NONNEGINT \b(?:[0-9]+)\b
|
||||
WORD \b\w+\b
|
||||
NOTSPACE \S+
|
||||
SPACE \s*
|
||||
DATA .*?
|
||||
GREEDYDATA .*
|
||||
QUOTEDSTRING (?>(?<!\\)(?>"(?>\\.|[^\\"]+)+"|""|(?>'(?>\\.|[^\\']+)+')|''|(?>`(?>\\.|[^\\`]+)+`)|``))
|
||||
UUID [A-Fa-f0-9]{8}-(?:[A-Fa-f0-9]{4}-){3}[A-Fa-f0-9]{12}
|
||||
|
||||
# Networking
|
||||
MAC (?:%{CISCOMAC}|%{WINDOWSMAC}|%{COMMONMAC})
|
||||
CISCOMAC (?:(?:[A-Fa-f0-9]{4}\.){2}[A-Fa-f0-9]{4})
|
||||
WINDOWSMAC (?:(?:[A-Fa-f0-9]{2}-){5}[A-Fa-f0-9]{2})
|
||||
COMMONMAC (?:(?:[A-Fa-f0-9]{2}:){5}[A-Fa-f0-9]{2})
|
||||
IPV6 ((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:)))(%.+)?
|
||||
IPV4 (?<![0-9])(?:(?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5]))(?![0-9])
|
||||
IP (?:%{IPV6}|%{IPV4})
|
||||
HOSTNAME \b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\.?|\b)
|
||||
IPORHOST (?:%{IP}|%{HOSTNAME})
|
||||
HOSTPORT %{IPORHOST}:%{POSINT}
|
||||
|
||||
# paths
|
||||
PATH (?:%{UNIXPATH}|%{WINPATH})
|
||||
UNIXPATH (/([\w_%!$@:.,~-]+|\\.)*)+
|
||||
TTY (?:/dev/(pts|tty([pq])?)(\w+)?/?(?:[0-9]+))
|
||||
WINPATH (?>[A-Za-z]+:|\\)(?:\\[^\\?*]*)+
|
||||
URIPROTO [A-Za-z]+(\+[A-Za-z+]+)?
|
||||
URIHOST %{IPORHOST}(?::%{POSINT:port})?
|
||||
# uripath comes loosely from RFC1738, but mostly from what Firefox
|
||||
# doesn't turn into %XX
|
||||
URIPATH (?:/[A-Za-z0-9$.+!*'(){},~:;=@#%_\-]*)+
|
||||
#URIPARAM \?(?:[A-Za-z0-9]+(?:=(?:[^&]*))?(?:&(?:[A-Za-z0-9]+(?:=(?:[^&]*))?)?)*)?
|
||||
URIPARAM \?[A-Za-z0-9$.+!*'|(){},~@#%&/=:;_?\-\[\]<>]*
|
||||
URIPATHPARAM %{URIPATH}(?:%{URIPARAM})?
|
||||
URI %{URIPROTO}://(?:%{USER}(?::[^@]*)?@)?(?:%{URIHOST})?(?:%{URIPATHPARAM})?
|
||||
|
||||
# Months: January, Feb, 3, 03, 12, December
|
||||
MONTH \b(?:Jan(?:uary|uar)?|Feb(?:ruary|ruar)?|M(?:a|ä)?r(?:ch|z)?|Apr(?:il)?|Ma(?:y|i)?|Jun(?:e|i)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|O(?:c|k)?t(?:ober)?|Nov(?:ember)?|De(?:c|z)(?:ember)?)\b
|
||||
MONTHNUM (?:0?[1-9]|1[0-2])
|
||||
MONTHNUM2 (?:0[1-9]|1[0-2])
|
||||
MONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])
|
||||
|
||||
# Days: Monday, Tue, Thu, etc...
|
||||
DAY (?:Mon(?:day)?|Tue(?:sday)?|Wed(?:nesday)?|Thu(?:rsday)?|Fri(?:day)?|Sat(?:urday)?|Sun(?:day)?)
|
||||
|
||||
# Years?
|
||||
YEAR (?>\d\d){1,2}
|
||||
HOUR (?:2[0123]|[01]?[0-9])
|
||||
MINUTE (?:[0-5][0-9])
|
||||
# '60' is a leap second in most time standards and thus is valid.
|
||||
SECOND (?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)
|
||||
TIME (?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9])
|
||||
# datestamp is YYYY/MM/DD-HH:MM:SS.UUUU (or something like it)
|
||||
DATE_US %{MONTHNUM}[/-]%{MONTHDAY}[/-]%{YEAR}
|
||||
DATE_EU %{MONTHDAY}[./-]%{MONTHNUM}[./-]%{YEAR}
|
||||
ISO8601_TIMEZONE (?:Z|[+-]%{HOUR}(?::?%{MINUTE}))
|
||||
ISO8601_SECOND (?:%{SECOND}|60)
|
||||
ISO8601_HOUR (?:2[0123]|[01][0-9])
|
||||
TIMESTAMP_ISO8601 %{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{ISO8601_HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE}?
|
||||
DATE %{DATE_US}|%{DATE_EU}
|
||||
DATESTAMP %{DATE}[- ]%{TIME}
|
||||
TZ (?:[PMCE][SD]T|UTC)
|
||||
DATESTAMP_RFC822 %{DAY} %{MONTH} %{MONTHDAY} %{YEAR} %{TIME} %{TZ}
|
||||
DATESTAMP_RFC2822 %{DAY}, %{MONTHDAY} %{MONTH} %{YEAR} %{TIME} %{ISO8601_TIMEZONE}
|
||||
DATESTAMP_OTHER %{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{TZ} %{YEAR}
|
||||
DATESTAMP_EVENTLOG %{YEAR}%{MONTHNUM2}%{MONTHDAY}%{HOUR}%{MINUTE}%{SECOND}
|
||||
HTTPDERROR_DATE %{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR}
|
||||
|
||||
# Syslog Dates: Month Day HH:MM:SS
|
||||
SYSLOGTIMESTAMP %{MONTH} +%{MONTHDAY} %{TIME}
|
||||
PROG [\x21-\x5a\x5c\x5e-\x7e]+
|
||||
SYSLOGPROG %{PROG:program}(?:\[%{POSINT:pid}\])?
|
||||
SYSLOGHOST %{IPORHOST}
|
||||
SYSLOGFACILITY <%{NONNEGINT:facility}.%{NONNEGINT:priority}>
|
||||
HTTPDATE %{MONTHDAY}/%{MONTH}/%{YEAR}:%{TIME} %{INT}
|
||||
|
||||
# Shortcuts
|
||||
QS %{QUOTEDSTRING}
|
||||
|
||||
# Log formats
|
||||
SYSLOGBASE %{SYSLOGTIMESTAMP:timestamp} (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}:
|
||||
COMMONAPACHELOG %{IPORHOST:clientip} %{HTTPDUSER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)
|
||||
COMBINEDAPACHELOG %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}
|
||||
HTTPD20_ERRORLOG \[%{HTTPDERROR_DATE:timestamp}\] \[%{LOGLEVEL:loglevel}\] (?:\[client %{IPORHOST:clientip}\] ){0,1}%{GREEDYDATA:errormsg}
|
||||
HTTPD24_ERRORLOG \[%{HTTPDERROR_DATE:timestamp}\] \[%{WORD:module}:%{LOGLEVEL:loglevel}\] \[pid %{POSINT:pid}:tid %{NUMBER:tid}\]( \(%{POSINT:proxy_errorcode}\)%{DATA:proxy_errormessage}:)?( \[client %{IPORHOST:client}:%{POSINT:clientport}\])? %{DATA:errorcode}: %{GREEDYDATA:message}
|
||||
HTTPD_ERRORLOG %{HTTPD20_ERRORLOG}|%{HTTPD24_ERRORLOG}
|
||||
|
||||
|
||||
# Log Levels
|
||||
LOGLEVEL ([Aa]lert|ALERT|[Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nfo|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|[Ee]rr?(?:or)?|ERR?(?:OR)?|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?)
|
||||
|
|
@ -0,0 +1,39 @@
|
|||
## These patterns were tested w/ haproxy-1.4.15
|
||||
|
||||
## Documentation of the haproxy log formats can be found at the following links:
|
||||
## http://code.google.com/p/haproxy-docs/wiki/HTTPLogFormat
|
||||
## http://code.google.com/p/haproxy-docs/wiki/TCPLogFormat
|
||||
|
||||
HAPROXYTIME (?!<[0-9])%{HOUR:haproxy_hour}:%{MINUTE:haproxy_minute}(?::%{SECOND:haproxy_second})(?![0-9])
|
||||
HAPROXYDATE %{MONTHDAY:haproxy_monthday}/%{MONTH:haproxy_month}/%{YEAR:haproxy_year}:%{HAPROXYTIME:haproxy_time}.%{INT:haproxy_milliseconds}
|
||||
|
||||
# Override these default patterns to parse out what is captured in your haproxy.cfg
|
||||
HAPROXYCAPTUREDREQUESTHEADERS %{DATA:captured_request_headers}
|
||||
HAPROXYCAPTUREDRESPONSEHEADERS %{DATA:captured_response_headers}
|
||||
|
||||
# Example:
|
||||
# These haproxy config lines will add data to the logs that are captured
|
||||
# by the patterns below. Place them in your custom patterns directory to
|
||||
# override the defaults.
|
||||
#
|
||||
# capture request header Host len 40
|
||||
# capture request header X-Forwarded-For len 50
|
||||
# capture request header Accept-Language len 50
|
||||
# capture request header Referer len 200
|
||||
# capture request header User-Agent len 200
|
||||
#
|
||||
# capture response header Content-Type len 30
|
||||
# capture response header Content-Encoding len 10
|
||||
# capture response header Cache-Control len 200
|
||||
# capture response header Last-Modified len 200
|
||||
#
|
||||
# HAPROXYCAPTUREDREQUESTHEADERS %{DATA:request_header_host}\|%{DATA:request_header_x_forwarded_for}\|%{DATA:request_header_accept_language}\|%{DATA:request_header_referer}\|%{DATA:request_header_user_agent}
|
||||
# HAPROXYCAPTUREDRESPONSEHEADERS %{DATA:response_header_content_type}\|%{DATA:response_header_content_encoding}\|%{DATA:response_header_cache_control}\|%{DATA:response_header_last_modified}
|
||||
|
||||
# parse a haproxy 'httplog' line
|
||||
HAPROXYHTTPBASE %{IP:client_ip}:%{INT:client_port} \[%{HAPROXYDATE:accept_date}\] %{NOTSPACE:frontend_name} %{NOTSPACE:backend_name}/%{NOTSPACE:server_name} %{INT:time_request}/%{INT:time_queue}/%{INT:time_backend_connect}/%{INT:time_backend_response}/%{NOTSPACE:time_duration} %{INT:http_status_code} %{NOTSPACE:bytes_read} %{DATA:captured_request_cookie} %{DATA:captured_response_cookie} %{NOTSPACE:termination_state} %{INT:actconn}/%{INT:feconn}/%{INT:beconn}/%{INT:srvconn}/%{NOTSPACE:retries} %{INT:srv_queue}/%{INT:backend_queue} (\{%{HAPROXYCAPTUREDREQUESTHEADERS}\})?( )?(\{%{HAPROXYCAPTUREDRESPONSEHEADERS}\})?( )?"(<BADREQ>|(%{WORD:http_verb} (%{URIPROTO:http_proto}://)?(?:%{USER:http_user}(?::[^@]*)?@)?(?:%{URIHOST:http_host})?(?:%{URIPATHPARAM:http_request})?( HTTP/%{NUMBER:http_version})?))?"
|
||||
|
||||
HAPROXYHTTP (?:%{SYSLOGTIMESTAMP:syslog_timestamp}|%{TIMESTAMP_ISO8601:timestamp8601}) %{IPORHOST:syslog_server} %{SYSLOGPROG}: %{HAPROXYHTTPBASE}
|
||||
|
||||
# parse a haproxy 'tcplog' line
|
||||
HAPROXYTCP (?:%{SYSLOGTIMESTAMP:syslog_timestamp}|%{TIMESTAMP_ISO8601:timestamp8601}) %{IPORHOST:syslog_server} %{SYSLOGPROG}: %{IP:client_ip}:%{INT:client_port} \[%{HAPROXYDATE:accept_date}\] %{NOTSPACE:frontend_name} %{NOTSPACE:backend_name}/%{NOTSPACE:server_name} %{INT:time_queue}/%{INT:time_backend_connect}/%{NOTSPACE:time_duration} %{NOTSPACE:bytes_read} %{NOTSPACE:termination_state} %{INT:actconn}/%{INT:feconn}/%{INT:beconn}/%{INT:srvconn}/%{NOTSPACE:retries} %{INT:srv_queue}/%{INT:backend_queue}
|
||||
|
|
@ -0,0 +1,20 @@
|
|||
JAVACLASS (?:[a-zA-Z$_][a-zA-Z$_0-9]*\.)*[a-zA-Z$_][a-zA-Z$_0-9]*
|
||||
#Space is an allowed character to match special cases like 'Native Method' or 'Unknown Source'
|
||||
JAVAFILE (?:[A-Za-z0-9_. -]+)
|
||||
#Allow special <init> method
|
||||
JAVAMETHOD (?:(<init>)|[a-zA-Z$_][a-zA-Z$_0-9]*)
|
||||
#Line number is optional in special cases 'Native method' or 'Unknown source'
|
||||
JAVASTACKTRACEPART %{SPACE}at %{JAVACLASS:class}\.%{JAVAMETHOD:method}\(%{JAVAFILE:file}(?::%{NUMBER:line})?\)
|
||||
# Java Logs
|
||||
JAVATHREAD (?:[A-Z]{2}-Processor[\d]+)
|
||||
JAVACLASS (?:[a-zA-Z0-9-]+\.)+[A-Za-z0-9$]+
|
||||
JAVAFILE (?:[A-Za-z0-9_.-]+)
|
||||
JAVASTACKTRACEPART at %{JAVACLASS:class}\.%{WORD:method}\(%{JAVAFILE:file}:%{NUMBER:line}\)
|
||||
JAVALOGMESSAGE (.*)
|
||||
# MMM dd, yyyy HH:mm:ss eg: Jan 9, 2014 7:13:13 AM
|
||||
CATALINA_DATESTAMP %{MONTH} %{MONTHDAY}, 20%{YEAR} %{HOUR}:?%{MINUTE}(?::?%{SECOND}) (?:AM|PM)
|
||||
# yyyy-MM-dd HH:mm:ss,SSS ZZZ eg: 2014-01-09 17:32:25,527 -0800
|
||||
TOMCAT_DATESTAMP 20%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:?%{MINUTE}(?::?%{SECOND}) %{ISO8601_TIMEZONE}
|
||||
CATALINALOG %{CATALINA_DATESTAMP:timestamp} %{JAVACLASS:class} %{JAVALOGMESSAGE:logmessage}
|
||||
# 2014-01-09 20:03:28,269 -0800 | ERROR | com.example.service.ExampleService - something compeletely unexpected happened...
|
||||
TOMCATLOG %{TOMCAT_DATESTAMP:timestamp} \| %{LOGLEVEL:level} \| %{JAVACLASS:class} - %{JAVALOGMESSAGE:logmessage}
|
||||
|
|
@ -0,0 +1,9 @@
|
|||
# JUNOS 11.4 RT_FLOW patterns
|
||||
RT_FLOW_EVENT (RT_FLOW_SESSION_CREATE|RT_FLOW_SESSION_CLOSE|RT_FLOW_SESSION_DENY)
|
||||
|
||||
RT_FLOW1 %{RT_FLOW_EVENT:event}: %{GREEDYDATA:close-reason}: %{IP:src-ip}/%{INT:src-port}->%{IP:dst-ip}/%{INT:dst-port} %{DATA:service} %{IP:nat-src-ip}/%{INT:nat-src-port}->%{IP:nat-dst-ip}/%{INT:nat-dst-port} %{DATA:src-nat-rule-name} %{DATA:dst-nat-rule-name} %{INT:protocol-id} %{DATA:policy-name} %{DATA:from-zone} %{DATA:to-zone} %{INT:session-id} \d+\(%{DATA:sent}\) \d+\(%{DATA:received}\) %{INT:elapsed-time} .*
|
||||
|
||||
RT_FLOW2 %{RT_FLOW_EVENT:event}: session created %{IP:src-ip}/%{INT:src-port}->%{IP:dst-ip}/%{INT:dst-port} %{DATA:service} %{IP:nat-src-ip}/%{INT:nat-src-port}->%{IP:nat-dst-ip}/%{INT:nat-dst-port} %{DATA:src-nat-rule-name} %{DATA:dst-nat-rule-name} %{INT:protocol-id} %{DATA:policy-name} %{DATA:from-zone} %{DATA:to-zone} %{INT:session-id} .*
|
||||
|
||||
RT_FLOW3 %{RT_FLOW_EVENT:event}: session denied %{IP:src-ip}/%{INT:src-port}->%{IP:dst-ip}/%{INT:dst-port} %{DATA:service} %{INT:protocol-id}\(\d\) %{DATA:policy-name} %{DATA:from-zone} %{DATA:to-zone} .*
|
||||
|
||||
|
|
@ -0,0 +1,16 @@
|
|||
SYSLOG5424PRINTASCII [!-~]+
|
||||
|
||||
SYSLOGBASE2 (?:%{SYSLOGTIMESTAMP:timestamp}|%{TIMESTAMP_ISO8601:timestamp8601}) (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource}+(?: %{SYSLOGPROG}:|)
|
||||
SYSLOGPAMSESSION %{SYSLOGBASE} (?=%{GREEDYDATA:message})%{WORD:pam_module}\(%{DATA:pam_caller}\): session %{WORD:pam_session_state} for user %{USERNAME:username}(?: by %{GREEDYDATA:pam_by})?
|
||||
|
||||
CRON_ACTION [A-Z ]+
|
||||
CRONLOG %{SYSLOGBASE} \(%{USER:user}\) %{CRON_ACTION:action} \(%{DATA:message}\)
|
||||
|
||||
SYSLOGLINE %{SYSLOGBASE2} %{GREEDYDATA:message}
|
||||
|
||||
# IETF 5424 syslog(8) format (see http://www.rfc-editor.org/info/rfc5424)
|
||||
SYSLOG5424PRI <%{NONNEGINT:syslog5424_pri}>
|
||||
SYSLOG5424SD \[%{DATA}\]+
|
||||
SYSLOG5424BASE %{SYSLOG5424PRI}%{NONNEGINT:syslog5424_ver} +(?:%{TIMESTAMP_ISO8601:syslog5424_ts}|-) +(?:%{HOSTNAME:syslog5424_host}|-) +(-|%{SYSLOG5424PRINTASCII:syslog5424_app}) +(-|%{SYSLOG5424PRINTASCII:syslog5424_proc}) +(-|%{SYSLOG5424PRINTASCII:syslog5424_msgid}) +(?:%{SYSLOG5424SD:syslog5424_sd}|-|)
|
||||
|
||||
SYSLOG5424LINE %{SYSLOG5424BASE} +%{GREEDYDATA:syslog5424_msg}
|
||||
|
|
@ -0,0 +1,4 @@
|
|||
# Remember, these can be multi-line events.
|
||||
MCOLLECTIVE ., \[%{TIMESTAMP_ISO8601:timestamp} #%{POSINT:pid}\]%{SPACE}%{LOGLEVEL:event_level}
|
||||
|
||||
MCOLLECTIVEAUDIT %{TIMESTAMP_ISO8601:timestamp}:
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
MONGO_LOG %{SYSLOGTIMESTAMP:timestamp} \[%{WORD:component}\] %{GREEDYDATA:message}
|
||||
MONGO_QUERY \{ (?<={ ).*(?= } ntoreturn:) \}
|
||||
MONGO_SLOWQUERY %{WORD} %{MONGO_WORDDASH:database}\.%{MONGO_WORDDASH:collection} %{WORD}: %{MONGO_QUERY:query} %{WORD}:%{NONNEGINT:ntoreturn} %{WORD}:%{NONNEGINT:ntoskip} %{WORD}:%{NONNEGINT:nscanned}.*nreturned:%{NONNEGINT:nreturned}..+ (?<duration>[0-9]+)ms
|
||||
MONGO_WORDDASH \b[\w-]+\b
|
||||
MONGO3_SEVERITY \w
|
||||
MONGO3_COMPONENT %{WORD}|-
|
||||
MONGO3_LOG %{TIMESTAMP_ISO8601:timestamp} %{MONGO3_SEVERITY:severity} %{MONGO3_COMPONENT:component}%{SPACE}(?:\[%{DATA:context}\])? %{GREEDYDATA:message}
|
||||
|
|
@ -0,0 +1,124 @@
|
|||
##################################################################################
|
||||
##################################################################################
|
||||
# Chop Nagios log files to smithereens!
|
||||
#
|
||||
# A set of GROK filters to process logfiles generated by Nagios.
|
||||
# While it does not, this set intends to cover all possible Nagios logs.
|
||||
#
|
||||
# Some more work needs to be done to cover all External Commands:
|
||||
# http://old.nagios.org/developerinfo/externalcommands/commandlist.php
|
||||
#
|
||||
# If you need some support on these rules please contact:
|
||||
# Jelle Smet http://smetj.net
|
||||
#
|
||||
#################################################################################
|
||||
#################################################################################
|
||||
|
||||
NAGIOSTIME \[%{NUMBER:nagios_epoch}\]
|
||||
|
||||
###############################################
|
||||
######## Begin nagios log types
|
||||
###############################################
|
||||
NAGIOS_TYPE_CURRENT_SERVICE_STATE CURRENT SERVICE STATE
|
||||
NAGIOS_TYPE_CURRENT_HOST_STATE CURRENT HOST STATE
|
||||
|
||||
NAGIOS_TYPE_SERVICE_NOTIFICATION SERVICE NOTIFICATION
|
||||
NAGIOS_TYPE_HOST_NOTIFICATION HOST NOTIFICATION
|
||||
|
||||
NAGIOS_TYPE_SERVICE_ALERT SERVICE ALERT
|
||||
NAGIOS_TYPE_HOST_ALERT HOST ALERT
|
||||
|
||||
NAGIOS_TYPE_SERVICE_FLAPPING_ALERT SERVICE FLAPPING ALERT
|
||||
NAGIOS_TYPE_HOST_FLAPPING_ALERT HOST FLAPPING ALERT
|
||||
|
||||
NAGIOS_TYPE_SERVICE_DOWNTIME_ALERT SERVICE DOWNTIME ALERT
|
||||
NAGIOS_TYPE_HOST_DOWNTIME_ALERT HOST DOWNTIME ALERT
|
||||
|
||||
NAGIOS_TYPE_PASSIVE_SERVICE_CHECK PASSIVE SERVICE CHECK
|
||||
NAGIOS_TYPE_PASSIVE_HOST_CHECK PASSIVE HOST CHECK
|
||||
|
||||
NAGIOS_TYPE_SERVICE_EVENT_HANDLER SERVICE EVENT HANDLER
|
||||
NAGIOS_TYPE_HOST_EVENT_HANDLER HOST EVENT HANDLER
|
||||
|
||||
NAGIOS_TYPE_EXTERNAL_COMMAND EXTERNAL COMMAND
|
||||
NAGIOS_TYPE_TIMEPERIOD_TRANSITION TIMEPERIOD TRANSITION
|
||||
###############################################
|
||||
######## End nagios log types
|
||||
###############################################
|
||||
|
||||
###############################################
|
||||
######## Begin external check types
|
||||
###############################################
|
||||
NAGIOS_EC_DISABLE_SVC_CHECK DISABLE_SVC_CHECK
|
||||
NAGIOS_EC_ENABLE_SVC_CHECK ENABLE_SVC_CHECK
|
||||
NAGIOS_EC_DISABLE_HOST_CHECK DISABLE_HOST_CHECK
|
||||
NAGIOS_EC_ENABLE_HOST_CHECK ENABLE_HOST_CHECK
|
||||
NAGIOS_EC_PROCESS_SERVICE_CHECK_RESULT PROCESS_SERVICE_CHECK_RESULT
|
||||
NAGIOS_EC_PROCESS_HOST_CHECK_RESULT PROCESS_HOST_CHECK_RESULT
|
||||
NAGIOS_EC_SCHEDULE_SERVICE_DOWNTIME SCHEDULE_SERVICE_DOWNTIME
|
||||
NAGIOS_EC_SCHEDULE_HOST_DOWNTIME SCHEDULE_HOST_DOWNTIME
|
||||
NAGIOS_EC_DISABLE_HOST_SVC_NOTIFICATIONS DISABLE_HOST_SVC_NOTIFICATIONS
|
||||
NAGIOS_EC_ENABLE_HOST_SVC_NOTIFICATIONS ENABLE_HOST_SVC_NOTIFICATIONS
|
||||
NAGIOS_EC_DISABLE_HOST_NOTIFICATIONS DISABLE_HOST_NOTIFICATIONS
|
||||
NAGIOS_EC_ENABLE_HOST_NOTIFICATIONS ENABLE_HOST_NOTIFICATIONS
|
||||
NAGIOS_EC_DISABLE_SVC_NOTIFICATIONS DISABLE_SVC_NOTIFICATIONS
|
||||
NAGIOS_EC_ENABLE_SVC_NOTIFICATIONS ENABLE_SVC_NOTIFICATIONS
|
||||
###############################################
|
||||
######## End external check types
|
||||
###############################################
|
||||
NAGIOS_WARNING Warning:%{SPACE}%{GREEDYDATA:nagios_message}
|
||||
|
||||
NAGIOS_CURRENT_SERVICE_STATE %{NAGIOS_TYPE_CURRENT_SERVICE_STATE:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_service};%{DATA:nagios_state};%{DATA:nagios_statetype};%{DATA:nagios_statecode};%{GREEDYDATA:nagios_message}
|
||||
NAGIOS_CURRENT_HOST_STATE %{NAGIOS_TYPE_CURRENT_HOST_STATE:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_state};%{DATA:nagios_statetype};%{DATA:nagios_statecode};%{GREEDYDATA:nagios_message}
|
||||
|
||||
NAGIOS_SERVICE_NOTIFICATION %{NAGIOS_TYPE_SERVICE_NOTIFICATION:nagios_type}: %{DATA:nagios_notifyname};%{DATA:nagios_hostname};%{DATA:nagios_service};%{DATA:nagios_state};%{DATA:nagios_contact};%{GREEDYDATA:nagios_message}
|
||||
NAGIOS_HOST_NOTIFICATION %{NAGIOS_TYPE_HOST_NOTIFICATION:nagios_type}: %{DATA:nagios_notifyname};%{DATA:nagios_hostname};%{DATA:nagios_state};%{DATA:nagios_contact};%{GREEDYDATA:nagios_message}
|
||||
|
||||
NAGIOS_SERVICE_ALERT %{NAGIOS_TYPE_SERVICE_ALERT:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_service};%{DATA:nagios_state};%{DATA:nagios_statelevel};%{NUMBER:nagios_attempt};%{GREEDYDATA:nagios_message}
|
||||
NAGIOS_HOST_ALERT %{NAGIOS_TYPE_HOST_ALERT:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_state};%{DATA:nagios_statelevel};%{NUMBER:nagios_attempt};%{GREEDYDATA:nagios_message}
|
||||
|
||||
NAGIOS_SERVICE_FLAPPING_ALERT %{NAGIOS_TYPE_SERVICE_FLAPPING_ALERT:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_service};%{DATA:nagios_state};%{GREEDYDATA:nagios_message}
|
||||
NAGIOS_HOST_FLAPPING_ALERT %{NAGIOS_TYPE_HOST_FLAPPING_ALERT:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_state};%{GREEDYDATA:nagios_message}
|
||||
|
||||
NAGIOS_SERVICE_DOWNTIME_ALERT %{NAGIOS_TYPE_SERVICE_DOWNTIME_ALERT:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_service};%{DATA:nagios_state};%{GREEDYDATA:nagios_comment}
|
||||
NAGIOS_HOST_DOWNTIME_ALERT %{NAGIOS_TYPE_HOST_DOWNTIME_ALERT:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_state};%{GREEDYDATA:nagios_comment}
|
||||
|
||||
NAGIOS_PASSIVE_SERVICE_CHECK %{NAGIOS_TYPE_PASSIVE_SERVICE_CHECK:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_service};%{DATA:nagios_state};%{GREEDYDATA:nagios_comment}
|
||||
NAGIOS_PASSIVE_HOST_CHECK %{NAGIOS_TYPE_PASSIVE_HOST_CHECK:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_state};%{GREEDYDATA:nagios_comment}
|
||||
|
||||
NAGIOS_SERVICE_EVENT_HANDLER %{NAGIOS_TYPE_SERVICE_EVENT_HANDLER:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_service};%{DATA:nagios_state};%{DATA:nagios_statelevel};%{DATA:nagios_event_handler_name}
|
||||
NAGIOS_HOST_EVENT_HANDLER %{NAGIOS_TYPE_HOST_EVENT_HANDLER:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_state};%{DATA:nagios_statelevel};%{DATA:nagios_event_handler_name}
|
||||
|
||||
NAGIOS_TIMEPERIOD_TRANSITION %{NAGIOS_TYPE_TIMEPERIOD_TRANSITION:nagios_type}: %{DATA:nagios_service};%{DATA:nagios_unknown1};%{DATA:nagios_unknown2}
|
||||
|
||||
####################
|
||||
#### External checks
|
||||
####################
|
||||
|
||||
#Disable host & service check
|
||||
NAGIOS_EC_LINE_DISABLE_SVC_CHECK %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_DISABLE_SVC_CHECK:nagios_command};%{DATA:nagios_hostname};%{DATA:nagios_service}
|
||||
NAGIOS_EC_LINE_DISABLE_HOST_CHECK %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_DISABLE_HOST_CHECK:nagios_command};%{DATA:nagios_hostname}
|
||||
|
||||
#Enable host & service check
|
||||
NAGIOS_EC_LINE_ENABLE_SVC_CHECK %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_ENABLE_SVC_CHECK:nagios_command};%{DATA:nagios_hostname};%{DATA:nagios_service}
|
||||
NAGIOS_EC_LINE_ENABLE_HOST_CHECK %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_ENABLE_HOST_CHECK:nagios_command};%{DATA:nagios_hostname}
|
||||
|
||||
#Process host & service check
|
||||
NAGIOS_EC_LINE_PROCESS_SERVICE_CHECK_RESULT %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_PROCESS_SERVICE_CHECK_RESULT:nagios_command};%{DATA:nagios_hostname};%{DATA:nagios_service};%{DATA:nagios_state};%{GREEDYDATA:nagios_check_result}
|
||||
NAGIOS_EC_LINE_PROCESS_HOST_CHECK_RESULT %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_PROCESS_HOST_CHECK_RESULT:nagios_command};%{DATA:nagios_hostname};%{DATA:nagios_state};%{GREEDYDATA:nagios_check_result}
|
||||
|
||||
#Disable host & service notifications
|
||||
NAGIOS_EC_LINE_DISABLE_HOST_SVC_NOTIFICATIONS %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_DISABLE_HOST_SVC_NOTIFICATIONS:nagios_command};%{GREEDYDATA:nagios_hostname}
|
||||
NAGIOS_EC_LINE_DISABLE_HOST_NOTIFICATIONS %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_DISABLE_HOST_NOTIFICATIONS:nagios_command};%{GREEDYDATA:nagios_hostname}
|
||||
NAGIOS_EC_LINE_DISABLE_SVC_NOTIFICATIONS %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_DISABLE_SVC_NOTIFICATIONS:nagios_command};%{DATA:nagios_hostname};%{GREEDYDATA:nagios_service}
|
||||
|
||||
#Enable host & service notifications
|
||||
NAGIOS_EC_LINE_ENABLE_HOST_SVC_NOTIFICATIONS %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_ENABLE_HOST_SVC_NOTIFICATIONS:nagios_command};%{GREEDYDATA:nagios_hostname}
|
||||
NAGIOS_EC_LINE_ENABLE_HOST_NOTIFICATIONS %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_ENABLE_HOST_NOTIFICATIONS:nagios_command};%{GREEDYDATA:nagios_hostname}
|
||||
NAGIOS_EC_LINE_ENABLE_SVC_NOTIFICATIONS %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_ENABLE_SVC_NOTIFICATIONS:nagios_command};%{DATA:nagios_hostname};%{GREEDYDATA:nagios_service}
|
||||
|
||||
#Schedule host & service downtime
|
||||
NAGIOS_EC_LINE_SCHEDULE_HOST_DOWNTIME %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_SCHEDULE_HOST_DOWNTIME:nagios_command};%{DATA:nagios_hostname};%{NUMBER:nagios_start_time};%{NUMBER:nagios_end_time};%{NUMBER:nagios_fixed};%{NUMBER:nagios_trigger_id};%{NUMBER:nagios_duration};%{DATA:author};%{DATA:comment}
|
||||
|
||||
#End matching line
|
||||
NAGIOSLOGLINE %{NAGIOSTIME} (?:%{NAGIOS_WARNING}|%{NAGIOS_CURRENT_SERVICE_STATE}|%{NAGIOS_CURRENT_HOST_STATE}|%{NAGIOS_SERVICE_NOTIFICATION}|%{NAGIOS_HOST_NOTIFICATION}|%{NAGIOS_SERVICE_ALERT}|%{NAGIOS_HOST_ALERT}|%{NAGIOS_SERVICE_FLAPPING_ALERT}|%{NAGIOS_HOST_FLAPPING_ALERT}|%{NAGIOS_SERVICE_DOWNTIME_ALERT}|%{NAGIOS_HOST_DOWNTIME_ALERT}|%{NAGIOS_PASSIVE_SERVICE_CHECK}|%{NAGIOS_PASSIVE_HOST_CHECK}|%{NAGIOS_SERVICE_EVENT_HANDLER}|%{NAGIOS_HOST_EVENT_HANDLER}|%{NAGIOS_TIMEPERIOD_TRANSITION}|%{NAGIOS_EC_LINE_DISABLE_SVC_CHECK}|%{NAGIOS_EC_LINE_ENABLE_SVC_CHECK}|%{NAGIOS_EC_LINE_DISABLE_HOST_CHECK}|%{NAGIOS_EC_LINE_ENABLE_HOST_CHECK}|%{NAGIOS_EC_LINE_PROCESS_HOST_CHECK_RESULT}|%{NAGIOS_EC_LINE_PROCESS_SERVICE_CHECK_RESULT}|%{NAGIOS_EC_LINE_SCHEDULE_HOST_DOWNTIME}|%{NAGIOS_EC_LINE_DISABLE_HOST_SVC_NOTIFICATIONS}|%{NAGIOS_EC_LINE_ENABLE_HOST_SVC_NOTIFICATIONS}|%{NAGIOS_EC_LINE_DISABLE_HOST_NOTIFICATIONS}|%{NAGIOS_EC_LINE_ENABLE_HOST_NOTIFICATIONS}|%{NAGIOS_EC_LINE_DISABLE_SVC_NOTIFICATIONS}|%{NAGIOS_EC_LINE_ENABLE_SVC_NOTIFICATIONS})
|
||||
|
|
@ -0,0 +1,3 @@
|
|||
# Default postgresql pg_log format pattern
|
||||
POSTGRESQL %{DATESTAMP:timestamp} %{TZ} %{DATA:user_id} %{GREEDYDATA:connection_id} %{POSINT:pid}
|
||||
|
||||
|
|
@ -0,0 +1,13 @@
|
|||
RUUID \h{32}
|
||||
# rails controller with action
|
||||
RCONTROLLER (?<controller>[^#]+)#(?<action>\w+)
|
||||
|
||||
# this will often be the only line:
|
||||
RAILS3HEAD (?m)Started %{WORD:verb} "%{URIPATHPARAM:request}" for %{IPORHOST:clientip} at (?<timestamp>%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND} %{ISO8601_TIMEZONE})
|
||||
# for some a strange reason, params are stripped of {} - not sure that's a good idea.
|
||||
RPROCESSING \W*Processing by %{RCONTROLLER} as (?<format>\S+)(?:\W*Parameters: {%{DATA:params}}\W*)?
|
||||
RAILS3FOOT Completed %{NUMBER:response}%{DATA} in %{NUMBER:totalms}ms %{RAILS3PROFILE}%{GREEDYDATA}
|
||||
RAILS3PROFILE (?:\(Views: %{NUMBER:viewms}ms \| ActiveRecord: %{NUMBER:activerecordms}ms|\(ActiveRecord: %{NUMBER:activerecordms}ms)?
|
||||
|
||||
# putting it all together
|
||||
RAILS3 %{RAILS3HEAD}(?:%{RPROCESSING})?(?<context>(?:%{DATA}\n)*)(?:%{RAILS3FOOT})?
|
||||
|
|
@ -0,0 +1,3 @@
|
|||
REDISTIMESTAMP %{MONTHDAY} %{MONTH} %{TIME}
|
||||
REDISLOG \[%{POSINT:pid}\] %{REDISTIMESTAMP:timestamp} \*
|
||||
|
||||
|
|
@ -0,0 +1,2 @@
|
|||
RUBY_LOGLEVEL (?:DEBUG|FATAL|ERROR|WARN|INFO)
|
||||
RUBY_LOGGER [DFEWI], \[%{TIMESTAMP_ISO8601:timestamp} #%{POSINT:pid}\] *%{RUBY_LOGLEVEL:loglevel} -- +%{DATA:progname}: %{GREEDYDATA:message}
|
||||
|
|
@ -19,6 +19,8 @@
|
|||
|
||||
package org.elasticsearch.ingest;
|
||||
|
||||
import org.elasticsearch.action.admin.indices.mapping.put.PutMappingResponse;
|
||||
import org.elasticsearch.common.xcontent.XContentBuilder;
|
||||
import org.elasticsearch.plugin.ingest.IngestPlugin;
|
||||
import org.elasticsearch.plugin.ingest.transport.delete.DeletePipelineAction;
|
||||
import org.elasticsearch.plugin.ingest.transport.delete.DeletePipelineRequestBuilder;
|
||||
|
|
@ -30,12 +32,15 @@ import org.elasticsearch.plugin.ingest.transport.put.PutPipelineAction;
|
|||
import org.elasticsearch.plugin.ingest.transport.put.PutPipelineRequestBuilder;
|
||||
import org.elasticsearch.plugins.Plugin;
|
||||
import org.elasticsearch.test.ESIntegTestCase;
|
||||
|
||||
import java.util.Collection;
|
||||
import java.util.Map;
|
||||
|
||||
import static org.elasticsearch.common.xcontent.XContentFactory.jsonBuilder;
|
||||
import static org.hamcrest.Matchers.*;
|
||||
import static org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertAcked;
|
||||
import static org.hamcrest.Matchers.equalTo;
|
||||
import static org.hamcrest.Matchers.nullValue;
|
||||
import static org.hamcrest.core.Is.is;
|
||||
import static org.hamcrest.core.IsNull.notNullValue;
|
||||
|
||||
public class IngestClientIT extends ESIntegTestCase {
|
||||
|
||||
|
|
@ -47,22 +52,21 @@ public class IngestClientIT extends ESIntegTestCase {
|
|||
@Override
|
||||
protected Collection<Class<? extends Plugin>> transportClientPlugins() {
|
||||
return nodePlugins();
|
||||
|
||||
}
|
||||
|
||||
public void testBasics() throws Exception {
|
||||
public void test() throws Exception {
|
||||
new PutPipelineRequestBuilder(client(), PutPipelineAction.INSTANCE)
|
||||
.setId("_id")
|
||||
.setSource(jsonBuilder().startObject()
|
||||
.field("description", "my_pipeline")
|
||||
.startArray("processors")
|
||||
.startObject()
|
||||
.startObject("simple")
|
||||
.field("path", "field2")
|
||||
.field("expected_value", "abc")
|
||||
.field("add_field", "field3")
|
||||
.field("add_field_value", "xyz")
|
||||
.endObject()
|
||||
.endObject()
|
||||
.startObject()
|
||||
.startObject("grok")
|
||||
.field("field", "field1")
|
||||
.field("pattern", "%{NUMBER:val:float} %{NUMBER:status:int} <%{WORD:msg}>")
|
||||
.endObject()
|
||||
.endObject()
|
||||
.endArray()
|
||||
.endObject().bytes())
|
||||
.get();
|
||||
|
|
@ -78,7 +82,15 @@ public class IngestClientIT extends ESIntegTestCase {
|
|||
});
|
||||
|
||||
createIndex("test");
|
||||
client().prepareIndex("test", "type", "1").setSource("field2", "abc")
|
||||
XContentBuilder updateMappingBuilder = jsonBuilder().startObject().startObject("properties")
|
||||
.startObject("status").field("type", "integer").endObject()
|
||||
.startObject("val").field("type", "float").endObject()
|
||||
.endObject();
|
||||
PutMappingResponse putMappingResponse = client().admin().indices()
|
||||
.preparePutMapping("test").setType("type").setSource(updateMappingBuilder).get();
|
||||
assertAcked(putMappingResponse);
|
||||
|
||||
client().prepareIndex("test", "type", "1").setSource("field1", "123.42 400 <foo>")
|
||||
.putHeader("ingest", "_id")
|
||||
.get();
|
||||
|
||||
|
|
@ -87,21 +99,25 @@ public class IngestClientIT extends ESIntegTestCase {
|
|||
public void run() {
|
||||
Map<String, Object> doc = client().prepareGet("test", "type", "1")
|
||||
.get().getSourceAsMap();
|
||||
assertThat(doc.get("field3"), equalTo("xyz"));
|
||||
assertThat(doc.get("val"), equalTo(123.42));
|
||||
assertThat(doc.get("status"), equalTo(400));
|
||||
assertThat(doc.get("msg"), equalTo("foo"));
|
||||
}
|
||||
});
|
||||
|
||||
client().prepareBulk().add(
|
||||
client().prepareIndex("test", "type", "2").setSource("field2", "abc")
|
||||
client().prepareIndex("test", "type", "2").setSource("field1", "123.42 400 <foo>")
|
||||
).putHeader("ingest", "_id").get();
|
||||
assertBusy(new Runnable() {
|
||||
@Override
|
||||
public void run() {
|
||||
Map<String, Object> doc = client().prepareGet("test", "type", "2").get().getSourceAsMap();
|
||||
assertThat(doc.get("field3"), equalTo("xyz"));
|
||||
assertThat(doc.get("val"), equalTo(123.42));
|
||||
assertThat(doc.get("status"), equalTo(400));
|
||||
assertThat(doc.get("msg"), equalTo("foo"));
|
||||
}
|
||||
});
|
||||
|
||||
|
||||
DeletePipelineResponse response = new DeletePipelineRequestBuilder(client(), DeletePipelineAction.INSTANCE)
|
||||
.setId("_id")
|
||||
.get();
|
||||
|
|
|
|||
|
|
@ -0,0 +1,284 @@
|
|||
/*
|
||||
* Licensed to Elasticsearch under one or more contributor
|
||||
* license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright
|
||||
* ownership. Elasticsearch licenses this file to you under
|
||||
* the Apache License, Version 2.0 (the "License"); you may
|
||||
* not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing,
|
||||
* software distributed under the License is distributed on an
|
||||
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
* KIND, either express or implied. See the License for the
|
||||
* specific language governing permissions and limitations
|
||||
* under the License.
|
||||
*/
|
||||
|
||||
package org.elasticsearch.ingest.processor.grok;
|
||||
|
||||
import org.elasticsearch.test.ESTestCase;
|
||||
import org.junit.Before;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.lang.Object;
|
||||
import java.lang.String;
|
||||
import java.util.*;
|
||||
|
||||
import static org.hamcrest.Matchers.equalTo;
|
||||
import static org.hamcrest.Matchers.is;
|
||||
import static org.hamcrest.Matchers.nullValue;
|
||||
|
||||
|
||||
public class GrokTests extends ESTestCase {
|
||||
private Map<String, String> basePatterns;
|
||||
|
||||
private Map<String, String> newBankFromStreams(InputStream... inputStreams) throws IOException {
|
||||
Map<String, String> patternBank = new HashMap<>();
|
||||
|
||||
for (InputStream is : inputStreams) {
|
||||
PatternUtils.loadBankFromStream(patternBank, is);
|
||||
}
|
||||
|
||||
return patternBank;
|
||||
}
|
||||
|
||||
@Before
|
||||
public void setup() throws IOException {
|
||||
basePatterns = newBankFromStreams(
|
||||
getClass().getResourceAsStream("/grok/patterns/grok-patterns"),
|
||||
getClass().getResourceAsStream("/grok/patterns/linux-syslog")
|
||||
);
|
||||
}
|
||||
|
||||
public void testSimpleSyslogLine() {
|
||||
String line = "Mar 16 00:01:25 evita postfix/smtpd[1713]: connect from camomile.cloud9.net[168.100.1.3]";
|
||||
Grok grok = new Grok(basePatterns, "%{SYSLOGLINE}");
|
||||
Map<String, Object> matches = grok.captures(line);
|
||||
assertEquals("evita", matches.get("logsource"));
|
||||
assertEquals("Mar 16 00:01:25", matches.get("timestamp"));
|
||||
assertEquals("connect from camomile.cloud9.net[168.100.1.3]", matches.get("message"));
|
||||
assertEquals("postfix/smtpd", matches.get("program"));
|
||||
assertEquals("1713", matches.get("pid"));
|
||||
}
|
||||
|
||||
public void testSyslog5424Line() {
|
||||
String line = "<191>1 2009-06-30T18:30:00+02:00 paxton.local grokdebug 4123 - [id1 foo=\\\"bar\\\"][id2 baz=\\\"something\\\"] Hello, syslog.";
|
||||
Grok grok = new Grok(basePatterns, "%{SYSLOG5424LINE}");
|
||||
Map<String, Object> matches = grok.captures(line);
|
||||
assertEquals("191", matches.get("syslog5424_pri"));
|
||||
assertEquals("1", matches.get("syslog5424_ver"));
|
||||
assertEquals("2009-06-30T18:30:00+02:00", matches.get("syslog5424_ts"));
|
||||
assertEquals("paxton.local", matches.get("syslog5424_host"));
|
||||
assertEquals("grokdebug", matches.get("syslog5424_app"));
|
||||
assertEquals("4123", matches.get("syslog5424_proc"));
|
||||
assertEquals(null, matches.get("syslog5424_msgid"));
|
||||
assertEquals("[id1 foo=\\\"bar\\\"][id2 baz=\\\"something\\\"]", matches.get("syslog5424_sd"));
|
||||
assertEquals("Hello, syslog.", matches.get("syslog5424_msg"));
|
||||
}
|
||||
|
||||
public void testDatePattern() {
|
||||
String line = "fancy 12-12-12 12:12:12";
|
||||
Grok grok = new Grok(basePatterns, "(?<timestamp>%{DATE_EU} %{TIME})");
|
||||
Map<String, Object> matches = grok.captures(line);
|
||||
assertEquals("12-12-12 12:12:12", matches.get("timestamp"));
|
||||
}
|
||||
|
||||
public void testNilCoercedValues() {
|
||||
Grok grok = new Grok(basePatterns, "test (N/A|%{BASE10NUM:duration:float}ms)");
|
||||
Map<String, Object> matches = grok.captures("test 28.4ms");
|
||||
assertEquals(28.4f, matches.get("duration"));
|
||||
matches = grok.captures("test N/A");
|
||||
assertEquals(null, matches.get("duration"));
|
||||
}
|
||||
|
||||
public void testNilWithNoCoercion() {
|
||||
Grok grok = new Grok(basePatterns, "test (N/A|%{BASE10NUM:duration}ms)");
|
||||
Map<String, Object> matches = grok.captures("test 28.4ms");
|
||||
assertEquals("28.4", matches.get("duration"));
|
||||
matches = grok.captures("test N/A");
|
||||
assertEquals(null, matches.get("duration"));
|
||||
}
|
||||
|
||||
public void testUnicodeSyslog() {
|
||||
Grok grok = new Grok(basePatterns, "<%{POSINT:syslog_pri}>%{SPACE}%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{PROG:syslog_program}(:?)(?:\\[%{GREEDYDATA:syslog_pid}\\])?(:?) %{GREEDYDATA:syslog_message}");
|
||||
Map<String, Object> matches = grok.captures("<22>Jan 4 07:50:46 mailmaster postfix/policy-spf[9454]: : SPF permerror (Junk encountered in record 'v=spf1 mx a:mail.domain.no ip4:192.168.0.4 <20>all'): Envelope-from: email@domain.no");
|
||||
assertThat(matches.get("syslog_pri"), equalTo("22"));
|
||||
assertThat(matches.get("syslog_program"), equalTo("postfix/policy-spf"));
|
||||
assertThat(matches.get("tags"), nullValue());
|
||||
}
|
||||
|
||||
public void testNamedFieldsWithWholeTextMatch() {
|
||||
Grok grok = new Grok(basePatterns, "%{DATE_EU:stimestamp}");
|
||||
Map<String, Object> matches = grok.captures("11/01/01");
|
||||
assertThat(matches.get("stimestamp"), equalTo("11/01/01"));
|
||||
}
|
||||
|
||||
public void testWithOniguramaNamedCaptures() {
|
||||
Grok grok = new Grok(basePatterns, "(?<foo>\\w+)");
|
||||
Map<String, Object> matches = grok.captures("hello world");
|
||||
assertThat(matches.get("foo"), equalTo("hello"));
|
||||
}
|
||||
|
||||
public void testISO8601() {
|
||||
Grok grok = new Grok(basePatterns, "^%{TIMESTAMP_ISO8601}$");
|
||||
List<String> timeMessages = Arrays.asList(
|
||||
"2001-01-01T00:00:00",
|
||||
"1974-03-02T04:09:09",
|
||||
"2010-05-03T08:18:18+00:00",
|
||||
"2004-07-04T12:27:27-00:00",
|
||||
"2001-09-05T16:36:36+0000",
|
||||
"2001-11-06T20:45:45-0000",
|
||||
"2001-12-07T23:54:54Z",
|
||||
"2001-01-01T00:00:00.123456",
|
||||
"1974-03-02T04:09:09.123456",
|
||||
"2010-05-03T08:18:18.123456+00:00",
|
||||
"2004-07-04T12:27:27.123456-00:00",
|
||||
"2001-09-05T16:36:36.123456+0000",
|
||||
"2001-11-06T20:45:45.123456-0000",
|
||||
"2001-12-07T23:54:54.123456Z",
|
||||
"2001-12-07T23:54:60.123456Z" // '60' second is a leap second.
|
||||
);
|
||||
for (String msg : timeMessages) {
|
||||
assertThat(grok.match(msg), is(true));
|
||||
}
|
||||
}
|
||||
|
||||
public void testNotISO8601() {
|
||||
Grok grok = new Grok(basePatterns, "^%{TIMESTAMP_ISO8601}$");
|
||||
List<String> timeMessages = Arrays.asList(
|
||||
"2001-13-01T00:00:00", // invalid month
|
||||
"2001-00-01T00:00:00", // invalid month
|
||||
"2001-01-00T00:00:00", // invalid day
|
||||
"2001-01-32T00:00:00", // invalid day
|
||||
"2001-01-aT00:00:00", // invalid day
|
||||
"2001-01-1aT00:00:00", // invalid day
|
||||
"2001-01-01Ta0:00:00", // invalid hour
|
||||
"2001-01-01T0:00:00", // invalid hour
|
||||
"2001-01-01T25:00:00", // invalid hour
|
||||
"2001-01-01T01:60:00", // invalid minute
|
||||
"2001-01-01T00:aa:00", // invalid minute
|
||||
"2001-01-01T00:00:aa", // invalid second
|
||||
"2001-01-01T00:00:-1", // invalid second
|
||||
"2001-01-01T00:00:61", // invalid second
|
||||
"2001-01-01T00:00:00A", // invalid timezone
|
||||
"2001-01-01T00:00:00+", // invalid timezone
|
||||
"2001-01-01T00:00:00+25", // invalid timezone
|
||||
"2001-01-01T00:00:00+2500", // invalid timezone
|
||||
"2001-01-01T00:00:00+25:00", // invalid timezone
|
||||
"2001-01-01T00:00:00-25", // invalid timezone
|
||||
"2001-01-01T00:00:00-2500", // invalid timezone
|
||||
"2001-01-01T00:00:00-00:61" // invalid timezone
|
||||
);
|
||||
for (String msg : timeMessages) {
|
||||
assertThat(grok.match(msg), is(false));
|
||||
}
|
||||
}
|
||||
|
||||
public void testNoNamedCaptures() {
|
||||
Map<String, String> bank = new HashMap<>();
|
||||
|
||||
bank.put("NAME", "Tal");
|
||||
bank.put("EXCITED_NAME", "!!!%{NAME:name}!!!");
|
||||
bank.put("TEST", "hello world");
|
||||
|
||||
String text = "wowza !!!Tal!!! - Tal";
|
||||
String pattern = "%{EXCITED_NAME} - %{NAME}";
|
||||
Grok g = new Grok(bank, pattern, false);
|
||||
|
||||
assertEquals("(?<EXCITED_NAME_0>!!!(?<NAME_21>Tal)!!!) - (?<NAME_22>Tal)", g.toRegex(pattern));
|
||||
assertEquals(true, g.match(text));
|
||||
|
||||
Object actual = g.captures(text);
|
||||
Map<String, Object> expected = new HashMap<>();
|
||||
expected.put("EXCITED_NAME_0", "!!!Tal!!!");
|
||||
expected.put("NAME_21", "Tal");
|
||||
expected.put("NAME_22", "Tal");
|
||||
assertEquals(expected, actual);
|
||||
}
|
||||
|
||||
public void testNumericCapturesCoercion() {
|
||||
Map<String, String> bank = new HashMap<>();
|
||||
bank.put("BASE10NUM", "(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+)))");
|
||||
bank.put("NUMBER", "(?:%{BASE10NUM})");
|
||||
|
||||
String pattern = "%{NUMBER:bytes:float} %{NUMBER:status} %{NUMBER}";
|
||||
Grok g = new Grok(bank, pattern);
|
||||
|
||||
String text = "12009.34 200 9032";
|
||||
Map<String, Object> expected = new HashMap<>();
|
||||
expected.put("bytes", 12009.34f);
|
||||
expected.put("status", "200");
|
||||
Map<String, Object> actual = g.captures(text);
|
||||
|
||||
assertEquals(expected, actual);
|
||||
}
|
||||
|
||||
public void testApacheLog() {
|
||||
String logLine = "31.184.238.164 - - [24/Jul/2014:05:35:37 +0530] \"GET /logs/access.log HTTP/1.0\" 200 69849 \"http://8rursodiol.enjin.com\" \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.12785 YaBrowser/13.12.1599.12785 Safari/537.36\" \"www.dlwindianrailways.com\"";
|
||||
Grok grok = new Grok(basePatterns, "%{COMBINEDAPACHELOG}");
|
||||
Map<String, Object> matches = grok.captures(logLine);
|
||||
|
||||
assertEquals("31.184.238.164", matches.get("clientip"));
|
||||
assertEquals("-", matches.get("ident"));
|
||||
assertEquals("-", matches.get("auth"));
|
||||
assertEquals("24/Jul/2014:05:35:37 +0530", matches.get("timestamp"));
|
||||
assertEquals("GET", matches.get("verb"));
|
||||
assertEquals("/logs/access.log", matches.get("request"));
|
||||
assertEquals("1.0", matches.get("httpversion"));
|
||||
assertEquals("200", matches.get("response"));
|
||||
assertEquals("69849", matches.get("bytes"));
|
||||
assertEquals("\"http://8rursodiol.enjin.com\"", matches.get("referrer"));
|
||||
assertEquals(null, matches.get("port"));
|
||||
assertEquals("\"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.12785 YaBrowser/13.12.1599.12785 Safari/537.36\"", matches.get("agent"));
|
||||
}
|
||||
|
||||
public void testComplete() {
|
||||
Map<String, String> bank = new HashMap<>();
|
||||
bank.put("MONTHDAY", "(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])");
|
||||
bank.put("MONTH", "\\b(?:Jan(?:uary|uar)?|Feb(?:ruary|ruar)?|M(?:a|ä)?r(?:ch|z)?|Apr(?:il)?|Ma(?:y|i)?|Jun(?:e|i)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|O(?:c|k)?t(?:ober)?|Nov(?:ember)?|De(?:c|z)(?:ember)?)\\b");
|
||||
bank.put("MINUTE", "(?:[0-5][0-9])");
|
||||
bank.put("YEAR", "(?>\\d\\d){1,2}");
|
||||
bank.put("HOUR", "(?:2[0123]|[01]?[0-9])");
|
||||
bank.put("SECOND", "(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)");
|
||||
bank.put("TIME", "(?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9])");
|
||||
bank.put("INT", "(?:[+-]?(?:[0-9]+))");
|
||||
bank.put("HTTPDATE", "%{MONTHDAY}/%{MONTH}/%{YEAR}:%{TIME} %{INT}");
|
||||
bank.put("WORD", "\\b\\w+\\b");
|
||||
bank.put("BASE10NUM", "(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+)))");
|
||||
bank.put("NUMBER", "(?:%{BASE10NUM})");
|
||||
bank.put("IPV6", "((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:)))(%.+)?");
|
||||
bank.put("IPV4", "(?<![0-9])(?:(?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5]))(?![0-9])");
|
||||
bank.put("IP", "(?:%{IPV6}|%{IPV4})");
|
||||
bank.put("HOSTNAME", "\\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\\.?|\\b)");
|
||||
bank.put("IPORHOST", "(?:%{IP}|%{HOSTNAME})");
|
||||
bank.put("USER", "[a-zA-Z0-9._-]+");
|
||||
bank.put("DATA", ".*?");
|
||||
bank.put("QS", "(?>(?<!\\\\)(?>\"(?>\\\\.|[^\\\\\"]+)+\"|\"\"|(?>'(?>\\\\.|[^\\\\']+)+')|''|(?>`(?>\\\\.|[^\\\\`]+)+`)|``))");
|
||||
|
||||
String text = "83.149.9.216 - - [19/Jul/2015:08:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-dashboard3.png HTTP/1.1\" 200 171717 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"";
|
||||
String pattern = "%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \\[%{HTTPDATE:timestamp}\\] \"%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int}) %{QS:referrer} %{QS:agent}";
|
||||
|
||||
Grok grok = new Grok(bank, pattern);
|
||||
|
||||
Map<String, Object> expected = new HashMap<>();
|
||||
expected.put("clientip", "83.149.9.216");
|
||||
expected.put("ident", "-");
|
||||
expected.put("auth", "-");
|
||||
expected.put("timestamp", "19/Jul/2015:08:13:42 +0000");
|
||||
expected.put("verb", "GET");
|
||||
expected.put("request", "/presentations/logstash-monitorama-2013/images/kibana-dashboard3.png");
|
||||
expected.put("httpversion", "1.1");
|
||||
expected.put("response", 200);
|
||||
expected.put("bytes", 171717);
|
||||
expected.put("referrer", "\"http://semicomplete.com/presentations/logstash-monitorama-2013/\"");
|
||||
expected.put("agent", "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"");
|
||||
|
||||
Map<String, Object> actual = grok.captures(text);
|
||||
|
||||
assertEquals(expected, actual);
|
||||
}
|
||||
}
|
||||
|
|
@ -23,6 +23,7 @@ import org.elasticsearch.cluster.ClusterService;
|
|||
import org.elasticsearch.common.bytes.BytesArray;
|
||||
import org.elasticsearch.common.settings.Settings;
|
||||
import org.elasticsearch.common.text.StringText;
|
||||
import org.elasticsearch.env.Environment;
|
||||
import org.elasticsearch.ingest.processor.simple.SimpleProcessor;
|
||||
import org.elasticsearch.search.SearchHit;
|
||||
import org.elasticsearch.search.internal.InternalSearchHit;
|
||||
|
|
@ -31,6 +32,7 @@ import org.elasticsearch.threadpool.ThreadPool;
|
|||
import org.junit.After;
|
||||
import org.junit.Before;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Collections;
|
||||
import java.util.Comparator;
|
||||
|
|
@ -54,7 +56,8 @@ public class PipelineStoreTests extends ESTestCase {
|
|||
threadPool = new ThreadPool("test");
|
||||
ClusterService clusterService = mock(ClusterService.class);
|
||||
client = mock(PipelineStoreClient.class);
|
||||
store = new PipelineStore(Settings.EMPTY, threadPool, clusterService, client, Collections.singletonMap(SimpleProcessor.TYPE, new SimpleProcessor.Builder.Factory()));
|
||||
Environment environment = mock(Environment.class);
|
||||
store = new PipelineStore(Settings.EMPTY, threadPool, environment, clusterService, client, Collections.singletonMap(SimpleProcessor.TYPE, new SimpleProcessor.Builder.Factory()));
|
||||
store.start();
|
||||
}
|
||||
|
||||
|
|
@ -65,7 +68,7 @@ public class PipelineStoreTests extends ESTestCase {
|
|||
}
|
||||
|
||||
|
||||
public void testUpdatePipeline() {
|
||||
public void testUpdatePipeline() throws IOException {
|
||||
List<SearchHit> hits = new ArrayList<>();
|
||||
hits.add(new InternalSearchHit(0, "1", new StringText("type"), Collections.emptyMap())
|
||||
.sourceRef(new BytesArray("{\"description\": \"_description1\"}"))
|
||||
|
|
@ -126,7 +129,7 @@ public class PipelineStoreTests extends ESTestCase {
|
|||
});
|
||||
}
|
||||
|
||||
public void testGetReference() {
|
||||
public void testGetReference() throws IOException {
|
||||
// fill the store up for the test:
|
||||
List<SearchHit> hits = new ArrayList<>();
|
||||
hits.add(new InternalSearchHit(0, "foo", new StringText("type"), Collections.emptyMap()).sourceRef(new BytesArray("{\"description\": \"_description\"}")));
|
||||
|
|
|
|||
|
|
@ -0,0 +1,48 @@
|
|||
---
|
||||
"Test Grok Pipeline":
|
||||
- do:
|
||||
cluster.health:
|
||||
wait_for_status: green
|
||||
|
||||
- do:
|
||||
ingest.put_pipeline:
|
||||
id: "my_pipeline"
|
||||
body: >
|
||||
{
|
||||
"description": "_description",
|
||||
"processors": [
|
||||
{
|
||||
"grok" : {
|
||||
"field" : "field1",
|
||||
"pattern" : "%{NUMBER:val:float} %{NUMBER:status:int} <%{WORD:msg}>"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
- match: { _id: "my_pipeline" }
|
||||
|
||||
# Simulate a Thread.sleep(), because pipeline are updated in the background
|
||||
- do:
|
||||
catch: request_timeout
|
||||
cluster.health:
|
||||
wait_for_nodes: 99
|
||||
timeout: 2s
|
||||
- match: { "timed_out": true }
|
||||
|
||||
- do:
|
||||
ingest.index:
|
||||
index: test
|
||||
type: test
|
||||
id: 1
|
||||
ingest: "my_pipeline"
|
||||
body: {field1: "123.42 400 <foo>"}
|
||||
|
||||
- do:
|
||||
get:
|
||||
index: test
|
||||
type: test
|
||||
id: 1
|
||||
- match: { _source.val: 123.42 }
|
||||
- match: { _source.status: 400 }
|
||||
- match: { _source.msg: "foo" }
|
||||
|
||||
Loading…
Reference in New Issue