Merge pull request #14132 from talevy/ingest/grok
[WIP] Introduce the GrokProcessor
This commit is contained in:
Tal Levy
commit
6e99c71330
|
|
@ -3,6 +3,145 @@
|
|||
|
||||
TODO
|
||||
|
||||
=== Processors
|
||||
|
||||
==== Grok Processor
|
||||
|
||||
The Grok Processor extracts structured fields out of a single text field within a document. You choose which field to
|
||||
extract matched fields from, as well as the Grok Pattern you expect will match. A Grok Pattern is like a regular
|
||||
expression that supports aliased expressions that can be reused.
|
||||
|
||||
This tool is perfect for syslog logs, apache and other webserver logs, mysql logs, and in general, any log format
|
||||
that is generally written for humans and not computer consumption.
|
||||
|
||||
The processor comes packaged with over 120 reusable patterns that are located at `$ES_HOME/config/ingest/grok/patterns`.
|
||||
Here, you can add your own custom grok pattern files with custom grok expressions to be used by the processor.
|
||||
|
||||
If you need help building patterns to match your logs, you will find the <http://grokdebug.herokuapp.com> and
|
||||
<http://grokconstructor.appspot.com/> applications quite useful!
|
||||
|
||||
===== Grok Basics
|
||||
|
||||
Grok sits on top of regular expressions, so any regular expressions are valid in grok as well.
|
||||
The regular expression library is Oniguruma, and you can see the full supported regexp syntax
|
||||
https://github.com/kkos/oniguruma/blob/master/doc/RE[on the Onigiruma site].
|
||||
|
||||
Grok works by leveraging this regular expression language to allow naming existing patterns and combining them into more
|
||||
complex patterns that match your fields.
|
||||
|
||||
The syntax for re-using a grok pattern comes in three forms: `%{SYNTAX:SEMANTIC}`, `%{SYNTAX}`, `%{SYNTAX:SEMANTIC:TYPE}`.
|
||||
|
||||
The `SYNTAX` is the name of the pattern that will match your text. For example, `3.44` will be matched by the `NUMBER`
|
||||
pattern and `55.3.244.1` will be matched by the `IP` pattern. The syntax is how you match. `NUMBER` and `IP` are both
|
||||
patterns that are provided within the default patterns set.
|
||||
|
||||
The `SEMANTIC` is the identifier you give to the piece of text being matched. For example, `3.44` could be the
|
||||
duration of an event, so you could call it simply `duration`. Further, a string `55.3.244.1` might identify
|
||||
the `client` making a request.
|
||||
|
||||
The `TYPE` is the type you wish to cast your named field. `int` and `float` are currently the only types supported for coercion.
|
||||
|
||||
For example, here is a grok pattern that would match the above example given. We would like to match a text with the following
|
||||
contents:
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
3.44 55.3.244.1
|
||||
--------------------------------------------------
|
||||
|
||||
We may know that the above message is a number followed by an IP-address. We can match this text with the following
|
||||
Grok expression.
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
%{NUMBER:duration} %{IP:client}
|
||||
--------------------------------------------------
|
||||
|
||||
===== Custom Patterns and Pattern Files
|
||||
|
||||
The Grok Processor comes pre-packaged with a base set of pattern files. These patterns may not always have
|
||||
what you are looking for. These pattern files have a very basic format. Each line describes a named pattern with
|
||||
the following format:
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
NAME ' '+ PATTERN '\n'
|
||||
--------------------------------------------------
|
||||
|
||||
You can add this pattern to an existing file, or add your own file in the patterns directory here: `$ES_HOME/config/ingest/grok/patterns`.
|
||||
The Ingest Plugin will pick up files in this directory to be loaded into the grok processor's known patterns. These patterns are loaded
|
||||
at startup, so you will need to do a restart your ingest node if you wish to update these files while running.
|
||||
|
||||
Example snippet of pattern definitions found in the `grok-patterns` patterns file:
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
YEAR (?>\d\d){1,2}
|
||||
HOUR (?:2[0123]|[01]?[0-9])
|
||||
MINUTE (?:[0-5][0-9])
|
||||
SECOND (?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)
|
||||
TIME (?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9])
|
||||
--------------------------------------------------
|
||||
|
||||
===== Using Grok Processor in a Pipeline
|
||||
|
||||
[[grok-options]]
|
||||
.Grok Options
|
||||
[options="header"]
|
||||
|======
|
||||
| Name | Required | Default | Description
|
||||
| `match_field` | yes | - | The field to use for grok expression parsing
|
||||
| `match_pattern` | yes | - | The grok expression to match and extract named captures with
|
||||
|======
|
||||
|
||||
Here is an example of using the provided patterns to extract out and name structured fields from a string field in
|
||||
a document.
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
{
|
||||
"message": "55.3.244.1 GET /index.html 15824 0.043"
|
||||
}
|
||||
--------------------------------------------------
|
||||
|
||||
The pattern for this could be
|
||||
|
||||
[source]
|
||||
--------------------------------------------------
|
||||
%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}
|
||||
--------------------------------------------------
|
||||
|
||||
An example pipeline for processing the above document using Grok:
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
{
|
||||
"description" : "...",
|
||||
"processors": [
|
||||
{
|
||||
"grok": {
|
||||
"match_field": "message",
|
||||
"match_pattern": "%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
--------------------------------------------------
|
||||
|
||||
This pipeline will insert these named captures as new fields within the document, like so:
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
{
|
||||
"message": "55.3.244.1 GET /index.html 15824 0.043",
|
||||
"client": "55.3.244.1",
|
||||
"method": "GET",
|
||||
"request": "/index.html",
|
||||
"bytes": 15824,
|
||||
"duration": "0.043"
|
||||
}
|
||||
--------------------------------------------------
|
||||
|
||||
=== Put pipeline API
|
||||
|
||||
The put pipeline api adds pipelines and updates existing pipelines in the cluster.
|
||||
|
|
|
|||
|
|
@ -23,8 +23,17 @@ esplugin {
|
|||
}
|
||||
|
||||
dependencies {
|
||||
compile 'org.jruby.joni:joni:2.1.6'
|
||||
testCompile 'org.elasticsearch:securemock:1.1'
|
||||
}
|
||||
|
||||
sourceSets {
|
||||
test {
|
||||
resources {
|
||||
srcDir "src/main/packaging/config"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
compileJava.options.compilerArgs << "-Xlint:-rawtypes,-unchecked"
|
||||
compileTestJava.options.compilerArgs << "-Xlint:-rawtypes,-unchecked"
|
||||
|
|
|
|||
|
|
@ -0,0 +1 @@
|
|||
6bc17079fcaa8823ea8cd0d4c66516335b558db8
|
||||
|
|
@ -0,0 +1,17 @@
|
|||
Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
this software and associated documentation files (the "Software"), to deal in
|
||||
the Software without restriction, including without limitation the rights to
|
||||
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
|
||||
of the Software, and to permit persons to whom the Software is furnished to do
|
||||
so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
|
|
@ -0,0 +1 @@
|
|||
JCodings is released under the MIT License.
|
||||
|
|
@ -0,0 +1 @@
|
|||
0f23c95a06eaecbc8c74c7458a8bfd13e4fd2d3a
|
||||
|
|
@ -0,0 +1,17 @@
|
|||
Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
this software and associated documentation files (the "Software"), to deal in
|
||||
the Software without restriction, including without limitation the rights to
|
||||
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
|
||||
of the Software, and to permit persons to whom the Software is furnished to do
|
||||
so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
|
|
@ -0,0 +1 @@
|
|||
Joni is released under the MIT License.
|
||||
|
|
@ -1 +0,0 @@
|
|||
This plugin has no third party dependencies
|
||||
|
|
@ -22,6 +22,7 @@ package org.elasticsearch.ingest;
|
|||
|
||||
import org.elasticsearch.ingest.processor.Processor;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Collections;
|
||||
import java.util.List;
|
||||
|
|
@ -82,7 +83,7 @@ public final class Pipeline {
|
|||
this.id = id;
|
||||
}
|
||||
|
||||
public void fromMap(Map<String, Object> config, Map<String, Processor.Builder.Factory> processorRegistry) {
|
||||
public void fromMap(Map<String, Object> config, Map<String, Processor.Builder.Factory> processorRegistry) throws IOException {
|
||||
description = (String) config.get("description");
|
||||
@SuppressWarnings("unchecked")
|
||||
List<Map<String, Map<String, Object>>> processors = (List<Map<String, Map<String, Object>>>) config.get("processors");
|
||||
|
|
@ -105,7 +106,7 @@ public final class Pipeline {
|
|||
this.description = description;
|
||||
}
|
||||
|
||||
public void addProcessors(Processor.Builder... processors) {
|
||||
public void addProcessors(Processor.Builder... processors) throws IOException {
|
||||
for (Processor.Builder processor : processors) {
|
||||
this.processors.add(processor.build());
|
||||
}
|
||||
|
|
|
|||
|
|
@ -22,6 +22,8 @@ package org.elasticsearch.ingest.processor;
|
|||
|
||||
import org.elasticsearch.ingest.Data;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.nio.file.Path;
|
||||
import java.util.Map;
|
||||
|
||||
/**
|
||||
|
|
@ -48,7 +50,7 @@ public interface Processor {
|
|||
/**
|
||||
* Builds the processor based on previous set settings.
|
||||
*/
|
||||
Processor build();
|
||||
Processor build() throws IOException;
|
||||
|
||||
/**
|
||||
* A factory that creates a processor builder when processor instances for pipelines are being created.
|
||||
|
|
@ -60,6 +62,11 @@ public interface Processor {
|
|||
*/
|
||||
Builder create();
|
||||
|
||||
/**
|
||||
*/
|
||||
default void setConfigDirectory(Path configDirectory) {
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
}
|
||||
|
|
|
|||
|
|
@ -0,0 +1,160 @@
|
|||
/*
|
||||
* Licensed to Elasticsearch under one or more contributor
|
||||
* license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright
|
||||
* ownership. Elasticsearch licenses this file to you under
|
||||
* the Apache License, Version 2.0 (the "License"); you may
|
||||
* not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing,
|
||||
* software distributed under the License is distributed on an
|
||||
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
* KIND, either express or implied. See the License for the
|
||||
* specific language governing permissions and limitations
|
||||
* under the License.
|
||||
*/
|
||||
|
||||
package org.elasticsearch.ingest.processor.grok;
|
||||
|
||||
import java.io.BufferedReader;
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.io.InputStreamReader;
|
||||
import java.lang.Object;
|
||||
import java.lang.String;
|
||||
import java.lang.StringIndexOutOfBoundsException;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.util.HashMap;
|
||||
import java.util.Iterator;
|
||||
import java.util.Locale;
|
||||
import java.util.Map;
|
||||
|
||||
import org.jcodings.specific.UTF8Encoding;
|
||||
import org.joni.*;
|
||||
import org.joni.exception.ValueException;
|
||||
|
||||
public class Grok {
|
||||
|
||||
private static final String NAME_GROUP = "name";
|
||||
private static final String SUBNAME_GROUP = "subname";
|
||||
private static final String PATTERN_GROUP = "pattern";
|
||||
private static final String DEFINITION_GROUP = "definition";
|
||||
private static final String GROK_PATTERN =
|
||||
"%\\{" +
|
||||
"(?<name>" +
|
||||
"(?<pattern>[A-z0-9]+)" +
|
||||
"(?::(?<subname>[A-z0-9_:.-]+))?" +
|
||||
")" +
|
||||
"(?:=(?<definition>" +
|
||||
"(?:" +
|
||||
"(?:[^{}]+|\\.+)+" +
|
||||
")+" +
|
||||
")" +
|
||||
")?" + "\\}";
|
||||
private static final Regex GROK_PATTERN_REGEX = new Regex(GROK_PATTERN.getBytes(StandardCharsets.UTF_8), 0, GROK_PATTERN.getBytes(StandardCharsets.UTF_8).length, Option.NONE, UTF8Encoding.INSTANCE, Syntax.DEFAULT);
|
||||
private final Map<String, String> patternBank;
|
||||
private final boolean namedCaptures;
|
||||
private final Regex compiledExpression;
|
||||
private final String expression;
|
||||
|
||||
|
||||
public Grok(Map<String, String> patternBank, String grokPattern) {
|
||||
this(patternBank, grokPattern, true);
|
||||
}
|
||||
|
||||
@SuppressWarnings("unchecked")
|
||||
public Grok(Map<String, String> patternBank, String grokPattern, boolean namedCaptures) {
|
||||
this.patternBank = patternBank;
|
||||
this.namedCaptures = namedCaptures;
|
||||
|
||||
this.expression = toRegex(grokPattern);
|
||||
byte[] expressionBytes = expression.getBytes(StandardCharsets.UTF_8);
|
||||
this.compiledExpression = new Regex(expressionBytes, 0, expressionBytes.length, Option.DEFAULT, UTF8Encoding.INSTANCE);
|
||||
}
|
||||
|
||||
|
||||
public String groupMatch(String name, Region region, String pattern) {
|
||||
try {
|
||||
int number = GROK_PATTERN_REGEX.nameToBackrefNumber(name.getBytes(StandardCharsets.UTF_8), 0, name.getBytes(StandardCharsets.UTF_8).length, region);
|
||||
int begin = region.beg[number];
|
||||
int end = region.end[number];
|
||||
return new String(pattern.getBytes(StandardCharsets.UTF_8), begin, end - begin, StandardCharsets.UTF_8);
|
||||
} catch (StringIndexOutOfBoundsException e) {
|
||||
return null;
|
||||
} catch (ValueException e) {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* converts a grok expression into a named regex expression
|
||||
*
|
||||
* @return named regex expression
|
||||
*/
|
||||
public String toRegex(String grokPattern) {
|
||||
byte[] grokPatternBytes = grokPattern.getBytes(StandardCharsets.UTF_8);
|
||||
Matcher matcher = GROK_PATTERN_REGEX.matcher(grokPatternBytes);
|
||||
|
||||
int result = matcher.search(0, grokPatternBytes.length, Option.NONE);
|
||||
if (result != -1) {
|
||||
Region region = matcher.getEagerRegion();
|
||||
String namedPatternRef = groupMatch(NAME_GROUP, region, grokPattern);
|
||||
String subName = groupMatch(SUBNAME_GROUP, region, grokPattern);
|
||||
// TODO(tal): Support definitions
|
||||
String definition = groupMatch(DEFINITION_GROUP, region, grokPattern);
|
||||
String patternName = groupMatch(PATTERN_GROUP, region, grokPattern);
|
||||
String pattern = patternBank.get(patternName);
|
||||
|
||||
String grokPart;
|
||||
if (namedCaptures && subName != null) {
|
||||
grokPart = String.format(Locale.US, "(?<%s>%s)", namedPatternRef, pattern);
|
||||
} else if (!namedCaptures) {
|
||||
grokPart = String.format(Locale.US, "(?<%s>%s)", patternName + "_" + String.valueOf(result), pattern);
|
||||
} else {
|
||||
grokPart = String.format(Locale.US, "(?:%s)", pattern);
|
||||
}
|
||||
|
||||
String start = new String(grokPatternBytes, 0, result, StandardCharsets.UTF_8);
|
||||
String rest = new String(grokPatternBytes, region.end[0], grokPatternBytes.length - region.end[0], StandardCharsets.UTF_8);
|
||||
return start + toRegex(grokPart + rest);
|
||||
}
|
||||
|
||||
return grokPattern;
|
||||
}
|
||||
|
||||
public boolean match(String text) {
|
||||
Matcher matcher = compiledExpression.matcher(text.getBytes(StandardCharsets.UTF_8));
|
||||
int result = matcher.search(0, text.length(), Option.DEFAULT);
|
||||
return (result != -1);
|
||||
}
|
||||
|
||||
public Map<String, Object> captures(String text) {
|
||||
byte[] textAsBytes = text.getBytes(StandardCharsets.UTF_8);
|
||||
Map<String, Object> fields = new HashMap<>();
|
||||
Matcher matcher = compiledExpression.matcher(textAsBytes);
|
||||
int result = matcher.search(0, textAsBytes.length, Option.DEFAULT);
|
||||
if (result != -1) {
|
||||
Region region = matcher.getEagerRegion();
|
||||
for (Iterator<NameEntry> entry = compiledExpression.namedBackrefIterator(); entry.hasNext();) {
|
||||
NameEntry e = entry.next();
|
||||
int number = e.getBackRefs()[0];
|
||||
|
||||
String groupName = new String(e.name, e.nameP, e.nameEnd - e.nameP, StandardCharsets.UTF_8);
|
||||
String matchValue = null;
|
||||
if (region.beg[number] >= 0) {
|
||||
matchValue = new String(textAsBytes, region.beg[number], region.end[number] - region.beg[number], StandardCharsets.UTF_8);
|
||||
}
|
||||
GrokMatchGroup match = new GrokMatchGroup(groupName, matchValue);
|
||||
fields.put(match.getName(), match.getValue());
|
||||
}
|
||||
} else {
|
||||
return null;
|
||||
}
|
||||
|
||||
return fields;
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -0,0 +1,67 @@
|
|||
/*
|
||||
* Licensed to Elasticsearch under one or more contributor
|
||||
* license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright
|
||||
* ownership. Elasticsearch licenses this file to you under
|
||||
* the Apache License, Version 2.0 (the "License"); you may
|
||||
* not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing,
|
||||
* software distributed under the License is distributed on an
|
||||
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
* KIND, either express or implied. See the License for the
|
||||
* specific language governing permissions and limitations
|
||||
* under the License.
|
||||
*/
|
||||
|
||||
package org.elasticsearch.ingest.processor.grok;
|
||||
|
||||
import java.lang.Float;
|
||||
import java.lang.Integer;
|
||||
import java.lang.Object;
|
||||
import java.lang.String;
|
||||
|
||||
public class GrokMatchGroup {
|
||||
private static final String DEFAULT_TYPE = "string";
|
||||
private final String patternName;
|
||||
private final String fieldName;
|
||||
private final String type;
|
||||
private final String groupValue;
|
||||
|
||||
public GrokMatchGroup(String groupName, String groupValue) {
|
||||
String[] parts = groupName.split(":");
|
||||
patternName = parts[0];
|
||||
if (parts.length >= 2) {
|
||||
fieldName = parts[1];
|
||||
} else {
|
||||
fieldName = null;
|
||||
}
|
||||
|
||||
if (parts.length == 3) {
|
||||
type = parts[2];
|
||||
} else {
|
||||
type = DEFAULT_TYPE;
|
||||
}
|
||||
this.groupValue = groupValue;
|
||||
}
|
||||
|
||||
public String getName() {
|
||||
return (fieldName == null) ? patternName : fieldName;
|
||||
}
|
||||
|
||||
public Object getValue() {
|
||||
if (groupValue == null) { return null; }
|
||||
|
||||
switch(type) {
|
||||
case "int":
|
||||
return Integer.parseInt(groupValue);
|
||||
case "float":
|
||||
return Float.parseFloat(groupValue);
|
||||
default:
|
||||
return groupValue;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,119 @@
|
|||
/*
|
||||
* Licensed to Elasticsearch under one or more contributor
|
||||
* license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright
|
||||
* ownership. Elasticsearch licenses this file to you under
|
||||
* the Apache License, Version 2.0 (the "License"); you may
|
||||
* not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing,
|
||||
* software distributed under the License is distributed on an
|
||||
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
* KIND, either express or implied. See the License for the
|
||||
* specific language governing permissions and limitations
|
||||
* under the License.
|
||||
*/
|
||||
|
||||
package org.elasticsearch.ingest.processor.grok;
|
||||
|
||||
import org.elasticsearch.common.inject.Inject;
|
||||
import org.elasticsearch.env.Environment;
|
||||
import org.elasticsearch.ingest.Data;
|
||||
import org.elasticsearch.ingest.processor.Processor;
|
||||
|
||||
import java.io.BufferedReader;
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.io.InputStreamReader;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.nio.file.DirectoryStream;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.nio.file.StandardOpenOption;
|
||||
import java.util.HashMap;
|
||||
import java.util.Map;
|
||||
|
||||
public final class GrokProcessor implements Processor {
|
||||
|
||||
public static final String TYPE = "grok";
|
||||
|
||||
private final String matchField;
|
||||
private final String matchPattern;
|
||||
private final Grok grok;
|
||||
|
||||
public GrokProcessor(Grok grok, String matchField, String matchPattern) throws IOException {
|
||||
this.matchField = matchField;
|
||||
this.matchPattern = matchPattern;
|
||||
this.grok = grok;
|
||||
}
|
||||
|
||||
@Override
|
||||
public void execute(Data data) {
|
||||
Object field = data.getProperty(matchField);
|
||||
// TODO(talevy): handle invalid field types
|
||||
if (field instanceof String) {
|
||||
Map<String, Object> matches = grok.captures((String) field);
|
||||
if (matches != null) {
|
||||
matches.forEach((k, v) -> data.addField(k, v));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
public static class Builder implements Processor.Builder {
|
||||
|
||||
private Path grokConfigDirectory;
|
||||
private String matchField;
|
||||
private String matchPattern;
|
||||
|
||||
public Builder(Path grokConfigDirectory) {
|
||||
this.grokConfigDirectory = grokConfigDirectory;
|
||||
}
|
||||
|
||||
public void setMatchField(String matchField) {
|
||||
this.matchField = matchField;
|
||||
}
|
||||
|
||||
public void setMatchPattern(String matchPattern) {
|
||||
this.matchPattern = matchPattern;
|
||||
}
|
||||
|
||||
public void fromMap(Map<String, Object> config) {
|
||||
this.matchField = (String) config.get("field");
|
||||
this.matchPattern = (String) config.get("pattern");
|
||||
}
|
||||
|
||||
@Override
|
||||
public Processor build() throws IOException {
|
||||
Map<String, String> patternBank = new HashMap<>();
|
||||
Path patternsDirectory = grokConfigDirectory.resolve("patterns");
|
||||
try (DirectoryStream<Path> stream = Files.newDirectoryStream(patternsDirectory)) {
|
||||
for (Path patternFilePath : stream) {
|
||||
try(InputStream is = Files.newInputStream(patternFilePath, StandardOpenOption.READ)) {
|
||||
PatternUtils.loadBankFromStream(patternBank, is);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Grok grok = new Grok(patternBank, matchPattern);
|
||||
return new GrokProcessor(grok, matchField, matchPattern);
|
||||
}
|
||||
|
||||
public static class Factory implements Processor.Builder.Factory {
|
||||
private Path grokConfigDirectory;
|
||||
|
||||
@Override
|
||||
public Processor.Builder create() {
|
||||
return new Builder(grokConfigDirectory);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void setConfigDirectory(Path configDirectory) {
|
||||
this.grokConfigDirectory = configDirectory.resolve("ingest").resolve("grok");
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,47 @@
|
|||
/*
|
||||
* Licensed to Elasticsearch under one or more contributor
|
||||
* license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright
|
||||
* ownership. Elasticsearch licenses this file to you under
|
||||
* the Apache License, Version 2.0 (the "License"); you may
|
||||
* not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing,
|
||||
* software distributed under the License is distributed on an
|
||||
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
* KIND, either express or implied. See the License for the
|
||||
* specific language governing permissions and limitations
|
||||
* under the License.
|
||||
*/
|
||||
package org.elasticsearch.ingest.processor.grok;
|
||||
|
||||
import java.io.BufferedReader;
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.io.InputStreamReader;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.util.Map;
|
||||
|
||||
public final class PatternUtils {
|
||||
private PatternUtils() {}
|
||||
|
||||
public static void loadBankFromStream(Map<String, String> patternBank, InputStream inputStream) throws IOException {
|
||||
String line;
|
||||
BufferedReader br = new BufferedReader(new InputStreamReader(inputStream, StandardCharsets.UTF_8));
|
||||
while ((line = br.readLine()) != null) {
|
||||
String trimmedLine = line.replaceAll("^\\s+", "");
|
||||
if (trimmedLine.startsWith("#") || trimmedLine.length() == 0) {
|
||||
continue;
|
||||
}
|
||||
|
||||
String[] parts = trimmedLine.split("\\s+", 2);
|
||||
if (parts.length == 2) {
|
||||
patternBank.put(parts[0], parts[1]);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
|
|
@ -22,6 +22,7 @@ package org.elasticsearch.plugin.ingest;
|
|||
import org.elasticsearch.common.inject.AbstractModule;
|
||||
import org.elasticsearch.common.inject.multibindings.MapBinder;
|
||||
import org.elasticsearch.ingest.processor.Processor;
|
||||
import org.elasticsearch.ingest.processor.grok.GrokProcessor;
|
||||
import org.elasticsearch.ingest.processor.simple.SimpleProcessor;
|
||||
import org.elasticsearch.plugin.ingest.rest.IngestRestFilter;
|
||||
|
||||
|
|
@ -40,6 +41,7 @@ public class IngestModule extends AbstractModule {
|
|||
binder().bind(PipelineStoreClient.class).asEagerSingleton();
|
||||
|
||||
registerProcessor(SimpleProcessor.TYPE, SimpleProcessor.Builder.Factory.class);
|
||||
registerProcessor(GrokProcessor.TYPE, GrokProcessor.Builder.Factory.class);
|
||||
|
||||
MapBinder<String, Processor.Builder.Factory> mapBinder = MapBinder.newMapBinder(binder(), String.class, Processor.Builder.Factory.class);
|
||||
for (Map.Entry<String, Class<? extends Processor.Builder.Factory>> entry : processors.entrySet()) {
|
||||
|
|
|
|||
|
|
@ -29,12 +29,14 @@ import org.elasticsearch.common.inject.Inject;
|
|||
import org.elasticsearch.common.regex.Regex;
|
||||
import org.elasticsearch.common.settings.Settings;
|
||||
import org.elasticsearch.common.unit.TimeValue;
|
||||
import org.elasticsearch.env.Environment;
|
||||
import org.elasticsearch.gateway.GatewayService;
|
||||
import org.elasticsearch.ingest.Pipeline;
|
||||
import org.elasticsearch.ingest.processor.Processor;
|
||||
import org.elasticsearch.search.SearchHit;
|
||||
import org.elasticsearch.threadpool.ThreadPool;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.util.*;
|
||||
|
||||
public class PipelineStore extends AbstractLifecycleComponent {
|
||||
|
|
@ -51,12 +53,15 @@ public class PipelineStore extends AbstractLifecycleComponent {
|
|||
private volatile Map<String, PipelineReference> pipelines = new HashMap<>();
|
||||
|
||||
@Inject
|
||||
public PipelineStore(Settings settings, ThreadPool threadPool, ClusterService clusterService, PipelineStoreClient client, Map<String, Processor.Builder.Factory> processors) {
|
||||
public PipelineStore(Settings settings, ThreadPool threadPool, Environment environment, ClusterService clusterService, PipelineStoreClient client, Map<String, Processor.Builder.Factory> processors) {
|
||||
super(settings);
|
||||
this.threadPool = threadPool;
|
||||
this.clusterService = clusterService;
|
||||
this.pipelineUpdateInterval = settings.getAsTime("ingest.pipeline.store.update.interval", TimeValue.timeValueSeconds(1));
|
||||
this.client = client;
|
||||
for (Processor.Builder.Factory factory : processors.values()) {
|
||||
factory.setConfigDirectory(environment.configFile());
|
||||
}
|
||||
this.processorFactoryRegistry = Collections.unmodifiableMap(processors);
|
||||
clusterService.add(new PipelineStoreListener());
|
||||
}
|
||||
|
|
@ -101,7 +106,7 @@ public class PipelineStore extends AbstractLifecycleComponent {
|
|||
return result;
|
||||
}
|
||||
|
||||
void updatePipelines() {
|
||||
void updatePipelines() throws IOException {
|
||||
// note: this process isn't fast or smart, but the idea is that there will not be many pipelines,
|
||||
// so for that reason the goal is to keep the update logic simple.
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,11 @@
|
|||
S3_REQUEST_LINE (?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})
|
||||
|
||||
S3_ACCESS_LOG %{WORD:owner} %{NOTSPACE:bucket} \[%{HTTPDATE:timestamp}\] %{IP:clientip} %{NOTSPACE:requester} %{NOTSPACE:request_id} %{NOTSPACE:operation} %{NOTSPACE:key} (?:"%{S3_REQUEST_LINE}"|-) (?:%{INT:response:int}|-) (?:-|%{NOTSPACE:error_code}) (?:%{INT:bytes:int}|-) (?:%{INT:object_size:int}|-) (?:%{INT:request_time_ms:int}|-) (?:%{INT:turnaround_time_ms:int}|-) (?:%{QS:referrer}|-) (?:"?%{QS:agent}"?|-) (?:-|%{NOTSPACE:version_id})
|
||||
|
||||
ELB_URIPATHPARAM %{URIPATH:path}(?:%{URIPARAM:params})?
|
||||
|
||||
ELB_URI %{URIPROTO:proto}://(?:%{USER}(?::[^@]*)?@)?(?:%{URIHOST:urihost})?(?:%{ELB_URIPATHPARAM})?
|
||||
|
||||
ELB_REQUEST_LINE (?:%{WORD:verb} %{ELB_URI:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})
|
||||
|
||||
ELB_ACCESS_LOG %{TIMESTAMP_ISO8601:timestamp} %{NOTSPACE:elb} %{IP:clientip}:%{INT:clientport:int} (?:(%{IP:backendip}:?:%{INT:backendport:int})|-) %{NUMBER:request_processing_time:float} %{NUMBER:backend_processing_time:float} %{NUMBER:response_processing_time:float} %{INT:response:int} %{INT:backend_response:int} %{INT:received_bytes:int} %{INT:bytes:int} "%{ELB_REQUEST_LINE}"
|
||||
|
|
@ -0,0 +1,50 @@
|
|||
BACULA_TIMESTAMP %{MONTHDAY}-%{MONTH} %{HOUR}:%{MINUTE}
|
||||
BACULA_HOST [a-zA-Z0-9-]+
|
||||
BACULA_VOLUME %{USER}
|
||||
BACULA_DEVICE %{USER}
|
||||
BACULA_DEVICEPATH %{UNIXPATH}
|
||||
BACULA_CAPACITY %{INT}{1,3}(,%{INT}{3})*
|
||||
BACULA_VERSION %{USER}
|
||||
BACULA_JOB %{USER}
|
||||
|
||||
BACULA_LOG_MAX_CAPACITY User defined maximum volume capacity %{BACULA_CAPACITY} exceeded on device \"%{BACULA_DEVICE:device}\" \(%{BACULA_DEVICEPATH}\)
|
||||
BACULA_LOG_END_VOLUME End of medium on Volume \"%{BACULA_VOLUME:volume}\" Bytes=%{BACULA_CAPACITY} Blocks=%{BACULA_CAPACITY} at %{MONTHDAY}-%{MONTH}-%{YEAR} %{HOUR}:%{MINUTE}.
|
||||
BACULA_LOG_NEW_VOLUME Created new Volume \"%{BACULA_VOLUME:volume}\" in catalog.
|
||||
BACULA_LOG_NEW_LABEL Labeled new Volume \"%{BACULA_VOLUME:volume}\" on device \"%{BACULA_DEVICE:device}\" \(%{BACULA_DEVICEPATH}\).
|
||||
BACULA_LOG_WROTE_LABEL Wrote label to prelabeled Volume \"%{BACULA_VOLUME:volume}\" on device \"%{BACULA_DEVICE}\" \(%{BACULA_DEVICEPATH}\)
|
||||
BACULA_LOG_NEW_MOUNT New volume \"%{BACULA_VOLUME:volume}\" mounted on device \"%{BACULA_DEVICE:device}\" \(%{BACULA_DEVICEPATH}\) at %{MONTHDAY}-%{MONTH}-%{YEAR} %{HOUR}:%{MINUTE}.
|
||||
BACULA_LOG_NOOPEN \s+Cannot open %{DATA}: ERR=%{GREEDYDATA:berror}
|
||||
BACULA_LOG_NOOPENDIR \s+Could not open directory %{DATA}: ERR=%{GREEDYDATA:berror}
|
||||
BACULA_LOG_NOSTAT \s+Could not stat %{DATA}: ERR=%{GREEDYDATA:berror}
|
||||
BACULA_LOG_NOJOBS There are no more Jobs associated with Volume \"%{BACULA_VOLUME:volume}\". Marking it purged.
|
||||
BACULA_LOG_ALL_RECORDS_PRUNED All records pruned from Volume \"%{BACULA_VOLUME:volume}\"; marking it \"Purged\"
|
||||
BACULA_LOG_BEGIN_PRUNE_JOBS Begin pruning Jobs older than %{INT} month %{INT} days .
|
||||
BACULA_LOG_BEGIN_PRUNE_FILES Begin pruning Files.
|
||||
BACULA_LOG_PRUNED_JOBS Pruned %{INT} Jobs* for client %{BACULA_HOST:client} from catalog.
|
||||
BACULA_LOG_PRUNED_FILES Pruned Files from %{INT} Jobs* for client %{BACULA_HOST:client} from catalog.
|
||||
BACULA_LOG_ENDPRUNE End auto prune.
|
||||
BACULA_LOG_STARTJOB Start Backup JobId %{INT}, Job=%{BACULA_JOB:job}
|
||||
BACULA_LOG_STARTRESTORE Start Restore Job %{BACULA_JOB:job}
|
||||
BACULA_LOG_USEDEVICE Using Device \"%{BACULA_DEVICE:device}\"
|
||||
BACULA_LOG_DIFF_FS \s+%{UNIXPATH} is a different filesystem. Will not descend from %{UNIXPATH} into it.
|
||||
BACULA_LOG_JOBEND Job write elapsed time = %{DATA:elapsed}, Transfer rate = %{NUMBER} (K|M|G)? Bytes/second
|
||||
BACULA_LOG_NOPRUNE_JOBS No Jobs found to prune.
|
||||
BACULA_LOG_NOPRUNE_FILES No Files found to prune.
|
||||
BACULA_LOG_VOLUME_PREVWRITTEN Volume \"%{BACULA_VOLUME:volume}\" previously written, moving to end of data.
|
||||
BACULA_LOG_READYAPPEND Ready to append to end of Volume \"%{BACULA_VOLUME:volume}\" size=%{INT}
|
||||
BACULA_LOG_CANCELLING Cancelling duplicate JobId=%{INT}.
|
||||
BACULA_LOG_MARKCANCEL JobId %{INT}, Job %{BACULA_JOB:job} marked to be canceled.
|
||||
BACULA_LOG_CLIENT_RBJ shell command: run ClientRunBeforeJob \"%{GREEDYDATA:runjob}\"
|
||||
BACULA_LOG_VSS (Generate )?VSS (Writer)?
|
||||
BACULA_LOG_MAXSTART Fatal error: Job canceled because max start delay time exceeded.
|
||||
BACULA_LOG_DUPLICATE Fatal error: JobId %{INT:duplicate} already running. Duplicate job not allowed.
|
||||
BACULA_LOG_NOJOBSTAT Fatal error: No Job status returned from FD.
|
||||
BACULA_LOG_FATAL_CONN Fatal error: bsock.c:133 Unable to connect to (Client: %{BACULA_HOST:client}|Storage daemon) on %{HOSTNAME}:%{POSINT}. ERR=(?<berror>%{GREEDYDATA})
|
||||
BACULA_LOG_NO_CONNECT Warning: bsock.c:127 Could not connect to (Client: %{BACULA_HOST:client}|Storage daemon) on %{HOSTNAME}:%{POSINT}. ERR=(?<berror>%{GREEDYDATA})
|
||||
BACULA_LOG_NO_AUTH Fatal error: Unable to authenticate with File daemon at %{HOSTNAME}. Possible causes:
|
||||
BACULA_LOG_NOSUIT No prior or suitable Full backup found in catalog. Doing FULL backup.
|
||||
BACULA_LOG_NOPRIOR No prior Full backup Job record found.
|
||||
|
||||
BACULA_LOG_JOB (Error: )?Bacula %{BACULA_HOST} %{BACULA_VERSION} \(%{BACULA_VERSION}\):
|
||||
|
||||
BACULA_LOGLINE %{BACULA_TIMESTAMP:bts} %{BACULA_HOST:hostname} JobId %{INT:jobid}: (%{BACULA_LOG_MAX_CAPACITY}|%{BACULA_LOG_END_VOLUME}|%{BACULA_LOG_NEW_VOLUME}|%{BACULA_LOG_NEW_LABEL}|%{BACULA_LOG_WROTE_LABEL}|%{BACULA_LOG_NEW_MOUNT}|%{BACULA_LOG_NOOPEN}|%{BACULA_LOG_NOOPENDIR}|%{BACULA_LOG_NOSTAT}|%{BACULA_LOG_NOJOBS}|%{BACULA_LOG_ALL_RECORDS_PRUNED}|%{BACULA_LOG_BEGIN_PRUNE_JOBS}|%{BACULA_LOG_BEGIN_PRUNE_FILES}|%{BACULA_LOG_PRUNED_JOBS}|%{BACULA_LOG_PRUNED_FILES}|%{BACULA_LOG_ENDPRUNE}|%{BACULA_LOG_STARTJOB}|%{BACULA_LOG_STARTRESTORE}|%{BACULA_LOG_USEDEVICE}|%{BACULA_LOG_DIFF_FS}|%{BACULA_LOG_JOBEND}|%{BACULA_LOG_NOPRUNE_JOBS}|%{BACULA_LOG_NOPRUNE_FILES}|%{BACULA_LOG_VOLUME_PREVWRITTEN}|%{BACULA_LOG_READYAPPEND}|%{BACULA_LOG_CANCELLING}|%{BACULA_LOG_MARKCANCEL}|%{BACULA_LOG_CLIENT_RBJ}|%{BACULA_LOG_VSS}|%{BACULA_LOG_MAXSTART}|%{BACULA_LOG_DUPLICATE}|%{BACULA_LOG_NOJOBSTAT}|%{BACULA_LOG_FATAL_CONN}|%{BACULA_LOG_NO_CONNECT}|%{BACULA_LOG_NO_AUTH}|%{BACULA_LOG_NOSUIT}|%{BACULA_LOG_JOB}|%{BACULA_LOG_NOPRIOR})
|
||||
|
|
@ -0,0 +1,13 @@
|
|||
# https://www.bro.org/sphinx/script-reference/log-files.html
|
||||
|
||||
# http.log
|
||||
BRO_HTTP %{NUMBER:ts}\t%{NOTSPACE:uid}\t%{IP:orig_h}\t%{INT:orig_p}\t%{IP:resp_h}\t%{INT:resp_p}\t%{INT:trans_depth}\t%{GREEDYDATA:method}\t%{GREEDYDATA:domain}\t%{GREEDYDATA:uri}\t%{GREEDYDATA:referrer}\t%{GREEDYDATA:user_agent}\t%{NUMBER:request_body_len}\t%{NUMBER:response_body_len}\t%{GREEDYDATA:status_code}\t%{GREEDYDATA:status_msg}\t%{GREEDYDATA:info_code}\t%{GREEDYDATA:info_msg}\t%{GREEDYDATA:filename}\t%{GREEDYDATA:bro_tags}\t%{GREEDYDATA:username}\t%{GREEDYDATA:password}\t%{GREEDYDATA:proxied}\t%{GREEDYDATA:orig_fuids}\t%{GREEDYDATA:orig_mime_types}\t%{GREEDYDATA:resp_fuids}\t%{GREEDYDATA:resp_mime_types}
|
||||
|
||||
# dns.log
|
||||
BRO_DNS %{NUMBER:ts}\t%{NOTSPACE:uid}\t%{IP:orig_h}\t%{INT:orig_p}\t%{IP:resp_h}\t%{INT:resp_p}\t%{WORD:proto}\t%{INT:trans_id}\t%{GREEDYDATA:query}\t%{GREEDYDATA:qclass}\t%{GREEDYDATA:qclass_name}\t%{GREEDYDATA:qtype}\t%{GREEDYDATA:qtype_name}\t%{GREEDYDATA:rcode}\t%{GREEDYDATA:rcode_name}\t%{GREEDYDATA:AA}\t%{GREEDYDATA:TC}\t%{GREEDYDATA:RD}\t%{GREEDYDATA:RA}\t%{GREEDYDATA:Z}\t%{GREEDYDATA:answers}\t%{GREEDYDATA:TTLs}\t%{GREEDYDATA:rejected}
|
||||
|
||||
# conn.log
|
||||
BRO_CONN %{NUMBER:ts}\t%{NOTSPACE:uid}\t%{IP:orig_h}\t%{INT:orig_p}\t%{IP:resp_h}\t%{INT:resp_p}\t%{WORD:proto}\t%{GREEDYDATA:service}\t%{NUMBER:duration}\t%{NUMBER:orig_bytes}\t%{NUMBER:resp_bytes}\t%{GREEDYDATA:conn_state}\t%{GREEDYDATA:local_orig}\t%{GREEDYDATA:missed_bytes}\t%{GREEDYDATA:history}\t%{GREEDYDATA:orig_pkts}\t%{GREEDYDATA:orig_ip_bytes}\t%{GREEDYDATA:resp_pkts}\t%{GREEDYDATA:resp_ip_bytes}\t%{GREEDYDATA:tunnel_parents}
|
||||
|
||||
# files.log
|
||||
BRO_FILES %{NUMBER:ts}\t%{NOTSPACE:fuid}\t%{IP:tx_hosts}\t%{IP:rx_hosts}\t%{NOTSPACE:conn_uids}\t%{GREEDYDATA:source}\t%{GREEDYDATA:depth}\t%{GREEDYDATA:analyzers}\t%{GREEDYDATA:mime_type}\t%{GREEDYDATA:filename}\t%{GREEDYDATA:duration}\t%{GREEDYDATA:local_orig}\t%{GREEDYDATA:is_orig}\t%{GREEDYDATA:seen_bytes}\t%{GREEDYDATA:total_bytes}\t%{GREEDYDATA:missing_bytes}\t%{GREEDYDATA:overflow_bytes}\t%{GREEDYDATA:timedout}\t%{GREEDYDATA:parent_fuid}\t%{GREEDYDATA:md5}\t%{GREEDYDATA:sha1}\t%{GREEDYDATA:sha256}\t%{GREEDYDATA:extracted}
|
||||
|
|
@ -0,0 +1,13 @@
|
|||
EXIM_MSGID [0-9A-Za-z]{6}-[0-9A-Za-z]{6}-[0-9A-Za-z]{2}
|
||||
EXIM_FLAGS (<=|[-=>*]>|[*]{2}|==)
|
||||
EXIM_DATE %{YEAR:exim_year}-%{MONTHNUM:exim_month}-%{MONTHDAY:exim_day} %{TIME:exim_time}
|
||||
EXIM_PID \[%{POSINT}\]
|
||||
EXIM_QT ((\d+y)?(\d+w)?(\d+d)?(\d+h)?(\d+m)?(\d+s)?)
|
||||
EXIM_EXCLUDE_TERMS (Message is frozen|(Start|End) queue run| Warning: | retry time not reached | no (IP address|host name) found for (IP address|host) | unexpected disconnection while reading SMTP command | no immediate delivery: |another process is handling this message)
|
||||
EXIM_REMOTE_HOST (H=(%{NOTSPACE:remote_hostname} )?(\(%{NOTSPACE:remote_heloname}\) )?\[%{IP:remote_host}\])
|
||||
EXIM_INTERFACE (I=\[%{IP:exim_interface}\](:%{NUMBER:exim_interface_port}))
|
||||
EXIM_PROTOCOL (P=%{NOTSPACE:protocol})
|
||||
EXIM_MSG_SIZE (S=%{NUMBER:exim_msg_size})
|
||||
EXIM_HEADER_ID (id=%{NOTSPACE:exim_header_id})
|
||||
EXIM_SUBJECT (T=%{QS:exim_subject})
|
||||
|
||||
|
|
@ -0,0 +1,86 @@
|
|||
# NetScreen firewall logs
|
||||
NETSCREENSESSIONLOG %{SYSLOGTIMESTAMP:date} %{IPORHOST:device} %{IPORHOST}: NetScreen device_id=%{WORD:device_id}%{DATA}: start_time=%{QUOTEDSTRING:start_time} duration=%{INT:duration} policy_id=%{INT:policy_id} service=%{DATA:service} proto=%{INT:proto} src zone=%{WORD:src_zone} dst zone=%{WORD:dst_zone} action=%{WORD:action} sent=%{INT:sent} rcvd=%{INT:rcvd} src=%{IPORHOST:src_ip} dst=%{IPORHOST:dst_ip} src_port=%{INT:src_port} dst_port=%{INT:dst_port} src-xlated ip=%{IPORHOST:src_xlated_ip} port=%{INT:src_xlated_port} dst-xlated ip=%{IPORHOST:dst_xlated_ip} port=%{INT:dst_xlated_port} session_id=%{INT:session_id} reason=%{GREEDYDATA:reason}
|
||||
|
||||
#== Cisco ASA ==
|
||||
CISCO_TAGGED_SYSLOG ^<%{POSINT:syslog_pri}>%{CISCOTIMESTAMP:timestamp}( %{SYSLOGHOST:sysloghost})? ?: %%{CISCOTAG:ciscotag}:
|
||||
CISCOTIMESTAMP %{MONTH} +%{MONTHDAY}(?: %{YEAR})? %{TIME}
|
||||
CISCOTAG [A-Z0-9]+-%{INT}-(?:[A-Z0-9_]+)
|
||||
# Common Particles
|
||||
CISCO_ACTION Built|Teardown|Deny|Denied|denied|requested|permitted|denied by ACL|discarded|est-allowed|Dropping|created|deleted
|
||||
CISCO_REASON Duplicate TCP SYN|Failed to locate egress interface|Invalid transport field|No matching connection|DNS Response|DNS Query|(?:%{WORD}\s*)*
|
||||
CISCO_DIRECTION Inbound|inbound|Outbound|outbound
|
||||
CISCO_INTERVAL first hit|%{INT}-second interval
|
||||
CISCO_XLATE_TYPE static|dynamic
|
||||
# ASA-1-104001
|
||||
CISCOFW104001 \((?:Primary|Secondary)\) Switching to ACTIVE - %{GREEDYDATA:switch_reason}
|
||||
# ASA-1-104002
|
||||
CISCOFW104002 \((?:Primary|Secondary)\) Switching to STANDBY - %{GREEDYDATA:switch_reason}
|
||||
# ASA-1-104003
|
||||
CISCOFW104003 \((?:Primary|Secondary)\) Switching to FAILED\.
|
||||
# ASA-1-104004
|
||||
CISCOFW104004 \((?:Primary|Secondary)\) Switching to OK\.
|
||||
# ASA-1-105003
|
||||
CISCOFW105003 \((?:Primary|Secondary)\) Monitoring on [Ii]nterface %{GREEDYDATA:interface_name} waiting
|
||||
# ASA-1-105004
|
||||
CISCOFW105004 \((?:Primary|Secondary)\) Monitoring on [Ii]nterface %{GREEDYDATA:interface_name} normal
|
||||
# ASA-1-105005
|
||||
CISCOFW105005 \((?:Primary|Secondary)\) Lost Failover communications with mate on [Ii]nterface %{GREEDYDATA:interface_name}
|
||||
# ASA-1-105008
|
||||
CISCOFW105008 \((?:Primary|Secondary)\) Testing [Ii]nterface %{GREEDYDATA:interface_name}
|
||||
# ASA-1-105009
|
||||
CISCOFW105009 \((?:Primary|Secondary)\) Testing on [Ii]nterface %{GREEDYDATA:interface_name} (?:Passed|Failed)
|
||||
# ASA-2-106001
|
||||
CISCOFW106001 %{CISCO_DIRECTION:direction} %{WORD:protocol} connection %{CISCO_ACTION:action} from %{IP:src_ip}/%{INT:src_port} to %{IP:dst_ip}/%{INT:dst_port} flags %{GREEDYDATA:tcp_flags} on interface %{GREEDYDATA:interface}
|
||||
# ASA-2-106006, ASA-2-106007, ASA-2-106010
|
||||
CISCOFW106006_106007_106010 %{CISCO_ACTION:action} %{CISCO_DIRECTION:direction} %{WORD:protocol} (?:from|src) %{IP:src_ip}/%{INT:src_port}(\(%{DATA:src_fwuser}\))? (?:to|dst) %{IP:dst_ip}/%{INT:dst_port}(\(%{DATA:dst_fwuser}\))? (?:on interface %{DATA:interface}|due to %{CISCO_REASON:reason})
|
||||
# ASA-3-106014
|
||||
CISCOFW106014 %{CISCO_ACTION:action} %{CISCO_DIRECTION:direction} %{WORD:protocol} src %{DATA:src_interface}:%{IP:src_ip}(\(%{DATA:src_fwuser}\))? dst %{DATA:dst_interface}:%{IP:dst_ip}(\(%{DATA:dst_fwuser}\))? \(type %{INT:icmp_type}, code %{INT:icmp_code}\)
|
||||
# ASA-6-106015
|
||||
CISCOFW106015 %{CISCO_ACTION:action} %{WORD:protocol} \(%{DATA:policy_id}\) from %{IP:src_ip}/%{INT:src_port} to %{IP:dst_ip}/%{INT:dst_port} flags %{DATA:tcp_flags} on interface %{GREEDYDATA:interface}
|
||||
# ASA-1-106021
|
||||
CISCOFW106021 %{CISCO_ACTION:action} %{WORD:protocol} reverse path check from %{IP:src_ip} to %{IP:dst_ip} on interface %{GREEDYDATA:interface}
|
||||
# ASA-4-106023
|
||||
CISCOFW106023 %{CISCO_ACTION:action}( protocol)? %{WORD:protocol} src %{DATA:src_interface}:%{DATA:src_ip}(/%{INT:src_port})?(\(%{DATA:src_fwuser}\))? dst %{DATA:dst_interface}:%{DATA:dst_ip}(/%{INT:dst_port})?(\(%{DATA:dst_fwuser}\))?( \(type %{INT:icmp_type}, code %{INT:icmp_code}\))? by access-group "?%{DATA:policy_id}"? \[%{DATA:hashcode1}, %{DATA:hashcode2}\]
|
||||
# ASA-4-106100, ASA-4-106102, ASA-4-106103
|
||||
CISCOFW106100_2_3 access-list %{NOTSPACE:policy_id} %{CISCO_ACTION:action} %{WORD:protocol} for user '%{DATA:src_fwuser}' %{DATA:src_interface}/%{IP:src_ip}\(%{INT:src_port}\) -> %{DATA:dst_interface}/%{IP:dst_ip}\(%{INT:dst_port}\) hit-cnt %{INT:hit_count} %{CISCO_INTERVAL:interval} \[%{DATA:hashcode1}, %{DATA:hashcode2}\]
|
||||
# ASA-5-106100
|
||||
CISCOFW106100 access-list %{NOTSPACE:policy_id} %{CISCO_ACTION:action} %{WORD:protocol} %{DATA:src_interface}/%{IP:src_ip}\(%{INT:src_port}\)(\(%{DATA:src_fwuser}\))? -> %{DATA:dst_interface}/%{IP:dst_ip}\(%{INT:dst_port}\)(\(%{DATA:src_fwuser}\))? hit-cnt %{INT:hit_count} %{CISCO_INTERVAL:interval} \[%{DATA:hashcode1}, %{DATA:hashcode2}\]
|
||||
# ASA-6-110002
|
||||
CISCOFW110002 %{CISCO_REASON:reason} for %{WORD:protocol} from %{DATA:src_interface}:%{IP:src_ip}/%{INT:src_port} to %{IP:dst_ip}/%{INT:dst_port}
|
||||
# ASA-6-302010
|
||||
CISCOFW302010 %{INT:connection_count} in use, %{INT:connection_count_max} most used
|
||||
# ASA-6-302013, ASA-6-302014, ASA-6-302015, ASA-6-302016
|
||||
CISCOFW302013_302014_302015_302016 %{CISCO_ACTION:action}(?: %{CISCO_DIRECTION:direction})? %{WORD:protocol} connection %{INT:connection_id} for %{DATA:src_interface}:%{IP:src_ip}/%{INT:src_port}( \(%{IP:src_mapped_ip}/%{INT:src_mapped_port}\))?(\(%{DATA:src_fwuser}\))? to %{DATA:dst_interface}:%{IP:dst_ip}/%{INT:dst_port}( \(%{IP:dst_mapped_ip}/%{INT:dst_mapped_port}\))?(\(%{DATA:dst_fwuser}\))?( duration %{TIME:duration} bytes %{INT:bytes})?(?: %{CISCO_REASON:reason})?( \(%{DATA:user}\))?
|
||||
# ASA-6-302020, ASA-6-302021
|
||||
CISCOFW302020_302021 %{CISCO_ACTION:action}(?: %{CISCO_DIRECTION:direction})? %{WORD:protocol} connection for faddr %{IP:dst_ip}/%{INT:icmp_seq_num}(?:\(%{DATA:fwuser}\))? gaddr %{IP:src_xlated_ip}/%{INT:icmp_code_xlated} laddr %{IP:src_ip}/%{INT:icmp_code}( \(%{DATA:user}\))?
|
||||
# ASA-6-305011
|
||||
CISCOFW305011 %{CISCO_ACTION:action} %{CISCO_XLATE_TYPE:xlate_type} %{WORD:protocol} translation from %{DATA:src_interface}:%{IP:src_ip}(/%{INT:src_port})?(\(%{DATA:src_fwuser}\))? to %{DATA:src_xlated_interface}:%{IP:src_xlated_ip}/%{DATA:src_xlated_port}
|
||||
# ASA-3-313001, ASA-3-313004, ASA-3-313008
|
||||
CISCOFW313001_313004_313008 %{CISCO_ACTION:action} %{WORD:protocol} type=%{INT:icmp_type}, code=%{INT:icmp_code} from %{IP:src_ip} on interface %{DATA:interface}( to %{IP:dst_ip})?
|
||||
# ASA-4-313005
|
||||
CISCOFW313005 %{CISCO_REASON:reason} for %{WORD:protocol} error message: %{WORD:err_protocol} src %{DATA:err_src_interface}:%{IP:err_src_ip}(\(%{DATA:err_src_fwuser}\))? dst %{DATA:err_dst_interface}:%{IP:err_dst_ip}(\(%{DATA:err_dst_fwuser}\))? \(type %{INT:err_icmp_type}, code %{INT:err_icmp_code}\) on %{DATA:interface} interface\. Original IP payload: %{WORD:protocol} src %{IP:orig_src_ip}/%{INT:orig_src_port}(\(%{DATA:orig_src_fwuser}\))? dst %{IP:orig_dst_ip}/%{INT:orig_dst_port}(\(%{DATA:orig_dst_fwuser}\))?
|
||||
# ASA-5-321001
|
||||
CISCOFW321001 Resource '%{WORD:resource_name}' limit of %{POSINT:resource_limit} reached for system
|
||||
# ASA-4-402117
|
||||
CISCOFW402117 %{WORD:protocol}: Received a non-IPSec packet \(protocol= %{WORD:orig_protocol}\) from %{IP:src_ip} to %{IP:dst_ip}
|
||||
# ASA-4-402119
|
||||
CISCOFW402119 %{WORD:protocol}: Received an %{WORD:orig_protocol} packet \(SPI= %{DATA:spi}, sequence number= %{DATA:seq_num}\) from %{IP:src_ip} \(user= %{DATA:user}\) to %{IP:dst_ip} that failed anti-replay checking
|
||||
# ASA-4-419001
|
||||
CISCOFW419001 %{CISCO_ACTION:action} %{WORD:protocol} packet from %{DATA:src_interface}:%{IP:src_ip}/%{INT:src_port} to %{DATA:dst_interface}:%{IP:dst_ip}/%{INT:dst_port}, reason: %{GREEDYDATA:reason}
|
||||
# ASA-4-419002
|
||||
CISCOFW419002 %{CISCO_REASON:reason} from %{DATA:src_interface}:%{IP:src_ip}/%{INT:src_port} to %{DATA:dst_interface}:%{IP:dst_ip}/%{INT:dst_port} with different initial sequence number
|
||||
# ASA-4-500004
|
||||
CISCOFW500004 %{CISCO_REASON:reason} for protocol=%{WORD:protocol}, from %{IP:src_ip}/%{INT:src_port} to %{IP:dst_ip}/%{INT:dst_port}
|
||||
# ASA-6-602303, ASA-6-602304
|
||||
CISCOFW602303_602304 %{WORD:protocol}: An %{CISCO_DIRECTION:direction} %{GREEDYDATA:tunnel_type} SA \(SPI= %{DATA:spi}\) between %{IP:src_ip} and %{IP:dst_ip} \(user= %{DATA:user}\) has been %{CISCO_ACTION:action}
|
||||
# ASA-7-710001, ASA-7-710002, ASA-7-710003, ASA-7-710005, ASA-7-710006
|
||||
CISCOFW710001_710002_710003_710005_710006 %{WORD:protocol} (?:request|access) %{CISCO_ACTION:action} from %{IP:src_ip}/%{INT:src_port} to %{DATA:dst_interface}:%{IP:dst_ip}/%{INT:dst_port}
|
||||
# ASA-6-713172
|
||||
CISCOFW713172 Group = %{GREEDYDATA:group}, IP = %{IP:src_ip}, Automatic NAT Detection Status:\s+Remote end\s*%{DATA:is_remote_natted}\s*behind a NAT device\s+This\s+end\s*%{DATA:is_local_natted}\s*behind a NAT device
|
||||
# ASA-4-733100
|
||||
CISCOFW733100 \[\s*%{DATA:drop_type}\s*\] drop %{DATA:drop_rate_id} exceeded. Current burst rate is %{INT:drop_rate_current_burst} per second, max configured rate is %{INT:drop_rate_max_burst}; Current average rate is %{INT:drop_rate_current_avg} per second, max configured rate is %{INT:drop_rate_max_avg}; Cumulative total count is %{INT:drop_total_count}
|
||||
#== End Cisco ASA ==
|
||||
|
||||
# Shorewall firewall logs
|
||||
SHOREWALL (%{SYSLOGTIMESTAMP:timestamp}) (%{WORD:nf_host}) kernel:.*Shorewall:(%{WORD:nf_action1})?:(%{WORD:nf_action2})?.*IN=(%{USERNAME:nf_in_interface})?.*(OUT= *MAC=(%{COMMONMAC:nf_dst_mac}):(%{COMMONMAC:nf_src_mac})?|OUT=%{USERNAME:nf_out_interface}).*SRC=(%{IPV4:nf_src_ip}).*DST=(%{IPV4:nf_dst_ip}).*LEN=(%{WORD:nf_len}).?*TOS=(%{WORD:nf_tos}).?*PREC=(%{WORD:nf_prec}).?*TTL=(%{INT:nf_ttl}).?*ID=(%{INT:nf_id}).?*PROTO=(%{WORD:nf_protocol}).?*SPT=(%{INT:nf_src_port}?.*DPT=%{INT:nf_dst_port}?.*)
|
||||
#== End Shorewall
|
||||
|
|
@ -0,0 +1,102 @@
|
|||
USERNAME [a-zA-Z0-9._-]+
|
||||
USER %{USERNAME}
|
||||
EMAILLOCALPART [a-zA-Z][a-zA-Z0-9_.+-=:]+
|
||||
EMAILADDRESS %{EMAILLOCALPART}@%{HOSTNAME}
|
||||
HTTPDUSER %{EMAILADDRESS}|%{USER}
|
||||
INT (?:[+-]?(?:[0-9]+))
|
||||
BASE10NUM (?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+)))
|
||||
NUMBER (?:%{BASE10NUM})
|
||||
BASE16NUM (?<![0-9A-Fa-f])(?:[+-]?(?:0x)?(?:[0-9A-Fa-f]+))
|
||||
BASE16FLOAT \b(?<![0-9A-Fa-f.])(?:[+-]?(?:0x)?(?:(?:[0-9A-Fa-f]+(?:\.[0-9A-Fa-f]*)?)|(?:\.[0-9A-Fa-f]+)))\b
|
||||
|
||||
POSINT \b(?:[1-9][0-9]*)\b
|
||||
NONNEGINT \b(?:[0-9]+)\b
|
||||
WORD \b\w+\b
|
||||
NOTSPACE \S+
|
||||
SPACE \s*
|
||||
DATA .*?
|
||||
GREEDYDATA .*
|
||||
QUOTEDSTRING (?>(?<!\\)(?>"(?>\\.|[^\\"]+)+"|""|(?>'(?>\\.|[^\\']+)+')|''|(?>`(?>\\.|[^\\`]+)+`)|``))
|
||||
UUID [A-Fa-f0-9]{8}-(?:[A-Fa-f0-9]{4}-){3}[A-Fa-f0-9]{12}
|
||||
|
||||
# Networking
|
||||
MAC (?:%{CISCOMAC}|%{WINDOWSMAC}|%{COMMONMAC})
|
||||
CISCOMAC (?:(?:[A-Fa-f0-9]{4}\.){2}[A-Fa-f0-9]{4})
|
||||
WINDOWSMAC (?:(?:[A-Fa-f0-9]{2}-){5}[A-Fa-f0-9]{2})
|
||||
COMMONMAC (?:(?:[A-Fa-f0-9]{2}:){5}[A-Fa-f0-9]{2})
|
||||
IPV6 ((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:)))(%.+)?
|
||||
IPV4 (?<![0-9])(?:(?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5]))(?![0-9])
|
||||
IP (?:%{IPV6}|%{IPV4})
|
||||
HOSTNAME \b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\.?|\b)
|
||||
IPORHOST (?:%{IP}|%{HOSTNAME})
|
||||
HOSTPORT %{IPORHOST}:%{POSINT}
|
||||
|
||||
# paths
|
||||
PATH (?:%{UNIXPATH}|%{WINPATH})
|
||||
UNIXPATH (/([\w_%!$@:.,~-]+|\\.)*)+
|
||||
TTY (?:/dev/(pts|tty([pq])?)(\w+)?/?(?:[0-9]+))
|
||||
WINPATH (?>[A-Za-z]+:|\\)(?:\\[^\\?*]*)+
|
||||
URIPROTO [A-Za-z]+(\+[A-Za-z+]+)?
|
||||
URIHOST %{IPORHOST}(?::%{POSINT:port})?
|
||||
# uripath comes loosely from RFC1738, but mostly from what Firefox
|
||||
# doesn't turn into %XX
|
||||
URIPATH (?:/[A-Za-z0-9$.+!*'(){},~:;=@#%_\-]*)+
|
||||
#URIPARAM \?(?:[A-Za-z0-9]+(?:=(?:[^&]*))?(?:&(?:[A-Za-z0-9]+(?:=(?:[^&]*))?)?)*)?
|
||||
URIPARAM \?[A-Za-z0-9$.+!*'|(){},~@#%&/=:;_?\-\[\]<>]*
|
||||
URIPATHPARAM %{URIPATH}(?:%{URIPARAM})?
|
||||
URI %{URIPROTO}://(?:%{USER}(?::[^@]*)?@)?(?:%{URIHOST})?(?:%{URIPATHPARAM})?
|
||||
|
||||
# Months: January, Feb, 3, 03, 12, December
|
||||
MONTH \b(?:Jan(?:uary|uar)?|Feb(?:ruary|ruar)?|M(?:a|ä)?r(?:ch|z)?|Apr(?:il)?|Ma(?:y|i)?|Jun(?:e|i)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|O(?:c|k)?t(?:ober)?|Nov(?:ember)?|De(?:c|z)(?:ember)?)\b
|
||||
MONTHNUM (?:0?[1-9]|1[0-2])
|
||||
MONTHNUM2 (?:0[1-9]|1[0-2])
|
||||
MONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])
|
||||
|
||||
# Days: Monday, Tue, Thu, etc...
|
||||
DAY (?:Mon(?:day)?|Tue(?:sday)?|Wed(?:nesday)?|Thu(?:rsday)?|Fri(?:day)?|Sat(?:urday)?|Sun(?:day)?)
|
||||
|
||||
# Years?
|
||||
YEAR (?>\d\d){1,2}
|
||||
HOUR (?:2[0123]|[01]?[0-9])
|
||||
MINUTE (?:[0-5][0-9])
|
||||
# '60' is a leap second in most time standards and thus is valid.
|
||||
SECOND (?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)
|
||||
TIME (?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9])
|
||||
# datestamp is YYYY/MM/DD-HH:MM:SS.UUUU (or something like it)
|
||||
DATE_US %{MONTHNUM}[/-]%{MONTHDAY}[/-]%{YEAR}
|
||||
DATE_EU %{MONTHDAY}[./-]%{MONTHNUM}[./-]%{YEAR}
|
||||
ISO8601_TIMEZONE (?:Z|[+-]%{HOUR}(?::?%{MINUTE}))
|
||||
ISO8601_SECOND (?:%{SECOND}|60)
|
||||
ISO8601_HOUR (?:2[0123]|[01][0-9])
|
||||
TIMESTAMP_ISO8601 %{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{ISO8601_HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE}?
|
||||
DATE %{DATE_US}|%{DATE_EU}
|
||||
DATESTAMP %{DATE}[- ]%{TIME}
|
||||
TZ (?:[PMCE][SD]T|UTC)
|
||||
DATESTAMP_RFC822 %{DAY} %{MONTH} %{MONTHDAY} %{YEAR} %{TIME} %{TZ}
|
||||
DATESTAMP_RFC2822 %{DAY}, %{MONTHDAY} %{MONTH} %{YEAR} %{TIME} %{ISO8601_TIMEZONE}
|
||||
DATESTAMP_OTHER %{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{TZ} %{YEAR}
|
||||
DATESTAMP_EVENTLOG %{YEAR}%{MONTHNUM2}%{MONTHDAY}%{HOUR}%{MINUTE}%{SECOND}
|
||||
HTTPDERROR_DATE %{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR}
|
||||
|
||||
# Syslog Dates: Month Day HH:MM:SS
|
||||
SYSLOGTIMESTAMP %{MONTH} +%{MONTHDAY} %{TIME}
|
||||
PROG [\x21-\x5a\x5c\x5e-\x7e]+
|
||||
SYSLOGPROG %{PROG:program}(?:\[%{POSINT:pid}\])?
|
||||
SYSLOGHOST %{IPORHOST}
|
||||
SYSLOGFACILITY <%{NONNEGINT:facility}.%{NONNEGINT:priority}>
|
||||
HTTPDATE %{MONTHDAY}/%{MONTH}/%{YEAR}:%{TIME} %{INT}
|
||||
|
||||
# Shortcuts
|
||||
QS %{QUOTEDSTRING}
|
||||
|
||||
# Log formats
|
||||
SYSLOGBASE %{SYSLOGTIMESTAMP:timestamp} (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}:
|
||||
COMMONAPACHELOG %{IPORHOST:clientip} %{HTTPDUSER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)
|
||||
COMBINEDAPACHELOG %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}
|
||||
HTTPD20_ERRORLOG \[%{HTTPDERROR_DATE:timestamp}\] \[%{LOGLEVEL:loglevel}\] (?:\[client %{IPORHOST:clientip}\] ){0,1}%{GREEDYDATA:errormsg}
|
||||
HTTPD24_ERRORLOG \[%{HTTPDERROR_DATE:timestamp}\] \[%{WORD:module}:%{LOGLEVEL:loglevel}\] \[pid %{POSINT:pid}:tid %{NUMBER:tid}\]( \(%{POSINT:proxy_errorcode}\)%{DATA:proxy_errormessage}:)?( \[client %{IPORHOST:client}:%{POSINT:clientport}\])? %{DATA:errorcode}: %{GREEDYDATA:message}
|
||||
HTTPD_ERRORLOG %{HTTPD20_ERRORLOG}|%{HTTPD24_ERRORLOG}
|
||||
|
||||
|
||||
# Log Levels
|
||||
LOGLEVEL ([Aa]lert|ALERT|[Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nfo|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|[Ee]rr?(?:or)?|ERR?(?:OR)?|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?)
|
||||
|
|
@ -0,0 +1,39 @@
|
|||
## These patterns were tested w/ haproxy-1.4.15
|
||||
|
||||
## Documentation of the haproxy log formats can be found at the following links:
|
||||
## http://code.google.com/p/haproxy-docs/wiki/HTTPLogFormat
|
||||
## http://code.google.com/p/haproxy-docs/wiki/TCPLogFormat
|
||||
|
||||
HAPROXYTIME (?!<[0-9])%{HOUR:haproxy_hour}:%{MINUTE:haproxy_minute}(?::%{SECOND:haproxy_second})(?![0-9])
|
||||
HAPROXYDATE %{MONTHDAY:haproxy_monthday}/%{MONTH:haproxy_month}/%{YEAR:haproxy_year}:%{HAPROXYTIME:haproxy_time}.%{INT:haproxy_milliseconds}
|
||||
|
||||
# Override these default patterns to parse out what is captured in your haproxy.cfg
|
||||
HAPROXYCAPTUREDREQUESTHEADERS %{DATA:captured_request_headers}
|
||||
HAPROXYCAPTUREDRESPONSEHEADERS %{DATA:captured_response_headers}
|
||||
|
||||
# Example:
|
||||
# These haproxy config lines will add data to the logs that are captured
|
||||
# by the patterns below. Place them in your custom patterns directory to
|
||||
# override the defaults.
|
||||
#
|
||||
# capture request header Host len 40
|
||||
# capture request header X-Forwarded-For len 50
|
||||
# capture request header Accept-Language len 50
|
||||
# capture request header Referer len 200
|
||||
# capture request header User-Agent len 200
|
||||
#
|
||||
# capture response header Content-Type len 30
|
||||
# capture response header Content-Encoding len 10
|
||||
# capture response header Cache-Control len 200
|
||||
# capture response header Last-Modified len 200
|
||||
#
|
||||
# HAPROXYCAPTUREDREQUESTHEADERS %{DATA:request_header_host}\|%{DATA:request_header_x_forwarded_for}\|%{DATA:request_header_accept_language}\|%{DATA:request_header_referer}\|%{DATA:request_header_user_agent}
|
||||
# HAPROXYCAPTUREDRESPONSEHEADERS %{DATA:response_header_content_type}\|%{DATA:response_header_content_encoding}\|%{DATA:response_header_cache_control}\|%{DATA:response_header_last_modified}
|
||||
|
||||
# parse a haproxy 'httplog' line
|
||||
HAPROXYHTTPBASE %{IP:client_ip}:%{INT:client_port} \[%{HAPROXYDATE:accept_date}\] %{NOTSPACE:frontend_name} %{NOTSPACE:backend_name}/%{NOTSPACE:server_name} %{INT:time_request}/%{INT:time_queue}/%{INT:time_backend_connect}/%{INT:time_backend_response}/%{NOTSPACE:time_duration} %{INT:http_status_code} %{NOTSPACE:bytes_read} %{DATA:captured_request_cookie} %{DATA:captured_response_cookie} %{NOTSPACE:termination_state} %{INT:actconn}/%{INT:feconn}/%{INT:beconn}/%{INT:srvconn}/%{NOTSPACE:retries} %{INT:srv_queue}/%{INT:backend_queue} (\{%{HAPROXYCAPTUREDREQUESTHEADERS}\})?( )?(\{%{HAPROXYCAPTUREDRESPONSEHEADERS}\})?( )?"(<BADREQ>|(%{WORD:http_verb} (%{URIPROTO:http_proto}://)?(?:%{USER:http_user}(?::[^@]*)?@)?(?:%{URIHOST:http_host})?(?:%{URIPATHPARAM:http_request})?( HTTP/%{NUMBER:http_version})?))?"
|
||||
|
||||
HAPROXYHTTP (?:%{SYSLOGTIMESTAMP:syslog_timestamp}|%{TIMESTAMP_ISO8601:timestamp8601}) %{IPORHOST:syslog_server} %{SYSLOGPROG}: %{HAPROXYHTTPBASE}
|
||||
|
||||
# parse a haproxy 'tcplog' line
|
||||
HAPROXYTCP (?:%{SYSLOGTIMESTAMP:syslog_timestamp}|%{TIMESTAMP_ISO8601:timestamp8601}) %{IPORHOST:syslog_server} %{SYSLOGPROG}: %{IP:client_ip}:%{INT:client_port} \[%{HAPROXYDATE:accept_date}\] %{NOTSPACE:frontend_name} %{NOTSPACE:backend_name}/%{NOTSPACE:server_name} %{INT:time_queue}/%{INT:time_backend_connect}/%{NOTSPACE:time_duration} %{NOTSPACE:bytes_read} %{NOTSPACE:termination_state} %{INT:actconn}/%{INT:feconn}/%{INT:beconn}/%{INT:srvconn}/%{NOTSPACE:retries} %{INT:srv_queue}/%{INT:backend_queue}
|
||||
|
|
@ -0,0 +1,20 @@
|
|||
JAVACLASS (?:[a-zA-Z$_][a-zA-Z$_0-9]*\.)*[a-zA-Z$_][a-zA-Z$_0-9]*
|
||||
#Space is an allowed character to match special cases like 'Native Method' or 'Unknown Source'
|
||||
JAVAFILE (?:[A-Za-z0-9_. -]+)
|
||||
#Allow special <init> method
|
||||
JAVAMETHOD (?:(<init>)|[a-zA-Z$_][a-zA-Z$_0-9]*)
|
||||
#Line number is optional in special cases 'Native method' or 'Unknown source'
|
||||
JAVASTACKTRACEPART %{SPACE}at %{JAVACLASS:class}\.%{JAVAMETHOD:method}\(%{JAVAFILE:file}(?::%{NUMBER:line})?\)
|
||||
# Java Logs
|
||||
JAVATHREAD (?:[A-Z]{2}-Processor[\d]+)
|
||||
JAVACLASS (?:[a-zA-Z0-9-]+\.)+[A-Za-z0-9$]+
|
||||
JAVAFILE (?:[A-Za-z0-9_.-]+)
|
||||
JAVASTACKTRACEPART at %{JAVACLASS:class}\.%{WORD:method}\(%{JAVAFILE:file}:%{NUMBER:line}\)
|
||||
JAVALOGMESSAGE (.*)
|
||||
# MMM dd, yyyy HH:mm:ss eg: Jan 9, 2014 7:13:13 AM
|
||||
CATALINA_DATESTAMP %{MONTH} %{MONTHDAY}, 20%{YEAR} %{HOUR}:?%{MINUTE}(?::?%{SECOND}) (?:AM|PM)
|
||||
# yyyy-MM-dd HH:mm:ss,SSS ZZZ eg: 2014-01-09 17:32:25,527 -0800
|
||||
TOMCAT_DATESTAMP 20%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:?%{MINUTE}(?::?%{SECOND}) %{ISO8601_TIMEZONE}
|
||||
CATALINALOG %{CATALINA_DATESTAMP:timestamp} %{JAVACLASS:class} %{JAVALOGMESSAGE:logmessage}
|
||||
# 2014-01-09 20:03:28,269 -0800 | ERROR | com.example.service.ExampleService - something compeletely unexpected happened...
|
||||
TOMCATLOG %{TOMCAT_DATESTAMP:timestamp} \| %{LOGLEVEL:level} \| %{JAVACLASS:class} - %{JAVALOGMESSAGE:logmessage}
|
||||
|
|
@ -0,0 +1,9 @@
|
|||
# JUNOS 11.4 RT_FLOW patterns
|
||||
RT_FLOW_EVENT (RT_FLOW_SESSION_CREATE|RT_FLOW_SESSION_CLOSE|RT_FLOW_SESSION_DENY)
|
||||
|
||||
RT_FLOW1 %{RT_FLOW_EVENT:event}: %{GREEDYDATA:close-reason}: %{IP:src-ip}/%{INT:src-port}->%{IP:dst-ip}/%{INT:dst-port} %{DATA:service} %{IP:nat-src-ip}/%{INT:nat-src-port}->%{IP:nat-dst-ip}/%{INT:nat-dst-port} %{DATA:src-nat-rule-name} %{DATA:dst-nat-rule-name} %{INT:protocol-id} %{DATA:policy-name} %{DATA:from-zone} %{DATA:to-zone} %{INT:session-id} \d+\(%{DATA:sent}\) \d+\(%{DATA:received}\) %{INT:elapsed-time} .*
|
||||
|
||||
RT_FLOW2 %{RT_FLOW_EVENT:event}: session created %{IP:src-ip}/%{INT:src-port}->%{IP:dst-ip}/%{INT:dst-port} %{DATA:service} %{IP:nat-src-ip}/%{INT:nat-src-port}->%{IP:nat-dst-ip}/%{INT:nat-dst-port} %{DATA:src-nat-rule-name} %{DATA:dst-nat-rule-name} %{INT:protocol-id} %{DATA:policy-name} %{DATA:from-zone} %{DATA:to-zone} %{INT:session-id} .*
|
||||
|
||||
RT_FLOW3 %{RT_FLOW_EVENT:event}: session denied %{IP:src-ip}/%{INT:src-port}->%{IP:dst-ip}/%{INT:dst-port} %{DATA:service} %{INT:protocol-id}\(\d\) %{DATA:policy-name} %{DATA:from-zone} %{DATA:to-zone} .*
|
||||
|
||||
|
|
@ -0,0 +1,16 @@
|
|||
SYSLOG5424PRINTASCII [!-~]+
|
||||
|
||||
SYSLOGBASE2 (?:%{SYSLOGTIMESTAMP:timestamp}|%{TIMESTAMP_ISO8601:timestamp8601}) (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource}+(?: %{SYSLOGPROG}:|)
|
||||
SYSLOGPAMSESSION %{SYSLOGBASE} (?=%{GREEDYDATA:message})%{WORD:pam_module}\(%{DATA:pam_caller}\): session %{WORD:pam_session_state} for user %{USERNAME:username}(?: by %{GREEDYDATA:pam_by})?
|
||||
|
||||
CRON_ACTION [A-Z ]+
|
||||
CRONLOG %{SYSLOGBASE} \(%{USER:user}\) %{CRON_ACTION:action} \(%{DATA:message}\)
|
||||
|
||||
SYSLOGLINE %{SYSLOGBASE2} %{GREEDYDATA:message}
|
||||
|
||||
# IETF 5424 syslog(8) format (see http://www.rfc-editor.org/info/rfc5424)
|
||||
SYSLOG5424PRI <%{NONNEGINT:syslog5424_pri}>
|
||||
SYSLOG5424SD \[%{DATA}\]+
|
||||
SYSLOG5424BASE %{SYSLOG5424PRI}%{NONNEGINT:syslog5424_ver} +(?:%{TIMESTAMP_ISO8601:syslog5424_ts}|-) +(?:%{HOSTNAME:syslog5424_host}|-) +(-|%{SYSLOG5424PRINTASCII:syslog5424_app}) +(-|%{SYSLOG5424PRINTASCII:syslog5424_proc}) +(-|%{SYSLOG5424PRINTASCII:syslog5424_msgid}) +(?:%{SYSLOG5424SD:syslog5424_sd}|-|)
|
||||
|
||||
SYSLOG5424LINE %{SYSLOG5424BASE} +%{GREEDYDATA:syslog5424_msg}
|
||||
|
|
@ -0,0 +1,4 @@
|
|||
# Remember, these can be multi-line events.
|
||||
MCOLLECTIVE ., \[%{TIMESTAMP_ISO8601:timestamp} #%{POSINT:pid}\]%{SPACE}%{LOGLEVEL:event_level}
|
||||
|
||||
MCOLLECTIVEAUDIT %{TIMESTAMP_ISO8601:timestamp}:
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
MONGO_LOG %{SYSLOGTIMESTAMP:timestamp} \[%{WORD:component}\] %{GREEDYDATA:message}
|
||||
MONGO_QUERY \{ (?<={ ).*(?= } ntoreturn:) \}
|
||||
MONGO_SLOWQUERY %{WORD} %{MONGO_WORDDASH:database}\.%{MONGO_WORDDASH:collection} %{WORD}: %{MONGO_QUERY:query} %{WORD}:%{NONNEGINT:ntoreturn} %{WORD}:%{NONNEGINT:ntoskip} %{WORD}:%{NONNEGINT:nscanned}.*nreturned:%{NONNEGINT:nreturned}..+ (?<duration>[0-9]+)ms
|
||||
MONGO_WORDDASH \b[\w-]+\b
|
||||
MONGO3_SEVERITY \w
|
||||
MONGO3_COMPONENT %{WORD}|-
|
||||
MONGO3_LOG %{TIMESTAMP_ISO8601:timestamp} %{MONGO3_SEVERITY:severity} %{MONGO3_COMPONENT:component}%{SPACE}(?:\[%{DATA:context}\])? %{GREEDYDATA:message}
|
||||
|
|
@ -0,0 +1,124 @@
|
|||
##################################################################################
|
||||
##################################################################################
|
||||
# Chop Nagios log files to smithereens!
|
||||
#
|
||||
# A set of GROK filters to process logfiles generated by Nagios.
|
||||
# While it does not, this set intends to cover all possible Nagios logs.
|
||||
#
|
||||
# Some more work needs to be done to cover all External Commands:
|
||||
# http://old.nagios.org/developerinfo/externalcommands/commandlist.php
|
||||
#
|
||||
# If you need some support on these rules please contact:
|
||||
# Jelle Smet http://smetj.net
|
||||
#
|
||||
#################################################################################
|
||||
#################################################################################
|
||||
|
||||
NAGIOSTIME \[%{NUMBER:nagios_epoch}\]
|
||||
|
||||
###############################################
|
||||
######## Begin nagios log types
|
||||
###############################################
|
||||
NAGIOS_TYPE_CURRENT_SERVICE_STATE CURRENT SERVICE STATE
|
||||
NAGIOS_TYPE_CURRENT_HOST_STATE CURRENT HOST STATE
|
||||
|
||||
NAGIOS_TYPE_SERVICE_NOTIFICATION SERVICE NOTIFICATION
|
||||
NAGIOS_TYPE_HOST_NOTIFICATION HOST NOTIFICATION
|
||||
|
||||
NAGIOS_TYPE_SERVICE_ALERT SERVICE ALERT
|
||||
NAGIOS_TYPE_HOST_ALERT HOST ALERT
|
||||
|
||||
NAGIOS_TYPE_SERVICE_FLAPPING_ALERT SERVICE FLAPPING ALERT
|
||||
NAGIOS_TYPE_HOST_FLAPPING_ALERT HOST FLAPPING ALERT
|
||||
|
||||
NAGIOS_TYPE_SERVICE_DOWNTIME_ALERT SERVICE DOWNTIME ALERT
|
||||
NAGIOS_TYPE_HOST_DOWNTIME_ALERT HOST DOWNTIME ALERT
|
||||
|
||||
NAGIOS_TYPE_PASSIVE_SERVICE_CHECK PASSIVE SERVICE CHECK
|
||||
NAGIOS_TYPE_PASSIVE_HOST_CHECK PASSIVE HOST CHECK
|
||||
|
||||
NAGIOS_TYPE_SERVICE_EVENT_HANDLER SERVICE EVENT HANDLER
|
||||
NAGIOS_TYPE_HOST_EVENT_HANDLER HOST EVENT HANDLER
|
||||
|
||||
NAGIOS_TYPE_EXTERNAL_COMMAND EXTERNAL COMMAND
|
||||
NAGIOS_TYPE_TIMEPERIOD_TRANSITION TIMEPERIOD TRANSITION
|
||||
###############################################
|
||||
######## End nagios log types
|
||||
###############################################
|
||||
|
||||
###############################################
|
||||
######## Begin external check types
|
||||
###############################################
|
||||
NAGIOS_EC_DISABLE_SVC_CHECK DISABLE_SVC_CHECK
|
||||
NAGIOS_EC_ENABLE_SVC_CHECK ENABLE_SVC_CHECK
|
||||
NAGIOS_EC_DISABLE_HOST_CHECK DISABLE_HOST_CHECK
|
||||
NAGIOS_EC_ENABLE_HOST_CHECK ENABLE_HOST_CHECK
|
||||
NAGIOS_EC_PROCESS_SERVICE_CHECK_RESULT PROCESS_SERVICE_CHECK_RESULT
|
||||
NAGIOS_EC_PROCESS_HOST_CHECK_RESULT PROCESS_HOST_CHECK_RESULT
|
||||
NAGIOS_EC_SCHEDULE_SERVICE_DOWNTIME SCHEDULE_SERVICE_DOWNTIME
|
||||
NAGIOS_EC_SCHEDULE_HOST_DOWNTIME SCHEDULE_HOST_DOWNTIME
|
||||
NAGIOS_EC_DISABLE_HOST_SVC_NOTIFICATIONS DISABLE_HOST_SVC_NOTIFICATIONS
|
||||
NAGIOS_EC_ENABLE_HOST_SVC_NOTIFICATIONS ENABLE_HOST_SVC_NOTIFICATIONS
|
||||
NAGIOS_EC_DISABLE_HOST_NOTIFICATIONS DISABLE_HOST_NOTIFICATIONS
|
||||
NAGIOS_EC_ENABLE_HOST_NOTIFICATIONS ENABLE_HOST_NOTIFICATIONS
|
||||
NAGIOS_EC_DISABLE_SVC_NOTIFICATIONS DISABLE_SVC_NOTIFICATIONS
|
||||
NAGIOS_EC_ENABLE_SVC_NOTIFICATIONS ENABLE_SVC_NOTIFICATIONS
|
||||
###############################################
|
||||
######## End external check types
|
||||
###############################################
|
||||
NAGIOS_WARNING Warning:%{SPACE}%{GREEDYDATA:nagios_message}
|
||||
|
||||
NAGIOS_CURRENT_SERVICE_STATE %{NAGIOS_TYPE_CURRENT_SERVICE_STATE:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_service};%{DATA:nagios_state};%{DATA:nagios_statetype};%{DATA:nagios_statecode};%{GREEDYDATA:nagios_message}
|
||||
NAGIOS_CURRENT_HOST_STATE %{NAGIOS_TYPE_CURRENT_HOST_STATE:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_state};%{DATA:nagios_statetype};%{DATA:nagios_statecode};%{GREEDYDATA:nagios_message}
|
||||
|
||||
NAGIOS_SERVICE_NOTIFICATION %{NAGIOS_TYPE_SERVICE_NOTIFICATION:nagios_type}: %{DATA:nagios_notifyname};%{DATA:nagios_hostname};%{DATA:nagios_service};%{DATA:nagios_state};%{DATA:nagios_contact};%{GREEDYDATA:nagios_message}
|
||||
NAGIOS_HOST_NOTIFICATION %{NAGIOS_TYPE_HOST_NOTIFICATION:nagios_type}: %{DATA:nagios_notifyname};%{DATA:nagios_hostname};%{DATA:nagios_state};%{DATA:nagios_contact};%{GREEDYDATA:nagios_message}
|
||||
|
||||
NAGIOS_SERVICE_ALERT %{NAGIOS_TYPE_SERVICE_ALERT:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_service};%{DATA:nagios_state};%{DATA:nagios_statelevel};%{NUMBER:nagios_attempt};%{GREEDYDATA:nagios_message}
|
||||
NAGIOS_HOST_ALERT %{NAGIOS_TYPE_HOST_ALERT:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_state};%{DATA:nagios_statelevel};%{NUMBER:nagios_attempt};%{GREEDYDATA:nagios_message}
|
||||
|
||||
NAGIOS_SERVICE_FLAPPING_ALERT %{NAGIOS_TYPE_SERVICE_FLAPPING_ALERT:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_service};%{DATA:nagios_state};%{GREEDYDATA:nagios_message}
|
||||
NAGIOS_HOST_FLAPPING_ALERT %{NAGIOS_TYPE_HOST_FLAPPING_ALERT:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_state};%{GREEDYDATA:nagios_message}
|
||||
|
||||
NAGIOS_SERVICE_DOWNTIME_ALERT %{NAGIOS_TYPE_SERVICE_DOWNTIME_ALERT:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_service};%{DATA:nagios_state};%{GREEDYDATA:nagios_comment}
|
||||
NAGIOS_HOST_DOWNTIME_ALERT %{NAGIOS_TYPE_HOST_DOWNTIME_ALERT:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_state};%{GREEDYDATA:nagios_comment}
|
||||
|
||||
NAGIOS_PASSIVE_SERVICE_CHECK %{NAGIOS_TYPE_PASSIVE_SERVICE_CHECK:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_service};%{DATA:nagios_state};%{GREEDYDATA:nagios_comment}
|
||||
NAGIOS_PASSIVE_HOST_CHECK %{NAGIOS_TYPE_PASSIVE_HOST_CHECK:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_state};%{GREEDYDATA:nagios_comment}
|
||||
|
||||
NAGIOS_SERVICE_EVENT_HANDLER %{NAGIOS_TYPE_SERVICE_EVENT_HANDLER:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_service};%{DATA:nagios_state};%{DATA:nagios_statelevel};%{DATA:nagios_event_handler_name}
|
||||
NAGIOS_HOST_EVENT_HANDLER %{NAGIOS_TYPE_HOST_EVENT_HANDLER:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_state};%{DATA:nagios_statelevel};%{DATA:nagios_event_handler_name}
|
||||
|
||||
NAGIOS_TIMEPERIOD_TRANSITION %{NAGIOS_TYPE_TIMEPERIOD_TRANSITION:nagios_type}: %{DATA:nagios_service};%{DATA:nagios_unknown1};%{DATA:nagios_unknown2}
|
||||
|
||||
####################
|
||||
#### External checks
|
||||
####################
|
||||
|
||||
#Disable host & service check
|
||||
NAGIOS_EC_LINE_DISABLE_SVC_CHECK %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_DISABLE_SVC_CHECK:nagios_command};%{DATA:nagios_hostname};%{DATA:nagios_service}
|
||||
NAGIOS_EC_LINE_DISABLE_HOST_CHECK %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_DISABLE_HOST_CHECK:nagios_command};%{DATA:nagios_hostname}
|
||||
|
||||
#Enable host & service check
|
||||
NAGIOS_EC_LINE_ENABLE_SVC_CHECK %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_ENABLE_SVC_CHECK:nagios_command};%{DATA:nagios_hostname};%{DATA:nagios_service}
|
||||
NAGIOS_EC_LINE_ENABLE_HOST_CHECK %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_ENABLE_HOST_CHECK:nagios_command};%{DATA:nagios_hostname}
|
||||
|
||||
#Process host & service check
|
||||
NAGIOS_EC_LINE_PROCESS_SERVICE_CHECK_RESULT %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_PROCESS_SERVICE_CHECK_RESULT:nagios_command};%{DATA:nagios_hostname};%{DATA:nagios_service};%{DATA:nagios_state};%{GREEDYDATA:nagios_check_result}
|
||||
NAGIOS_EC_LINE_PROCESS_HOST_CHECK_RESULT %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_PROCESS_HOST_CHECK_RESULT:nagios_command};%{DATA:nagios_hostname};%{DATA:nagios_state};%{GREEDYDATA:nagios_check_result}
|
||||
|
||||
#Disable host & service notifications
|
||||
NAGIOS_EC_LINE_DISABLE_HOST_SVC_NOTIFICATIONS %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_DISABLE_HOST_SVC_NOTIFICATIONS:nagios_command};%{GREEDYDATA:nagios_hostname}
|
||||
NAGIOS_EC_LINE_DISABLE_HOST_NOTIFICATIONS %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_DISABLE_HOST_NOTIFICATIONS:nagios_command};%{GREEDYDATA:nagios_hostname}
|
||||
NAGIOS_EC_LINE_DISABLE_SVC_NOTIFICATIONS %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_DISABLE_SVC_NOTIFICATIONS:nagios_command};%{DATA:nagios_hostname};%{GREEDYDATA:nagios_service}
|
||||
|
||||
#Enable host & service notifications
|
||||
NAGIOS_EC_LINE_ENABLE_HOST_SVC_NOTIFICATIONS %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_ENABLE_HOST_SVC_NOTIFICATIONS:nagios_command};%{GREEDYDATA:nagios_hostname}
|
||||
NAGIOS_EC_LINE_ENABLE_HOST_NOTIFICATIONS %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_ENABLE_HOST_NOTIFICATIONS:nagios_command};%{GREEDYDATA:nagios_hostname}
|
||||
NAGIOS_EC_LINE_ENABLE_SVC_NOTIFICATIONS %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_ENABLE_SVC_NOTIFICATIONS:nagios_command};%{DATA:nagios_hostname};%{GREEDYDATA:nagios_service}
|
||||
|
||||
#Schedule host & service downtime
|
||||
NAGIOS_EC_LINE_SCHEDULE_HOST_DOWNTIME %{NAGIOS_TYPE_EXTERNAL_COMMAND:nagios_type}: %{NAGIOS_EC_SCHEDULE_HOST_DOWNTIME:nagios_command};%{DATA:nagios_hostname};%{NUMBER:nagios_start_time};%{NUMBER:nagios_end_time};%{NUMBER:nagios_fixed};%{NUMBER:nagios_trigger_id};%{NUMBER:nagios_duration};%{DATA:author};%{DATA:comment}
|
||||
|
||||
#End matching line
|
||||
NAGIOSLOGLINE %{NAGIOSTIME} (?:%{NAGIOS_WARNING}|%{NAGIOS_CURRENT_SERVICE_STATE}|%{NAGIOS_CURRENT_HOST_STATE}|%{NAGIOS_SERVICE_NOTIFICATION}|%{NAGIOS_HOST_NOTIFICATION}|%{NAGIOS_SERVICE_ALERT}|%{NAGIOS_HOST_ALERT}|%{NAGIOS_SERVICE_FLAPPING_ALERT}|%{NAGIOS_HOST_FLAPPING_ALERT}|%{NAGIOS_SERVICE_DOWNTIME_ALERT}|%{NAGIOS_HOST_DOWNTIME_ALERT}|%{NAGIOS_PASSIVE_SERVICE_CHECK}|%{NAGIOS_PASSIVE_HOST_CHECK}|%{NAGIOS_SERVICE_EVENT_HANDLER}|%{NAGIOS_HOST_EVENT_HANDLER}|%{NAGIOS_TIMEPERIOD_TRANSITION}|%{NAGIOS_EC_LINE_DISABLE_SVC_CHECK}|%{NAGIOS_EC_LINE_ENABLE_SVC_CHECK}|%{NAGIOS_EC_LINE_DISABLE_HOST_CHECK}|%{NAGIOS_EC_LINE_ENABLE_HOST_CHECK}|%{NAGIOS_EC_LINE_PROCESS_HOST_CHECK_RESULT}|%{NAGIOS_EC_LINE_PROCESS_SERVICE_CHECK_RESULT}|%{NAGIOS_EC_LINE_SCHEDULE_HOST_DOWNTIME}|%{NAGIOS_EC_LINE_DISABLE_HOST_SVC_NOTIFICATIONS}|%{NAGIOS_EC_LINE_ENABLE_HOST_SVC_NOTIFICATIONS}|%{NAGIOS_EC_LINE_DISABLE_HOST_NOTIFICATIONS}|%{NAGIOS_EC_LINE_ENABLE_HOST_NOTIFICATIONS}|%{NAGIOS_EC_LINE_DISABLE_SVC_NOTIFICATIONS}|%{NAGIOS_EC_LINE_ENABLE_SVC_NOTIFICATIONS})
|
||||
|
|
@ -0,0 +1,3 @@
|
|||
# Default postgresql pg_log format pattern
|
||||
POSTGRESQL %{DATESTAMP:timestamp} %{TZ} %{DATA:user_id} %{GREEDYDATA:connection_id} %{POSINT:pid}
|
||||
|
||||
|
|
@ -0,0 +1,13 @@
|
|||
RUUID \h{32}
|
||||
# rails controller with action
|
||||
RCONTROLLER (?<controller>[^#]+)#(?<action>\w+)
|
||||
|
||||
# this will often be the only line:
|
||||
RAILS3HEAD (?m)Started %{WORD:verb} "%{URIPATHPARAM:request}" for %{IPORHOST:clientip} at (?<timestamp>%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND} %{ISO8601_TIMEZONE})
|
||||
# for some a strange reason, params are stripped of {} - not sure that's a good idea.
|
||||
RPROCESSING \W*Processing by %{RCONTROLLER} as (?<format>\S+)(?:\W*Parameters: {%{DATA:params}}\W*)?
|
||||
RAILS3FOOT Completed %{NUMBER:response}%{DATA} in %{NUMBER:totalms}ms %{RAILS3PROFILE}%{GREEDYDATA}
|
||||
RAILS3PROFILE (?:\(Views: %{NUMBER:viewms}ms \| ActiveRecord: %{NUMBER:activerecordms}ms|\(ActiveRecord: %{NUMBER:activerecordms}ms)?
|
||||
|
||||
# putting it all together
|
||||
RAILS3 %{RAILS3HEAD}(?:%{RPROCESSING})?(?<context>(?:%{DATA}\n)*)(?:%{RAILS3FOOT})?
|
||||
|
|
@ -0,0 +1,3 @@
|
|||
REDISTIMESTAMP %{MONTHDAY} %{MONTH} %{TIME}
|
||||
REDISLOG \[%{POSINT:pid}\] %{REDISTIMESTAMP:timestamp} \*
|
||||
|
||||
|
|
@ -0,0 +1,2 @@
|
|||
RUBY_LOGLEVEL (?:DEBUG|FATAL|ERROR|WARN|INFO)
|
||||
RUBY_LOGGER [DFEWI], \[%{TIMESTAMP_ISO8601:timestamp} #%{POSINT:pid}\] *%{RUBY_LOGLEVEL:loglevel} -- +%{DATA:progname}: %{GREEDYDATA:message}
|
||||
|
|
@ -19,6 +19,8 @@
|
|||
|
||||
package org.elasticsearch.ingest;
|
||||
|
||||
import org.elasticsearch.action.admin.indices.mapping.put.PutMappingResponse;
|
||||
import org.elasticsearch.common.xcontent.XContentBuilder;
|
||||
import org.elasticsearch.plugin.ingest.IngestPlugin;
|
||||
import org.elasticsearch.plugin.ingest.transport.delete.DeletePipelineAction;
|
||||
import org.elasticsearch.plugin.ingest.transport.delete.DeletePipelineRequestBuilder;
|
||||
|
|
@ -30,12 +32,15 @@ import org.elasticsearch.plugin.ingest.transport.put.PutPipelineAction;
|
|||
import org.elasticsearch.plugin.ingest.transport.put.PutPipelineRequestBuilder;
|
||||
import org.elasticsearch.plugins.Plugin;
|
||||
import org.elasticsearch.test.ESIntegTestCase;
|
||||
|
||||
import java.util.Collection;
|
||||
import java.util.Map;
|
||||
|
||||
import static org.elasticsearch.common.xcontent.XContentFactory.jsonBuilder;
|
||||
import static org.hamcrest.Matchers.*;
|
||||
import static org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertAcked;
|
||||
import static org.hamcrest.Matchers.equalTo;
|
||||
import static org.hamcrest.Matchers.nullValue;
|
||||
import static org.hamcrest.core.Is.is;
|
||||
import static org.hamcrest.core.IsNull.notNullValue;
|
||||
|
||||
public class IngestClientIT extends ESIntegTestCase {
|
||||
|
||||
|
|
@ -47,22 +52,21 @@ public class IngestClientIT extends ESIntegTestCase {
|
|||
@Override
|
||||
protected Collection<Class<? extends Plugin>> transportClientPlugins() {
|
||||
return nodePlugins();
|
||||
|
||||
}
|
||||
|
||||
public void testBasics() throws Exception {
|
||||
public void test() throws Exception {
|
||||
new PutPipelineRequestBuilder(client(), PutPipelineAction.INSTANCE)
|
||||
.setId("_id")
|
||||
.setSource(jsonBuilder().startObject()
|
||||
.field("description", "my_pipeline")
|
||||
.startArray("processors")
|
||||
.startObject()
|
||||
.startObject("simple")
|
||||
.field("path", "field2")
|
||||
.field("expected_value", "abc")
|
||||
.field("add_field", "field3")
|
||||
.field("add_field_value", "xyz")
|
||||
.endObject()
|
||||
.endObject()
|
||||
.startObject()
|
||||
.startObject("grok")
|
||||
.field("field", "field1")
|
||||
.field("pattern", "%{NUMBER:val:float} %{NUMBER:status:int} <%{WORD:msg}>")
|
||||
.endObject()
|
||||
.endObject()
|
||||
.endArray()
|
||||
.endObject().bytes())
|
||||
.get();
|
||||
|
|
@ -78,7 +82,15 @@ public class IngestClientIT extends ESIntegTestCase {
|
|||
});
|
||||
|
||||
createIndex("test");
|
||||
client().prepareIndex("test", "type", "1").setSource("field2", "abc")
|
||||
XContentBuilder updateMappingBuilder = jsonBuilder().startObject().startObject("properties")
|
||||
.startObject("status").field("type", "integer").endObject()
|
||||
.startObject("val").field("type", "float").endObject()
|
||||
.endObject();
|
||||
PutMappingResponse putMappingResponse = client().admin().indices()
|
||||
.preparePutMapping("test").setType("type").setSource(updateMappingBuilder).get();
|
||||
assertAcked(putMappingResponse);
|
||||
|
||||
client().prepareIndex("test", "type", "1").setSource("field1", "123.42 400 <foo>")
|
||||
.putHeader("ingest", "_id")
|
||||
.get();
|
||||
|
||||
|
|
@ -87,21 +99,25 @@ public class IngestClientIT extends ESIntegTestCase {
|
|||
public void run() {
|
||||
Map<String, Object> doc = client().prepareGet("test", "type", "1")
|
||||
.get().getSourceAsMap();
|
||||
assertThat(doc.get("field3"), equalTo("xyz"));
|
||||
assertThat(doc.get("val"), equalTo(123.42));
|
||||
assertThat(doc.get("status"), equalTo(400));
|
||||
assertThat(doc.get("msg"), equalTo("foo"));
|
||||
}
|
||||
});
|
||||
|
||||
client().prepareBulk().add(
|
||||
client().prepareIndex("test", "type", "2").setSource("field2", "abc")
|
||||
client().prepareIndex("test", "type", "2").setSource("field1", "123.42 400 <foo>")
|
||||
).putHeader("ingest", "_id").get();
|
||||
assertBusy(new Runnable() {
|
||||
@Override
|
||||
public void run() {
|
||||
Map<String, Object> doc = client().prepareGet("test", "type", "2").get().getSourceAsMap();
|
||||
assertThat(doc.get("field3"), equalTo("xyz"));
|
||||
assertThat(doc.get("val"), equalTo(123.42));
|
||||
assertThat(doc.get("status"), equalTo(400));
|
||||
assertThat(doc.get("msg"), equalTo("foo"));
|
||||
}
|
||||
});
|
||||
|
||||
|
||||
DeletePipelineResponse response = new DeletePipelineRequestBuilder(client(), DeletePipelineAction.INSTANCE)
|
||||
.setId("_id")
|
||||
.get();
|
||||
|
|
|
|||
|
|
@ -0,0 +1,284 @@
|
|||
/*
|
||||
* Licensed to Elasticsearch under one or more contributor
|
||||
* license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright
|
||||
* ownership. Elasticsearch licenses this file to you under
|
||||
* the Apache License, Version 2.0 (the "License"); you may
|
||||
* not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing,
|
||||
* software distributed under the License is distributed on an
|
||||
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
* KIND, either express or implied. See the License for the
|
||||
* specific language governing permissions and limitations
|
||||
* under the License.
|
||||
*/
|
||||
|
||||
package org.elasticsearch.ingest.processor.grok;
|
||||
|
||||
import org.elasticsearch.test.ESTestCase;
|
||||
import org.junit.Before;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.lang.Object;
|
||||
import java.lang.String;
|
||||
import java.util.*;
|
||||
|
||||
import static org.hamcrest.Matchers.equalTo;
|
||||
import static org.hamcrest.Matchers.is;
|
||||
import static org.hamcrest.Matchers.nullValue;
|
||||
|
||||
|
||||
public class GrokTests extends ESTestCase {
|
||||
private Map<String, String> basePatterns;
|
||||
|
||||
private Map<String, String> newBankFromStreams(InputStream... inputStreams) throws IOException {
|
||||
Map<String, String> patternBank = new HashMap<>();
|
||||
|
||||
for (InputStream is : inputStreams) {
|
||||
PatternUtils.loadBankFromStream(patternBank, is);
|
||||
}
|
||||
|
||||
return patternBank;
|
||||
}
|
||||
|
||||
@Before
|
||||
public void setup() throws IOException {
|
||||
basePatterns = newBankFromStreams(
|
||||
getClass().getResourceAsStream("/grok/patterns/grok-patterns"),
|
||||
getClass().getResourceAsStream("/grok/patterns/linux-syslog")
|
||||
);
|
||||
}
|
||||
|
||||
public void testSimpleSyslogLine() {
|
||||
String line = "Mar 16 00:01:25 evita postfix/smtpd[1713]: connect from camomile.cloud9.net[168.100.1.3]";
|
||||
Grok grok = new Grok(basePatterns, "%{SYSLOGLINE}");
|
||||
Map<String, Object> matches = grok.captures(line);
|
||||
assertEquals("evita", matches.get("logsource"));
|
||||
assertEquals("Mar 16 00:01:25", matches.get("timestamp"));
|
||||
assertEquals("connect from camomile.cloud9.net[168.100.1.3]", matches.get("message"));
|
||||
assertEquals("postfix/smtpd", matches.get("program"));
|
||||
assertEquals("1713", matches.get("pid"));
|
||||
}
|
||||
|
||||
public void testSyslog5424Line() {
|
||||
String line = "<191>1 2009-06-30T18:30:00+02:00 paxton.local grokdebug 4123 - [id1 foo=\\\"bar\\\"][id2 baz=\\\"something\\\"] Hello, syslog.";
|
||||
Grok grok = new Grok(basePatterns, "%{SYSLOG5424LINE}");
|
||||
Map<String, Object> matches = grok.captures(line);
|
||||
assertEquals("191", matches.get("syslog5424_pri"));
|
||||
assertEquals("1", matches.get("syslog5424_ver"));
|
||||
assertEquals("2009-06-30T18:30:00+02:00", matches.get("syslog5424_ts"));
|
||||
assertEquals("paxton.local", matches.get("syslog5424_host"));
|
||||
assertEquals("grokdebug", matches.get("syslog5424_app"));
|
||||
assertEquals("4123", matches.get("syslog5424_proc"));
|
||||
assertEquals(null, matches.get("syslog5424_msgid"));
|
||||
assertEquals("[id1 foo=\\\"bar\\\"][id2 baz=\\\"something\\\"]", matches.get("syslog5424_sd"));
|
||||
assertEquals("Hello, syslog.", matches.get("syslog5424_msg"));
|
||||
}
|
||||
|
||||
public void testDatePattern() {
|
||||
String line = "fancy 12-12-12 12:12:12";
|
||||
Grok grok = new Grok(basePatterns, "(?<timestamp>%{DATE_EU} %{TIME})");
|
||||
Map<String, Object> matches = grok.captures(line);
|
||||
assertEquals("12-12-12 12:12:12", matches.get("timestamp"));
|
||||
}
|
||||
|
||||
public void testNilCoercedValues() {
|
||||
Grok grok = new Grok(basePatterns, "test (N/A|%{BASE10NUM:duration:float}ms)");
|
||||
Map<String, Object> matches = grok.captures("test 28.4ms");
|
||||
assertEquals(28.4f, matches.get("duration"));
|
||||
matches = grok.captures("test N/A");
|
||||
assertEquals(null, matches.get("duration"));
|
||||
}
|
||||
|
||||
public void testNilWithNoCoercion() {
|
||||
Grok grok = new Grok(basePatterns, "test (N/A|%{BASE10NUM:duration}ms)");
|
||||
Map<String, Object> matches = grok.captures("test 28.4ms");
|
||||
assertEquals("28.4", matches.get("duration"));
|
||||
matches = grok.captures("test N/A");
|
||||
assertEquals(null, matches.get("duration"));
|
||||
}
|
||||
|
||||
public void testUnicodeSyslog() {
|
||||
Grok grok = new Grok(basePatterns, "<%{POSINT:syslog_pri}>%{SPACE}%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{PROG:syslog_program}(:?)(?:\\[%{GREEDYDATA:syslog_pid}\\])?(:?) %{GREEDYDATA:syslog_message}");
|
||||
Map<String, Object> matches = grok.captures("<22>Jan 4 07:50:46 mailmaster postfix/policy-spf[9454]: : SPF permerror (Junk encountered in record 'v=spf1 mx a:mail.domain.no ip4:192.168.0.4 <20>all'): Envelope-from: email@domain.no");
|
||||
assertThat(matches.get("syslog_pri"), equalTo("22"));
|
||||
assertThat(matches.get("syslog_program"), equalTo("postfix/policy-spf"));
|
||||
assertThat(matches.get("tags"), nullValue());
|
||||
}
|
||||
|
||||
public void testNamedFieldsWithWholeTextMatch() {
|
||||
Grok grok = new Grok(basePatterns, "%{DATE_EU:stimestamp}");
|
||||
Map<String, Object> matches = grok.captures("11/01/01");
|
||||
assertThat(matches.get("stimestamp"), equalTo("11/01/01"));
|
||||
}
|
||||
|
||||
public void testWithOniguramaNamedCaptures() {
|
||||
Grok grok = new Grok(basePatterns, "(?<foo>\\w+)");
|
||||
Map<String, Object> matches = grok.captures("hello world");
|
||||
assertThat(matches.get("foo"), equalTo("hello"));
|
||||
}
|
||||
|
||||
public void testISO8601() {
|
||||
Grok grok = new Grok(basePatterns, "^%{TIMESTAMP_ISO8601}$");
|
||||
List<String> timeMessages = Arrays.asList(
|
||||
"2001-01-01T00:00:00",
|
||||
"1974-03-02T04:09:09",
|
||||
"2010-05-03T08:18:18+00:00",
|
||||
"2004-07-04T12:27:27-00:00",
|
||||
"2001-09-05T16:36:36+0000",
|
||||
"2001-11-06T20:45:45-0000",
|
||||
"2001-12-07T23:54:54Z",
|
||||
"2001-01-01T00:00:00.123456",
|
||||
"1974-03-02T04:09:09.123456",
|
||||
"2010-05-03T08:18:18.123456+00:00",
|
||||
"2004-07-04T12:27:27.123456-00:00",
|
||||
"2001-09-05T16:36:36.123456+0000",
|
||||
"2001-11-06T20:45:45.123456-0000",
|
||||
"2001-12-07T23:54:54.123456Z",
|
||||
"2001-12-07T23:54:60.123456Z" // '60' second is a leap second.
|
||||
);
|
||||
for (String msg : timeMessages) {
|
||||
assertThat(grok.match(msg), is(true));
|
||||
}
|
||||
}
|
||||
|
||||
public void testNotISO8601() {
|
||||
Grok grok = new Grok(basePatterns, "^%{TIMESTAMP_ISO8601}$");
|
||||
List<String> timeMessages = Arrays.asList(
|
||||
"2001-13-01T00:00:00", // invalid month
|
||||
"2001-00-01T00:00:00", // invalid month
|
||||
"2001-01-00T00:00:00", // invalid day
|
||||
"2001-01-32T00:00:00", // invalid day
|
||||
"2001-01-aT00:00:00", // invalid day
|
||||
"2001-01-1aT00:00:00", // invalid day
|
||||
"2001-01-01Ta0:00:00", // invalid hour
|
||||
"2001-01-01T0:00:00", // invalid hour
|
||||
"2001-01-01T25:00:00", // invalid hour
|
||||
"2001-01-01T01:60:00", // invalid minute
|
||||
"2001-01-01T00:aa:00", // invalid minute
|
||||
"2001-01-01T00:00:aa", // invalid second
|
||||
"2001-01-01T00:00:-1", // invalid second
|
||||
"2001-01-01T00:00:61", // invalid second
|
||||
"2001-01-01T00:00:00A", // invalid timezone
|
||||
"2001-01-01T00:00:00+", // invalid timezone
|
||||
"2001-01-01T00:00:00+25", // invalid timezone
|
||||
"2001-01-01T00:00:00+2500", // invalid timezone
|
||||
"2001-01-01T00:00:00+25:00", // invalid timezone
|
||||
"2001-01-01T00:00:00-25", // invalid timezone
|
||||
"2001-01-01T00:00:00-2500", // invalid timezone
|
||||
"2001-01-01T00:00:00-00:61" // invalid timezone
|
||||
);
|
||||
for (String msg : timeMessages) {
|
||||
assertThat(grok.match(msg), is(false));
|
||||
}
|
||||
}
|
||||
|
||||
public void testNoNamedCaptures() {
|
||||
Map<String, String> bank = new HashMap<>();
|
||||
|
||||
bank.put("NAME", "Tal");
|
||||
bank.put("EXCITED_NAME", "!!!%{NAME:name}!!!");
|
||||
bank.put("TEST", "hello world");
|
||||
|
||||
String text = "wowza !!!Tal!!! - Tal";
|
||||
String pattern = "%{EXCITED_NAME} - %{NAME}";
|
||||
Grok g = new Grok(bank, pattern, false);
|
||||
|
||||
assertEquals("(?<EXCITED_NAME_0>!!!(?<NAME_21>Tal)!!!) - (?<NAME_22>Tal)", g.toRegex(pattern));
|
||||
assertEquals(true, g.match(text));
|
||||
|
||||
Object actual = g.captures(text);
|
||||
Map<String, Object> expected = new HashMap<>();
|
||||
expected.put("EXCITED_NAME_0", "!!!Tal!!!");
|
||||
expected.put("NAME_21", "Tal");
|
||||
expected.put("NAME_22", "Tal");
|
||||
assertEquals(expected, actual);
|
||||
}
|
||||
|
||||
public void testNumericCapturesCoercion() {
|
||||
Map<String, String> bank = new HashMap<>();
|
||||
bank.put("BASE10NUM", "(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+)))");
|
||||
bank.put("NUMBER", "(?:%{BASE10NUM})");
|
||||
|
||||
String pattern = "%{NUMBER:bytes:float} %{NUMBER:status} %{NUMBER}";
|
||||
Grok g = new Grok(bank, pattern);
|
||||
|
||||
String text = "12009.34 200 9032";
|
||||
Map<String, Object> expected = new HashMap<>();
|
||||
expected.put("bytes", 12009.34f);
|
||||
expected.put("status", "200");
|
||||
Map<String, Object> actual = g.captures(text);
|
||||
|
||||
assertEquals(expected, actual);
|
||||
}
|
||||
|
||||
public void testApacheLog() {
|
||||
String logLine = "31.184.238.164 - - [24/Jul/2014:05:35:37 +0530] \"GET /logs/access.log HTTP/1.0\" 200 69849 \"http://8rursodiol.enjin.com\" \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.12785 YaBrowser/13.12.1599.12785 Safari/537.36\" \"www.dlwindianrailways.com\"";
|
||||
Grok grok = new Grok(basePatterns, "%{COMBINEDAPACHELOG}");
|
||||
Map<String, Object> matches = grok.captures(logLine);
|
||||
|
||||
assertEquals("31.184.238.164", matches.get("clientip"));
|
||||
assertEquals("-", matches.get("ident"));
|
||||
assertEquals("-", matches.get("auth"));
|
||||
assertEquals("24/Jul/2014:05:35:37 +0530", matches.get("timestamp"));
|
||||
assertEquals("GET", matches.get("verb"));
|
||||
assertEquals("/logs/access.log", matches.get("request"));
|
||||
assertEquals("1.0", matches.get("httpversion"));
|
||||
assertEquals("200", matches.get("response"));
|
||||
assertEquals("69849", matches.get("bytes"));
|
||||
assertEquals("\"http://8rursodiol.enjin.com\"", matches.get("referrer"));
|
||||
assertEquals(null, matches.get("port"));
|
||||
assertEquals("\"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.12785 YaBrowser/13.12.1599.12785 Safari/537.36\"", matches.get("agent"));
|
||||
}
|
||||
|
||||
public void testComplete() {
|
||||
Map<String, String> bank = new HashMap<>();
|
||||
bank.put("MONTHDAY", "(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])");
|
||||
bank.put("MONTH", "\\b(?:Jan(?:uary|uar)?|Feb(?:ruary|ruar)?|M(?:a|ä)?r(?:ch|z)?|Apr(?:il)?|Ma(?:y|i)?|Jun(?:e|i)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|O(?:c|k)?t(?:ober)?|Nov(?:ember)?|De(?:c|z)(?:ember)?)\\b");
|
||||
bank.put("MINUTE", "(?:[0-5][0-9])");
|
||||
bank.put("YEAR", "(?>\\d\\d){1,2}");
|
||||
bank.put("HOUR", "(?:2[0123]|[01]?[0-9])");
|
||||
bank.put("SECOND", "(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)");
|
||||
bank.put("TIME", "(?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9])");
|
||||
bank.put("INT", "(?:[+-]?(?:[0-9]+))");
|
||||
bank.put("HTTPDATE", "%{MONTHDAY}/%{MONTH}/%{YEAR}:%{TIME} %{INT}");
|
||||
bank.put("WORD", "\\b\\w+\\b");
|
||||
bank.put("BASE10NUM", "(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+)))");
|
||||
bank.put("NUMBER", "(?:%{BASE10NUM})");
|
||||
bank.put("IPV6", "((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:)))(%.+)?");
|
||||
bank.put("IPV4", "(?<![0-9])(?:(?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5]))(?![0-9])");
|
||||
bank.put("IP", "(?:%{IPV6}|%{IPV4})");
|
||||
bank.put("HOSTNAME", "\\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\\.?|\\b)");
|
||||
bank.put("IPORHOST", "(?:%{IP}|%{HOSTNAME})");
|
||||
bank.put("USER", "[a-zA-Z0-9._-]+");
|
||||
bank.put("DATA", ".*?");
|
||||
bank.put("QS", "(?>(?<!\\\\)(?>\"(?>\\\\.|[^\\\\\"]+)+\"|\"\"|(?>'(?>\\\\.|[^\\\\']+)+')|''|(?>`(?>\\\\.|[^\\\\`]+)+`)|``))");
|
||||
|
||||
String text = "83.149.9.216 - - [19/Jul/2015:08:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-dashboard3.png HTTP/1.1\" 200 171717 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"";
|
||||
String pattern = "%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \\[%{HTTPDATE:timestamp}\\] \"%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int}) %{QS:referrer} %{QS:agent}";
|
||||
|
||||
Grok grok = new Grok(bank, pattern);
|
||||
|
||||
Map<String, Object> expected = new HashMap<>();
|
||||
expected.put("clientip", "83.149.9.216");
|
||||
expected.put("ident", "-");
|
||||
expected.put("auth", "-");
|
||||
expected.put("timestamp", "19/Jul/2015:08:13:42 +0000");
|
||||
expected.put("verb", "GET");
|
||||
expected.put("request", "/presentations/logstash-monitorama-2013/images/kibana-dashboard3.png");
|
||||
expected.put("httpversion", "1.1");
|
||||
expected.put("response", 200);
|
||||
expected.put("bytes", 171717);
|
||||
expected.put("referrer", "\"http://semicomplete.com/presentations/logstash-monitorama-2013/\"");
|
||||
expected.put("agent", "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"");
|
||||
|
||||
Map<String, Object> actual = grok.captures(text);
|
||||
|
||||
assertEquals(expected, actual);
|
||||
}
|
||||
}
|
||||
|
|
@ -23,6 +23,7 @@ import org.elasticsearch.cluster.ClusterService;
|
|||
import org.elasticsearch.common.bytes.BytesArray;
|
||||
import org.elasticsearch.common.settings.Settings;
|
||||
import org.elasticsearch.common.text.StringText;
|
||||
import org.elasticsearch.env.Environment;
|
||||
import org.elasticsearch.ingest.processor.simple.SimpleProcessor;
|
||||
import org.elasticsearch.search.SearchHit;
|
||||
import org.elasticsearch.search.internal.InternalSearchHit;
|
||||
|
|
@ -31,6 +32,7 @@ import org.elasticsearch.threadpool.ThreadPool;
|
|||
import org.junit.After;
|
||||
import org.junit.Before;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Collections;
|
||||
import java.util.Comparator;
|
||||
|
|
@ -54,7 +56,8 @@ public class PipelineStoreTests extends ESTestCase {
|
|||
threadPool = new ThreadPool("test");
|
||||
ClusterService clusterService = mock(ClusterService.class);
|
||||
client = mock(PipelineStoreClient.class);
|
||||
store = new PipelineStore(Settings.EMPTY, threadPool, clusterService, client, Collections.singletonMap(SimpleProcessor.TYPE, new SimpleProcessor.Builder.Factory()));
|
||||
Environment environment = mock(Environment.class);
|
||||
store = new PipelineStore(Settings.EMPTY, threadPool, environment, clusterService, client, Collections.singletonMap(SimpleProcessor.TYPE, new SimpleProcessor.Builder.Factory()));
|
||||
store.start();
|
||||
}
|
||||
|
||||
|
|
@ -65,7 +68,7 @@ public class PipelineStoreTests extends ESTestCase {
|
|||
}
|
||||
|
||||
|
||||
public void testUpdatePipeline() {
|
||||
public void testUpdatePipeline() throws IOException {
|
||||
List<SearchHit> hits = new ArrayList<>();
|
||||
hits.add(new InternalSearchHit(0, "1", new StringText("type"), Collections.emptyMap())
|
||||
.sourceRef(new BytesArray("{\"description\": \"_description1\"}"))
|
||||
|
|
@ -126,7 +129,7 @@ public class PipelineStoreTests extends ESTestCase {
|
|||
});
|
||||
}
|
||||
|
||||
public void testGetReference() {
|
||||
public void testGetReference() throws IOException {
|
||||
// fill the store up for the test:
|
||||
List<SearchHit> hits = new ArrayList<>();
|
||||
hits.add(new InternalSearchHit(0, "foo", new StringText("type"), Collections.emptyMap()).sourceRef(new BytesArray("{\"description\": \"_description\"}")));
|
||||
|
|
|
|||
|
|
@ -0,0 +1,48 @@
|
|||
---
|
||||
"Test Grok Pipeline":
|
||||
- do:
|
||||
cluster.health:
|
||||
wait_for_status: green
|
||||
|
||||
- do:
|
||||
ingest.put_pipeline:
|
||||
id: "my_pipeline"
|
||||
body: >
|
||||
{
|
||||
"description": "_description",
|
||||
"processors": [
|
||||
{
|
||||
"grok" : {
|
||||
"field" : "field1",
|
||||
"pattern" : "%{NUMBER:val:float} %{NUMBER:status:int} <%{WORD:msg}>"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
- match: { _id: "my_pipeline" }
|
||||
|
||||
# Simulate a Thread.sleep(), because pipeline are updated in the background
|
||||
- do:
|
||||
catch: request_timeout
|
||||
cluster.health:
|
||||
wait_for_nodes: 99
|
||||
timeout: 2s
|
||||
- match: { "timed_out": true }
|
||||
|
||||
- do:
|
||||
ingest.index:
|
||||
index: test
|
||||
type: test
|
||||
id: 1
|
||||
ingest: "my_pipeline"
|
||||
body: {field1: "123.42 400 <foo>"}
|
||||
|
||||
- do:
|
||||
get:
|
||||
index: test
|
||||
type: test
|
||||
id: 1
|
||||
- match: { _source.val: 123.42 }
|
||||
- match: { _source.status: 400 }
|
||||
- match: { _source.msg: "foo" }
|
||||
|
||||
Loading…
Reference in New Issue