ingest: Renamed from `ingest-useragent` to `ingest-user-agent` and processor from `useragent` to `user_agent`
and on some other places did similar renaming. This is consistent with ES naming. Also made sure that the docs are navigable from the reference guide.
This commit is contained in:
Martijn van Groningen
parent
71b48fb16c
commit
b4defafcb2
|
|
@ -1,12 +1,12 @@
|
|||
[[ingest-useragent]]
|
||||
=== Ingest Useragent Processor Plugin
|
||||
[[ingest-user-agent]]
|
||||
=== Ingest user agent processor plugin
|
||||
|
||||
The Useragent processor extracts details from the user agent string a browser sends with its web requests.
|
||||
This processor adds this information by default under the `useragent` field.
|
||||
The `user_agent` processor extracts details from the user agent string a browser sends with its web requests.
|
||||
This processor adds this information by default under the `user_agent` field.
|
||||
|
||||
The ingest-useragent plugin ships by default with the regexes.yaml made available by uap-java with an Apache 2.0 license. For more details see https://github.com/ua-parser/uap-core.
|
||||
The ingest-user-agent plugin ships by default with the regexes.yaml made available by uap-java with an Apache 2.0 license. For more details see https://github.com/ua-parser/uap-core.
|
||||
|
||||
[[ingest-useragent-install]]
|
||||
[[ingest-user-agent-install]]
|
||||
[float]
|
||||
==== Installation
|
||||
|
||||
|
|
@ -14,13 +14,13 @@ This plugin can be installed using the plugin manager:
|
|||
|
||||
[source,sh]
|
||||
----------------------------------------------------------------
|
||||
sudo bin/elasticsearch-plugin install ingest-useragent
|
||||
sudo bin/elasticsearch-plugin install ingest-user-agent
|
||||
----------------------------------------------------------------
|
||||
|
||||
The plugin must be installed on every node in the cluster, and each node must
|
||||
be restarted after installation.
|
||||
|
||||
[[ingest-useragent-remove]]
|
||||
[[ingest-user-agent-remove]]
|
||||
[float]
|
||||
==== Removal
|
||||
|
||||
|
|
@ -28,26 +28,26 @@ The plugin can be removed with the following command:
|
|||
|
||||
[source,sh]
|
||||
----------------------------------------------------------------
|
||||
sudo bin/elasticsearch-plugin remove ingest-useragent
|
||||
sudo bin/elasticsearch-plugin remove ingest-user-agent
|
||||
----------------------------------------------------------------
|
||||
|
||||
The node must be stopped before removing the plugin.
|
||||
|
||||
[[using-ingest-useragent]]
|
||||
==== Using the Useragent Processor in a Pipeline
|
||||
[[using-ingest-user-agent]]
|
||||
==== Using the user_agent Processor in a Pipeline
|
||||
|
||||
[[ingest-useragent-options]]
|
||||
.Useragent options
|
||||
[[ingest-user-agent-options]]
|
||||
.User-agent options
|
||||
[options="header"]
|
||||
|======
|
||||
| Name | Required | Default | Description
|
||||
| `field` | yes | - | The field containing the user agent string.
|
||||
| `target_field` | no | useragent | The field that will be filled with the user agent details.
|
||||
| `regex_file` | no | - | The name of the file in the `config/ingest-useragent` directory containing the regular expressions for parsing the user agent string. Both the directory and the file have to be created before starting Elasticsearch. If not specified, ingest-useragent will use the regexes.yaml from uap-core it ships with (see below).
|
||||
| `target_field` | no | user_agent | The field that will be filled with the user agent details.
|
||||
| `regex_file` | no | - | The name of the file in the `config/ingest-user-agent` directory containing the regular expressions for parsing the user agent string. Both the directory and the file have to be created before starting Elasticsearch. If not specified, ingest-user-agent will use the regexes.yaml from uap-core it ships with (see below).
|
||||
| `properties` | no | [`name`, `major`, `minor`, `patch`, `build`, `os`, `os_name`, `os_major`, `os_minor`, `device`] | Controls what properties are added to `target_field`.
|
||||
|======
|
||||
|
||||
Here is an example that adds the user agent details to the `useragent` field based on the `agent` field:
|
||||
Here is an example that adds the user agent details to the `user_agent` field based on the `agent` field:
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
|
|
@ -55,7 +55,7 @@ Here is an example that adds the user agent details to the `useragent` field bas
|
|||
"description" : "...",
|
||||
"processors" : [
|
||||
{
|
||||
"useragent" : {
|
||||
"user_agent" : {
|
||||
"field" : "agent"
|
||||
}
|
||||
}
|
||||
|
|
@ -64,11 +64,11 @@ Here is an example that adds the user agent details to the `useragent` field bas
|
|||
--------------------------------------------------
|
||||
|
||||
===== Using a custom regex file
|
||||
To use a custom regex file for parsing the user agents, that file has to be put into the `config/ingest-useragent` directory and
|
||||
To use a custom regex file for parsing the user agents, that file has to be put into the `config/ingest-user-agent` directory and
|
||||
has to have a `.yaml` filename extension. The file has to be present at node startup, any changes to it or any new files added
|
||||
while the node is running will not have any effect.
|
||||
|
||||
In practice, it will make most sense for any custom regex file to be a variant of the default file, either a more recent version
|
||||
or a customised version.
|
||||
|
||||
The default file included in `ingest-useragent` is the `regexes.yaml` from uap-core: https://github.com/ua-parser/uap-core/blob/master/regexes.yaml
|
||||
The default file included in `ingest-user-agent` is the `regexes.yaml` from uap-core: https://github.com/ua-parser/uap-core/blob/master/regexes.yaml
|
||||
|
|
@ -19,8 +19,14 @@ The GeoIP processor adds information about the geographical location of IP addre
|
|||
This processor adds this information by default under the `geoip` field.
|
||||
+
|
||||
The ingest-geoip plugin ships by default with the GeoLite2 City and GeoLite2 Country geoip2 databases from Maxmind made available
|
||||
under the CCA-ShareAlike 3.0 license. For more details see, http://dev.maxmind.com/geoip/geoip2/geolite2/.
|
||||
under the CCA-ShareAlike 3.0 license. For more details see, http://dev.maxmind.com/geoip/geoip2/geolite2/.
|
||||
|
||||
<<ingest-user-agent>>::
|
||||
|
||||
A processor that extracts details from the User-Agent header value.
|
||||
|
||||
include::ingest-attachment.asciidoc[]
|
||||
|
||||
include::ingest-geoip.asciidoc[]
|
||||
include::ingest-geoip.asciidoc[]
|
||||
|
||||
include::ingest-user-agent.asciidoc[]
|
||||
|
|
@ -24,6 +24,6 @@ esplugin {
|
|||
|
||||
integTest {
|
||||
cluster {
|
||||
extraConfigFile 'ingest-useragent/test-regexes.yaml', 'test/test-regexes.yaml'
|
||||
extraConfigFile 'ingest-user-agent/test-regexes.yaml', 'test/test-regexes.yaml'
|
||||
}
|
||||
}
|
||||
|
|
@ -42,7 +42,7 @@ import java.util.stream.Stream;
|
|||
|
||||
public class IngestUserAgentPlugin extends Plugin implements IngestPlugin {
|
||||
|
||||
private final Setting<Long> CACHE_SIZE_SETTING = Setting.longSetting("ingest.useragent.cache_size", 1000, 0,
|
||||
private final Setting<Long> CACHE_SIZE_SETTING = Setting.longSetting("ingest.user_agent.cache_size", 1000, 0,
|
||||
Setting.Property.NodeScope);
|
||||
|
||||
static final String DEFAULT_PARSER_NAME = "_default_";
|
||||
|
|
@ -76,7 +76,7 @@ public class IngestUserAgentPlugin extends Plugin implements IngestPlugin {
|
|||
|
||||
if (Files.exists(userAgentConfigDirectory) && Files.isDirectory(userAgentConfigDirectory)) {
|
||||
PathMatcher pathMatcher = userAgentConfigDirectory.getFileSystem().getPathMatcher("glob:**.yaml");
|
||||
|
||||
|
||||
try (Stream<Path> regexFiles = Files.find(userAgentConfigDirectory, 1,
|
||||
(path, attr) -> attr.isRegularFile() && pathMatcher.matches(path))) {
|
||||
Iterable<Path> iterable = regexFiles::iterator;
|
||||
|
|
@ -39,7 +39,7 @@ import static org.elasticsearch.ingest.ConfigurationUtils.readStringProperty;
|
|||
|
||||
public class UserAgentProcessor extends AbstractProcessor {
|
||||
|
||||
public static final String TYPE = "useragent";
|
||||
public static final String TYPE = "user_agent";
|
||||
|
||||
private final String field;
|
||||
private final String targetField;
|
||||
|
|
@ -197,7 +197,7 @@ public class UserAgentProcessor extends AbstractProcessor {
|
|||
public UserAgentProcessor create(Map<String, Processor.Factory> factories, String processorTag,
|
||||
Map<String, Object> config) throws Exception {
|
||||
String field = readStringProperty(TYPE, processorTag, config, "field");
|
||||
String targetField = readStringProperty(TYPE, processorTag, config, "target_field", "useragent");
|
||||
String targetField = readStringProperty(TYPE, processorTag, config, "target_field", "user_agent");
|
||||
String regexFilename = readStringProperty(TYPE, processorTag, config, "regex_file", IngestUserAgentPlugin.DEFAULT_PARSER_NAME);
|
||||
List<String> propertyNames = readOptionalList(TYPE, processorTag, config, "properties");
|
||||
|
||||
|
|
@ -52,7 +52,7 @@ public class UserAgentProcessorFactoryTests extends ESTestCase {
|
|||
@BeforeClass
|
||||
public static void createUserAgentParsers() throws IOException {
|
||||
Path configDir = createTempDir();
|
||||
userAgentConfigDir = configDir.resolve("ingest-useragent");
|
||||
userAgentConfigDir = configDir.resolve("ingest-user-agent");
|
||||
Files.createDirectories(userAgentConfigDir);
|
||||
|
||||
// Copy file, leaving out the device parsers at the end
|
||||
|
|
@ -84,7 +84,7 @@ public class UserAgentProcessorFactoryTests extends ESTestCase {
|
|||
UserAgentProcessor processor = factory.create(null, processorTag, config);
|
||||
assertThat(processor.getTag(), equalTo(processorTag));
|
||||
assertThat(processor.getField(), equalTo("_field"));
|
||||
assertThat(processor.getTargetField(), equalTo("useragent"));
|
||||
assertThat(processor.getTargetField(), equalTo("user_agent"));
|
||||
assertThat(processor.getUaParser().getUaPatterns().size(), greaterThan(0));
|
||||
assertThat(processor.getUaParser().getOsPatterns().size(), greaterThan(0));
|
||||
assertThat(processor.getUaParser().getDevicePatterns().size(), greaterThan(0));
|
||||
|
|
@ -0,0 +1,11 @@
|
|||
"ingest-user-agent plugin installed":
|
||||
- do:
|
||||
cluster.state: {}
|
||||
|
||||
- set: {master_node: master}
|
||||
|
||||
- do:
|
||||
nodes.info: {}
|
||||
|
||||
- match: { nodes.$master.plugins.0.name: ingest-user-agent }
|
||||
- match: { nodes.$master.ingest.processors.0.type: user_agent }
|
||||
|
|
@ -8,7 +8,7 @@
|
|||
"description": "_description",
|
||||
"processors": [
|
||||
{
|
||||
"useragent" : {
|
||||
"user_agent" : {
|
||||
"field" : "field1"
|
||||
}
|
||||
}
|
||||
|
|
@ -30,15 +30,15 @@
|
|||
type: test
|
||||
id: 1
|
||||
- match: { _source.field1: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.149 Safari/537.36" }
|
||||
- match: { _source.useragent.name: "Chrome" }
|
||||
- match: { _source.useragent.os: "Mac OS X 10.9.2" }
|
||||
- match: { _source.useragent.os_name: "Mac OS X" }
|
||||
- match: { _source.useragent.os_major: "10" }
|
||||
- match: { _source.useragent.os_minor: "9" }
|
||||
- match: { _source.useragent.major: "33" }
|
||||
- match: { _source.useragent.minor: "0" }
|
||||
- match: { _source.useragent.patch: "1750" }
|
||||
- match: { _source.useragent.device: "Other" }
|
||||
- match: { _source.user_agent.name: "Chrome" }
|
||||
- match: { _source.user_agent.os: "Mac OS X 10.9.2" }
|
||||
- match: { _source.user_agent.os_name: "Mac OS X" }
|
||||
- match: { _source.user_agent.os_major: "10" }
|
||||
- match: { _source.user_agent.os_minor: "9" }
|
||||
- match: { _source.user_agent.major: "33" }
|
||||
- match: { _source.user_agent.minor: "0" }
|
||||
- match: { _source.user_agent.patch: "1750" }
|
||||
- match: { _source.user_agent.device: "Other" }
|
||||
|
||||
---
|
||||
"Test user agent processor with parameters":
|
||||
|
|
@ -50,7 +50,7 @@
|
|||
"description": "_description",
|
||||
"processors": [
|
||||
{
|
||||
"useragent" : {
|
||||
"user_agent" : {
|
||||
"field" : "field1",
|
||||
"target_field": "field2",
|
||||
"properties": ["os"]
|
||||
|
|
@ -75,7 +75,7 @@
|
|||
id: 1
|
||||
- match: { _source.field1: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.149 Safari/537.36" }
|
||||
- match: { _source.field2.os: "Mac OS X 10.9.2" }
|
||||
- is_false: _source.useragent
|
||||
- is_false: _source.user_agent
|
||||
- is_false: _source.field2.name
|
||||
- is_false: _source.field2.os_name
|
||||
- is_false: _source.field2.os_major
|
||||
|
|
@ -8,7 +8,7 @@
|
|||
"description": "_description",
|
||||
"processors": [
|
||||
{
|
||||
"useragent" : {
|
||||
"user_agent" : {
|
||||
"field": "field1",
|
||||
"regex_file": "test-regexes.yaml"
|
||||
}
|
||||
|
|
@ -31,12 +31,12 @@
|
|||
type: test
|
||||
id: 1
|
||||
- match: { _source.field1: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.149 Safari/537.36" }
|
||||
- match: { _source.useragent.name: "Test" }
|
||||
- match: { _source.useragent.os: "Other" }
|
||||
- match: { _source.useragent.os_name: "Other" }
|
||||
- match: { _source.useragent.device: "Other" }
|
||||
- is_false: _source.useragent.os_major
|
||||
- is_false: _source.useragent.os_minor
|
||||
- is_false: _source.useragent.major
|
||||
- is_false: _source.useragent.minor
|
||||
- is_false: _source.useragent.patch
|
||||
- match: { _source.user_agent.name: "Test" }
|
||||
- match: { _source.user_agent.os: "Other" }
|
||||
- match: { _source.user_agent.os_name: "Other" }
|
||||
- match: { _source.user_agent.device: "Other" }
|
||||
- is_false: _source.user_agent.os_major
|
||||
- is_false: _source.user_agent.os_minor
|
||||
- is_false: _source.user_agent.major
|
||||
- is_false: _source.user_agent.minor
|
||||
- is_false: _source.user_agent.patch
|
||||
|
|
@ -1,11 +0,0 @@
|
|||
"ingest-useragent plugin installed":
|
||||
- do:
|
||||
cluster.state: {}
|
||||
|
||||
- set: {master_node: master}
|
||||
|
||||
- do:
|
||||
nodes.info: {}
|
||||
|
||||
- match: { nodes.$master.plugins.0.name: ingest-useragent }
|
||||
- match: { nodes.$master.ingest.processors.0.type: useragent }
|
||||
|
|
@ -228,8 +228,8 @@ fi
|
|||
install_and_check_plugin ingest geoip geoip2-*.jar jackson-annotations-*.jar jackson-databind-*.jar maxmind-db-*.jar
|
||||
}
|
||||
|
||||
@test "[$GROUP] install ingest-useragent plugin" {
|
||||
install_and_check_plugin ingest useragent
|
||||
@test "[$GROUP] install ingest-user-agent plugin" {
|
||||
install_and_check_plugin ingest user-agent
|
||||
}
|
||||
|
||||
@test "[$GROUP] check ingest-common module" {
|
||||
|
|
@ -361,8 +361,8 @@ fi
|
|||
remove_plugin ingest-geoip
|
||||
}
|
||||
|
||||
@test "[$GROUP] remove ingest-useragent plugin" {
|
||||
remove_plugin ingest-useragent
|
||||
@test "[$GROUP] remove ingest-user0agent plugin" {
|
||||
remove_plugin ingest-user-agent
|
||||
}
|
||||
|
||||
@test "[$GROUP] remove javascript plugin" {
|
||||
|
|
|
|||
|
|
@ -36,7 +36,7 @@ List projects = [
|
|||
'plugins:discovery-gce',
|
||||
'plugins:ingest-geoip',
|
||||
'plugins:ingest-attachment',
|
||||
'plugins:ingest-useragent',
|
||||
'plugins:ingest-user-agent',
|
||||
'plugins:lang-javascript',
|
||||
'plugins:lang-python',
|
||||
'plugins:mapper-attachments',
|
||||
|
|
|
|||
Loading…
Reference in New Issue