1946 lines
52 KiB
Plaintext
1946 lines
52 KiB
Plaintext
[role="xpack"]
|
|
[testenv="basic"]
|
|
[[ml-find-file-structure]]
|
|
= Find file structure API
|
|
++++
|
|
<titleabbrev>Find file structure</titleabbrev>
|
|
++++
|
|
|
|
experimental[]
|
|
|
|
Finds the structure of a text file. The text file must contain data that is
|
|
suitable to be ingested into {es}.
|
|
|
|
|
|
[[ml-find-file-structure-request]]
|
|
== {api-request-title}
|
|
|
|
`POST _ml/find_file_structure`
|
|
|
|
|
|
[[ml-find-file-structure-prereqs]]
|
|
== {api-prereq-title}
|
|
|
|
* If the {es} {security-features} are enabled, you must have `monitor_ml` or
|
|
`monitor` cluster privileges to use this API. See
|
|
<<security-privileges>>.
|
|
|
|
|
|
[[ml-find-file-structure-desc]]
|
|
== {api-description-title}
|
|
|
|
This API provides a starting point for ingesting data into {es} in a format that
|
|
is suitable for subsequent use with other {ml} functionality.
|
|
|
|
Unlike other {es} endpoints, the data that is posted to this endpoint does not
|
|
need to be UTF-8 encoded and in JSON format. It must, however, be text; binary
|
|
file formats are not currently supported.
|
|
|
|
The response from the API contains:
|
|
|
|
* A couple of messages from the beginning of the file.
|
|
* Statistics that reveal the most common values for all fields detected within
|
|
the file and basic numeric statistics for numeric fields.
|
|
* Information about the structure of the file, which is useful when you write
|
|
ingest configurations to index the file contents.
|
|
* Appropriate mappings for an {es} index, which you could use to ingest the file
|
|
contents.
|
|
|
|
All this information can be calculated by the structure finder with no guidance.
|
|
However, you can optionally override some of the decisions about the file
|
|
structure by specifying one or more query parameters.
|
|
|
|
Details of the output can be seen in the
|
|
<<ml-find-file-structure-examples,examples>>.
|
|
|
|
If the structure finder produces unexpected results for a particular file,
|
|
specify the `explain` query parameter. It causes an `explanation` to appear in
|
|
the response, which should help in determining why the returned structure was
|
|
chosen.
|
|
|
|
|
|
[[ml-find-file-structure-query-parms]]
|
|
== {api-query-parms-title}
|
|
|
|
`charset`::
|
|
(Optional, string) The file's character set. It must be a character set that
|
|
is supported by the JVM that {es} uses. For example, `UTF-8`, `UTF-16LE`,
|
|
`windows-1252`, or `EUC-JP`. If this parameter is not specified, the structure
|
|
finder chooses an appropriate character set.
|
|
|
|
`column_names`::
|
|
(Optional, string) If you have set `format` to `delimited`, you can specify
|
|
the column names in a comma-separated list. If this parameter is not specified,
|
|
the structure finder uses the column names from the header row of the file. If
|
|
the file does not have a header role, columns are named "column1", "column2",
|
|
"column3", etc.
|
|
|
|
`delimiter`::
|
|
(Optional, string) If you have set `format` to `delimited`, you can specify
|
|
the character used to delimit the values in each row. Only a single character
|
|
is supported; the delimiter cannot have multiple characters. By default, the
|
|
API considers the following possibilities: comma, tab, semi-colon, and pipe
|
|
(`|`). In this default scenario, all rows must have the same number of fields
|
|
for the delimited format to be detected. If you specify a delimiter, up to 10%
|
|
of the rows can have a different number of columns than the first row.
|
|
|
|
`explain`::
|
|
(Optional, boolean) If this parameter is set to `true`, the response includes
|
|
a field named `explanation`, which is an array of strings that indicate how
|
|
the structure finder produced its result. The default value is `false`.
|
|
|
|
`format`::
|
|
(Optional, string) The high level structure of the file. Valid values are
|
|
`ndjson`, `xml`, `delimited`, and `semi_structured_text`. By default, the
|
|
API chooses the format. In this default scenario, all rows must
|
|
have the same number of fields for a delimited format to be detected. If the
|
|
`format` is set to `delimited` and the `delimiter` is not set, however, the
|
|
API tolerates up to 5% of rows that have a different number of
|
|
columns than the first row.
|
|
|
|
`grok_pattern`::
|
|
(Optional, string) If you have set `format` to `semi_structured_text`, you can
|
|
specify a Grok pattern that is used to extract fields from every message in
|
|
the file. The name of the timestamp field in the Grok pattern must match what
|
|
is specified in the `timestamp_field` parameter. If that parameter is not
|
|
specified, the name of the timestamp field in the Grok pattern must match
|
|
"timestamp". If `grok_pattern` is not specified, the structure finder creates
|
|
a Grok pattern.
|
|
|
|
`has_header_row`::
|
|
(Optional, boolean) If you have set `format` to `delimited`, you can use this
|
|
parameter to indicate whether the column names are in the first row of the
|
|
file. If this parameter is not specified, the structure finder guesses based
|
|
on the similarity of the first row of the file to other rows.
|
|
|
|
`line_merge_size_limit`::
|
|
(Optional, unsigned integer) The maximum number of characters in a message
|
|
when lines are merged to form messages while analyzing semi-structured files.
|
|
The default is `10000`. If you have extremely long messages you may need to
|
|
increase this, but be aware that this may lead to very long processing times
|
|
if the way to group lines into messages is misdetected.
|
|
|
|
`lines_to_sample`::
|
|
(Optional, unsigned integer) The number of lines to include in the structural
|
|
analysis, starting from the beginning of the file. The minimum is 2; the
|
|
default is `1000`. If the value of this parameter is greater than the number
|
|
of lines in the file, the analysis proceeds (as long as there are at least two
|
|
lines in the file) for all of the lines. +
|
|
+
|
|
--
|
|
NOTE: The number of lines and the variation of the lines affects the speed of
|
|
the analysis. For example, if you upload a log file where the first 1000 lines
|
|
are all variations on the same message, the analysis will find more commonality
|
|
than would be seen with a bigger sample. If possible, however, it is more
|
|
efficient to upload a sample file with more variety in the first 1000 lines than
|
|
to request analysis of 100000 lines to achieve some variety.
|
|
--
|
|
|
|
`quote`::
|
|
(Optional, string) If you have set `format` to `delimited`, you can specify
|
|
the character used to quote the values in each row if they contain newlines or
|
|
the delimiter character. Only a single character is supported. If this
|
|
parameter is not specified, the default value is a double quote (`"`). If your
|
|
delimited file format does not use quoting, a workaround is to set this
|
|
argument to a character that does not appear anywhere in the sample.
|
|
|
|
`should_trim_fields`::
|
|
(Optional, boolean) If you have set `format` to `delimited`, you can specify
|
|
whether values between delimiters should have whitespace trimmed from them. If
|
|
this parameter is not specified and the delimiter is pipe (`|`), the default
|
|
value is `true`. Otherwise, the default value is `false`.
|
|
|
|
`timeout`::
|
|
(Optional, <<time-units,time units>>) Sets the maximum amount of time that the
|
|
structure analysis make take. If the analysis is still running when the
|
|
timeout expires then it will be aborted. The default value is 25 seconds.
|
|
|
|
`timestamp_field`::
|
|
(Optional, string) The name of the field that contains the primary timestamp
|
|
of each record in the file. In particular, if the file were ingested into an
|
|
index, this is the field that would be used to populate the `@timestamp` field.
|
|
+
|
|
--
|
|
If the `format` is `semi_structured_text`, this field must match the name of the
|
|
appropriate extraction in the `grok_pattern`. Therefore, for semi-structured
|
|
file formats, it is best not to specify this parameter unless `grok_pattern` is
|
|
also specified.
|
|
|
|
For structured file formats, if you specify this parameter, the field must exist
|
|
within the file.
|
|
|
|
If this parameter is not specified, the structure finder makes a decision about
|
|
which field (if any) is the primary timestamp field. For structured file
|
|
formats, it is not compulsory to have a timestamp in the file.
|
|
--
|
|
|
|
`timestamp_format`::
|
|
(Optional, string) The Java time format of the timestamp field in the file.
|
|
+
|
|
--
|
|
Only a subset of Java time format letter groups are supported:
|
|
|
|
* `a`
|
|
* `d`
|
|
* `dd`
|
|
* `EEE`
|
|
* `EEEE`
|
|
* `H`
|
|
* `HH`
|
|
* `h`
|
|
* `M`
|
|
* `MM`
|
|
* `MMM`
|
|
* `MMMM`
|
|
* `mm`
|
|
* `ss`
|
|
* `XX`
|
|
* `XXX`
|
|
* `yy`
|
|
* `yyyy`
|
|
* `zzz`
|
|
|
|
Additionally `S` letter groups (fractional seconds) of length one to nine are
|
|
supported providing they occur after `ss` and separated from the `ss` by a `.`,
|
|
`,` or `:`. Spacing and punctuation is also permitted with the exception of `?`,
|
|
newline and carriage return, together with literal text enclosed in single
|
|
quotes. For example, `MM/dd HH.mm.ss,SSSSSS 'in' yyyy` is a valid override
|
|
format.
|
|
|
|
One valuable use case for this parameter is when the format is semi-structured
|
|
text, there are multiple timestamp formats in the file, and you know which
|
|
format corresponds to the primary timestamp, but you do not want to specify the
|
|
full `grok_pattern`. Another is when the timestamp format is one that the
|
|
structure finder does not consider by default.
|
|
|
|
If this parameter is not specified, the structure finder chooses the best
|
|
format from a built-in set.
|
|
|
|
The following table provides the appropriate `timeformat` values for some example timestamps:
|
|
|
|
|===
|
|
| Timeformat | Presentation
|
|
|
|
| yyyy-MM-dd HH:mm:ssZ | 2019-04-20 13:15:22+0000
|
|
| EEE, d MMM yyyy HH:mm:ss Z | Sat, 20 Apr 2019 13:15:22 +0000
|
|
| dd.MM.yy HH:mm:ss.SSS | 20.04.19 13:15:22.285
|
|
|===
|
|
|
|
See
|
|
https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html[the Java date/time format documentation]
|
|
for more information about date and time format syntax.
|
|
|
|
--
|
|
|
|
[[ml-find-file-structure-request-body]]
|
|
== {api-request-body-title}
|
|
|
|
The text file that you want to analyze. It must contain data that is suitable to
|
|
be ingested into {es}. It does not need to be in JSON format and it does not
|
|
need to be UTF-8 encoded. The size is limited to the {es} HTTP receive buffer
|
|
size, which defaults to 100 Mb.
|
|
|
|
[[ml-find-file-structure-examples]]
|
|
== {api-examples-title}
|
|
|
|
[[ml-find-file-structure-example-nld-json]]
|
|
=== Ingesting newline-delimited JSON
|
|
|
|
Suppose you have a newline-delimited JSON file that contains information about
|
|
some books. You can send the contents to the `find_file_structure` endpoint:
|
|
|
|
[source,console]
|
|
----
|
|
POST _ml/find_file_structure
|
|
{"name": "Leviathan Wakes", "author": "James S.A. Corey", "release_date": "2011-06-02", "page_count": 561}
|
|
{"name": "Hyperion", "author": "Dan Simmons", "release_date": "1989-05-26", "page_count": 482}
|
|
{"name": "Dune", "author": "Frank Herbert", "release_date": "1965-06-01", "page_count": 604}
|
|
{"name": "Dune Messiah", "author": "Frank Herbert", "release_date": "1969-10-15", "page_count": 331}
|
|
{"name": "Children of Dune", "author": "Frank Herbert", "release_date": "1976-04-21", "page_count": 408}
|
|
{"name": "God Emperor of Dune", "author": "Frank Herbert", "release_date": "1981-05-28", "page_count": 454}
|
|
{"name": "Consider Phlebas", "author": "Iain M. Banks", "release_date": "1987-04-23", "page_count": 471}
|
|
{"name": "Pandora's Star", "author": "Peter F. Hamilton", "release_date": "2004-03-02", "page_count": 768}
|
|
{"name": "Revelation Space", "author": "Alastair Reynolds", "release_date": "2000-03-15", "page_count": 585}
|
|
{"name": "A Fire Upon the Deep", "author": "Vernor Vinge", "release_date": "1992-06-01", "page_count": 613}
|
|
{"name": "Ender's Game", "author": "Orson Scott Card", "release_date": "1985-06-01", "page_count": 324}
|
|
{"name": "1984", "author": "George Orwell", "release_date": "1985-06-01", "page_count": 328}
|
|
{"name": "Fahrenheit 451", "author": "Ray Bradbury", "release_date": "1953-10-15", "page_count": 227}
|
|
{"name": "Brave New World", "author": "Aldous Huxley", "release_date": "1932-06-01", "page_count": 268}
|
|
{"name": "Foundation", "author": "Isaac Asimov", "release_date": "1951-06-01", "page_count": 224}
|
|
{"name": "The Giver", "author": "Lois Lowry", "release_date": "1993-04-26", "page_count": 208}
|
|
{"name": "Slaughterhouse-Five", "author": "Kurt Vonnegut", "release_date": "1969-06-01", "page_count": 275}
|
|
{"name": "The Hitchhiker's Guide to the Galaxy", "author": "Douglas Adams", "release_date": "1979-10-12", "page_count": 180}
|
|
{"name": "Snow Crash", "author": "Neal Stephenson", "release_date": "1992-06-01", "page_count": 470}
|
|
{"name": "Neuromancer", "author": "William Gibson", "release_date": "1984-07-01", "page_count": 271}
|
|
{"name": "The Handmaid's Tale", "author": "Margaret Atwood", "release_date": "1985-06-01", "page_count": 311}
|
|
{"name": "Starship Troopers", "author": "Robert A. Heinlein", "release_date": "1959-12-01", "page_count": 335}
|
|
{"name": "The Left Hand of Darkness", "author": "Ursula K. Le Guin", "release_date": "1969-06-01", "page_count": 304}
|
|
{"name": "The Moon is a Harsh Mistress", "author": "Robert A. Heinlein", "release_date": "1966-04-01", "page_count": 288}
|
|
----
|
|
|
|
If the request does not encounter errors, you receive the following result:
|
|
|
|
[source,console-result]
|
|
----
|
|
{
|
|
"num_lines_analyzed" : 24, <1>
|
|
"num_messages_analyzed" : 24, <2>
|
|
"sample_start" : "{\"name\": \"Leviathan Wakes\", \"author\": \"James S.A. Corey\", \"release_date\": \"2011-06-02\", \"page_count\": 561}\n{\"name\": \"Hyperion\", \"author\": \"Dan Simmons\", \"release_date\": \"1989-05-26\", \"page_count\": 482}\n", <3>
|
|
"charset" : "UTF-8", <4>
|
|
"has_byte_order_marker" : false, <5>
|
|
"format" : "ndjson", <6>
|
|
"timestamp_field" : "release_date", <7>
|
|
"joda_timestamp_formats" : [ <8>
|
|
"ISO8601"
|
|
],
|
|
"java_timestamp_formats" : [ <9>
|
|
"ISO8601"
|
|
],
|
|
"need_client_timezone" : true, <10>
|
|
"mappings" : { <11>
|
|
"@timestamp" : {
|
|
"type" : "date"
|
|
},
|
|
"author" : {
|
|
"type" : "keyword"
|
|
},
|
|
"name" : {
|
|
"type" : "keyword"
|
|
},
|
|
"page_count" : {
|
|
"type" : "long"
|
|
},
|
|
"release_date" : {
|
|
"type" : "date",
|
|
"format" : "iso8601"
|
|
}
|
|
},
|
|
"ingest_pipeline" : {
|
|
"description" : "Ingest pipeline created by file structure finder",
|
|
"processors" : [
|
|
{
|
|
"date" : {
|
|
"field" : "release_date",
|
|
"timezone" : "{{ event.timezone }}",
|
|
"formats" : [
|
|
"ISO8601"
|
|
]
|
|
}
|
|
}
|
|
]
|
|
},
|
|
"field_stats" : { <12>
|
|
"author" : {
|
|
"count" : 24,
|
|
"cardinality" : 20,
|
|
"top_hits" : [
|
|
{
|
|
"value" : "Frank Herbert",
|
|
"count" : 4
|
|
},
|
|
{
|
|
"value" : "Robert A. Heinlein",
|
|
"count" : 2
|
|
},
|
|
{
|
|
"value" : "Alastair Reynolds",
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : "Aldous Huxley",
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : "Dan Simmons",
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : "Douglas Adams",
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : "George Orwell",
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : "Iain M. Banks",
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : "Isaac Asimov",
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : "James S.A. Corey",
|
|
"count" : 1
|
|
}
|
|
]
|
|
},
|
|
"name" : {
|
|
"count" : 24,
|
|
"cardinality" : 24,
|
|
"top_hits" : [
|
|
{
|
|
"value" : "1984",
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : "A Fire Upon the Deep",
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : "Brave New World",
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : "Children of Dune",
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : "Consider Phlebas",
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : "Dune",
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : "Dune Messiah",
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : "Ender's Game",
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : "Fahrenheit 451",
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : "Foundation",
|
|
"count" : 1
|
|
}
|
|
]
|
|
},
|
|
"page_count" : {
|
|
"count" : 24,
|
|
"cardinality" : 24,
|
|
"min_value" : 180,
|
|
"max_value" : 768,
|
|
"mean_value" : 387.0833333333333,
|
|
"median_value" : 329.5,
|
|
"top_hits" : [
|
|
{
|
|
"value" : 180,
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : 208,
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : 224,
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : 227,
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : 268,
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : 271,
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : 275,
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : 288,
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : 304,
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : 311,
|
|
"count" : 1
|
|
}
|
|
]
|
|
},
|
|
"release_date" : {
|
|
"count" : 24,
|
|
"cardinality" : 20,
|
|
"earliest" : "1932-06-01",
|
|
"latest" : "2011-06-02",
|
|
"top_hits" : [
|
|
{
|
|
"value" : "1985-06-01",
|
|
"count" : 3
|
|
},
|
|
{
|
|
"value" : "1969-06-01",
|
|
"count" : 2
|
|
},
|
|
{
|
|
"value" : "1992-06-01",
|
|
"count" : 2
|
|
},
|
|
{
|
|
"value" : "1932-06-01",
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : "1951-06-01",
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : "1953-10-15",
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : "1959-12-01",
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : "1965-06-01",
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : "1966-04-01",
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : "1969-10-15",
|
|
"count" : 1
|
|
}
|
|
]
|
|
}
|
|
}
|
|
}
|
|
----
|
|
// TESTRESPONSE[s/"sample_start" : ".*",/"sample_start" : "$body.sample_start",/]
|
|
// The substitution is because the "file" is pre-processed by the test harness,
|
|
// so the fields may get reordered in the JSON the endpoint sees
|
|
|
|
<1> `num_lines_analyzed` indicates how many lines of the file were analyzed.
|
|
<2> `num_messages_analyzed` indicates how many distinct messages the lines contained.
|
|
For NDJSON, this value is the same as `num_lines_analyzed`. For other file
|
|
formats, messages can span several lines.
|
|
<3> `sample_start` reproduces the first two messages in the file verbatim. This
|
|
may help to diagnose parse errors or accidental uploads of the wrong file.
|
|
<4> `charset` indicates the character encoding used to parse the file.
|
|
<5> For UTF character encodings, `has_byte_order_marker` indicates whether the
|
|
file begins with a byte order marker.
|
|
<6> `format` is one of `ndjson`, `xml`, `delimited` or `semi_structured_text`.
|
|
<7> The `timestamp_field` names the field considered most likely to be the
|
|
primary timestamp of each document.
|
|
<8> `joda_timestamp_formats` are used to tell Logstash how to parse timestamps.
|
|
<9> `java_timestamp_formats` are the Java time formats recognized in the time
|
|
fields. Elasticsearch mappings and Ingest pipeline use this format.
|
|
<10> If a timestamp format is detected that does not include a timezone,
|
|
`need_client_timezone` will be `true`. The server that parses the file must
|
|
therefore be told the correct timezone by the client.
|
|
<11> `mappings` contains some suitable mappings for an index into which the data
|
|
could be ingested. In this case, the `release_date` field has been given a
|
|
`keyword` type as it is not considered specific enough to convert to the
|
|
`date` type.
|
|
<12> `field_stats` contains the most common values of each field, plus basic
|
|
numeric statistics for the numeric `page_count` field. This information
|
|
may provide clues that the data needs to be cleaned or transformed prior
|
|
to use by other {ml} functionality.
|
|
|
|
|
|
[[ml-find-file-structure-example-nyc]]
|
|
=== Finding the structure of NYC yellow cab example data
|
|
|
|
The next example shows how it's possible to find the structure of some New York
|
|
City yellow cab trip data. The first `curl` command downloads the data, the
|
|
first 20000 lines of which are then piped into the `find_file_structure`
|
|
endpoint. The `lines_to_sample` query parameter of the endpoint is set to 20000
|
|
to match what is specified in the `head` command.
|
|
|
|
[source,js]
|
|
----
|
|
curl -s "s3.amazonaws.com/nyc-tlc/trip+data/yellow_tripdata_2018-06.csv" | head -20000 | curl -s -H "Content-Type: application/json" -XPOST "localhost:9200/_ml/find_file_structure?pretty&lines_to_sample=20000" -T -
|
|
----
|
|
// NOTCONSOLE
|
|
// Not converting to console because this shows how curl can be used
|
|
|
|
--
|
|
NOTE: The `Content-Type: application/json` header must be set even though in
|
|
this case the data is not JSON. (Alternatively the `Content-Type` can be set
|
|
to any other supported by {es}, but it must be set.)
|
|
--
|
|
|
|
If the request does not encounter errors, you receive the following result:
|
|
|
|
[source,js]
|
|
----
|
|
{
|
|
"num_lines_analyzed" : 20000,
|
|
"num_messages_analyzed" : 19998, <1>
|
|
"sample_start" : "VendorID,tpep_pickup_datetime,tpep_dropoff_datetime,passenger_count,trip_distance,RatecodeID,store_and_fwd_flag,PULocationID,DOLocationID,payment_type,fare_amount,extra,mta_tax,tip_amount,tolls_amount,improvement_surcharge,total_amount\n\n1,2018-06-01 00:15:40,2018-06-01 00:16:46,1,.00,1,N,145,145,2,3,0.5,0.5,0,0,0.3,4.3\n",
|
|
"charset" : "UTF-8",
|
|
"has_byte_order_marker" : false,
|
|
"format" : "delimited", <2>
|
|
"multiline_start_pattern" : "^.*?,\"?\\d{4}-\\d{2}-\\d{2}[T ]\\d{2}:\\d{2}",
|
|
"exclude_lines_pattern" : "^\"?VendorID\"?,\"?tpep_pickup_datetime\"?,\"?tpep_dropoff_datetime\"?,\"?passenger_count\"?,\"?trip_distance\"?,\"?RatecodeID\"?,\"?store_and_fwd_flag\"?,\"?PULocationID\"?,\"?DOLocationID\"?,\"?payment_type\"?,\"?fare_amount\"?,\"?extra\"?,\"?mta_tax\"?,\"?tip_amount\"?,\"?tolls_amount\"?,\"?improvement_surcharge\"?,\"?total_amount\"?",
|
|
"column_names" : [ <3>
|
|
"VendorID",
|
|
"tpep_pickup_datetime",
|
|
"tpep_dropoff_datetime",
|
|
"passenger_count",
|
|
"trip_distance",
|
|
"RatecodeID",
|
|
"store_and_fwd_flag",
|
|
"PULocationID",
|
|
"DOLocationID",
|
|
"payment_type",
|
|
"fare_amount",
|
|
"extra",
|
|
"mta_tax",
|
|
"tip_amount",
|
|
"tolls_amount",
|
|
"improvement_surcharge",
|
|
"total_amount"
|
|
],
|
|
"has_header_row" : true, <4>
|
|
"delimiter" : ",", <5>
|
|
"quote" : "\"", <6>
|
|
"timestamp_field" : "tpep_pickup_datetime", <7>
|
|
"joda_timestamp_formats" : [ <8>
|
|
"YYYY-MM-dd HH:mm:ss"
|
|
],
|
|
"java_timestamp_formats" : [ <9>
|
|
"yyyy-MM-dd HH:mm:ss"
|
|
],
|
|
"need_client_timezone" : true, <10>
|
|
"mappings" : {
|
|
"@timestamp" : {
|
|
"type" : "date"
|
|
},
|
|
"DOLocationID" : {
|
|
"type" : "long"
|
|
},
|
|
"PULocationID" : {
|
|
"type" : "long"
|
|
},
|
|
"RatecodeID" : {
|
|
"type" : "long"
|
|
},
|
|
"VendorID" : {
|
|
"type" : "long"
|
|
},
|
|
"extra" : {
|
|
"type" : "double"
|
|
},
|
|
"fare_amount" : {
|
|
"type" : "double"
|
|
},
|
|
"improvement_surcharge" : {
|
|
"type" : "double"
|
|
},
|
|
"mta_tax" : {
|
|
"type" : "double"
|
|
},
|
|
"passenger_count" : {
|
|
"type" : "long"
|
|
},
|
|
"payment_type" : {
|
|
"type" : "long"
|
|
},
|
|
"store_and_fwd_flag" : {
|
|
"type" : "keyword"
|
|
},
|
|
"tip_amount" : {
|
|
"type" : "double"
|
|
},
|
|
"tolls_amount" : {
|
|
"type" : "double"
|
|
},
|
|
"total_amount" : {
|
|
"type" : "double"
|
|
},
|
|
"tpep_dropoff_datetime" : {
|
|
"type" : "date",
|
|
"format" : "yyyy-MM-dd HH:mm:ss"
|
|
},
|
|
"tpep_pickup_datetime" : {
|
|
"type" : "date",
|
|
"format" : "yyyy-MM-dd HH:mm:ss"
|
|
},
|
|
"trip_distance" : {
|
|
"type" : "double"
|
|
}
|
|
},
|
|
"ingest_pipeline" : {
|
|
"description" : "Ingest pipeline created by file structure finder",
|
|
"processors" : [
|
|
{
|
|
"csv" : {
|
|
"field" : "message",
|
|
"target_fields" : [
|
|
"VendorID",
|
|
"tpep_pickup_datetime",
|
|
"tpep_dropoff_datetime",
|
|
"passenger_count",
|
|
"trip_distance",
|
|
"RatecodeID",
|
|
"store_and_fwd_flag",
|
|
"PULocationID",
|
|
"DOLocationID",
|
|
"payment_type",
|
|
"fare_amount",
|
|
"extra",
|
|
"mta_tax",
|
|
"tip_amount",
|
|
"tolls_amount",
|
|
"improvement_surcharge",
|
|
"total_amount"
|
|
]
|
|
}
|
|
},
|
|
{
|
|
"date" : {
|
|
"field" : "tpep_pickup_datetime",
|
|
"timezone" : "{{ event.timezone }}",
|
|
"formats" : [
|
|
"yyyy-MM-dd HH:mm:ss"
|
|
]
|
|
}
|
|
},
|
|
{
|
|
"convert" : {
|
|
"field" : "DOLocationID",
|
|
"type" : "long"
|
|
}
|
|
},
|
|
{
|
|
"convert" : {
|
|
"field" : "PULocationID",
|
|
"type" : "long"
|
|
}
|
|
},
|
|
{
|
|
"convert" : {
|
|
"field" : "RatecodeID",
|
|
"type" : "long"
|
|
}
|
|
},
|
|
{
|
|
"convert" : {
|
|
"field" : "VendorID",
|
|
"type" : "long"
|
|
}
|
|
},
|
|
{
|
|
"convert" : {
|
|
"field" : "extra",
|
|
"type" : "double"
|
|
}
|
|
},
|
|
{
|
|
"convert" : {
|
|
"field" : "fare_amount",
|
|
"type" : "double"
|
|
}
|
|
},
|
|
{
|
|
"convert" : {
|
|
"field" : "improvement_surcharge",
|
|
"type" : "double"
|
|
}
|
|
},
|
|
{
|
|
"convert" : {
|
|
"field" : "mta_tax",
|
|
"type" : "double"
|
|
}
|
|
},
|
|
{
|
|
"convert" : {
|
|
"field" : "passenger_count",
|
|
"type" : "long"
|
|
}
|
|
},
|
|
{
|
|
"convert" : {
|
|
"field" : "payment_type",
|
|
"type" : "long"
|
|
}
|
|
},
|
|
{
|
|
"convert" : {
|
|
"field" : "tip_amount",
|
|
"type" : "double"
|
|
}
|
|
},
|
|
{
|
|
"convert" : {
|
|
"field" : "tolls_amount",
|
|
"type" : "double"
|
|
}
|
|
},
|
|
{
|
|
"convert" : {
|
|
"field" : "total_amount",
|
|
"type" : "double"
|
|
}
|
|
},
|
|
{
|
|
"convert" : {
|
|
"field" : "trip_distance",
|
|
"type" : "double"
|
|
}
|
|
},
|
|
{
|
|
"remove" : {
|
|
"field" : "message"
|
|
}
|
|
}
|
|
]
|
|
},
|
|
"field_stats" : {
|
|
"DOLocationID" : {
|
|
"count" : 19998,
|
|
"cardinality" : 240,
|
|
"min_value" : 1,
|
|
"max_value" : 265,
|
|
"mean_value" : 150.26532653265312,
|
|
"median_value" : 148,
|
|
"top_hits" : [
|
|
{
|
|
"value" : 79,
|
|
"count" : 760
|
|
},
|
|
{
|
|
"value" : 48,
|
|
"count" : 683
|
|
},
|
|
{
|
|
"value" : 68,
|
|
"count" : 529
|
|
},
|
|
{
|
|
"value" : 170,
|
|
"count" : 506
|
|
},
|
|
{
|
|
"value" : 107,
|
|
"count" : 468
|
|
},
|
|
{
|
|
"value" : 249,
|
|
"count" : 457
|
|
},
|
|
{
|
|
"value" : 230,
|
|
"count" : 441
|
|
},
|
|
{
|
|
"value" : 186,
|
|
"count" : 432
|
|
},
|
|
{
|
|
"value" : 141,
|
|
"count" : 409
|
|
},
|
|
{
|
|
"value" : 263,
|
|
"count" : 386
|
|
}
|
|
]
|
|
},
|
|
"PULocationID" : {
|
|
"count" : 19998,
|
|
"cardinality" : 154,
|
|
"min_value" : 1,
|
|
"max_value" : 265,
|
|
"mean_value" : 153.4042404240424,
|
|
"median_value" : 148,
|
|
"top_hits" : [
|
|
{
|
|
"value" : 79,
|
|
"count" : 1067
|
|
},
|
|
{
|
|
"value" : 230,
|
|
"count" : 949
|
|
},
|
|
{
|
|
"value" : 148,
|
|
"count" : 940
|
|
},
|
|
{
|
|
"value" : 132,
|
|
"count" : 897
|
|
},
|
|
{
|
|
"value" : 48,
|
|
"count" : 853
|
|
},
|
|
{
|
|
"value" : 161,
|
|
"count" : 820
|
|
},
|
|
{
|
|
"value" : 234,
|
|
"count" : 750
|
|
},
|
|
{
|
|
"value" : 249,
|
|
"count" : 722
|
|
},
|
|
{
|
|
"value" : 164,
|
|
"count" : 663
|
|
},
|
|
{
|
|
"value" : 114,
|
|
"count" : 646
|
|
}
|
|
]
|
|
},
|
|
"RatecodeID" : {
|
|
"count" : 19998,
|
|
"cardinality" : 5,
|
|
"min_value" : 1,
|
|
"max_value" : 5,
|
|
"mean_value" : 1.0656565656565653,
|
|
"median_value" : 1,
|
|
"top_hits" : [
|
|
{
|
|
"value" : 1,
|
|
"count" : 19311
|
|
},
|
|
{
|
|
"value" : 2,
|
|
"count" : 468
|
|
},
|
|
{
|
|
"value" : 5,
|
|
"count" : 195
|
|
},
|
|
{
|
|
"value" : 4,
|
|
"count" : 17
|
|
},
|
|
{
|
|
"value" : 3,
|
|
"count" : 7
|
|
}
|
|
]
|
|
},
|
|
"VendorID" : {
|
|
"count" : 19998,
|
|
"cardinality" : 2,
|
|
"min_value" : 1,
|
|
"max_value" : 2,
|
|
"mean_value" : 1.59005900590059,
|
|
"median_value" : 2,
|
|
"top_hits" : [
|
|
{
|
|
"value" : 2,
|
|
"count" : 11800
|
|
},
|
|
{
|
|
"value" : 1,
|
|
"count" : 8198
|
|
}
|
|
]
|
|
},
|
|
"extra" : {
|
|
"count" : 19998,
|
|
"cardinality" : 3,
|
|
"min_value" : -0.5,
|
|
"max_value" : 0.5,
|
|
"mean_value" : 0.4815981598159816,
|
|
"median_value" : 0.5,
|
|
"top_hits" : [
|
|
{
|
|
"value" : 0.5,
|
|
"count" : 19281
|
|
},
|
|
{
|
|
"value" : 0,
|
|
"count" : 698
|
|
},
|
|
{
|
|
"value" : -0.5,
|
|
"count" : 19
|
|
}
|
|
]
|
|
},
|
|
"fare_amount" : {
|
|
"count" : 19998,
|
|
"cardinality" : 208,
|
|
"min_value" : -100,
|
|
"max_value" : 300,
|
|
"mean_value" : 13.937719771977209,
|
|
"median_value" : 9.5,
|
|
"top_hits" : [
|
|
{
|
|
"value" : 6,
|
|
"count" : 1004
|
|
},
|
|
{
|
|
"value" : 6.5,
|
|
"count" : 935
|
|
},
|
|
{
|
|
"value" : 5.5,
|
|
"count" : 909
|
|
},
|
|
{
|
|
"value" : 7,
|
|
"count" : 903
|
|
},
|
|
{
|
|
"value" : 5,
|
|
"count" : 889
|
|
},
|
|
{
|
|
"value" : 7.5,
|
|
"count" : 854
|
|
},
|
|
{
|
|
"value" : 4.5,
|
|
"count" : 802
|
|
},
|
|
{
|
|
"value" : 8.5,
|
|
"count" : 790
|
|
},
|
|
{
|
|
"value" : 8,
|
|
"count" : 789
|
|
},
|
|
{
|
|
"value" : 9,
|
|
"count" : 711
|
|
}
|
|
]
|
|
},
|
|
"improvement_surcharge" : {
|
|
"count" : 19998,
|
|
"cardinality" : 3,
|
|
"min_value" : -0.3,
|
|
"max_value" : 0.3,
|
|
"mean_value" : 0.29915991599159913,
|
|
"median_value" : 0.3,
|
|
"top_hits" : [
|
|
{
|
|
"value" : 0.3,
|
|
"count" : 19964
|
|
},
|
|
{
|
|
"value" : -0.3,
|
|
"count" : 22
|
|
},
|
|
{
|
|
"value" : 0,
|
|
"count" : 12
|
|
}
|
|
]
|
|
},
|
|
"mta_tax" : {
|
|
"count" : 19998,
|
|
"cardinality" : 3,
|
|
"min_value" : -0.5,
|
|
"max_value" : 0.5,
|
|
"mean_value" : 0.4962246224622462,
|
|
"median_value" : 0.5,
|
|
"top_hits" : [
|
|
{
|
|
"value" : 0.5,
|
|
"count" : 19868
|
|
},
|
|
{
|
|
"value" : 0,
|
|
"count" : 109
|
|
},
|
|
{
|
|
"value" : -0.5,
|
|
"count" : 21
|
|
}
|
|
]
|
|
},
|
|
"passenger_count" : {
|
|
"count" : 19998,
|
|
"cardinality" : 7,
|
|
"min_value" : 0,
|
|
"max_value" : 6,
|
|
"mean_value" : 1.6201620162016201,
|
|
"median_value" : 1,
|
|
"top_hits" : [
|
|
{
|
|
"value" : 1,
|
|
"count" : 14219
|
|
},
|
|
{
|
|
"value" : 2,
|
|
"count" : 2886
|
|
},
|
|
{
|
|
"value" : 5,
|
|
"count" : 1047
|
|
},
|
|
{
|
|
"value" : 3,
|
|
"count" : 804
|
|
},
|
|
{
|
|
"value" : 6,
|
|
"count" : 523
|
|
},
|
|
{
|
|
"value" : 4,
|
|
"count" : 406
|
|
},
|
|
{
|
|
"value" : 0,
|
|
"count" : 113
|
|
}
|
|
]
|
|
},
|
|
"payment_type" : {
|
|
"count" : 19998,
|
|
"cardinality" : 4,
|
|
"min_value" : 1,
|
|
"max_value" : 4,
|
|
"mean_value" : 1.315631563156316,
|
|
"median_value" : 1,
|
|
"top_hits" : [
|
|
{
|
|
"value" : 1,
|
|
"count" : 13936
|
|
},
|
|
{
|
|
"value" : 2,
|
|
"count" : 5857
|
|
},
|
|
{
|
|
"value" : 3,
|
|
"count" : 160
|
|
},
|
|
{
|
|
"value" : 4,
|
|
"count" : 45
|
|
}
|
|
]
|
|
},
|
|
"store_and_fwd_flag" : {
|
|
"count" : 19998,
|
|
"cardinality" : 2,
|
|
"top_hits" : [
|
|
{
|
|
"value" : "N",
|
|
"count" : 19910
|
|
},
|
|
{
|
|
"value" : "Y",
|
|
"count" : 88
|
|
}
|
|
]
|
|
},
|
|
"tip_amount" : {
|
|
"count" : 19998,
|
|
"cardinality" : 717,
|
|
"min_value" : 0,
|
|
"max_value" : 128,
|
|
"mean_value" : 2.010959095909593,
|
|
"median_value" : 1.45,
|
|
"top_hits" : [
|
|
{
|
|
"value" : 0,
|
|
"count" : 6917
|
|
},
|
|
{
|
|
"value" : 1,
|
|
"count" : 1178
|
|
},
|
|
{
|
|
"value" : 2,
|
|
"count" : 624
|
|
},
|
|
{
|
|
"value" : 3,
|
|
"count" : 248
|
|
},
|
|
{
|
|
"value" : 1.56,
|
|
"count" : 206
|
|
},
|
|
{
|
|
"value" : 1.46,
|
|
"count" : 205
|
|
},
|
|
{
|
|
"value" : 1.76,
|
|
"count" : 196
|
|
},
|
|
{
|
|
"value" : 1.45,
|
|
"count" : 195
|
|
},
|
|
{
|
|
"value" : 1.36,
|
|
"count" : 191
|
|
},
|
|
{
|
|
"value" : 1.5,
|
|
"count" : 187
|
|
}
|
|
]
|
|
},
|
|
"tolls_amount" : {
|
|
"count" : 19998,
|
|
"cardinality" : 26,
|
|
"min_value" : 0,
|
|
"max_value" : 35,
|
|
"mean_value" : 0.2729697969796978,
|
|
"median_value" : 0,
|
|
"top_hits" : [
|
|
{
|
|
"value" : 0,
|
|
"count" : 19107
|
|
},
|
|
{
|
|
"value" : 5.76,
|
|
"count" : 791
|
|
},
|
|
{
|
|
"value" : 10.5,
|
|
"count" : 36
|
|
},
|
|
{
|
|
"value" : 2.64,
|
|
"count" : 21
|
|
},
|
|
{
|
|
"value" : 11.52,
|
|
"count" : 8
|
|
},
|
|
{
|
|
"value" : 5.54,
|
|
"count" : 4
|
|
},
|
|
{
|
|
"value" : 8.5,
|
|
"count" : 4
|
|
},
|
|
{
|
|
"value" : 17.28,
|
|
"count" : 4
|
|
},
|
|
{
|
|
"value" : 2,
|
|
"count" : 2
|
|
},
|
|
{
|
|
"value" : 2.16,
|
|
"count" : 2
|
|
}
|
|
]
|
|
},
|
|
"total_amount" : {
|
|
"count" : 19998,
|
|
"cardinality" : 1267,
|
|
"min_value" : -100.3,
|
|
"max_value" : 389.12,
|
|
"mean_value" : 17.499898989898995,
|
|
"median_value" : 12.35,
|
|
"top_hits" : [
|
|
{
|
|
"value" : 7.3,
|
|
"count" : 478
|
|
},
|
|
{
|
|
"value" : 8.3,
|
|
"count" : 443
|
|
},
|
|
{
|
|
"value" : 8.8,
|
|
"count" : 420
|
|
},
|
|
{
|
|
"value" : 6.8,
|
|
"count" : 406
|
|
},
|
|
{
|
|
"value" : 7.8,
|
|
"count" : 405
|
|
},
|
|
{
|
|
"value" : 6.3,
|
|
"count" : 371
|
|
},
|
|
{
|
|
"value" : 9.8,
|
|
"count" : 368
|
|
},
|
|
{
|
|
"value" : 5.8,
|
|
"count" : 362
|
|
},
|
|
{
|
|
"value" : 9.3,
|
|
"count" : 332
|
|
},
|
|
{
|
|
"value" : 10.3,
|
|
"count" : 332
|
|
}
|
|
]
|
|
},
|
|
"tpep_dropoff_datetime" : {
|
|
"count" : 19998,
|
|
"cardinality" : 9066,
|
|
"earliest" : "2018-05-31 06:18:15",
|
|
"latest" : "2018-06-02 02:25:44",
|
|
"top_hits" : [
|
|
{
|
|
"value" : "2018-06-01 01:12:12",
|
|
"count" : 10
|
|
},
|
|
{
|
|
"value" : "2018-06-01 00:32:15",
|
|
"count" : 9
|
|
},
|
|
{
|
|
"value" : "2018-06-01 00:44:27",
|
|
"count" : 9
|
|
},
|
|
{
|
|
"value" : "2018-06-01 00:46:42",
|
|
"count" : 9
|
|
},
|
|
{
|
|
"value" : "2018-06-01 01:03:22",
|
|
"count" : 9
|
|
},
|
|
{
|
|
"value" : "2018-06-01 01:05:13",
|
|
"count" : 9
|
|
},
|
|
{
|
|
"value" : "2018-06-01 00:11:20",
|
|
"count" : 8
|
|
},
|
|
{
|
|
"value" : "2018-06-01 00:16:03",
|
|
"count" : 8
|
|
},
|
|
{
|
|
"value" : "2018-06-01 00:19:47",
|
|
"count" : 8
|
|
},
|
|
{
|
|
"value" : "2018-06-01 00:25:17",
|
|
"count" : 8
|
|
}
|
|
]
|
|
},
|
|
"tpep_pickup_datetime" : {
|
|
"count" : 19998,
|
|
"cardinality" : 8760,
|
|
"earliest" : "2018-05-31 06:08:31",
|
|
"latest" : "2018-06-02 01:21:21",
|
|
"top_hits" : [
|
|
{
|
|
"value" : "2018-06-01 00:01:23",
|
|
"count" : 12
|
|
},
|
|
{
|
|
"value" : "2018-06-01 00:04:31",
|
|
"count" : 10
|
|
},
|
|
{
|
|
"value" : "2018-06-01 00:05:38",
|
|
"count" : 10
|
|
},
|
|
{
|
|
"value" : "2018-06-01 00:09:50",
|
|
"count" : 10
|
|
},
|
|
{
|
|
"value" : "2018-06-01 00:12:01",
|
|
"count" : 10
|
|
},
|
|
{
|
|
"value" : "2018-06-01 00:14:17",
|
|
"count" : 10
|
|
},
|
|
{
|
|
"value" : "2018-06-01 00:00:34",
|
|
"count" : 9
|
|
},
|
|
{
|
|
"value" : "2018-06-01 00:00:40",
|
|
"count" : 9
|
|
},
|
|
{
|
|
"value" : "2018-06-01 00:02:53",
|
|
"count" : 9
|
|
},
|
|
{
|
|
"value" : "2018-06-01 00:05:40",
|
|
"count" : 9
|
|
}
|
|
]
|
|
},
|
|
"trip_distance" : {
|
|
"count" : 19998,
|
|
"cardinality" : 1687,
|
|
"min_value" : 0,
|
|
"max_value" : 64.63,
|
|
"mean_value" : 3.6521062106210715,
|
|
"median_value" : 2.16,
|
|
"top_hits" : [
|
|
{
|
|
"value" : 0.9,
|
|
"count" : 335
|
|
},
|
|
{
|
|
"value" : 0.8,
|
|
"count" : 320
|
|
},
|
|
{
|
|
"value" : 1.1,
|
|
"count" : 316
|
|
},
|
|
{
|
|
"value" : 0.7,
|
|
"count" : 304
|
|
},
|
|
{
|
|
"value" : 1.2,
|
|
"count" : 303
|
|
},
|
|
{
|
|
"value" : 1,
|
|
"count" : 296
|
|
},
|
|
{
|
|
"value" : 1.3,
|
|
"count" : 280
|
|
},
|
|
{
|
|
"value" : 1.5,
|
|
"count" : 268
|
|
},
|
|
{
|
|
"value" : 1.6,
|
|
"count" : 268
|
|
},
|
|
{
|
|
"value" : 0.6,
|
|
"count" : 256
|
|
}
|
|
]
|
|
}
|
|
}
|
|
}
|
|
----
|
|
// NOTCONSOLE
|
|
|
|
<1> `num_messages_analyzed` is 2 lower than `num_lines_analyzed` because only
|
|
data records count as messages. The first line contains the column names
|
|
and in this sample the second line is blank.
|
|
<2> Unlike the first example, in this case the `format` has been identified as
|
|
`delimited`.
|
|
<3> Because the `format` is `delimited`, the `column_names` field in the output
|
|
lists the column names in the order they appear in the sample.
|
|
<4> `has_header_row` indicates that for this sample the column names were in
|
|
the first row of the sample. (If they hadn't been then it would have been
|
|
a good idea to specify them in the `column_names` query parameter.)
|
|
<5> The `delimiter` for this sample is a comma, as it's a CSV file.
|
|
<6> The `quote` character is the default double quote. (The structure finder
|
|
does not attempt to deduce any other quote character, so if you have a
|
|
delimited file that's quoted with some other character you must specify it
|
|
using the `quote` query parameter.)
|
|
<7> The `timestamp_field` has been chosen to be `tpep_pickup_datetime`.
|
|
`tpep_dropoff_datetime` would work just as well, but `tpep_pickup_datetime`
|
|
was chosen because it comes first in the column order. If you prefer
|
|
`tpep_dropoff_datetime` then force it to be chosen using the
|
|
`timestamp_field` query parameter.
|
|
<8> `joda_timestamp_formats` are used to tell Logstash how to parse timestamps.
|
|
<9> `java_timestamp_formats` are the Java time formats recognized in the time
|
|
fields. Elasticsearch mappings and Ingest pipeline use this format.
|
|
<10> The timestamp format in this sample doesn't specify a timezone, so to
|
|
accurately convert them to UTC timestamps to store in Elasticsearch it's
|
|
necessary to supply the timezone they relate to. `need_client_timezone`
|
|
will be `false` for timestamp formats that include the timezone.
|
|
|
|
|
|
[[ml-find-file-structure-example-timeout]]
|
|
=== Setting the timeout parameter
|
|
|
|
If you try to analyze a lot of data then the analysis will take a long time.
|
|
If you want to limit the amount of processing your {es} cluster performs for
|
|
a request, use the `timeout` query parameter. The analysis will be aborted and
|
|
an error returned when the timeout expires. For example, you can replace 20000
|
|
lines in the previous example with 200000 and set a 1 second timeout on the
|
|
analysis:
|
|
|
|
[source,js]
|
|
----
|
|
curl -s "s3.amazonaws.com/nyc-tlc/trip+data/yellow_tripdata_2018-06.csv" | head -200000 | curl -s -H "Content-Type: application/json" -XPOST "localhost:9200/_ml/find_file_structure?pretty&lines_to_sample=200000&timeout=1s" -T -
|
|
----
|
|
// NOTCONSOLE
|
|
// Not converting to console because this shows how curl can be used
|
|
|
|
Unless you are using an incredibly fast computer you'll receive a timeout error:
|
|
|
|
[source,js]
|
|
----
|
|
{
|
|
"error" : {
|
|
"root_cause" : [
|
|
{
|
|
"type" : "timeout_exception",
|
|
"reason" : "Aborting structure analysis during [delimited record parsing] as it has taken longer than the timeout of [1s]"
|
|
}
|
|
],
|
|
"type" : "timeout_exception",
|
|
"reason" : "Aborting structure analysis during [delimited record parsing] as it has taken longer than the timeout of [1s]"
|
|
},
|
|
"status" : 500
|
|
}
|
|
----
|
|
// NOTCONSOLE
|
|
|
|
--
|
|
NOTE: If you try the example above yourself you will note that the overall
|
|
running time of the `curl` commands is considerably longer than 1 second. This
|
|
is because it takes a while to download 200000 lines of CSV from the internet,
|
|
and the timeout is measured from the time this endpoint starts to process the
|
|
data.
|
|
--
|
|
|
|
|
|
[[ml-find-file-structure-example-eslog]]
|
|
=== Analyzing {es} log files
|
|
|
|
This is an example of analyzing {es}'s own log file:
|
|
|
|
[source,js]
|
|
----
|
|
curl -s -H "Content-Type: application/json" -XPOST "localhost:9200/_ml/find_file_structure?pretty" -T "$ES_HOME/logs/elasticsearch.log"
|
|
----
|
|
// NOTCONSOLE
|
|
// Not converting to console because this shows how curl can be used
|
|
|
|
If the request does not encounter errors, the result will look something like
|
|
this:
|
|
|
|
[source,js]
|
|
----
|
|
{
|
|
"num_lines_analyzed" : 53,
|
|
"num_messages_analyzed" : 53,
|
|
"sample_start" : "[2018-09-27T14:39:28,518][INFO ][o.e.e.NodeEnvironment ] [node-0] using [1] data paths, mounts [[/ (/dev/disk1)]], net usable_space [165.4gb], net total_space [464.7gb], types [hfs]\n[2018-09-27T14:39:28,521][INFO ][o.e.e.NodeEnvironment ] [node-0] heap size [494.9mb], compressed ordinary object pointers [true]\n",
|
|
"charset" : "UTF-8",
|
|
"has_byte_order_marker" : false,
|
|
"format" : "semi_structured_text", <1>
|
|
"multiline_start_pattern" : "^\\[\\b\\d{4}-\\d{2}-\\d{2}[T ]\\d{2}:\\d{2}", <2>
|
|
"grok_pattern" : "\\[%{TIMESTAMP_ISO8601:timestamp}\\]\\[%{LOGLEVEL:loglevel}.*", <3>
|
|
"timestamp_field" : "timestamp",
|
|
"joda_timestamp_formats" : [
|
|
"ISO8601"
|
|
],
|
|
"java_timestamp_formats" : [
|
|
"ISO8601"
|
|
],
|
|
"need_client_timezone" : true,
|
|
"mappings" : {
|
|
"@timestamp" : {
|
|
"type" : "date"
|
|
},
|
|
"loglevel" : {
|
|
"type" : "keyword"
|
|
},
|
|
"message" : {
|
|
"type" : "text"
|
|
}
|
|
},
|
|
"ingest_pipeline" : {
|
|
"description" : "Ingest pipeline created by file structure finder",
|
|
"processors" : [
|
|
{
|
|
"grok" : {
|
|
"field" : "message",
|
|
"patterns" : [
|
|
"\\[%{TIMESTAMP_ISO8601:timestamp}\\]\\[%{LOGLEVEL:loglevel}.*"
|
|
]
|
|
}
|
|
},
|
|
{
|
|
"date" : {
|
|
"field" : "timestamp",
|
|
"timezone" : "{{ event.timezone }}",
|
|
"formats" : [
|
|
"ISO8601"
|
|
]
|
|
}
|
|
},
|
|
{
|
|
"remove" : {
|
|
"field" : "timestamp"
|
|
}
|
|
}
|
|
]
|
|
},
|
|
"field_stats" : {
|
|
"loglevel" : {
|
|
"count" : 53,
|
|
"cardinality" : 3,
|
|
"top_hits" : [
|
|
{
|
|
"value" : "INFO",
|
|
"count" : 51
|
|
},
|
|
{
|
|
"value" : "DEBUG",
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : "WARN",
|
|
"count" : 1
|
|
}
|
|
]
|
|
},
|
|
"timestamp" : {
|
|
"count" : 53,
|
|
"cardinality" : 28,
|
|
"earliest" : "2018-09-27T14:39:28,518",
|
|
"latest" : "2018-09-27T14:39:37,012",
|
|
"top_hits" : [
|
|
{
|
|
"value" : "2018-09-27T14:39:29,859",
|
|
"count" : 10
|
|
},
|
|
{
|
|
"value" : "2018-09-27T14:39:29,860",
|
|
"count" : 9
|
|
},
|
|
{
|
|
"value" : "2018-09-27T14:39:29,858",
|
|
"count" : 6
|
|
},
|
|
{
|
|
"value" : "2018-09-27T14:39:28,523",
|
|
"count" : 3
|
|
},
|
|
{
|
|
"value" : "2018-09-27T14:39:34,234",
|
|
"count" : 2
|
|
},
|
|
{
|
|
"value" : "2018-09-27T14:39:28,518",
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : "2018-09-27T14:39:28,521",
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : "2018-09-27T14:39:28,522",
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : "2018-09-27T14:39:29,861",
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : "2018-09-27T14:39:32,786",
|
|
"count" : 1
|
|
}
|
|
]
|
|
}
|
|
}
|
|
}
|
|
----
|
|
// NOTCONSOLE
|
|
|
|
<1> This time the `format` has been identified as `semi_structured_text`.
|
|
<2> The `multiline_start_pattern` is set on the basis that the timestamp appears
|
|
in the first line of each multi-line log message.
|
|
<3> A very simple `grok_pattern` has been created, which extracts the timestamp
|
|
and recognizable fields that appear in every analyzed message. In this case
|
|
the only field that was recognized beyond the timestamp was the log level.
|
|
|
|
|
|
[[ml-find-file-structure-example-grok]]
|
|
=== Specifying `grok_pattern` as query parameter
|
|
|
|
If you recognize more fields than the simple `grok_pattern` produced by the
|
|
structure finder unaided then you can resubmit the request specifying a more
|
|
advanced `grok_pattern` as a query parameter and the structure finder will
|
|
calculate `field_stats` for your additional fields.
|
|
|
|
In the case of the {es} log a more complete Grok pattern is
|
|
`\[%{TIMESTAMP_ISO8601:timestamp}\]\[%{LOGLEVEL:loglevel} *\]\[%{JAVACLASS:class} *\] \[%{HOSTNAME:node}\] %{JAVALOGMESSAGE:message}`.
|
|
You can analyze the same log file again, submitting this `grok_pattern` as a
|
|
query parameter (appropriately URL escaped):
|
|
|
|
[source,js]
|
|
----
|
|
curl -s -H "Content-Type: application/json" -XPOST "localhost:9200/_ml/find_file_structure?pretty&format=semi_structured_text&grok_pattern=%5C%5B%25%7BTIMESTAMP_ISO8601:timestamp%7D%5C%5D%5C%5B%25%7BLOGLEVEL:loglevel%7D%20*%5C%5D%5C%5B%25%7BJAVACLASS:class%7D%20*%5C%5D%20%5C%5B%25%7BHOSTNAME:node%7D%5C%5D%20%25%7BJAVALOGMESSAGE:message%7D" -T "$ES_HOME/logs/elasticsearch.log"
|
|
----
|
|
// NOTCONSOLE
|
|
// Not converting to console because this shows how curl can be used
|
|
|
|
If the request does not encounter errors, the result will look something like
|
|
this:
|
|
|
|
[source,js]
|
|
----
|
|
{
|
|
"num_lines_analyzed" : 53,
|
|
"num_messages_analyzed" : 53,
|
|
"sample_start" : "[2018-09-27T14:39:28,518][INFO ][o.e.e.NodeEnvironment ] [node-0] using [1] data paths, mounts [[/ (/dev/disk1)]], net usable_space [165.4gb], net total_space [464.7gb], types [hfs]\n[2018-09-27T14:39:28,521][INFO ][o.e.e.NodeEnvironment ] [node-0] heap size [494.9mb], compressed ordinary object pointers [true]\n",
|
|
"charset" : "UTF-8",
|
|
"has_byte_order_marker" : false,
|
|
"format" : "semi_structured_text",
|
|
"multiline_start_pattern" : "^\\[\\b\\d{4}-\\d{2}-\\d{2}[T ]\\d{2}:\\d{2}",
|
|
"grok_pattern" : "\\[%{TIMESTAMP_ISO8601:timestamp}\\]\\[%{LOGLEVEL:loglevel} *\\]\\[%{JAVACLASS:class} *\\] \\[%{HOSTNAME:node}\\] %{JAVALOGMESSAGE:message}", <1>
|
|
"timestamp_field" : "timestamp",
|
|
"joda_timestamp_formats" : [
|
|
"ISO8601"
|
|
],
|
|
"java_timestamp_formats" : [
|
|
"ISO8601"
|
|
],
|
|
"need_client_timezone" : true,
|
|
"mappings" : {
|
|
"@timestamp" : {
|
|
"type" : "date"
|
|
},
|
|
"class" : {
|
|
"type" : "keyword"
|
|
},
|
|
"loglevel" : {
|
|
"type" : "keyword"
|
|
},
|
|
"message" : {
|
|
"type" : "text"
|
|
},
|
|
"node" : {
|
|
"type" : "keyword"
|
|
}
|
|
},
|
|
"ingest_pipeline" : {
|
|
"description" : "Ingest pipeline created by file structure finder",
|
|
"processors" : [
|
|
{
|
|
"grok" : {
|
|
"field" : "message",
|
|
"patterns" : [
|
|
"\\[%{TIMESTAMP_ISO8601:timestamp}\\]\\[%{LOGLEVEL:loglevel} *\\]\\[%{JAVACLASS:class} *\\] \\[%{HOSTNAME:node}\\] %{JAVALOGMESSAGE:message}"
|
|
]
|
|
}
|
|
},
|
|
{
|
|
"date" : {
|
|
"field" : "timestamp",
|
|
"timezone" : "{{ event.timezone }}",
|
|
"formats" : [
|
|
"ISO8601"
|
|
]
|
|
}
|
|
},
|
|
{
|
|
"remove" : {
|
|
"field" : "timestamp"
|
|
}
|
|
}
|
|
]
|
|
},
|
|
"field_stats" : { <2>
|
|
"class" : {
|
|
"count" : 53,
|
|
"cardinality" : 14,
|
|
"top_hits" : [
|
|
{
|
|
"value" : "o.e.p.PluginsService",
|
|
"count" : 26
|
|
},
|
|
{
|
|
"value" : "o.e.c.m.MetadataIndexTemplateService",
|
|
"count" : 8
|
|
},
|
|
{
|
|
"value" : "o.e.n.Node",
|
|
"count" : 7
|
|
},
|
|
{
|
|
"value" : "o.e.e.NodeEnvironment",
|
|
"count" : 2
|
|
},
|
|
{
|
|
"value" : "o.e.a.ActionModule",
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : "o.e.c.s.ClusterApplierService",
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : "o.e.c.s.MasterService",
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : "o.e.d.DiscoveryModule",
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : "o.e.g.GatewayService",
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : "o.e.l.LicenseService",
|
|
"count" : 1
|
|
}
|
|
]
|
|
},
|
|
"loglevel" : {
|
|
"count" : 53,
|
|
"cardinality" : 3,
|
|
"top_hits" : [
|
|
{
|
|
"value" : "INFO",
|
|
"count" : 51
|
|
},
|
|
{
|
|
"value" : "DEBUG",
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : "WARN",
|
|
"count" : 1
|
|
}
|
|
]
|
|
},
|
|
"message" : {
|
|
"count" : 53,
|
|
"cardinality" : 53,
|
|
"top_hits" : [
|
|
{
|
|
"value" : "Using REST wrapper from plugin org.elasticsearch.xpack.security.Security",
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : "adding template [.monitoring-alerts] for index patterns [.monitoring-alerts-6]",
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : "adding template [.monitoring-beats] for index patterns [.monitoring-beats-6-*]",
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : "adding template [.monitoring-es] for index patterns [.monitoring-es-6-*]",
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : "adding template [.monitoring-kibana] for index patterns [.monitoring-kibana-6-*]",
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : "adding template [.monitoring-logstash] for index patterns [.monitoring-logstash-6-*]",
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : "adding template [.triggered_watches] for index patterns [.triggered_watches*]",
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : "adding template [.watch-history-9] for index patterns [.watcher-history-9*]",
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : "adding template [.watches] for index patterns [.watches*]",
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : "starting ...",
|
|
"count" : 1
|
|
}
|
|
]
|
|
},
|
|
"node" : {
|
|
"count" : 53,
|
|
"cardinality" : 1,
|
|
"top_hits" : [
|
|
{
|
|
"value" : "node-0",
|
|
"count" : 53
|
|
}
|
|
]
|
|
},
|
|
"timestamp" : {
|
|
"count" : 53,
|
|
"cardinality" : 28,
|
|
"earliest" : "2018-09-27T14:39:28,518",
|
|
"latest" : "2018-09-27T14:39:37,012",
|
|
"top_hits" : [
|
|
{
|
|
"value" : "2018-09-27T14:39:29,859",
|
|
"count" : 10
|
|
},
|
|
{
|
|
"value" : "2018-09-27T14:39:29,860",
|
|
"count" : 9
|
|
},
|
|
{
|
|
"value" : "2018-09-27T14:39:29,858",
|
|
"count" : 6
|
|
},
|
|
{
|
|
"value" : "2018-09-27T14:39:28,523",
|
|
"count" : 3
|
|
},
|
|
{
|
|
"value" : "2018-09-27T14:39:34,234",
|
|
"count" : 2
|
|
},
|
|
{
|
|
"value" : "2018-09-27T14:39:28,518",
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : "2018-09-27T14:39:28,521",
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : "2018-09-27T14:39:28,522",
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : "2018-09-27T14:39:29,861",
|
|
"count" : 1
|
|
},
|
|
{
|
|
"value" : "2018-09-27T14:39:32,786",
|
|
"count" : 1
|
|
}
|
|
]
|
|
}
|
|
}
|
|
}
|
|
----
|
|
// NOTCONSOLE
|
|
|
|
<1> The `grok_pattern` in the output is now the overridden one supplied in the
|
|
query parameter.
|
|
<2> The returned `field_stats` include entries for the fields from the
|
|
overridden `grok_pattern`.
|
|
|
|
The URL escaping is hard, so if you are working interactively it is best to use
|
|
the {ml} UI!
|