Commit Graph

204 Commits

Author SHA1 Message Date
Martijn van Groningen 555b630160
Merge remote-tracking branch 'es/7.x' into enrich-7.x 2019-09-02 09:16:55 +02:00
Tal Levy a356bcff41
Add Circle Processor (#43851) (#46097)
add circle-processor that translates circles to polygons
2019-08-28 14:44:08 -07:00
Martijn van Groningen 1157224a6b
Merge remote-tracking branch 'es/7.x' into enrich-7.x 2019-08-28 10:14:07 +02:00
James Rodewig f3825767f4 [DOCS] Relocate Ingest API docs to REST API section (#45812) 2019-08-23 11:55:01 -04:00
Martijn van Groningen cb42e19a32
Change how type is stored in an enrich policy. (#45789)
A policy type controls how the enrich index is created and
the query executed against the match field. Currently there
is a single policy type (`exact_match`). In the near future
more policy types will be added and different policy may have
different configuration options.

For this reason type should be a json object instead of a string field:

```
{
   "exact_match": {
      ...
   }
}
```

instead of:

```
{
  "type": "exact_match",
  ...
}
```

This will make streaming parsing of enrich policies easier as in the
new format, the parsing code can know ahead what configuration fields
to expect. In the latter format that is not possible if the type field
appears not as the first field.

Relates to #32789
2019-08-23 13:43:38 +02:00
Martijn van Groningen 33972423e9
Enrich processor configuration changes (#45466)
Enrich processor configuration changes:
* Renamed `enrich_key` option to `field` option.
* Replaced `set_from` and `targets` options with `target_field`.

The `target_field` option behaves different to how `set_from` and
`targets` worked. The `target_field` is the field that will contain
the looked up document.

Relates to #32789
2019-08-22 09:49:22 +02:00
Michael Basnight e3373d349b Consolidate enrich list all and get by name APIs (#45705)
The get and list APIs are a single API in this commit. Whether
requesting one named policy or all policies, a list of policies is
returened. The list API code has all been removed and the GET api is
what remains, which contains much of the list response code.
2019-08-20 10:29:59 -05:00
Martijn van Groningen 5ea0985711
Merge remote-tracking branch 'es/7.x' into enrich-7.x 2019-08-16 09:47:11 +02:00
Michael Basnight 52a094b177 Fail delete policy if pipeline exists (#44438)
If a pipeline that refrences the policy exists, we should not allow the
policy to be deleted. The user will need to remove the processor from
the pipeline before deleting the policy. This commit adds a check to
ensure that the policy cannot be deleted if it is referenced by any
pipeline in the system.
2019-08-14 13:51:10 -05:00
Martijn van Groningen 43b8ab607d
Improve naming of enrich policy fields. (#45494)
Renamed `enrich_key` to `match_field` and
renamed `enrich_values` to `enrich_fields`.

Relates #32789
2019-08-14 11:45:22 +02:00
István Zoltán Szabó 356a632b95 [DOCS] Reformats cluster node info API (#45446)
Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2019-08-13 13:33:15 +02:00
István Zoltán Szabó 4ee7ac25ae [DOCS] Reformats cluster node stats API (#45441)
Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2019-08-13 12:48:07 +02:00
Martijn van Groningen 04626de6ae
Add initial version of enrich processor docs. (#45084)
Relates to #32789
2019-08-12 20:36:54 +02:00
Alexander Reelsen 210593d8e5 Add back lowercase processor in docs (#45090)
This got lost in a refactoring in 9137d92ca6
2019-08-06 09:23:13 -04:00
Jason Tedor bf74d38782
Fix GeoIP custom database directory in docs (#43383)
These docs were misleading for package installations of
Elasticsearch. Instead, we should refer to $ES_CONFIG/ingest-geoip as
the path to place the custom database files. For non-package
installations, this is the same as $ES_HOME/config, but for package
installations this is not the case as the config directory for package
installations is /etc/elasticsearch, and is not relative to
$ES_HOME. This commit corrects the docs.
2019-06-19 13:26:07 -04:00
Marios Trivyzas 3b42dde64f
[Docs] Add note for date patterns used for index search. (#42810)
Add an explanatory NOTE section to draw attention to the difference
between small and capital letters used for the index date patterns.
e.g.: HH vs hh, MM vs mm.

Closes: #22322
(cherry picked from commit c8125417dc33215651f9bb76c9b1ffaf25f41caf)
2019-06-03 22:27:19 +02:00
Jack Conradson 813db163d8 Reorganize Painless doc structure (#42303) 2019-05-21 10:50:21 -07:00
Alexander Reelsen 8e33a5292a Add HTML strip processor (#41888)
This processor uses the lucene HTMLStripCharFilter class to remove HTML
entities from a field. This adds to the char filter, so that there is
possibility to store the stripped version as well.

Note, that the characeter filter replaces tags with a newline, so that
the produced HTML will look slightly different than the incoming HTML
with regards to newlines.
2019-05-09 13:01:07 +02:00
Flavio Pompermaier 83fef23fd1
Fix wrong property name (#40636) 2019-05-09 08:53:05 +02:00
James Rodewig b65ceb36bc [DOCS] Escape quotes to avoid smart quotes in Asciidoctor (#41603) 2019-04-30 16:31:20 -04:00
James Rodewig 53702efddd [DOCS] Add anchors for Asciidoctor migration (#41648) 2019-04-30 10:20:17 -04:00
Jason Tedor ac58b9bded
Fix date index name processor default date_formats (#40915)
This commit is a correction of a doc bug in the docs for the ingest
date-index-name processor. The correct pattern is
yyyy-MM-dd'T'HH:mm:ss.SSSXX. This is due to the transition from Joda
time to Java time where Z does not mean the same thing between the two.
2019-04-05 17:45:57 -04:00
Tal Levy 9ab2410436
Adding an example in the Set processor documentation to address #30604 (#39941) (#39969)
* Added an example of using set to copy values from one field to another.

* Modified the document type to match the test.
2019-03-12 11:14:41 -07:00
Jake Landis 797d6b8a66
Execute ingest node pipeline before creating the index (#39607) (#39796)
Prior to this commit (and after 6.5.0), if an ingest node changes
the _index in a pipeline, the original target index would be created.
For daily indexes this could create an extra, empty index per day.

This commit changes the TransportBulkAction to execute the ingest node
pipeline before attempting to create the index. This ensures that the 
only index created is the original or one set by the ingest node pipeline. 
This was the execution order prior to 6.5.0 (#32786). 

The execution order was changed in 6.5 to better support default pipelines. 
Specifically the execution order was changed to be able to read the settings
from the index meta data. This commit also includes a change in logic such 
that if the target index does not exist when ingest node pipeline runs, it 
will now pull the default pipeline (if one exists) from the settings of the 
best matched of the index template. 

Relates #32786
Relates #32758 
Closes #36545
2019-03-07 13:31:41 -06:00
Alexander Reelsen 8e5e48319e
Add documentation about breaking java time changes (#38886)
In addition remove joda time mentions across the docs, make 
sure links are updated to java time javadocs.

Forward port of #38720
2019-02-14 10:18:12 +01:00
Jake Landis 46bb663a09
Make 7.x like 6.7 user agent ecs, but default to true (#38828)
Forward port of https://github.com/elastic/elasticsearch/pull/38757

This change reverts the initial 7.0 commits and replaces them
with the 6.7 variant that still allows for the ecs flag. 
This commit differs from the 6.7 variants in that ecs flag will 
now default to true. 

6.7: `ecs` : default `false`
7.x: `ecs` : default `true`
8.0: no option, but behaves as `true`

* Revert "Ingest node - user agent, move device to an object (#38115)"
This reverts commit 5b008a34aa.

* Revert "Add ECS schema for user-agent ingest processor (#37727) (#37984)"
This reverts commit cac6b8e06f.

* cherry-pick 5dfe1935345da3799931fd4a3ebe0b6aa9c17f57 
Add ECS schema for user-agent ingest processor (#37727)

* cherry-pick ec8ddc890a34853ee8db6af66f608b0ad0cd1099 
Ingest node - user agent, move device to an object (#38115) (#38121)
  
* cherry-pick f63cbdb9b426ba24ee4d987ca767ca05a22f2fbb (with manual merge fixes)
Dep. check for ECS changes to User Agent processor (#38362)

* make true the default for the ecs option, and update 7.0 references and tests
2019-02-13 10:28:01 -06:00
Jake Landis 46bd04959e
fix dissect doc "ip" --> "clientip" (#38544)
Forward port of #38512.
2019-02-08 16:51:58 -06:00
Lee Hinman 70956f6f34
bad formatted JSON object (#38515) (#38526)
It just need to replace the wrong " , " to " : "

Backport of #38515
2019-02-06 13:01:45 -07:00
Gordon Brown 292e0f6fb7
Deprecate `_type` in simulate pipeline requests (#37949)
As mapping types are being removed throughout Elasticsearch, the use of
`_type` in pipeline simulation requests is deprecated. Additionally, the
default `_type` used if one is not supplied has been changed to `_doc` for
consistency with the rest of Elasticsearch.
2019-02-04 16:11:44 -07:00
Jake Landis 5b008a34aa
Ingest node - user agent, move device to an object (#38115)
When the ingest node user agent parses the device field, it
will result in a string value. To match the ecs schema
this commit moves the value of the parsed device to an
object with an inner field named 'name'. There are not
any passivity concerns since this modifies an unreleased change.

closes #38094
relates #37329
2019-01-31 13:54:34 -06:00
Lee Hinman cac6b8e06f
Add ECS schema for user-agent ingest processor (#37727) (#37984)
* Add ECS schema for user-agent ingest processor (#37727)

This switches the format of the user agent processor to use the schema from [ECS](https://github.com/elastic/ecs).
So rather than something like this:

```
{
  "patch" : "3538",
  "major" : "70",
  "minor" : "0",
  "os" : "Mac OS X 10.14.1",
  "os_minor" : "14",
  "os_major" : "10",
  "name" : "Chrome",
  "os_name" : "Mac OS X",
  "device" : "Other"
}
```

The structure is now like this:

```
{
  "name" : "Chrome",
  "original" : "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36",
  "os" : {
    "name" : "Mac OS X",
    "version" : "10.14.1",
    "full" : "Mac OS X 10.14.1"
  },
  "device" : "Other",
  "version" : "70.0.3538.102"
}
```

This is now the default for 7.0. The deprecated `ecs` setting in 6.x is not
supported.

Resolves #37329

* Remove `ecs` setting from docs
2019-01-30 11:24:18 -07:00
Christoph Büscher 34f2d2ec91
Remove remaining occurances of "include_type_name=true" in docs (#37646) 2019-01-22 15:13:52 +01:00
Julie Tibshirani 36a3b84fc9
Update the default for include_type_name to false. (#37285)
* Default include_type_name to false for get and put mappings.

* Default include_type_name to false for get field mappings.

* Add a constant for the default include_type_name value.

* Default include_type_name to false for get and put index templates.

* Default include_type_name to false for create index.

* Update create index calls in REST documentation to use include_type_name=true.

* Some minor clean-ups around the get index API.

* In REST tests, use include_type_name=true by default for index creation.

* Make sure to use 'expression == false'.

* Clarify the different IndexTemplateMetaData toXContent methods.

* Fix FullClusterRestartIT#testSnapshotRestore.

* Fix the ml_anomalies_default_mappings test.

* Fix GetFieldMappingsResponseTests and GetIndexTemplateResponseTests.

We make sure to specify include_type_name=true during xContent parsing,
so we continue to test the legacy typed responses. XContent generation
for the typeless responses is currently only covered by REST tests,
but we will be adding unit test coverage for these as we implement
each typeless API in the Java HLRC.

This commit also refactors GetMappingsResponse to follow the same appraoch
as the other mappings-related responses, where we read include_type_name
out of the xContent params, instead of creating a second toXContent method.
This gives better consistency in the response parsing code.

* Fix more REST tests.

* Improve some wording in the create index documentation.

* Add a note about types removal in the create index docs.

* Fix SmokeTestMonitoringWithSecurityIT#testHTTPExporterWithSSL.

* Make sure to mention include_type_name in the REST docs for affected APIs.

* Make sure to use 'expression == false' in FullClusterRestartIT.

* Mention include_type_name in the REST templates docs.
2019-01-14 13:08:01 -08:00
Josh Soref edb48321ba [DOCS] Various spelling corrections (#37046) 2019-01-07 14:44:12 +01:00
Adam Thomson ac4aecc92d [Docs] Update ingest-node.asciidoc (#37116) 2019-01-04 19:33:06 +01:00
Jason Tedor 9137d92ca6
Refactor ingest node API docs (#36962)
This commit is a simple refactoring of the ingest node API docs,
breaking each API into a single file for ease of maintaining.
2018-12-23 08:59:18 -05:00
Jason Tedor 7562768bd6
Fix ingest cross-doc links
This commit fixes some cross-doc links from the old ingest plugins page
to the new ingest processor pages that arose after converting
ingest-geoip and ingest-user-agent to modules.
2018-12-22 20:51:18 -05:00
Jason Tedor e14f27c033
Fix titles of GeoIP and User Agent processor docs
This commit makes the titles of the new GeoIP and User Agent processor
docs look more like the titles of the docs for other processors.
2018-12-22 20:31:07 -05:00
Jason Tedor 1f574bd17a
Package ingest-user-agent as a module (#36956)
This commit moves ingest-user-agent from being a plugin to being a
module that is packaged with Elasticsearch distributions.
2018-12-22 20:20:53 -05:00
Jason Tedor 434021c3ec
Add placeholder ingest-geoip plugin page (#36958)
This commit adds a placeholder ingest-geoip plugin page as there are
other components in the Elastic Stack that still refer to these
pages. These docs would be broken without this placeholder page forcing
teams responsible for those docs to scramble to fix the build over the
weekend before a holiday period. Instead, we add a placeholder page so
the docs build continues to function, and those teams can fix their docs
without the constraint of a broken build. We also cleanup a few minor
docs issues that were missed during the initial changes to convert
ingest-geoip to a module.
2018-12-22 09:49:56 -05:00
Jason Tedor e1717df0ac
Package ingest-geoip as a module (#36898)
This commit moves ingest-geoip from being a plugin to being a module
that is packaged with Elasticsearch distributions.
2018-12-22 07:21:49 -05:00
Jason Tedor 35911d8dd7
Split the ingest processor docs into multiple files (#36887)
This commit breaks the single ingest docs file into multiple files,
factoring out the processor docs into a documentation file per
processor. This will help make this content easier to maintain.
2018-12-20 08:04:54 -05:00
Boaz Leskes e356b8cb95
Add doc's sequence number + primary term to GetResult and use it for updates (#36680)
This commit adds the last sequence number and primary term of the last operation that have
modified a document to `GetResult` and uses it to power the Update API.

Relates #36148 
Relates #10708
2018-12-17 15:22:13 +01:00
Jake Landis 4b99a663c1
ingest: fix broken doc link 2018-11-26 10:34:42 -06:00
Jake Landis 7f7b31723e
ingest: extended `if` documentation (#35044)
part of #33188
2018-11-26 09:35:45 -06:00
Chris Cho e572a21c4b [Docs] Improve Convert Processor description (#35280)
Sometimes users are confused about whether they can use the Convert Processor
for changing an existing fields type to other types even if the existing one is already
ingested. This confusion is from the first line of description. Changing this and also
adding a some detail to the code snippet.
2018-11-07 17:01:35 +01:00
Jake Landis c2766b65cf
ingest: raise visibility of ingest plugin documentation (#35048)
* move the set security user processor to the main documentation
* link to plugin processors

part of #33188
2018-11-05 11:44:10 -06:00
Jake Landis 77fab62ebe
ingest: add common options to each processor's documentation (#35091)
* adds `if`, `on_failure`, `tag`, and `ignore_failure` to table for each processor

part of #33188

* added ingore_failure

* fix whitespace noise
2018-11-01 11:08:04 -05:00
Armin Braun f79bdec58a INGEST: Document Pipeline Processor (#33418)
* Added documentation for Pipeline Processor
* Relates #33188
2018-10-23 15:36:57 -05:00
Jake Landis a8e1ee34ca
ingest: document fields that support templating (#34536)
This change also updates many of the examples to use ecs as the example.
Some additional minor improvements are also included.

Part of #33188
2018-10-23 13:28:44 -05:00
Jake Landis c447fc258a
ingest: documentation for the drop processor (#34570) 2018-10-23 12:30:23 -05:00
Armin Braun f0f732908e
INGEST: Document Processor Conditional (#33388)
* INGEST: Document Processor Conditional

Relates #33188
2018-10-23 17:37:30 +02:00
Jake Landis 79b507dbf5
ingest: Introduce the dissect processor (#32884)
* ingest: Introduce the dissect processor

The ingest node dissect processor is an alternative to Grok
to split a string based on a pattern. Dissect differs from
Grok such that regular expressions are not used to split the
string.

Dissect can be used to parse a source text field with a
simpler pattern, and is often faster the Grok for basic string
parsing. This processor uses the dissect library which
does most of the work.
2018-08-28 07:11:20 -07:00
Jake Landis 3d4c84f7ca
ingest: doc: move Dot Expander Processor doc to correct position (#31743)
No changes to the content.
2018-08-03 07:21:05 -07:00
Armin Braun 7aa8a0a927
INGEST: Extend KV Processor (#31789) (#32232)
* INGEST: Extend KV Processor (#31789)

Added more capabilities supported by LS to the KV processor:
* Stripping of brackets and quotes from values (`include_brackets` in corresponding LS filter)
* Adding key prefixes
* Trimming specified chars from keys and values

Refactored the way the filter is configured to avoid conditionals during execution.
Refactored Tests a little to not have to add more redundant getters for new parameters.

Relates #31786
* Add documentation
2018-07-20 22:32:50 +02:00
Armin Braun e46ed73379
Ingest: Add ignore_missing option to RemoveProc (#31693)
Added `ignore_missing` setting to the RemoveProcessor to fix #23086
2018-07-09 10:24:34 +02:00
Jake Landis c0056cddd8
ingest: Introduction of a bytes processor (#31733)
ingest: Introduction of a bytes processor

This processor allows for human readable byte values (e.g. 1kb) to be converted to value in bytes (e.g. 1024). Internally this processor re-uses "ByteSizeValue.parseBytesSizeValue" which supports conversions up to Long.MAX_VALUE and the following units: "b", "kb", "mb", "gb", "tb", pb".

This change also introduces a generic return type for the AbstractStringProcessor to allow for code reuse while supporting a String -> T conversion. (String -> Long in this case).
2018-07-03 10:40:56 -05:00
Armin Braun 13e1cf6191
ingest: Add ignore_missing property to foreach filter (#22147) (#31578) 2018-06-26 20:04:41 +02:00
Martijn van Groningen 6030d4be1e
[INGEST] Interrupt the current thread if evaluation grok expressions take too long (#31024)
This adds a thread interrupter that allows us to encapsulate calls to org.joni.Matcher#search()
This method can hang forever if the regex expression is too complex.

The thread interrupter in the background checks every 3 seconds whether there are threads
execution the org.joni.Matcher#search() method for longer than 5 seconds and
if so interrupts these threads.

Joni has checks that that for every 30k iterations it checks if the current thread is interrupted and
if so returns org.joni.Matcher#INTERRUPTED

Closes #28731
2018-06-12 07:49:03 +02:00
Tanguy Leroux 42608881b0
[Docs] Remove mention pattern files in Grok processor (#31170)
Pattern files have been removed in 
16fa3e546e
2018-06-11 09:32:12 +02:00
rzmf 080cefec73 Fix missing comma in ingest-node.asciidoc (#29343) 2018-04-03 11:33:44 +01:00
Nik Everett 762226bee9
Docs: Support triple quotes (#28915)
Adds support for triple quoted strings to the documentation test
generator. Kibana's CONSOLE tool has supported them for a year but we
were unable to use them in Elasticsearch's docs because the process that
converts example snippets into tests couldn't handle this. This change
adds code to convert them into standard JSON so we can pass them to
Elasticsearch.
2018-03-16 12:46:39 -04:00
Jiri Tyr c713d62f88 [Docs] Fix link to Grok patterns (#29088) 2018-03-16 14:13:17 +01:00
Devin Young e8a78df555 Fix markdown formatting (#28392) 2018-01-26 08:15:16 -07:00
Sian Lerk Lau 5e3ba8a88d Enable convert processor to support Long and Double. (#27957)
Closes #23085
2018-01-03 11:27:55 +01:00
Sian Lerk Lau 47eefbe889 Enable grok processor to support long, double and boolean (#27896) 2017-12-20 11:19:49 -08:00
David Pilato 3ca39186d1
Fix missing comma in examples (#27904) 2017-12-19 18:28:39 +01:00
Adrien Grand 1b660821a2
Allow `_doc` as a type. (#27816)
Allowing `_doc` as a type will enable users to make the transition to 7.0
smoother since the index APIs will be `PUT index/_doc/id` and `POST index/_doc`.
This also moves most of the documentation to `_doc` as a type name.

Closes #27750
Closes #27751
2017-12-14 17:47:53 +01:00
Tal Levy 5c34533761
add json-processor support for non-map json types (#27335)
The Json Processor originally only supported parsing field values into Maps even
though the JSON spec specifies that strings, null-values, numbers, booleans, and arrays
are also valid JSON types. This commit enables parsing these values now.

response to #25972.
2017-11-13 10:28:19 -08:00
Tal Levy d22fd4ea58
Introduce templating support to timezone/locale in DateProcessor (#27089)
Sometimes systems like Beats would want to extract the date's timezone and/or locale
from a value in a field of the document. This PR adds support for mustache templating
to extract these values.

Closes #24024.
2017-11-09 09:45:32 -08:00
Loek van Gool 67e677f443
Add an example of dynamic field names (#27255) 2017-11-03 23:20:58 +01:00
Christoph Büscher c7c6443b10 [Docs] "The the" is a great band, but ... (#26644)
Removing several occurrences of this typo in the docs and javadocs, seems to be
a common mistake. Corrections turn up once in a while in PRs, better to correct
some of this in one sweep.
2017-09-14 15:08:20 +02:00
Atothendrew c30d6ebcbb [Docs] Correct json example in ingest-node.asciidoc (#26221) 2017-08-21 11:07:44 +02:00
Tal Levy 872526cad3 add URL-Decode Processor to Ingest (#26045)
closes #25837

Adds a URL Decoder Processor to Ingest

this will decode urls like:

https%3a%2f%2felastic.co%2 to https://elastic.co/
2017-08-07 10:26:11 -07:00
Clinton Gormley ff4a2519f2 Update experimental labels in the docs (#25727)
Relates https://github.com/elastic/elasticsearch/issues/19798

Removed experimental label from:
* Painless
* Diversified Sampler Agg
* Sampler Agg
* Significant Terms Agg
* Terms Agg document count error and execution_hint
* Cardinality Agg precision_threshold
* Pipeline Aggregations
* index.shard.check_on_startup
* index.store.type (added warning)
* Preloading data into the file system cache
* foreach ingest processor
* Field caps API
* Profile API

Added experimental label to:
* Moving Average Agg Prediction


Changed experimental to beta for:
* Adjacency matrix agg
* Normalizers
* Tasks API
* Index sorting

Labelled experimental in Lucene:
* ICU plugin custom rules file
* Flatten graph token filter
* Synonym graph token filter
* Word delimiter graph token filter
* Simple pattern tokenizer
* Simple pattern split tokenizer

Replaced experimental label with warning that details may change in the future:
* Analysis explain output format
* Segments verbose output format
* Percentile Agg compression and HDR Histogram
* Percentile Rank Agg HDR Histogram
2017-07-18 14:06:22 +02:00
Glen Smith e9dfb2a215 Fix another simulate example in ingest docs
When simulating an ingest pipeline against an existing pipeline, the
_source field is required to wrap each doc. This commit fixes another
example in the docs that is missing this.
    
Relates #25743, relates e3a0c11239
2017-07-17 15:17:42 +09:00
Glen Smith e3a0c11239 Fix simulate example in ingest docs
When simulating an ingest pipeline against an existing pipeline, the
_source field is required to wrap each doc. This commit fixes an example
in the docs that is missing this.

Relates #25742
2017-07-17 14:17:41 +09:00
olcbean 2ba9fd2aec Remove deprecated created and found from index, delete and bulk (#25516)
The created and found fields in index and delete responses became obsolete after the introduction of the result field in index, update and delete responses (#19566).

After deprecating the created and found fields in 5.x (#19633), now they are removed.

Fixes #19630
2017-07-07 13:58:46 -04:00
Clinton Gormley 0170e0e8d3 Remove usage of multi-types from the docs and added a page explaining type removal (#25543)
Closes #25401
2017-07-05 12:30:19 +02:00
DeDe Morton 6442d1f75e [Docs] Add link to grok debugger docs (#25412) 2017-06-28 16:14:16 -07:00
Alexander Kazakov 53b74348ff Fix documentation for script processor (#25299) 2017-06-26 12:14:23 -07:00
Alexander Kazakov a7dafdaa05 Add target_field parameter to gsub, join, lowercase, sort, split, trim, uppercase (#24133)
Closes #23682 #23228
2017-06-13 09:40:44 -07:00
Ryan Ernst a03b6c2fa5 Scripting: Change keys for inline/stored scripts to source/id (#25127)
This commit adds back "id" as the key within a script to specify a
stored script (which with file scripts now gone is no longer ambiguous).
It also adds "source" as a replacement for "code". This is in an attempt
to normalize how scripts are specified across both put stored scripts and script usages, including search template requests. This also deprecates the old inline/stored keys.
2017-06-09 08:29:25 -07:00
Tal Levy a771912a22 Add Ingest-Processor specific Rest Endpoints & Add Grok endpoint (#25059)
This PR enables Ingest plugins to leverage processor-scoped REST
endpoints. First of which being the Grok endpoint that retrieves
Grok Patterns for users to retrieve all the built-in patterns.
Example usage: Kibana Grok Autocomplete!
2017-06-08 15:24:35 -07:00
Guillaume Le Floch 3f6d80aa66 Allow removing multiple fields in ingest processor (#24750)
* Allow removing multiple fields in ingest processor

* Iteration 2

* Few fixes
2017-06-08 13:17:44 -07:00
Tal Levy e51246023a add `exclude_keys` option to KeyValueProcessor (#24876)
and modify data-structure of `include_keys` and `exclude_keys` to be
backed by a HashSet
2017-06-05 14:12:48 -07:00
Tal Levy dfe2ecaa28 add docs example for Ingest scripts manipulating document metadata (#24875)
It may not be clear to users that the Ingest ScriptProcessor context object `ctx` can 
manipulate document metadata like `_index` and `_type`.
2017-05-25 07:45:19 -07:00
Ryan Ernst 463fe2f4d4 Scripting: Remove file scripts (#24627)
This commit removes file scripts, which were deprecated in 5.5.

closes #21798
2017-05-17 14:42:25 -07:00
Nik Everett a01f846226 CONSOLEify a few more docs
Adds CONSOLE to cross-cluster-search docs but skips them for testing
because we don't have a second cluster set up. This gets us the
`VIEW IN CONSOLE` and `COPY AS CURL` links and makes sure that they
are valid yaml (not json, technically) but doesn't get testing.
Which is better than we had before.

Adds CONSOLE to the dynamic templates docs and ingest-node docs.
The ingest-node docs contain a *ton* of non-console snippets. We
might want to convert them to full examples later, but that can be
a separate thing.

Relates to #18160
2017-05-04 21:01:14 -04:00
Jason Tedor 4796557a30 Add primary term to doc write response
This commit adds the primary term to the doc write response.

Relates #24171
2017-04-19 14:44:22 -04:00
Glen Smith 3ff014d07d ingest-node.asciidoc - Clarify json processor (#21876)
Add examples for the json processor.
2017-04-18 23:27:26 -04:00
Clinton Gormley 5cf13f29bb Update ingest-node.asciidoc
Fixed docs typo
2017-03-22 10:44:11 +01:00
Gameldar e3eb363882 Link directly to the attachments in arrays section
The link should be made to the relevant section of the ingest attachments documentation, rather than the top of the page.
2016-12-22 20:52:08 +08:00
gameldar d404ee3533 Add ingest-attachment-with-arrays section to ingest attachments doc
Added a new section detailing how to use the attachment processor
within an array.

This reverts commit #22296 and instead links to the foreach processor.
2016-12-22 00:18:33 +08:00
Tal Levy c53b2ee9cd introduce KV Processor in Ingest Node (#22272)
Now you can parse field values of the `key=value` variety and have
`key` be inserted as a field name in an ingest document.

Closes #22222.
2016-12-20 13:26:17 -08:00
Tal Levy ad4b1ecdeb [docs] update ingest-node delete docs to mention wildcarding (#22270) 2016-12-20 10:52:17 -08:00
Tal Levy bb37167946 Enables the ability to inject serialized json fields into root of document. (#22179)
The JSON processor has an optional field called "target_field".
If you don't specify target_field then target_field becomes what you specified as "field".
There isn't anyway to add the fields to the root of a document. By
setting `add_to_root`, now serialized fields will be inserted into the
top-level fields of the ingest document.

Closes #21898.
2016-12-16 10:17:27 -08:00
Pablo Musa 152efe95e6 Small typo fix in the docs. (#22180)
There is a small typo in the convert processor code example.
2016-12-16 14:50:06 +01:00
Jason Tedor d06a8903fd Merge branch 'master' into feature/seq_no
* master: (22 commits)
  Add proper toString() method to UpdateTask (#21582)
  Fix `InternalEngine#isThrottled` to not always return `false`. (#21592)
  add `ignore_missing` option to SplitProcessor (#20982)
  fix trace_match behavior for when there is only one grok pattern (#21413)
  Remove dead code from GetResponse.java
  Fixes date range query using epoch with timezone (#21542)
  Do not cache term queries. (#21566)
  Updated dynamic mapper section
  Docs: Clarify date_histogram bucket sizes for DST time zones
  Handle release of 5.0.1
  Fix skip reason for stats API parameters test
  Reduce skip version for stats API parameter tests
  Strict level parsing for indices stats
  Remove cluster update task when task times out (#21578)
  [DOCS] Mention "all-fields" mode doesn't search across nested documents
  InternalTestCluster: when restarting a node we should validate the cluster is formed via the node we just restarted
  Fixed bad asciidoc in boolean mapping docs
  Fixed bad asciidoc ID in node stats
  Be strict when parsing values searching for booleans (#21555)
  Fix time zone rounding edge case for DST overlaps
  ...
2016-11-16 09:10:35 -05:00
Tal Levy 6796464f16 add `ignore_missing` option to SplitProcessor (#20982)
Closes #20840.
2016-11-16 15:46:09 +02:00