Commit Graph

7356 Commits

Author SHA1 Message Date
Przemysław Witek 7512337922
[7.x] Allow the user to specify 'query' in Evaluate Data Frame request (#45775) (#45825) 2019-08-22 11:14:26 +02:00
debadair 9b180314e3 [DOCS] Streamlined GS indexing topic. (#45714)
* Streamlined GS indexing topic.

* Incorporated review feedback

* Applied formatting per the style guidelines.
2019-08-21 16:59:56 -07:00
Tal Levy 9b14b7298b
[7.x] Add is_write_index column to cat.aliases (#45798)
* Add is_write_index column to cat.aliases (#44772)

Aliases have had the option to set `is_write_index` since 6.4,
but the cat.aliases action was never updated.

* correct version bounds to 7.4
2019-08-21 14:15:49 -07:00
Armin Braun 6aaee8aa0a
Repository Cleanup Endpoint (#43900) (#45780)
* Repository Cleanup Endpoint (#43900)

* Snapshot cleanup functionality via transport/REST endpoint.
* Added all the infrastructure for this with the HLRC and node client
* Made use of it in tests and resolved relevant TODO
* Added new `Custom` CS element that tracks the cleanup logic.
Kept it similar to the delete and in progress classes and gave it
some (for now) redundant way of handling multiple cleanups but only allow one
* Use the exact same mechanism used by deletes to have the combination
of CS entry and increment in repository state ID provide some
concurrency safety (the initial approach of just an entry in the CS
was not enough, we must increment the repository state ID to be safe
against concurrent modifications, otherwise we run the risk of "cleaning up"
blobs that just got created without noticing)
* Isolated the logic to the transport action class as much as I could.
It's not ideal, but we don't need to keep any state and do the same
for other repository operations
(like getting the detailed snapshot shard status)
2019-08-21 17:59:49 +02:00
Jim Ferenczi fe2a7523ec Add support for inlined user dictionary in the Kuromoji plugin (#45489)
This change adds a new option called user_dictionary_rules to
Kuromoji's tokenizer. It can be used to set additional tokenization rules
to the Japanese tokenizer directly in the settings (instead of using a file).
This commit also adds a check that no rules are duplicated since this is not allowed
in the UserDictionary.

Closes #25343
2019-08-21 16:28:30 +02:00
James Rodewig 3318c91fea [DOCS] Reformat get mapping API. Reformat and reuse multi-index params. (#45699) 2019-08-21 09:44:00 -04:00
Przemysław Witek 5faa012fd6
[7.x] Add docs for HLRC for Estimate memory usage API (#45538) (#45783) 2019-08-21 14:27:36 +02:00
Russ Cam 1d9b2d57ce Use the elasticsearch.bat file in zip distribution (#45596)
This commit updates the documentation to

- use the batch file included with the zip distribution; the exe file is included in the MSI only.
- introduce a space between the -E arguments and their values. Without a space (or quoted, but adding a space is cleaner), the argument will fail with PowerShell

(cherry picked from commit 5c8dbcedb0edf3a48ca1ec52aad9ea41fa941f8a)
2019-08-21 18:48:30 +10:00
James Rodewig a27759a624 [DOCS] Document `indices.lifecycle.poll_interval` ILM cluster setting (#45744) 2019-08-20 12:59:25 -04:00
James Rodewig a219a0f819 [DOCS] Replace placeholder anchors for create rollup job API docs. (#45712) 2019-08-20 10:45:28 -04:00
Jonathan Hult 041385559c [DOCS] Fix typo in highlighting doc (#45707) 2019-08-20 07:28:05 -04:00
markharwood 7d5ab17bb2
Search enhancement: pinned queries (#44345) (#45657)
* Search enhancement: pinned queries (#44345)

Search enhancement: - new query type allows selected documents to be promoted above any "organic” search results.
This is the first feature in a new module `search-business-rules` which will house licensed (non OSS) logic for rewriting queries according to business rules.
The PinnedQueryBuilder class offers a new `pinned` query in the DSL that takes an array of promoted IDs and an “organic” query and ensures the documents with the promoted IDs rank higher than the organic matches.

Closes #44074
2019-08-20 11:38:22 +01:00
James Rodewig 5dcc00a8b3 [DOCS] Add placeholder for 7.3.1 release notes (#45710) 2019-08-19 17:04:02 -04:00
LHearen 8f86faca5c [DOCS] Correct conditional clause in histogram agg docs (#45643) 2019-08-19 10:09:46 -04:00
LHearen da0a785685 [DOCS] Fix a 'value' -> 'values' typo in histogram aggregation docs (#45642) 2019-08-19 10:02:59 -04:00
James Rodewig b0ef313817 [DOCS] Add examples to the mapping docs (#45520) 2019-08-19 09:32:59 -04:00
debadair 74de6d2c71 [DOCS] Fixed TOC heading level issue. (#45680) 2019-08-16 14:56:26 -07:00
Igor Motov 98c850c08b
Geo: Change order of parameter in Geometries to lon, lat 7.x (#45618)
Changes the order of parameters in Geometries from lat, lon to lon, lat
and moves all Geometry classes are moved to the
org.elasticsearch.geomtery package.

Backport of #45332

Closes #45048
2019-08-16 14:42:02 -04:00
Jack Conradson e243bbdc2a Fix Watcher Examples in Painless (#45631)
This fixes the mappings and types required to run watcher and other 
examples. A new set of seat data will be updated and available for 
download to go with this change.
2019-08-16 09:08:19 -07:00
Andrey Ershov dbc90653dc transport.publish_address should contain CNAME (#45626)
This commit adds CNAME reporting for transport.publish_address same way
it's done for http.publish_address.

Relates #32806
Relates #39970

(cherry picked from commit e0a2558a4c3a6b6fbfc6cd17ed34a6f6ef7b15a9)
2019-08-16 17:42:00 +02:00
István Zoltán Szabó cab2eeeafe [DOCS] Reformats voting configuration exclusion API. (#45644) 2019-08-16 15:42:07 +02:00
István Zoltán Szabó 6350020401 [DOCS] Reformats cluster allocation explain API (#45603)
Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2019-08-16 15:39:07 +02:00
István Zoltán Szabó 95bd264449 [DOCS] Reformats nodes hot_threads API (#45597)
Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2019-08-16 15:35:29 +02:00
István Zoltán Szabó da03a5d8d1 [DOCS] Reformats task management API (#45549)
This PR updates the task management API to align with the new API reference template.
2019-08-16 10:23:32 +02:00
James Rodewig a635eca5f8 Retitle and relocate cross-cluster search docs (#45608) 2019-08-15 16:28:04 -04:00
James Rodewig d64c31e43d [DOCS] Rewrite cross-cluster seach docs (#45583) 2019-08-15 13:23:40 -04:00
James Rodewig c75fd40f2c [DOCS] Add diagrams to cross-cluster search documentation (#45569) 2019-08-15 11:00:25 -04:00
debadair ad3d8c59bd Restructure the GS topics & add redirects. (#45527)
* Restructure the GS topics & add redirects.

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

* Update docs/reference/getting-started.asciidoc to fix tests
2019-08-14 16:56:31 -07:00
debadair 0f780f43b9 [DOCS] Streamlined install of 3 node cluster for GS tutorial (#45009)
* [DOCS] Revise GS intro and remove redundant conceptual content. Closes #43846.

* [DOCS] Incorporated feedback.

* [DOCS] Consolidated archive install for all platforms.

* [DOCS] Added powershell command.

* [DOCS] Added health check.
2019-08-14 16:36:59 -07:00
Nick Knize 647a8308c3
[SPATIAL] Backport new ShapeFieldMapper and ShapeQueryBuilder to 7x (#45363)
* Introduce Spatial Plugin (#44389)

Introduce a skeleton Spatial plugin that holds new licensed features coming to 
Geo/Spatial land!

* [GEO] Refactor DeprecatedParameters in AbstractGeometryFieldMapper (#44923)

Refactor DeprecatedParameters specific to legacy geo_shape out of
AbstractGeometryFieldMapper.TypeParser#parse.

* [SPATIAL] New ShapeFieldMapper for indexing cartesian geometries (#44980)

Add a new ShapeFieldMapper to the xpack spatial module for
indexing arbitrary cartesian geometries using a new field type called shape.
The indexing approach leverages lucene's new XYShape field type which is
backed by BKD in the same manner as LatLonShape but without the WGS84
latitude longitude restrictions. The new field mapper builds on and
extends the refactoring effort in AbstractGeometryFieldMapper and accepts
shapes in either GeoJSON or WKT format (both of which support non geospatial
geometries).

Tests are provided in the ShapeFieldMapperTest class in the same manner
as GeoShapeFieldMapperTests and LegacyGeoShapeFieldMapperTests.
Documentation for how to use the new field type and what parameters are
accepted is included. The QueryBuilder for searching indexed shapes is
provided in a separate commit.

* [SPATIAL] New ShapeQueryBuilder for querying indexed cartesian geometry (#45108)

Add a new ShapeQueryBuilder to the xpack spatial module for
querying arbitrary Cartesian geometries indexed using the new shape field
type.

The query builder extends AbstractGeometryQueryBuilder and leverages the
ShapeQueryProcessor added in the previous field mapper commit.

Tests are provided in ShapeQueryTests in the same manner as
GeoShapeQueryTests and docs are updated to explain how the query works.
2019-08-14 16:35:10 -05:00
Chris Dean deab736aad
[DOCS] - Updating chunk_size values to fix size value notation. Chunksize41591 (#45552) (#45579)
* changes to chunk_size #41591

* update to chunk size to include ` `

* Update docs/plugins/repository-azure.asciidoc

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

* Update docs/reference/modules/snapshots.asciidoc

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

* Update docs/plugins/repository-azure.asciidoc

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

* Update docs/plugins/repository-s3.asciidoc

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

* edits to fix passive voice
2019-08-14 15:59:36 -05:00
Chris Dean caa2a7738f Revert "[DOCS] - Updating chunk_size values to fix size value notation. Chunksize41591 (#45552)"
This reverts commit 8fdbcd7395.
2019-08-14 15:14:10 -05:00
Chris Dean 8fdbcd7395 [DOCS] - Updating chunk_size values to fix size value notation. Chunksize41591 (#45552)
* changes to chunk_size #41591

* update to chunk size to include ` `

* Update docs/plugins/repository-azure.asciidoc

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

* Update docs/reference/modules/snapshots.asciidoc

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

* Update docs/plugins/repository-azure.asciidoc

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

* Update docs/plugins/repository-s3.asciidoc

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

* edits to fix passive voice
2019-08-14 14:15:22 -05:00
Benjamin Trent 0c343d8443
[7.x] [ML][Transforms] adjusting stats.progress for cont. transforms (#45361) (#45551)
* [ML][Transforms] adjusting stats.progress for cont. transforms (#45361)

* [ML][Transforms] adjusting stats.progress for cont. transforms

* addressing PR comments

* rename fix

* Adjusting bwc serialization versions
2019-08-14 13:08:27 -05:00
James Rodewig 6904778a14 [DOCS] Rewrite `fuzzy` query docs (#42078) 2019-08-14 13:07:29 -04:00
Jim Ferenczi 79a1390935 Add mapper-extras and the RankFeatureQuery in the hlrc (#43713)
This change adds the support for the RankFeatureQuery in the HLRC by
providing an extra dependency on mapper-extras-client. It also removes
the dependency on lang-painless in mapper-extras which is not needed
anymore since the move of the vector field into a dedicated module.

Closes #43634
2019-08-14 18:41:39 +02:00
István Zoltán Szabó 8142ca82f1 [DOCS] Reformats nodes feature usage API (#45539)
Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2019-08-14 18:20:19 +02:00
István Zoltán Szabó 0db5af5f65 [DOCS] Reformats cluster remote info API (#45545)
Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2019-08-14 17:08:52 +02:00
Ioannis Kakavas 3eb0e59a83 Remove reference to `accept_default_password` (#45533)
`xpack.security.authc.accept_default_password` has not been
 used since 6.0 but we still referenced it in our docs.
2019-08-14 11:33:57 +03:00
Przemysław Witek df574e5168
[7.x] Implement ml/data_frame/analytics/_estimate_memory_usage API endpoint (#45188) (#45510) 2019-08-14 08:26:03 +02:00
James Rodewig 109358ca3f [DOCS] Reformat cat API titles (#45407) 2019-08-13 08:37:10 -04:00
James Rodewig df0df9402d [DOCS] Correct anchor for cat segments API examples 2019-08-13 08:30:20 -04:00
James Rodewig 2543e9eca5 [DOCS] Reformats cat shards API (#45392) 2019-08-13 08:30:14 -04:00
István Zoltán Szabó 0ff723852f [DOCS] Reformats cluster update settings API (#45337) 2019-08-13 14:24:18 +02:00
István Zoltán Szabó f4b001dc4b [DOCS] Adds heading between examples to better separate them. (#45499) 2019-08-13 13:55:15 +02:00
István Zoltán Szabó 356a632b95 [DOCS] Reformats cluster node info API (#45446)
Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2019-08-13 13:33:15 +02:00
István Zoltán Szabó 4ee7ac25ae [DOCS] Reformats cluster node stats API (#45441)
Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2019-08-13 12:48:07 +02:00
Chris Dean 82d48cfcc9
[DOCS] Added cross-link to snapshot lifecycle management. Closes #44588. (#45408) (#45468)
merging #44588 changes into 7.x
2019-08-12 15:13:11 -05:00
Ryan Ernst 620cd4cb20 Fix old name of script context in docs (#45410)
The docs contain an example how to limit the scripting contexts allowed,
but the example used an outdated name for the scoring context.

closes #44232
2019-08-12 09:29:13 -07:00
James Rodewig e3f618e1d3 [DOCS] Reformat query string query (#45296) 2019-08-12 11:18:00 -04:00
James Rodewig bff4ef2d95 [DOCS] Reformats cat templates API (#45403) 2019-08-12 08:45:45 -04:00
James Rodewig c1c3bd011a [DOCS] Reformats cat snapshots API (#45402) 2019-08-12 08:41:57 -04:00
James Rodewig 7556d37925 [DOCS] Reformats cat thread pool API (#45385) 2019-08-12 08:36:57 -04:00
James Rodewig e8acf6b34f [DOCS] Reformats cat repositories API (#45382) 2019-08-12 08:30:46 -04:00
James Rodewig fb81346055 [DOCS] Reformats cat segments API (#45397) 2019-08-12 08:25:32 -04:00
James Rodewig 4e79e04c0e [DOCS] Reformats cat recovery API (#45345) 2019-08-12 08:17:52 -04:00
Bogdan Pintea 499be8398f SQL: ODBC: document newest connection string parameters (#44185)
* SQL: ODBC: document newest conn string parameters

This commit adds the documentation for two newly added connection string
parameters: AutoEscapePVA and IndexIncludeFrozen.

It also removes the recommended OSes from the prerequisites list and
places the recommendation distinctively: the unmet prerequisites will
fail the installation, while the driver would install on other OSes than
those recommended.

* address review suggestions.

- adjust phrasing for clearer message.

(cherry picked from commit e18ac10c6e163a04f5b7cf7fa72f262882ffb711)
2019-08-12 13:49:11 +02:00
Emmanuel DEMEY f8c08c537b Add snippet for the search_type query parameter (#43540) 2019-08-11 18:36:52 -04:00
James Rodewig 846928a52a [DOCS] Reformats interval query (#45350) 2019-08-09 08:53:47 -04:00
James Rodewig 1506d4436b [DOCS] Reformats cat plugins API (#45344) 2019-08-09 08:49:00 -04:00
James Rodewig eec87ffab8 [DOCS] Reformats simple query string query (#45343) 2019-08-09 08:33:05 -04:00
David Roberts 14545f8958
[ML-DataFrame] Combine task_state and indexer_state in _stats (#45324)
This commit replaces task_state and indexer_state in the
data frame _stats output with a single top level state
that combines the two. It is defined as:

- failed if what's currently reported as task_state is failed
- stopped if there is no persistent task
- Otherwise what's currently reported as indexer_state

Backport of #45276
2019-08-08 16:24:26 +01:00
Mayya Sharipova f0f2294695 Add filters in examples of vector functions (#45327) 2019-08-08 09:44:59 -04:00
István Zoltán Szabó 4e32470827 [DOCS] Reformats cluster reroute API. (#45328) 2019-08-08 15:27:54 +02:00
István Zoltán Szabó 4d96c83854 [DOCS] Reformats cluster pending tasks API (#45280)
Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2019-08-08 14:49:32 +02:00
James Rodewig e72bda1703 [DOCS] Reformats cat nodes API (#45285) 2019-08-08 08:38:02 -04:00
James Rodewig b806e6edde [DOCS] Reformats cat pending tasks API (#45287) 2019-08-08 08:32:04 -04:00
István Zoltán Szabó 276e9c6697 [DOCS] Adds supported time units ref to the ML and DF API params. (#45322) 2019-08-08 14:26:19 +02:00
David Turner ddcc38cf1c
More read-only-allow-delete docs (#45320)
Adds to the `index.blocks.read_only_allow_delete` docs the information that
this block may be added or removed automatically, and rewords the
breaking-changes docs to mention the blocks explicitly and to recommend using a
different block.

Relates #42559
2019-08-08 09:58:23 +01:00
István Zoltán Szabó 9f62c04637 [DOCS] Reformats cluster health and cluster state APIs (#45206)
Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2019-08-08 10:25:05 +02:00
Gordon Brown e3599fded7
Add warning about versions to Deprecation API docs (#36624)
Add a note that the Deprecation API may not be up to date with all
breaking changes until the last minor version in a major version series.
2019-08-07 14:11:05 -06:00
Benjamin Trent 5db9982f71
[7.x] [ML][Data Frame] Add update transform api endpoint (#45154) (#45279)
* [ML][Data Frame] Add update transform api endpoint (#45154)

This adds the ability to `_update` stored data frame transforms. All mutable fields are applied when the next checkpoint starts. The exception being `description`.

This PR contains all that is necessary for this addition:
* HLRC
* Docs
* Server side
2019-08-07 10:37:35 -05:00
István Zoltán Szabó 95d3a8e8ad [DOCS] Reformats cluster stats API and expands common params (#45270)
Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2019-08-07 16:49:58 +02:00
István Zoltán Szabó 9384774b4c [DOCS] Adds supported time units ref to the frequency and delay params. (#45283) 2019-08-07 16:19:59 +02:00
James Rodewig 46fc989ca2 [DOCS] Reformats cat nodeattrs API (#45255) 2019-08-07 09:31:37 -04:00
James Rodewig 5ade756275 [DOCS] Reformats cat indices API (#45239) 2019-08-07 09:08:35 -04:00
James Rodewig d8423f0a80 [DOCS] Reformat cat master API (#45240) 2019-08-07 09:04:04 -04:00
Bas ten Berge 15d41a7508 [DOCS] Remove outdated 5.x community plugins (#40306)
Our docs previously included several community plugins that are only supported for versions 5.x and earlier. This removes those plugins for our 6.6+ docs.
2019-08-07 08:45:05 -04:00
Bukhtawar cd304c4def Auto-release flood-stage write block (#42559)
If a node exceeds the flood-stage disk watermark then we add a block to all of
its indices to prevent further writes as a last-ditch attempt to prevent the
node completely exhausting its disk space. However today this block remains in
place until manually removed, and this block is a source of confusion for users
who current have ample disk space and did not even realise they nearly ran out
at some point in the past.

This commit changes our behaviour to automatically remove this block when a
node drops below the high watermark again. The expectation is that the high
watermark is some distance below the flood-stage watermark and therefore the
disk space problem is truly resolved.

Fixes #39334
2019-08-07 11:03:53 +01:00
Julie Tibshirani 9318192578 Correct a code snippet in removal_of_types. (#45118)
Previously, the reindex examples did not include `_doc` as the destination type.
This would result in the reindex failing with the error "Rejecting mapping
update to [users] as the final mapping would have more than 1 type: [_doc,
user]".

Relates to #43100.
2019-08-06 14:09:21 -07:00
Lisa Cawley 7c9c9a9cc4 [DOCS] Reformats ML update APIs (#45253) 2019-08-06 11:16:29 -07:00
James Rodewig 51c1abc112 [DOCS] Reformat match phrase prefix query (#45209) 2019-08-06 14:02:07 -04:00
Jason Tedor 9a142ff25c
Introduce formal node ML role (#45174)
This commit builds on the ability for plugins to introduce new roles to
add a formal node ML role.
2019-08-06 13:00:05 -04:00
Lisa Cawley bb22b2e45f [DOCS] Reformats get rollup jobs API (#45114) 2019-08-06 09:40:18 -07:00
James Rodewig b09b874801 [DOCS] Reformat cat fielddata API (#45202) 2019-08-06 10:33:51 -04:00
Alexander Reelsen 210593d8e5 Add back lowercase processor in docs (#45090)
This got lost in a refactoring in 9137d92ca6
2019-08-06 09:23:13 -04:00
James Rodewig 1c96e8a6af [DOCS] Reformat cat health API (#45218) 2019-08-06 08:41:18 -04:00
James Rodewig 2489508e1f [DOCS] Reformat cat count API (#45160) 2019-08-06 08:37:09 -04:00
Yannick Welsch 7aeb2fe73c Add per-socket keepalive options (#44055)
Uses JDK 11's per-socket configuration of TCP keepalive (supported on Linux and Mac), see
https://bugs.openjdk.java.net/browse/JDK-8194298, and exposes these as transport settings.
By default, these options are disabled for now (i.e. fall-back to OS behavior), but we would like
to explore whether we can enable them by default, in particular to force keepalive configurations
that are better tuned for running ES.
2019-08-06 10:45:44 +02:00
David Turner f91487baff Relax recovery duration in docs test (#45186)
Sometimes the recovery in this docs test takes long enough that it is expressed
in `s` rather than `ms`. This commit relaxes the assertion to account for this.
2019-08-06 09:30:53 +01:00
Gordon Brown d0d808776f
Add guide for using ILM with existing indices (#43667)
In many cases, including migration from previous versions of data
shippers (e.g. Beats), it is useful to use ILM to manage historical
indices, which are no longer being written to. This commit adds a guide
which gives an example of how to do that.
2019-08-05 17:28:59 -06:00
Zachary Tong 3df1c76f9b Allow pipeline aggs to select specific buckets from multi-bucket aggs (#44179)
This adjusts the `buckets_path` parser so that pipeline aggs can
select specific buckets (via their bucket keys) instead of fetching
the entire set of buckets.  This is useful for bucket_script in
particular, which might want specific buckets for calculations.

It's possible to workaround this with `filter` aggs, but the workaround
is hacky and probably less performant.

- Adjusts documentation
- Adds a barebones AggregatorTestCase for bucket_script
- Tweaks AggTestCase to use getMockScriptService() for reductions and
pipelines.  Previously pipelines could just pass in a script service
for testing, but this didnt work for regular aggs.  The new
getMockScriptService() method fixes that issue, but needs to be used
for pipelines too.  This had a knock-on effect of touching MovFn,
AvgBucket and ScriptedMetric
2019-08-05 12:18:40 -04:00
Zachary Tong e5079ac288
[7.x backport] Add more flexibility to MovingFunction window alignment (#45159)
Introduce shift field to MovingFunction aggregation.

By default, shift = 0. Behavior, in this case, is the same as before.
Increasing shift by 1 moves starting window position by 1 to the right.

    To simply include current bucket to the window, use shift = 1
    For center alignment (n/2 values before and after the current bucket), use shift = window / 2
    For right alignment (n values after the current bucket), use shift = window.
2019-08-05 11:56:52 -04:00
James Rodewig adaa54c066 [DOCS] Reformats cat allocation API (#45158) 2019-08-05 11:24:27 -04:00
James Rodewig 2f4c342fff [DOCS] Reformat cat alias API (#45119) 2019-08-05 08:29:55 -04:00
István Zoltán Szabó dae648eb32 [DOCS] Makes clearer the note under freq_rare. (#45193) 2019-08-05 13:29:43 +02:00
Zachary Tong ffbe047c32 Revert "Add more flexibility to MovingFunction window alignment (#44360)"
This reverts commit 1a58a487f0.
2019-08-02 15:16:04 -04:00
Nikita Glashenko 1a58a487f0 Add more flexibility to MovingFunction window alignment (#44360)
Introduce shift field to MovingFunction aggregation.

By default, shift = 0. Behavior, in this case, is the same as before.
Increasing shift by 1 moves starting window position by 1 to the right.

    To simply include current bucket to the window, use shift = 1
    For center alignment (n/2 values before and after the current bucket), use shift = window / 2
    For right alignment (n values after the current bucket), use shift = window.
2019-08-02 15:10:21 -04:00
James Rodewig 4b8ae97e9d [DOCS] Update relevance score cross-references (#45092) 2019-08-02 14:15:34 -04:00
James Rodewig 8dd74dfe0b Rename "indices APIs" to "index APIs" (#44863) 2019-08-02 14:10:09 -04:00
Lisa Cawley 09bd6c4692 [DOCS] Clarifies bucket span in overall buckets API (#45110) 2019-08-02 08:42:39 -07:00
James Rodewig f657f3914d [DOCS] Create common parameters file for REST APIs (#45117) 2019-08-02 08:43:12 -04:00
James Rodewig 138865a58e [DOCS] Reformat script score query (#45087) 2019-08-01 12:23:35 -04:00
Lisa Cawley 7f74790db2
[DOCS] Adds 7.3.0 ml-cpp release notes (#45019) 2019-08-01 09:12:33 -07:00
lcawl 4e00e119ed [DOCS] Fixes missing discrete attribute 2019-08-01 09:11:14 -07:00
James Rodewig 187f1e776b [DOCS] Offset clone index API headings to correct nav (#45097) 2019-08-01 11:22:41 -04:00
Benjamin Trent 2fc4b76ba8
[ML][Data Frame] adding breaking changes to doc (#45051)
* [ML][Data Frame] adding breaking changes to doc

* [DOCS] Edits data frame transform breaking change

* Update migrate_7_3.asciidoc
2019-08-01 07:30:41 -05:00
David Turner 532ade7816 More logging for slow cluster state application (#45007)
Today the lag detector may remove nodes from the cluster if they fail to apply
a cluster state within a reasonable timeframe, but it is rather unclear from
the default logging that this has occurred and there is very little extra
information beyond the fact that the removed node was lagging. Moreover the
only forewarning that the lag detector might be invoked is a message indicating
that cluster state publication took unreasonably long, which does not contain
enough information to investigate the problem further.

This commit adds a good deal more detail to make the issues of slow nodes more
prominent:

- after 10 seconds (by default) we log an INFO message indicating that a
  publication is still waiting for responses from some nodes, including the
  identities of the problematic nodes.

- when the publication times out after 30 seconds (by default) we log a WARN
  message identifying the nodes that are still pending.

- the lag detector logs a more detailed warning when a fatally-lagging node is
  detected.

- if applying a cluster state takes too long then the cluster applier service
  logs a breakdown of all the tasks it ran as part of that process.
2019-08-01 13:20:46 +01:00
Lisa Cawley 9e901c5f94
[DOCS] Edits 7.3 release highlights (#45066) 2019-07-31 15:49:27 -07:00
Lisa Cawley 78c254a9c3
[DOCS] Adds notable 7.3 breaking changes (#45057) 2019-07-31 11:47:10 -07:00
James Rodewig 3c4150cf72 [DOCS] Reformat rank feature query. Add relevance score section. (#44975) 2019-07-31 14:31:50 -04:00
James Rodewig 728b0cf9ff [DOCS] Update parameter format (#44703) 2019-07-31 14:18:55 -04:00
Flavio Pompermaier f1bab2fa89 [DOCS] Correct sum_other_doc_count value in terms agg example (#45028)
Closes issue #41902
2019-07-31 14:10:36 -04:00
Jack Conradson 5202d2624e Add several context examples for Painless date documentation (#44985) 2019-07-31 08:23:17 -07:00
Zachary Tong 6ed2a6e9dc [DOCS] Add breaking change re: direct alias modification on followers
Relates: #43017
2019-07-31 11:11:56 -04:00
Daniel Mitterdorfer 5dd0e74e79 Clarify which circuit breaker settings are static (#44992)
Most of the circuit breaker settings are dynamically configurable.
However, `indices.breaker.total.use_real_memory` is not. With this
commit we add a clarifying note that this specific setting is static.

Closes #44974
2019-07-31 13:15:33 +02:00
Jesse Wright f19f2adbe6 [Docs] Fix typo in rank-eval.asciidoc (#44978) 2019-07-31 12:37:49 +02:00
Lisa Cawley 899c62ad02 [DOCS] Remove xpack terminology from installation pages (#44973) 2019-07-30 15:29:59 -07:00
Lisa Cawley e4b7ae211b [DOCS] Updates terms in machine learning get APIs (#44986) 2019-07-30 11:29:25 -07:00
István Zoltán Szabó 19426f9cdf [DOCS] Adds allow no jobs param to the GET, GET stats and Close APIs (#44503) 2019-07-30 14:27:27 +02:00
Lisa Cawley 0914c04b8e
[DOCS] Updates to data frame transforms release highlight (#44907) 2019-07-29 09:45:09 -07:00
James Rodewig 3839840b0c [DOCS] Reformat script query (#44882) 2019-07-29 09:59:36 -04:00
James Rodewig 601521f4c1 [DOCS] Rewrite `prefix` query docs (#41955) 2019-07-29 08:37:14 -04:00
James Rodewig 652f943f30 [DOCS] Reformat distance feature query (#44916) 2019-07-29 08:35:18 -04:00
István Zoltán Szabó ad129f7947 [DOCS] Amends REST API TOC. (#44948) 2019-07-29 14:08:57 +02:00
Sandeep Kanabar 8f1a3ab70a [Docs] Update daterange-aggregation.asciidoc (#44730)
Correcting the value to be the same as that specified for "missing".
2019-07-29 12:50:33 +02:00
István Zoltán Szabó dc26521b0f
[DOCS] Updates to outlier detection release highlight (#44911) 2019-07-29 08:03:11 +02:00
Lisa Cawley a041d1eacf [DOCS] Updates anomaly detection terminology (#44888) 2019-07-26 11:10:49 -07:00
Lisa Cawley cef375f883 [DOCS] Updates terms in machine learning datafeed APIs (#44883) 2019-07-26 10:48:28 -07:00
Gordon Brown d4b2d21339
Add option to filter ILM explain response (#44777)
In order to make it easier to interpret the output of the ILM Explain
API, this commit adds two request parameters to that API:

- `only_managed`, which causes the response to only contain indices
  which have `index.lifecycle.name` set
- `only_errors`, which causes the response to contain only indices in an
  ILM error state

"Error state" is defined as either being in the `ERROR` step or having
`index.lifecycle.name` set to a policy that does not exist.
2019-07-26 11:57:38 -04:00
Adrien Grand 97177a3b4f Update 7.3 release notes. 2019-07-26 17:47:35 +02:00
István Zoltán Szabó cd7ba9f302 [DOCS] Amends data frame analytics resources, GET, and PUT API docs (#44806)
This PR addresses the feedback in  https://github.com/elastic/ml-team/issues/175#issuecomment-512215731.

* Adds an example to `analyzed_fields`
* Includes `source` and `dest` objects inline in the resource page
* Lists `model_memory_limit` in the PUT API page
* Amends the `analysis` section in the resource page
* Removes Properties headings in subsections
2019-07-26 11:52:43 +02:00
Nhat Nguyen d128188c28 Return seq_no and primary_term in noop update (#44603)
With this change, we will return primary_term and seq_no of the current
document if an update is detected as a noop. We already return the
version; hence we should also return seq_no and primary_term.

Relates #42497
2019-07-25 19:16:56 -04:00
lcawl 03605169f3 [DOCS] Clarifies dataframe transform validations 2019-07-25 15:05:42 -07:00
Yannick Welsch 0ce841915c Add Clone Index API (#44267)
Adds an API to clone an index. This is similar to the index split and shrink APIs, just with the
difference that the number of primary shards is kept the same. In case where the filesystem
provides hard-linking capabilities, this is a very cheap operation.

Indexing cloning can be done by running `POST my_source_index/_clone/my_target_index` and it
supports the same options as the split and shrink APIs.

Closes #44128
2019-07-25 22:02:28 +02:00
Lisa Cawley 21971feae8 [DOCS] Updates terms in machine learning calendar APIs (#44866) 2019-07-25 11:50:34 -07:00
Lisa Cawley c9909b09b5 [DOCS] Adds command reference for elasticsearch-croneval (#43946) 2019-07-25 11:41:05 -07:00
lcawl 6a60fd6d30 [DOCS] Fixes terms in HLRC data frame transform APIs (#44838) (#44865) 2019-07-25 10:08:38 -07:00
Lisa Cawley a79adca7e3 [DOCS] Updates terms in anomaly detection job APIs (#44839) 2019-07-25 09:06:52 -07:00
István Zoltán Szabó 4a31c426e6 [DOCS] Adds allow no datafeeds query param to the GET, GET stats and STOP datafeed APIs (#44499) 2019-07-25 17:08:01 +02:00
Andrei Stefan 2633d11eb7
Switch from using docvalue_fields to extracting values from _source (#44062) (#44804)
* Switch from using docvalue_fields to extracting values from _source
where applicable. Doing this means parsing the _source and handling the
numbers parsing just like Elasticsearch is doing it when it's indexing
a document.
* This also introduces a minor limitation: aliases type of fields that
are NOT part of a tree of sub-fields will not be able to be retrieved
anymore. field_caps API doesn't shed any light into a field being an
alias or not and at _source parsing time there is no way to know if a
root field is an alias or not. Fields of the type "a.b.c.alias" can be
extracted from docvalue_fields, only if the field they point to can be
extracted from docvalue_fields. Also, not all fields in a hierarchy of
fields can be evaluated to being an alias.

(cherry picked from commit 8bf8a055e38f00df5f49c8d97f632f69d6e00c2c)
2019-07-25 10:02:41 +03:00
Lisa Cawley 8bac13d5b2 [DOCS] Adds descriptions for put and start data frame transforms (#44724) 2019-07-24 11:12:56 -07:00
Lee Hinman fde6fcdfac Order ILM actions in policy definition documentation (#44773)
We already have a note that the order of actions is up to ILM for each
phase, this commit puts the actions in the same order as they will be
executed.

Resolves #41729
2019-07-24 11:28:38 -06:00
Tanguy Leroux a8905ef142
[7.x] Add CloseIndexResponse to HLRC (#44349) (#44788)
The CloseIndexResponse was improved in #39687; this commit
exposes it in the HLRC.

Backport of #44349 to 7.x.
2019-07-24 15:51:01 +02:00
Adrien Grand 0418eb35a7
Add 7.3 release highlights. (#44153)
Add 7.3 release highlights.
2019-07-24 15:05:23 +02:00
James Rodewig ad7c164dd0 [DOCS] Rewrite `regexp` query (#42711) 2019-07-24 08:38:41 -04:00
Jake Landis 27440b7692
update 7.2.1 release notes (#44774) 2019-07-23 16:33:31 -05:00
James Rodewig 9524d71159 [DOCS] Document // TEARDOWN test snippet. (#44762)
The `// TEARDOWN` test snippet was added with #34716. You can use this
snippet to end and clean up a test series started with `// TESTSETUP` or
`// TEST[setup:name]`.
2019-07-23 14:05:14 -04:00
David Roberts caf9411a72
[ML] Improve response format of data frame stats endpoint (#44743)
This change adjusts the data frame transforms stats
endpoint to return a structure that is easier to
understand.

This is a breaking change for clients of the data frame
transforms stats endpoint, but the feature is in beta so
stability is not guaranteed.

Backport of #44350
2019-07-23 18:00:50 +01:00
Przemysław Witek 16c8e18013
Deprecate the ability to update datafeed's job_id. (#44691) (#44742) 2019-07-23 14:48:56 +02:00
David Turner 8516fb0f3b Expand docs on force-merge and global ordinals (#44684)
Some small clarifications about force-merging and global ordinals, particularly
that global ordinals are cheap on a single-segment index and how this relates
to frozen indices.

Fixes #41687
2019-07-23 07:33:33 +01:00
Lisa Cawley ed8f75c990 [DOCS] Adds more validation details for data frame transforms (#44714) 2019-07-22 15:41:15 -07:00
Benjamin Trent 4456850a8e
[7.x] [ML][Data Frame] Add optional defer_validation param to PUT (#44455) (#44697)
* [ML][Data Frame] Add optional defer_validation param to PUT (#44455)

* [ML][Data Frame] Add optional defer_validation param to PUT

* addressing PR comments

* reverting bad replace

* addressing pr comments

* Update put-transform.asciidoc

* Update put-transform.asciidoc

* Update put-transform.asciidoc

* adjusting for backport

* fixing imports

* [DOCS] Fixes formatting in  create data frame transform API
2019-07-22 15:12:55 -05:00
Benjamin Trent 06e21f7902
[7.x] [ML][Data Frame] adding force delete (#44590) (#44696)
* [ML][Data Frame] adding force delete (#44590)

* [ML][Data Frame] adding force delete

* Update delete-transform.asciidoc

* adjusting for backport
2019-07-22 13:13:25 -05:00
David Roberts 2a4380e0aa [DOCS] Clarifies data frame transform frequency is also for retries (#44229)
In data frame transforms the same scheduler controls both
retries in the event of search failures and gaps between
checks for changes when the transform is running continuously.

Co-Authored-By: Lisa Cawley <lcawley@elastic.co>
2019-07-22 17:53:49 +01:00
James Rodewig 8d7392de35 [DOCS] Make field datatype titles consistent (#43933)
* [DOCS] Make field datatype titles consistent

* Add titleabbrev for array
2019-07-22 08:52:23 -04:00
Benjamin Trent 2e303fc5f7
[ML][Data Frame] adding dynamic cluster setting for failure retries (#44577) (#44639)
This adds a new dynamic cluster setting `xpack.data_frame.num_transform_failure_retries`.

This setting indicates how many times non-critical failures should be retried before a data frame transform is marked as failed and should stop executing. At the time of this commit; Min: 0, Max: 100, Default: 10
2019-07-19 16:17:39 -05:00
James Rodewig a63f60b776 [DOCS] Remove heading offsets for REST APIs (#44568)
Several files in the REST APIs nav section are included using
:leveloffset: tags. This increments headings (h2 -> h3, h3 -> h4, etc.)
in those files and removes the :leveloffset: tags.

Other supporting changes:
* Alphabetizes top-level REST API nav items.
* Change 'indices APIs' heading to 'index APIs.'
* Changes 'Snapshot lifecycle management' heading to sentence case.
2019-07-19 14:36:06 -04:00
David Turner c3b6ab4dec Fix docs line length
Followup to #44617 that fixes some unexpected wrapping.
2019-07-19 17:04:42 +01:00
David Turner 87f246f6bd Clarify repository-s3 defaulting to instance role (#44617)
Specifies more clearly that you can leave the client credential settings unset
if you want to use the instance or container role.
2019-07-19 16:11:23 +01:00
James Rodewig 1f5a2ac3ad [DOCS] Reformat `parent_id` query docs (#44449) 2019-07-19 11:03:45 -04:00
James Rodewig 51aefbdd3d [DOCS] Reformat `has_parent` query docs (#44443) 2019-07-19 10:51:11 -04:00
James Rodewig d46545f729 [DOCS] Update anchors and links for Elasticsearch API relocation (#44500) 2019-07-19 09:18:23 -04:00
James Rodewig 76c7e3a05f [DOCS] Replace `_meta` with `metadata` for snapshot APIs. (#44596)
elastic/elasticsearch#41281 added custom metadata parameter to
snapshots. During review, the parameter name was changed from '_meta' to
'metadata,' but the documentation wasn't updated. This corrects the
documentation to use the 'metadata' name.
2019-07-19 08:40:57 -04:00
Lisa Cawley 5a05bd76b4 [DOCS] Reformat info and deprecation APIs to use new API format (#44599) 2019-07-18 16:11:48 -07:00
Lee Hinman fe2ef66e45 Expose index age in ILM explain output (#44457)
* Expose index age in ILM explain output

This adds the index's age to the ILM explain output, for example:

```
{
  "indices" : {
    "ilm-000001" : {
      "index" : "ilm-000001",
      "managed" : true,
      "policy" : "full-lifecycle",
      "lifecycle_date" : "2019-07-16T19:48:22.294Z",
      "lifecycle_date_millis" : 1563306502294,
      "age" : "1.34m",
      "phase" : "hot",
      "phase_time" : "2019-07-16T19:48:22.487Z",
      ... etc ...
    }
  }
}
```

This age can be used to tell when ILM will transition the index to the
next phase, based on that phase's `min_age`.

Resolves #38988

* Expose age in getters and in HLRC
2019-07-18 15:33:45 -06:00
Lisa Cawley 8445c41004
[DOCS] Moves content to ML anomaly-detection folder (#44520) (#44530) 2019-07-18 08:44:52 -07:00
Lisa Cawley 213af8411f [DOCS] Fixes query default value (#44572) 2019-07-18 08:18:58 -07:00
Lisa Cawley fe23dd1073 [DOCS] Clarifies data transform source definition (#44504) 2019-07-18 08:04:56 -07:00
Armin Braun 07cf2cb665
Add disable_chunked_encoding Setting to S3 Repo (#44052) (#44562)
* Add disable_chunked_encoding setting to S3 repo plugin to support S3 implementations that don't support chunked encoding
2019-07-18 16:57:56 +02:00
Lisa Cawley 94ffa5a93c [DOCS] Improves description of dest data frame transform object (#44509) 2019-07-18 07:45:55 -07:00
James Rodewig 661f70183e [DOCS] Make Query DSL titles consistent (#43935) 2019-07-18 10:33:10 -04:00
David Turner 452f7f67a0
Defer reroute when starting shards (#44539)
Today we reroute the cluster as part of the process of starting a shard, which
runs at `URGENT` priority. In large clusters, rerouting may take some time to
complete, and this means that a mere trickle of shard-started events can cause
starvation for other, lower-priority, tasks that are pending on the master.

However, it isn't really necessary to perform a reroute when starting a shard,
as long as one occurs eventually. This commit removes the inline reroute from
the process of starting a shard and replaces it with a deferred one that runs
at `NORMAL` priority, avoiding starvation of higher-priority tasks.

Backport of #44433 and #44543.
2019-07-18 14:10:40 +01:00
Igor Motov a19c7977ac Docs: fix WKTToSQL function example (#44377)
Fixes wrong example snippet in WKTToSQL documentation.

Closes #44299
2019-07-17 12:24:28 -04:00
Lisa Cawley 51d8e6e3de [DOCS] Adds sync to data frame transform API (#44254) 2019-07-17 08:56:58 -07:00
Yannick Welsch d98b3e4760
Move frozen indices to x-pack module (#44490)
Backport of #44408 and #44286.
2019-07-17 16:53:10 +02:00
James Rodewig 34725e20fb [DOCS] Move Elasticsearch APIs to REST APIs section. (#44238) (#44372)
Moves the following API sections under the REST APIs navigations:
- API Conventions
- Document APIs
- Search APIs
- Index APIs (previously named Indices APIs)
- cat APIs
- Cluster APIs

Other supporting changes:
- Removes the previous index APIs page under REST APIs. Adds a redirect for the removed page.
- Removes several [partintro] macros so the docs build correctly.
- Changes anchors for pages that become sections of a parent page.
- Adds several redirects for existing pages that become sections of a parent page.

This commit re-applies changes from #44238. Changes from that PR were reverted due to broken links in several repos. This commit adds redirects for those broken links.
2019-07-17 09:18:31 -04:00
Lisa Cawley c18a6402cd [DOCS] Renames data frame APIs section (#44458) 2019-07-16 15:10:46 -07:00
Lisa Cawley 53514b0477 [DOCS] Separates data frame analytics APIs (#44451) 2019-07-16 13:33:23 -07:00
Mayya Sharipova 3220709b0a Add positions info into term_vector doc (#44379) 2019-07-16 16:24:50 -04:00
István Zoltán Szabó 642e9019ff [DOCS] Adds max_page_search_size to data frame transform pivot properties (#44126) 2019-07-16 21:33:19 +02:00
James Rodewig ac07eef86c [DOCS] Remove :edit_url: overrides. (#44445)
These overrides do not work in Asciidoctor and are no longer needed.
2019-07-16 15:04:44 -04:00
Yu 563a78829f Do not allow version in Rest Update API (#43516)
The versioning of Update API doesn't rely on version number anymore (and
rather on sequence number). But in rest api level we ignored the
"version" and "version_type" parameter, so that the server cannot raise
the exception when whey were set.

This PR restores "version" and "version_type" parsing in Update Rest API
so that we can get the appropriate errors.

Relates to #42497
2019-07-16 13:19:07 -04:00
Hendrik Muhs 5ea89a4e7b Remove PR 41491 from release notes (#44417)
PR 41491 hasn't been merged into 7.2 but got closed and superseeded by #41800 (merged via feature branch merge into 7.3)
2019-07-16 10:03:40 -07:00
Lee Hinman fb0461ac76
[7.x] Add Snapshot Lifecycle Management (#44382)
* Add Snapshot Lifecycle Management (#43934)

* Add SnapshotLifecycleService and related CRUD APIs

This commit adds `SnapshotLifecycleService` as a new service under the ilm
plugin. This service handles snapshot lifecycle policies by scheduling based on
the policies defined schedule.

This also includes the get, put, and delete APIs for these policies

Relates to #38461

* Make scheduledJobIds return an immutable set

* Use Object.equals for SnapshotLifecyclePolicy

* Remove unneeded TODO

* Implement ToXContentFragment on SnapshotLifecyclePolicyItem

* Copy contents of the scheduledJobIds

* Handle snapshot lifecycle policy updates and deletions (#40062)

(Note this is a PR against the `snapshot-lifecycle-management` feature branch)

This adds logic to `SnapshotLifecycleService` to handle updates and deletes for
snapshot policies. Policies with incremented versions have the old policy
cancelled and the new one scheduled. Deleted policies have their schedules
cancelled when they are no longer present in the cluster state metadata.

Relates to #38461

* Take a snapshot for the policy when the SLM policy is triggered (#40383)

(This is a PR for the `snapshot-lifecycle-management` branch)

This commit fills in `SnapshotLifecycleTask` to actually perform the
snapshotting when the policy is triggered. Currently there is no handling of the
results (other than logging) as that will be added in subsequent work.

This also adds unit tests and an integration test that schedules a policy and
ensures that a snapshot is correctly taken.

Relates to #38461

* Record most recent snapshot policy success/failure (#40619)

Keeping a record of the results of the successes and failures will aid
troubleshooting of policies and make users more confident that their
snapshots are being taken as expected.

This is the first step toward writing history in a more permanent
fashion.

* Validate snapshot lifecycle policies (#40654)

(This is a PR against the `snapshot-lifecycle-management` branch)

With the commit, we now validate the content of snapshot lifecycle policies when
the policy is being created or updated. This checks for the validity of the id,
name, schedule, and repository. Additionally, cluster state is checked to ensure
that the repository exists prior to the lifecycle being added to the cluster
state.

Part of #38461

* Hook SLM into ILM's start and stop APIs (#40871)

(This pull request is for the `snapshot-lifecycle-management` branch)

This change allows the existing `/_ilm/stop` and `/_ilm/start` APIs to also
manage snapshot lifecycle scheduling. When ILM is stopped all scheduled jobs are
cancelled.

Relates to #38461

* Add tests for SnapshotLifecyclePolicyItem (#40912)

Adds serialization tests for SnapshotLifecyclePolicyItem.

* Fix improper import in build.gradle after master merge

* Add human readable version of modified date for snapshot lifecycle policy (#41035)

* Add human readable version of modified date for snapshot lifecycle policy

This small change changes it from:

```
...
"modified_date": 1554843903242,
...
```

To

```
...
"modified_date" : "2019-04-09T21:05:03.242Z",
"modified_date_millis" : 1554843903242,
...
```

Including the `"modified_date"` field when the `?human` field is used.

Relates to #38461

* Fix test

* Add API to execute SLM policy on demand (#41038)

This commit adds the ability to perform a snapshot on demand for a policy. This
can be useful to take a snapshot immediately prior to performing some sort of
maintenance.

```json
PUT /_ilm/snapshot/<policy>/_execute
```

And it returns the response with the generated snapshot name:

```json
{
  "snapshot_name" : "production-snap-2019.04.09-rfyv3j9qreixkdbnfuw0ug"
}
```

Note that this does not allow waiting for the snapshot, and the snapshot could
still fail. It *does* record this information into the cluster state similar to
a regularly trigged SLM job.

Relates to #38461

* Add next_execution to SLM policy metadata (#41221)

* Add next_execution to SLM policy metadata

This adds the next time a snapshot lifecycle policy will be executed when
retriving a policy's metadata, for example:

```json
GET /_ilm/snapshot?human
{
  "production" : {
    "version" : 1,
    "modified_date" : "2019-04-15T21:16:21.865Z",
    "modified_date_millis" : 1555362981865,
    "policy" : {
      "name" : "<production-snap-{now/d}>",
      "schedule" : "*/30 * * * * ?",
      "repository" : "repo",
      "config" : {
        "indices" : [
          "foo-*",
          "important"
        ],
        "ignore_unavailable" : true,
        "include_global_state" : false
      }
    },
    "next_execution" : "2019-04-15T21:16:30.000Z",
    "next_execution_millis" : 1555362990000
  },
  "other" : {
    "version" : 1,
    "modified_date" : "2019-04-15T21:12:19.959Z",
    "modified_date_millis" : 1555362739959,
    "policy" : {
      "name" : "<other-snap-{now/d}>",
      "schedule" : "0 30 2 * * ?",
      "repository" : "repo",
      "config" : {
        "indices" : [
          "other"
        ],
        "ignore_unavailable" : false,
        "include_global_state" : true
      }
    },
    "next_execution" : "2019-04-16T02:30:00.000Z",
    "next_execution_millis" : 1555381800000
  }
}
```

Relates to #38461

* Fix and enhance tests

* Figured out how to Cron

* Change SLM endpoint from /_ilm/* to /_slm/* (#41320)

This commit changes the endpoint for snapshot lifecycle management from:

```
GET /_ilm/snapshot/<policy>
```

to:

```
GET /_slm/policy/<policy>
```

It mimics the ILM path only using `slm` instead of `ilm`.

Relates to #38461

* Add initial documentation for SLM (#41510)

* Add initial documentation for SLM

This adds the initial documentation for snapshot lifecycle management.

It also includes the REST spec API json files since they're sort of
documentation.

Relates to #38461

* Add `manage_slm` and `read_slm` roles (#41607)

* Add `manage_slm` and `read_slm` roles

This adds two more built in roles -

`manage_slm` which has permission to perform any of the SLM actions, as well as
stopping, starting, and retrieving the operation status of ILM.

`read_slm` which has permission to retrieve snapshot lifecycle policies as well
as retrieving the operation status of ILM.

Relates to #38461

* Add execute to the test

* Fix ilm -> slm typo in test

* Record SLM history into an index (#41707)

It is useful to have a record of the actions that Snapshot Lifecycle
Management takes, especially for the purposes of alerting when a
snapshot fails or has not been taken successfully for a certain amount of
time.

This adds the infrastructure to record SLM actions into an index that
can be queried at leisure, along with a lifecycle policy so that this
history does not grow without bound.

Additionally,
SLM automatically setting up an index + lifecycle policy leads to
`index_lifecycle` custom metadata in the cluster state, which some of
the ML tests don't know how to deal with due to setting up custom
`NamedXContentRegistry`s.  Watcher would cause the same problem, but it
is already disabled (for the same reason).

* High Level Rest Client support for SLM (#41767)

* High Level Rest Client support for SLM

This commit add HLRC support for SLM.

Relates to #38461

* Fill out documentation tests with tags

* Add more callouts and asciidoc for HLRC

* Update javadoc links to real locations

* Add security test testing SLM cluster privileges (#42678)

* Add security test testing SLM cluster privileges

This adds a test to `PermissionsIT` that uses the `manage_slm` and `read_slm`
cluster privileges.

Relates to #38461

* Don't redefine vars

*  Add Getting Started Guide for SLM  (#42878)

This commit adds a basic Getting Started Guide for SLM.

* Include SLM policy name in Snapshot metadata (#43132)

Keep track of which SLM policy in the metadata field of the Snapshots
taken by SLM. This allows users to more easily understand where the
snapshot came from, and will enable future SLM features such as
retention policies.

* Fix compilation after master merge

* [TEST] Move exception wrapping for devious exception throwing

Fixes an issue where an exception was created from one line and thrown in another.

* Fix SLM for the change to AcknowledgedResponse

* Add Snapshot Lifecycle Management Package Docs (#43535)

* Fix compilation for transport actions now that task is required

* Add a note mentioning the privileges needed for SLM (#43708)

* Add a note mentioning the privileges needed for SLM

This adds a note to the top of the "getting started with SLM"
documentation mentioning that there are two built-in privileges to
assist with creating roles for SLM users and administrators.

Relates to #38461

* Mention that you can create snapshots for indices you can't read

* Fix REST tests for new number of cluster privileges

* Mute testThatNonExistingTemplatesAreAddedImmediately (#43951)

* Fix SnapshotHistoryStoreTests after merge

* Remove overridden newResponse functions that have been removed

* Fix compilation for backport

* Fix get snapshot output parsing in test

* [DOCS] Add redirects for removed autogen anchors (#44380)

* Switch <tt>...</tt> in javadocs for {@code ...}
2019-07-16 07:37:13 -06:00
Lucas Groenendaal aa9dd313cf Fix incorrect node name in docs (#43062)
After starting up elasticsearch the documentation said that their node
name was "6-bjhwl" but in the documentation's output I did not see that
node name. Instead I saw the node name as `localhost.localdomain`
2019-07-16 14:58:42 +02:00
david raistrick ae5a917efe Add clarification around TESTSETUP docs and error message (#43306) 2019-07-16 14:58:16 +02:00
Mark Walkom 4a5215d22a [DOCS] Update id-field.asciidoc (#42482)
Adding a note around the size limit for `_id`
2019-07-16 14:57:33 +02:00
Dan Fey 8a2d23671a [DOCS] Update split-index.asciidoc: fix shards example (#41382)
The max value should be 640 instead of 740 in the shard example:
2019-07-16 14:54:27 +02:00
Tanguy Buchier 078efc9ec4 [DOCS] Clarify refresh_interval new behavior (#43726)
Update indexing-speed.asciidoc to clarify refresh_interval new behavior
2019-07-16 14:53:46 +02:00
magnusram05 096c03945c [Docs] Small update to getting-started.asciidoc (#40393) 2019-07-16 13:40:54 +02:00
Tim Vernum 4b50de2e2e
Document xpack.security.dls.bitset.cache settings (#44400)
Two new settings were introduced in #43669 to control the
behaviour of the Document Level Security BitSet cache.

This change adds documentation for these 2 settings.

Backport of: #44100
2019-07-16 16:22:25 +10:00
Ryan Ernst 7e06888bae
Convert testclusters to use distro download plugin (#44253) (#44362)
Test clusters currently has its own set of logic for dealing with
finding different versions of Elasticsearch, downloading them, and
extracting them. This commit converts testclusters to use the
DistributionDownloadPlugin.
2019-07-15 17:53:05 -07:00
Jake Landis c00b082701
add 7.2.1 release notes (#44367) 2019-07-15 15:02:56 -05:00
Adrien Grand 3734356955 Update release notes. 2019-07-15 20:01:23 +02:00
Lisa Cawley e7ea49e32f [DOCS] Removes unnecessary resource definition pages (#44289) 2019-07-15 10:03:53 -07:00
Julie Tibshirani 141d09ee15 Correct a formatting mistake in the _field_caps docs. (#44303)
The 'indices' block that was recently added should appear in the top-level of
the response, as opposed to being nested under 'fields'.
2019-07-15 09:46:02 -07:00
Lisa Cawley 6c7f7d4a10 [DOCS] Adds ml-cpp PRs to release notes (#44354) 2019-07-15 09:22:36 -07:00
David Kyle 2382701547 Wait for pending tasks in docs tests cleanup (#44123)
ML and Data Frame tests should wait for pending tasks
2019-07-15 12:04:27 +01:00
Armin Braun d73e2f9c56
HLRC: Fix '+' Not Correctly Encoded in GET Req. (#33164) (#44324)
* HLRC: Fix '+' Not Correctly Encoded in GET Req.

* Encode `+` correctly as `%2B` in URL paths
* Keep encoding `+` as space in URL parameters
* Closes #33077
2019-07-15 10:21:54 +02:00
Nikita Glashenko d187fcb9de Support WKT point conversion to geo_point type (#44107)
This PR adds support for parsing geo_point values from WKT POINT format.
Also, a few minor bugs in geo_point parsing were fixed.

Closes #41821
2019-07-12 14:31:07 -04:00
Lisa Cawley 4e0cc1be8f [DOCS] Fixes attribute in data frame transform API (#44249) 2019-07-12 09:56:35 -07:00
Zachary Tong 3fa677ce79 Document that pipeline aggs are not compatible with composite agg (#44180) 2019-07-12 12:35:18 -04:00
Lisa Cawley 8fdcf28fac [DOCS] Reformats API parameter details (#44194) 2019-07-12 08:28:49 -07:00
Lisa Cawley 4d8bf1c3e3 [DOCS] Removes links to ML tutorial (#44251) 2019-07-12 08:28:36 -07:00
James Rodewig 9ff8600d46 Revert "[DOCS] Relocate several APIs to REST APIs section (#44238)"
This reverts commit 6ebd59791afe2e0d55be2989fdbb594972237340.
2019-07-11 17:01:32 -04:00
James Rodewig 62b5b81fd2 [DOCS] Relocate several APIs to REST APIs section (#44238) 2019-07-11 16:24:28 -04:00
John Murphy 8030d8f6dc [DOCS] Add `lowercase` filter to phrase suggester example so searches are case insensitive (#44186) 2019-07-11 15:27:31 -04:00
Mayya Sharipova 32cb47b91c Add l1norm and l2norm distances for vectors (#44116)
Add L1norm - Manhattan distance
Add L2norm - Euclidean distance
relates to #37947
2019-07-11 14:30:02 -04:00
István Zoltán Szabó 2171b6b47f [DOCS] Adds data frame analytics API and evaluate API resource documentation (#43972)
This PR adds the resource documentation of the data frame analytics APIs and the evaluate API to the ML API doc pool.
2019-07-11 18:12:48 +02:00
Nick Knize 374030a53f
Upgrade to lucene-8.2.0-snapshot-860e0be5378 (#44171) (#44184)
Upgrades lucene library to lucene-8.2.0-snapshot-860e0be5378
2019-07-11 09:17:22 -05:00
James Rodewig f01a9eeb34 [DOCS] Rewrite `has_child` query to use new format (#44190) 2019-07-11 09:11:26 -04:00
surprisingb eace735d24 Update discovery-ec2 docs (#43693)
Fix `discovery.ec2.tag.TAGNAME` example with the correct parameter.
2019-07-11 12:59:38 +01:00
lcawl 4e6cbc2890 [DOCS] Fixes formatting in data frame analytics API 2019-07-10 18:01:47 -07:00
Lisa Cawley 00b16e332d [DOCS] Reformat rollup APIs to use new API format (#44131) 2019-07-10 15:15:02 -07:00
Lisa Cawley fa36f82277 [DOCS] Minor edits to data frame APIs (#44138) 2019-07-10 14:46:03 -07:00
Lisa Cawley aaf8ba9cb4 [DOCS] Adds frequency option to data frame transform resource (#44177) 2019-07-10 14:45:33 -07:00
James Rodewig 4cbd028960 [DOCS] Correct `ignore_unmapped` parm typo for nested query 2019-07-10 10:10:14 -04:00
Andrei Stefan bb3e5351b5 SQL: double quotes escaping bug fix (#43829)
(cherry picked from commit d589dcad18c3708913e13c757b91c846aeb35bb4)
2019-07-10 16:05:22 +03:00
James Rodewig 1ae0db7053 [DOCS] Rewrite nested query to use new format (#44130) 2019-07-10 08:52:04 -04:00
Albert Zaharovits 018d946bba [DOC] Backup & Restore Security Configuration (#42970)
This commit documents the backup and restore of a cluster's
security configuration.

It is not possible to only backup (or only restore) security
configuration, independent to the rest of the cluster's conf,
so this describes how a full configuration backup&restore
will include security as well. Moreover, it explains how part
of the security conf data resides on the special .security
index and how to backup that using regular data snapshot API.

Co-Authored-By: Lisa Cawley <lcawley@elastic.co>
Co-Authored-By: Tim Vernum <tim@adjective.org>
2019-07-10 14:53:56 +03:00
Przemysław Witek 44781e415e
[7.x] [ML] Add DatafeedTimingStats to datafeed GetDatafeedStatsAction.Response (#43045) (#44118) 2019-07-10 11:51:44 +02:00
David Roberts cb62d4acdf [ML-DataFrame] Add a frequency option to transform config, default 1m (#44120)
Previously a data frame transform would check whether the
source index was changed every 10 seconds. Sometimes it
may be desirable for the check to be done less frequently.
This commit increases the default to 60 seconds but also
allows the frequency to be overridden by a setting in the
data frame transform config.
2019-07-10 09:59:00 +01:00
Adrien Grand 64ff895a32 Add 7.3 release notes. (#44010) 2019-07-10 09:36:51 +02:00
David Kyle 23d7e309da Mute put job docs test
Relates to #43271
2019-07-09 13:23:31 +01:00
Armin Braun f1ebb82031
Update the gcs chunk_size documentation. (#38749) (#44098)
Remove `1g` from the examples, as the GCS repository chunk_size can be at most 100m.
2019-07-09 09:18:03 +02:00
Lisa Cawley 94578a8b47 [DOCS] Defines data frame transform resources (#43996)
Co-Authored-By: István Zoltán Szabó <istvan.szabo@elastic.co>
2019-07-08 17:53:00 -07:00
lcawl cd4021274a [DOCS] Enables testing for create job ML API (#44022) 2019-07-08 11:43:18 -07:00
Lisa Cawley 117f14e0ed
[DOCS] Updates 7.x version in data frame analytics API (#44026) 2019-07-08 11:20:57 -07:00
Lisa Cawley efddbcc1d1 [DOCS] Fixes earliest_record_timestamp data type (#44030) 2019-07-08 10:16:07 -07:00
Lisa Cawley 4b3f1003b0 [DOCS] Reformat freeze unfreeze APis to use new API format (#43948) 2019-07-08 09:01:06 -07:00
James Rodewig 4390d4a8af [DOCS] Clarify array is not a field datatype (#43931) 2019-07-08 08:58:10 -04:00
Armin Braun 2176d09c37
Provide an Option to Use Path-Style-Access with S3 Repo (#41966) (#44046)
* Provide an Option to Use Path-Style-Access with S3 Repo

* As discussed, added the option to use path style access back again and
deprecated it.
* Defaulted to `false`
* Added warning to docs

* Closes #41816
2019-07-08 08:10:01 +02:00
Mayya Sharipova 37e1ad7062 Forbid empty doc values on vector functions (#43944)
Currently when a document misses a vector value, vector function
returns 0 as a score for this document. We think this is incorrect
behaviour.
With this change, an error will be thrown if vector functions are
used with docs that are missing vector doc values.
Also VectorScriptDocValues is modified to allow size() function,
which can be used to check if a document has a value for the
vector field.
2019-07-05 18:09:06 -04:00
Nhat Nguyen 8bfe18477e Clarify consequence of translog async setting (#44020)
Relates #43915
2019-07-05 13:56:42 -04:00
lcawl a831d4707c [DOCS] Temporarily disables data frame API testing 2019-07-05 10:56:09 -07:00
István Zoltán Szabó 5aeb736801 Merge branch '7.x' of github.com:elastic/elasticsearch into 7.x 2019-07-05 14:26:47 +02:00
István Zoltán Szabó 7242267f5d [DOCS] Adds data frame analytics APIs to the ML APIs (#43875)
This PR adds the reference documentation pages of the data frame analytics APIs (PUT, START, STOP, GET, GET stats, DELETE, Evaluate) to the ML APIs pool.
2019-07-05 14:25:54 +02:00
Akshesh Doshi 01b982fd31 Draw attention to transport layer in remote cluster docs (#43883)
Closes #43858
2019-07-05 13:44:36 +02:00
István Zoltán Szabó 4c3e71b61a [DOCS] Adds description to the preview data frame transform API (#43745) 2019-07-05 09:53:24 +02:00
lcawl 688bf1b388 [DOCS] Fixes broken link 2019-07-04 09:13:56 -07:00
Lisa Cawley a030e3e513 [DOCS] Reformat CCR APIs to use new API format (#43952) 2019-07-04 08:29:54 -07:00
Alan Woodward 4b99255fed Add name() method to TokenizerFactory (#43909)
This brings TokenizerFactory into line with CharFilterFactory and TokenFilterFactory,
and removes the need to pass around tokenizer names when building custom analyzers.

As this means that TokenizerFactory is no longer a functional interface, the commit also
adds a factory method to TokenizerFactory to make construction simpler.
2019-07-04 11:28:55 +01:00
Lisa Cawley 50e96f9f0e
[DOCS] Updates documentation version (#43937) 2019-07-03 11:09:34 -07:00
Zachary Tong f8fd4321f8 Link rare_terms docs from index page (#43882)
Docs for rare_terms were added in #35718, but neglected to
link it from the bucket index page
2019-07-03 09:32:01 -04:00
James Rodewig e2a9a787fc [DOCS] Rewrite dis max query (#43586) 2019-07-03 08:56:18 -04:00
Christoph Büscher 662f517f4e Add _reload_search_analyzers endpoint to HLRC (#43733)
This change adds the new endpoint that allows reloading of search analyzers to
the high-level java rest client.

Relates to #43313
2019-07-03 12:05:59 +02:00
Dimitris Athanasiou 96b0b27f18
[7.x][ML] Set df-analytics task state to failed when appropriate (#43880) (#43906)
This introduces a `failed` state to which the data frame analytics
persistent task is set to when something unexpected fails. It could
be the process crashing, the results processor hitting some error,
etc. The failure message is then captured and set on the task state.
From there, it becomes available via the _stats API as `failure_reason`.

The df-analytics stop API now has a `force` boolean parameter. This allows
the user to call it for a failed task in order to reset it to `stopped` after
we have ensured the failure has been communicated to the user.

This commit also adds the analytics version in the persistent task
params as this allows us to prevent tasks to run on unsuitable nodes in
the future.
2019-07-03 12:41:56 +03:00
Jay Modi 1e0f67fb38 Deprecate transport profile security type setting (#43237)
This commit deprecates the `transport.profiles.*.xpack.security.type`
setting. This setting is used to configure a profile that would only
allow client actions. With the upcoming removal of the transport client
the setting should also be deprecated so that it may be removed in
a future version.
2019-07-03 19:31:55 +10:00
Tim Vernum 2a8f30eb9a
Support builtin privileges in get privileges API (#43901)
Adds a new "/_security/privilege/_builtin" endpoint so that builtin
index and cluster privileges can be retrieved via the Rest API

Backport of: #42134
2019-07-03 19:08:28 +10:00
Deb Adair a4e518b640 [DOCS] Revise GS intro and remove redundant conceptual content. Closes #43846. 2019-07-02 18:28:13 -07:00
Mayya Sharipova 756c42f99f
Add dims parameter to dense_vector mapping (#43444) (#43895)
Typically, dense vectors of both documents and queries must have the same
number of dimensions. Different number of dimensions among documents
or query vector indicate an error. This PR enforces that all vectors
for the same field have the same number of dimensions. It also enforces
that query vectors have the same number of dimensions.
2019-07-02 21:14:16 -04:00
Benjamin Trent fb825a6470
[7.x] [ML][Data Frame] add node attr to GET _stats (#43842) (#43894)
* [ML][Data Frame] add node attr to GET _stats (#43842)

* [ML][Data Frame] add node attr to GET _stats

* addressing testing issues with node.attributes

* adjusting for backport
2019-07-02 19:35:37 -05:00
Jack Conradson 8755448a18 Add Datetime Now to Painless Documentation (#43852)
This change explains why Painless doesn't natively support datetime now, and 
gives examples of how to create a version of now through user-defined 
parameters.
2019-07-02 15:43:34 -07:00
Christoph Büscher 31cf96e7bf Return reloaded analyzers in _reload_search_ananlyzer response (#43813)
Currently the repsonse of the "_reload_search_analyzer" endpoint contains the
index names and nodeIds of indices were analyzers reloading was triggered. This
change add the names of the search-time analyzers that were reloaded.

Closes #43804
2019-07-02 18:51:15 +02:00
Yannick Welsch cc7c5ab2c0 Clarify voting-only master node docs (#43857)
Clarifies the roles of a dedicated voting-only master-eligible node.

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
Co-Authored-By: David Turner <david.turner@elastic.co>
2019-07-02 18:49:40 +02:00
Benjamin Trent 82c1ddc117
[7.x] [ML][Data Frame] Add deduced mappings to _preview response payload (#43742) (#43849)
* [ML][Data Frame] Add deduced mappings to _preview response payload (#43742)

* [ML][Data Frame] Add deduced mappings to _preview response payload

* updating preview docs

* fixing code for backport
2019-07-02 06:52:14 -05:00
Alexander Reelsen ac7e1476a0 Update docs to refer to 6.8 instead of 6.7 (#43685)
A few places in the documentation had mentioned 6.7 as the version to
upgrade from, when doing an upgrade to 7.0. While this is technically
possible, this commit will replace all those mentions to 6.8, as this is
the latest version with the latest bugfixes, deprecation checks and
ugprade assistant features - which should be the one used for upgrades.

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2019-07-02 09:35:04 +02:00
Jack Conradson 81f60652d8 Add timezone documentation for Painless datetimes (#43768) 2019-07-01 21:31:13 -07:00
Christoph Büscher fe3f9f0c6b Yet another `the the` cleanup (#43815) 2019-07-01 20:22:19 +02:00
Zachary Tong ea1794832f Add RareTerms aggregation (#35718)
This adds a `rare_terms` aggregation.  It is an aggregation designed
to identify the long-tail of keywords, e.g. terms that are "rare" or
have low doc counts.

This aggregation is designed to be more memory efficient than the
alternative, which is setting a terms aggregation to size: LONG_MAX
(or worse, ordering a terms agg by count ascending, which has
unbounded error).

This aggregation works by maintaining a map of terms that have
been seen. A counter associated with each value is incremented
when we see the term again.  If the counter surpasses a predefined
threshold, the term is removed from the map and inserted into a cuckoo
filter.  If a future term is found in the cuckoo filter we assume it
was previously removed from the map and is "common".

The map keys are the "rare" terms after collection is done.
2019-07-01 10:30:02 -04:00
Tanguy Leroux 7554420581 Update docs for Open/Close API (#43809)
Relates #43530
2019-07-01 15:20:36 +02:00
Dimitrios Liappis 9aa6f7c434
Update TLS configuration in Docker docs (#43816)
Following the removal of the `unzip` package from the Elasticsearch 
Docker image in #39040, update setup instructions for TLS in Docker.

Also avoid cross-platform ownership+permission issues by not relying
on local bind mounts for storing generated certs and don't require 
`curl` locally installed.

Backport of #43748
2019-07-01 15:33:34 +03:00
David Turner 40d43e3f87 Avoid IP addresses for bootstrapping in setup docs (#43802)
Removes the suggestion to use IP addresses for `cluster.initial_master_nodes`
in the "important settings" discovery docs, leaving only the suggestion to use
node names.

Relates #41179, #41569
2019-07-01 12:39:54 +01:00
Julie Tibshirani ffa5919d7c
Add support for 'flattened object' fields. (#43762)
This commit merges the `object-fields` feature branch. The new 'flattened
object' field type allows an entire JSON object to be indexed into a field, and
provides limited search functionality over the field's contents.
2019-07-01 12:08:50 +03:00
weizijun 28358fdbed
show a full ingest example in the index page, to let user fast understand ingest node. (#43476) 2019-07-01 08:04:57 +02:00
James Rodewig d8fe0f5c13 [DOCS] Rewrite `terms_set` query (#43060) 2019-06-28 12:57:22 -04:00
Alan Woodward 81dbcfb268 Wildcard intervals (#43691)
This commit adds a wildcard intervals source, similar to the prefix. It
also changes the term parameter in prefix to read prefix, to bring it
in to line with the pattern parameter in wildcard.

Closes #43198
2019-06-28 14:04:03 +01:00
James Rodewig 74dd6e49fc [DOCS] Rewrite boosting query (#43647) 2019-06-28 08:35:55 -04:00
Henning Andersen 632da7f2c8 Enabled cannot be updated (#43701)
Removed the invalid tip that enabled can be updated for existing fields
and clarified instead that it cannot.

Related to #33566 and #33933
2019-06-28 12:59:00 +02:00
Christoph Büscher 2cc7f5a744
Allow reloading of search time analyzers (#43313)
Currently changing resources (like dictionaries, synonym files etc...) of search
time analyzers is only possible by closing an index, changing the underlying
resource (e.g. synonym files) and then re-opening the index for the change to
take effect.

This PR adds a new API endpoint that allows triggering reloading of certain
analysis resources (currently token filters) that will then pick up changes in
underlying file resources. To achieve this we introduce a new type of custom
analyzer (ReloadableCustomAnalyzer) that uses a ReuseStrategy that allows
swapping out analysis components. Custom analyzers that contain filters that are
markes as "updateable" will automatically choose this implementation. This PR
also adds this capability to `synonym` token filters for use in search time
analyzers.

Relates to #29051
2019-06-28 09:55:40 +02:00
Lisa Cawley 1b7bcdc3a0 [DOCS] Adds data frame API response codes for allow_no_match (#43666) 2019-06-27 15:17:58 -07:00
Lisa Cawley 42cb59f7b4 [DOCS] Updates ML APIs to use new API template (#43711) 2019-06-27 15:05:51 -07:00
lcawl d46e2bb26a [DOCS] Adds anchors and attributes to ML APIs 2019-06-27 09:44:56 -07:00
Alan Woodward 05a7333eca Require [articles] setting in elision filter (#43083)
We should throw an exception at construction time if a list of
articles is not provided, otherwise we can get random NPEs during
indexing.

Relates to #43002
2019-06-27 09:02:36 +01:00
Julie Tibshirani bed7e68014 Make the ignore_above docs tests more robust. (#43349)
It is possible for internal ML indices like `.data-frame-notifications-1` to leak,
causing other docs tests to fail when they accidentally search over these
indices. This PR updates the ignore_above tests to only search a specific index.
2019-06-27 10:50:55 +03:00
Lisa Cawley ad84059db6 [DOCS] Updates data frame APIs to use API template (#43610) 2019-06-26 13:49:37 -07:00
James Rodewig 87566c9324 [DOCS] Change 'X-Pack APIs' section to 'REST APIs' (#43451) 2019-06-26 13:46:12 -04:00
Alan Woodward 76d0edd1a4 Add prefix intervals source (#43635)
This commit adds a prefix intervals source, allowing you to search
for intervals that contain terms starting with a given prefix. The source
can make use of the index_prefixes mapping option.

Relates to #43198
2019-06-26 16:22:12 +01:00
Benjamin Trent c121b00c98
[7.x] [ML][Data Frame] Add support for allow_no_match for endpoints (#43490) (#43637)
* [ML][Data Frame] Add support for allow_no_match for endpoints (#43490)

* [ML][Data Frame] Add support for allow_no_match parameter in endpoints

Adds support for:
* Get Transforms
* Get Transforms stats
* stop transforms

* Update DataFrameTransformDocumentationIT.java
2019-06-26 10:09:56 -05:00
Stuart Tettemer 500205e8c5
Add painless method getByPath, get value from nested collections with dotted path (#43170) (#43606)
Given a nested structure composed of Lists and Maps, getByPath will return the value
keyed by path.  getByPath is a method on Lists and Maps.

The path is string Map keys and integer List indices separated by dot. An optional third
argument returns a default value if the path lookup fails due to a missing value.

Eg.
['key0': ['a', 'b'], 'key1': ['c', 'd']].getByPath('key1') = ['c', 'd']
['key0': ['a', 'b'], 'key1': ['c', 'd']].getByPath('key1.0') = 'c'
['key0': ['a', 'b'], 'key1': ['c', 'd']].getByPath('key2', 'x') = 'x'
[['key0': 'value0'], ['key1': 'value1']].getByPath('1.key1') = 'value1'

Throws IllegalArgumentException if an item cannot be found and a default is not given.
Throws NumberFormatException if a path element operating on a List is not an integer.

Fixes #42769
2019-06-26 09:06:34 -06:00
Jake Landis 51161a4b0e
add 7.2.0 release notes 2019-06-26 08:50:11 -05:00
Armin Braun 83067968ca
Add SAS Token Authentication Support to Azure Repo Plugin (#42982) (#43618)
* Added setting for SAS token
* Added support for the token in tests
* Relates #42117
2019-06-26 13:43:32 +02:00
David Roberts 558e323c89 [ML] Introduce a setting for the process connect timeout (#43234)
This change introduces a new setting,
xpack.ml.process_connect_timeout, to enable
the timeout for one of the external ML processes
to connect to the ES JVM to be increased.

The timeout may need to be increased if many
processes are being started simultaneously on
the same machine. This is unlikely in clusters
with many ML nodes, as we balance the processes
across the ML nodes, but can happen in clusters
with a single ML node and a high value for
xpack.ml.node_concurrent_job_allocations.
2019-06-26 09:22:04 +01:00
Yannick Welsch 2049f715b3 Add voting-only master node (#43410)
A voting-only master-eligible node is a node that can participate in master elections but will not act
as a master in the cluster. In particular, a voting-only node can help elect another master-eligible
node as master, and can serve as a tiebreaker in elections. High availability (HA) clusters require at
least three master-eligible nodes, so that if one of the three nodes is down, then the remaining two
can still elect a master amongst them-selves. This only requires one of the two remaining nodes to
have the capability to act as master, but both need to have voting powers. This means that one of
the three master-eligible nodes can be made as voting-only. If this voting-only node is a dedicated
master, a less powerful machine or a smaller heap-size can be chosen for this node. Alternatively, a
voting-only non-dedicated master node can play the role of the third master-eligible node, which
allows running an HA cluster with only two dedicated master nodes.

Closes #14340

Co-authored-by: David Turner <david.turner@elastic.co>
2019-06-26 08:07:56 +02:00
James Rodewig 50eac875e4 [DOCS] Rewrite `range` query (#43282) 2019-06-25 15:25:48 -04:00
Dimitris Athanasiou 126c2fd2d5
[7.x][ML] Machine learning data frame analytics (#43544) (#43592)
This merges the initial work that adds a framework for performing
machine learning analytics on data frames. The feature is currently experimental
and requires a platinum license. Note that the original commits can be
found in the `feature-ml-data-frame-analytics` branch.

A new set of APIs is added which allows the creation of data frame analytics
jobs. Configuration allows specifying different types of analysis to be performed
on a data frame. At first there is support for outlier detection.

The APIs are:

- PUT _ml/data_frame/analysis/{id}
- GET _ml/data_frame/analysis/{id}
- GET _ml/data_frame/analysis/{id}/_stats
- POST _ml/data_frame/analysis/{id}/_start
- POST _ml/data_frame/analysis/{id}/_stop
- DELETE _ml/data_frame/analysis/{id}

When a data frame analytics job is started a persistent task is created and started.
The main steps of the task are:

1. reindex the source index into the dest index
2. analyze the data through the data_frame_analyzer c++ process
3. merge the results of the process back into the destination index

In addition, an evaluation API is added which packages commonly used metrics
that provide evaluation of various analysis:

- POST _ml/data_frame/_evaluate
2019-06-25 20:29:11 +03:00
James Rodewig b598701198 [DOCS] Add redirect for painless examples anchor 2019-06-25 12:34:18 -04:00
rbayet 66693c2706
Fixing backquote in fail_on_unsupported_field (#43572) 2019-06-25 16:34:38 +02:00
Ernesto Reig c594a956e2
Default number of shards is now 1 instead of 5 (#43573)
As specified in the [Breaking changes for 7.X](https://www.elastic.co/guide/en/elasticsearch/reference/7.1/breaking-changes-7.0.html#breaking_70_indices_changes), the default number of shards for an index is now `1` instead of `5`.
2019-06-25 14:51:07 +02:00
debadair df42fac9ac [DOCS] Edited title/subtitle. (#43552) 2019-06-24 15:31:19 -07:00
Lisa Cawley 8ffd9c6981 [DOCS] Adds administering section (#43493) 2019-06-24 10:15:25 -07:00
David Roberts 6728e63619 [DOCS] Rename "job" to "transform" in data frame transform docs (#43534) 2019-06-24 09:11:24 -07:00
Tanguy Leroux 9794409ca0 Fix broken link 2019-06-24 16:19:57 +02:00
Tanguy Leroux a4dfa7c29b Add release highlight for replicated closed indices on 7.2.0 (#43530) 2019-06-24 15:54:36 +02:00
Matthew Adams 0bcadbf846 Clarify storage location of ML Snapshots (#43437)
The existing language was misleading about the model snapshots and where they are located. Saying "to disk" sounds like files external to Elasticsearch IMO. It raises the obvious question, where on disk? which node? Is it in the Elasticsearch snapshot repo? The model snapshots are held in an internal index.
2019-06-24 09:14:12 +01:00
Igor Motov 6162471d2e Docs: Add description of the coerce parameter in geo_shape mapper (#43340)
Explains the effect of the coerce parameter on the geo_shape field.

Relates #35059
2019-06-21 12:30:20 -04:00
James Rodewig 014fd19abd [DOCS] Rewrite `constant_score` query (#43374) 2019-06-21 12:04:00 -04:00
James Rodewig 359b103f87 [DOCS] Rewrite term-level queries overview (#43337) 2019-06-21 11:55:02 -04:00
Luiz Guilherme Pais dos Santos eeb1812510 Example of how to set slow logs dynamically per-index (#42384)
* Example of how to set slow logs dynamically per-index

* Make _settings API example more explicit

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

* Add TEST directive to fix CI

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2019-06-21 09:30:53 -04:00