Commit Graph

7922 Commits

Author SHA1 Message Date
James Rodewig 261566154b [DOCS] Fix search request body link (#50498) 2019-12-26 12:34:09 -05:00
James Rodewig ef467cc6f5 [DOCS] Remove unneeded redirects (#50476)
The docs/reference/redirects.asciidoc file stores a list of relocated or
deleted pages for the Elasticsearch Reference documentation.

This prunes several older redirects that are no longer needed and
don't require work to fix broken links in other repositories.
2019-12-26 08:29:28 -05:00
James Rodewig 1cec87c0e6 [DOCS] Document `transport` and `http` node stats (#50473)
Documents the `transport` and `http` parameters returned by the
`_nodes/stats` API.
2019-12-26 07:43:14 -05:00
Lisa Cawley d479e0563a
[7.x][DOCS] Augments ML shared definitions (#50487) 2019-12-24 10:22:05 -08:00
Martijn van Groningen 10ed1ae1d2
Add remote info to the HLRC (#50483)
The additional change to the original PR (#49657), is that `org.elasticsearch.client.cluster.RemoteConnectionInfo` now parses the initial_connect_timeout field as a string instead of a TimeValue instance.

The reason that this is needed is because that the initial_connect_timeout field in the remote connection api is serialized for human consumption, but not for parsing purposes.
Therefore the HLRC can't parse it correctly (which caused test failures in CI, but not in the PR CI
:( ). The way this field is serialized needs to be changed in the remote connection api, but that is a breaking change. We should wait making this change until rest api versioning is introduced.

Co-Authored-By: j-bean <anton.shuvaev91@gmail.com>

Co-authored-by: j-bean <anton.shuvaev91@gmail.com>
2019-12-24 15:11:58 +01:00
Orhan Toy 6a3d1a077e [DOCS] Fixes "enables you to" typos (#50225) 2019-12-23 14:39:14 -05:00
James Rodewig 694b119f0a [DOCS] Percentile aggs are non-deterministic (#50468)
Percentile aggregations are non-deterministic. A percentile aggregation
can produce different results even when using the same data.

Based on [this discuss post][0], the non-deterministic property stems
from processes in Lucene that can affect the order in which docs are
provided to the aggregation.

This adds a warning stating that the aggregation is non-deterministic
and what that means.

[0]: https://discuss.elastic.co/t/different-results-for-same-query/111757
2019-12-23 13:13:34 -05:00
Nik Everett 01293ebad5
Fix docs typos (#50365) (#50464)
Fixes a few typos in the docs.

Co-authored-by: Xiang Dai <764524258@qq.com>
2019-12-23 12:38:17 -05:00
Igor Motov 339d10c16f Geo: Switch generated GeoJson type names to camel case (#50400)
Switches generated GeoJson type names to camel case
to conform to the standard.

Closes #49568
2019-12-20 15:37:22 -05:00
James Rodewig 27ae9a1435 [DOCS] Remove outdated file scripts refererence (#50437)
File scripts were removed in 6.0 with #24627.

This removes an outdated file scripts reference from the conditional clauses section of the search templates docs.
2019-12-20 14:53:40 -05:00
Lee Hinman c3c9ccf61f
[7.x] Add ILM histore store index (#50287) (#50345)
* Add ILM histore store index (#50287)

* Add ILM histore store index

This commit adds an ILM history store that tracks the lifecycle
execution state as an index progresses through its ILM policy. ILM
history documents store output similar to what the ILM explain API
returns.

An example document with ALL fields (not all documents will have all
fields) would look like:

```json
{
  "@timestamp": 1203012389,
  "policy": "my-ilm-policy",
  "index": "index-2019.1.1-000023",
  "index_age":123120,
  "success": true,
  "state": {
    "phase": "warm",
    "action": "allocate",
    "step": "ERROR",
    "failed_step": "update-settings",
    "is_auto-retryable_error": true,
    "creation_date": 12389012039,
    "phase_time": 12908389120,
    "action_time": 1283901209,
    "step_time": 123904107140,
    "phase_definition": "{\"policy\":\"ilm-history-ilm-policy\",\"phase_definition\":{\"min_age\":\"0ms\",\"actions\":{\"rollover\":{\"max_size\":\"50gb\",\"max_age\":\"30d\"}}},\"version\":1,\"modified_date_in_millis\":1576517253463}",
    "step_info": "{... etc step info here as json ...}"
  },
  "error_details": "java.lang.RuntimeException: etc\n\tcaused by:etc etc etc full stacktrace"
}
```

These documents go into the `ilm-history-1-00000N` index to provide an
audit trail of the operations ILM has performed.

This history storage is enabled by default but can be disabled by setting
`index.lifecycle.history_index_enabled` to `false.`

Resolves #49180

* Make ILMHistoryStore.putAsync truly async (#50403)

This moves the `putAsync` method in `ILMHistoryStore` never to block.
Previously due to the way that the `BulkProcessor` works, it was possible
for `BulkProcessor#add` to block executing a bulk request. This was bad
as we may be adding things to the history store in cluster state update
threads.

This also moves the index creation to be done prior to the bulk request
execution, rather than being checked every time an operation was added
to the queue. This lessens the chance of the index being created, then
deleted (by some external force), and then recreated via a bulk indexing
request.

Resolves #50353
2019-12-20 12:33:36 -07:00
Jack Conradson b81e072504 Document use of context in put stored script (#50446)
This documents how to test compile a stored script against a specific 
context when using PUT/POST.
2019-12-20 10:53:43 -08:00
Lisa Cawley 2106a7b02a
[7.x][DOCS] Updates ML links (#50387) (#50409) 2019-12-20 10:01:19 -08:00
Florian Kelbert af8bed13d3 [DOCS] Fix typo in bucket sum aggregation docs (#50431) 2019-12-20 08:48:25 -05:00
Jim Ferenczi 2acafd4b15
Optimize composite aggregation based on index sorting (#48399) (#50272)
Co-authored-by: Daniel Huang <danielhuang@tencent.com>

This is a spinoff of #48130 that generalizes the proposal to allow early termination with the composite aggregation when leading sources match a prefix or the entire index sort specification.
In such case the composite aggregation can use the index sort natural order to early terminate the collection when it reaches a composite key that is greater than the bottom of the queue.
The optimization is also applicable when a query other than match_all is provided. However the optimization is deactivated for sources that match the index sort in the following cases:
  * Multi-valued source, in such case early termination is not possible.
  * missing_bucket is set to true
2019-12-20 12:32:37 +01:00
Tim Brooks cb73fb0f9b
Backport remote proxy mode stats and naming (#50402)
* Update remote cluster stats to support simple mode (#49961)

Remote cluster stats API currently only returns useful information if
the strategy in use is the SNIFF mode. This PR modifies the API to
provide relevant information if the user is in the SIMPLE mode. This
information is the configured addresses, max socket connections, and
open socket connections.

* Send hostname in SNI header in simple remote mode (#50247)

Currently an intermediate proxy must route conncctions to the
appropriate remote cluster when using simple mode. This commit offers
a additional mechanism for the proxy to route the connections by
including the hostname in the TLS SNI header.

* Rename the remote connection mode simple to proxy (#50291)

This commit renames the simple connection mode to the proxy connection
mode for remote cluster connections. In order to do this, the mode specific
settings which we namespaced by their mode (ex: sniff.seed and
proxy.addresses) have been reverted.

* Modify proxy mode to support a single address (#50391)

Currently, the remote proxy connection mode uses a list setting for the
proxy address. This commit modifies this so that the setting is
proxy_address and only supports a single remote proxy address.
2019-12-19 18:02:48 -07:00
Stuart Tettemer 2e76865290
[DOCS] Deterministic scripted queries are cached (#50408) (#50411)
**Backport**

Refs: #49321
2019-12-19 16:30:34 -07:00
Przemysław Witek cc4bc797f9
[7.x] Implement `precision` and `recall` metrics for classification evaluation (#49671) (#50378) 2019-12-19 18:55:05 +01:00
István Zoltán Szabó 501ab83471 [DOCS] Adds inference processor documentation (#50204)
Co-Authored-By: Lisa Cawley <lcawley@elastic.co>
2019-12-19 12:21:04 +01:00
Igor Motov c77ca98928 Geo: Switch generated WKT to upper case (#50285)
Switches generated WKT to upper case to
conform to the standard recommendation.

Relates #49568
2019-12-18 17:29:08 -05:00
James Rodewig 73ca2e5826 [DOCS] Document `thread_pool` node stats (#50330) 2019-12-18 17:02:00 -05:00
lcawl 246d926412 [DOCS] Fixes security links 2019-12-18 11:52:00 -08:00
James Rodewig 498532a12d [DOCS] Clarify frozen indices are read-only (#50318)
The freeze index API docs state that frozen indices are blocked for
write operations.

While this implies frozen indices are read-only, it does not explicitly
use the term "read-only", which is found in other docs, such as the
force merge docs.

This adds the "ready-only" term to the freeze index API docs as well
as other clarification.
2019-12-18 12:18:38 -05:00
Christoph Büscher bf63f24209 [Docs] Remove `intervals` filter rule from allowed top-level rules (#50320)
The `filter` rule is not allowed on the top-level of the query, so removing it
from the list of allowed rules. Where it can be nested inside other rules, those
rules already mention it.
2019-12-18 17:37:27 +01:00
Adrien Grand ec11c04bd5 [DOCS] Add 7.5.1 release notes (#50196)
Co-Authored-By: Lisa Cawley <lcawley@elastic.co>
2019-12-18 11:09:53 -05:00
Kevin Woblick cbffd127d5 [DOCS] Add warning about Docker port exposure (#50169)
Docker bypasses the Uncomplicated Firewall (UFW) on Linux by editing the `iptables` config directly, which leads to the exposure of port 9200, even if you blocked it via UFW.

This adds a warning along with work-arounds to the docs.

Signed-off-by: Kovah <mail@kovah.de>
2019-12-18 09:16:36 -05:00
István Zoltán Szabó 5759a263cb [DOCS] Adds GET, GET stats and DELETE inference APIs (#50224)
Co-Authored-By: Lisa Cawley <lcawley@elastic.co>
2019-12-18 09:18:56 +01:00
Lisa Cawley 30d66828ae [DOCS] Move transform resource definitions into APIs (#50108) 2019-12-17 12:31:31 -08:00
James Rodewig 726c35dfd0 [DOCS] Add identifier mapping tip to numeric and keyword datatype docs (#49933)
Users often mistakenly map numeric IDs to numeric datatypes. However,
this is often slow for the `term` and other term-level queries.

The "Tune for search speed" docs includes advice for mapping numeric
IDs to `keyword` fields. However, this tip is not included in the
`numeric` or `keyword` field datatype doc pages.

This rewords the tip in the "Tune for search speed" docs, relocates it
to the `numeric` field docs, and reuses it using tagged regions.
2019-12-17 09:34:32 -05:00
James Rodewig 99fdea50dd
[DOCS] Correct cat snapshots API request example (#50274)
For 7.x and earlier branches, `_cat/repositories` API requests require a
repository name.

This removes an erroneous request example without a repository name
added with a8e0275.
2019-12-17 09:23:38 -05:00
Ignacio Vera 3717c733ff
"CONTAINS" support for BKD-backed geo_shape and shape fields (#50141) (#50213)
Lucene 8.4 added support for "CONTAINS", therefore in this commit those
changes are integrated in Elasticsearch. This commit contains as well a
bug fix when querying with a geometry collection with "DISJOINT" relation.
2019-12-16 09:17:51 +01:00
Henning Andersen a0044df78a
Reindex source types disregarded in 7.x (#49580)
Clarify that types in source index are disregarded.
2019-12-15 17:30:28 +01:00
James Rodewig bbb872a022 Revert "[DOCS] Add `index-extra-title-page.html` for direct HTML migration (#50189)"
This reverts commit 79c40728f7.
2019-12-13 12:58:11 -05:00
James Rodewig 79c40728f7 [DOCS] Add `index-extra-title-page.html` for direct HTML migration (#50189) 2019-12-13 12:47:13 -05:00
James Rodewig cd04021961 [DOCS] Reformat token count limit filter docs (#49835) 2019-12-13 08:44:39 -05:00
István Zoltán Szabó 8f36bfa37f
[7.x][DOCS] Changes hyperparam optimization section ID (#50173) 2019-12-13 12:22:50 +01:00
István Zoltán Szabó 7611b3c9be
[7.x][DOCS] Moves data frame analytics job resource definitions into APIs (#50165)
* [7.x][DOCS] Moves data frame analytics job resource definitions into APIs.
2019-12-13 11:48:21 +01:00
James Rodewig 249334f38d [DOCS] Document JVM node stats (#49500)
* [DOCS] Document JVM node stats

Documents the `jvm` parameters returned by the `_nodes/stats` API.

Co-Authored-By: James Baiera <james.baiera@gmail.com>
2019-12-12 15:42:18 -05:00
Lisa Cawley 2d19d0c166 [DOCS] Updates transform screenshots and text (#50059) 2019-12-12 09:30:56 -08:00
James Rodewig 1186a5dc09 [DOCS] Reformat lowercase token filter docs (#49935) 2019-12-12 09:50:12 -05:00
James Rodewig 364eb2d34c [DOCS] Correct percentile rank agg example response (#50052)
The example snippets in the percentile rank agg docs use a test dataset
named `latency`, which is generated from docs/gradle.build.

At some point the dataset and example snippets were updated, but the
text surrounding the snippets was not. This means the text and the
example snippets shown no longer match up.

This corrects that by changing the snippets using /TESTRESPONSE magic comments.
2019-12-12 09:06:41 -05:00
Przemko Robakowski 4619834b97
[7.x] CSV ingest processor (#49509) (#50083)
* CSV ingest processor (#49509)

This change adds new ingest processor that breaks line from CSV file into separate fields.
By default it conforms to RFC 4180 but can be tweaked.

Closes #49113
2019-12-11 23:06:05 +01:00
Wilder Pereira 8ff809af2d [DOCS] Replace interval notation with plain English in match query docs (#47334)
As we discussed in #36371, interval notation is confusing to some users. This makes the intention clearer by just explaining inclusivity and exclusivity in the docs.
2019-12-11 09:58:28 -05:00
Patryk Krawaczyński df558aa0ca [DOCS] Document `index.queries.cache.enabled` as a static setting (#49886) 2019-12-10 14:24:03 -05:00
Adrien Grand 87e72156ce
Upgrade to lucene 8.4.0-snapshot-662c455. (#50016) (#50039)
Lucene 8.4 is about to be released so we should check it doesn't cause problems
with Elasticsearch.
2019-12-10 18:04:58 +01:00
Peter Johnson 1a6e5bf220 [Docs] Fix typo in function-score-query.asciidoc (#50030) 2019-12-10 17:33:03 +01:00
Lisa Cawley 15f27d8c54 [DOCS] Removes realm type security setting (#50001) 2019-12-10 08:09:07 -08:00
James Rodewig 3f5678ca79 [DOCS] Remove shadow replica reference (#50029)
Removes a reference to shadow replicas from the cat shards API docs
and a comment in cluster/routing/UnassignedInfo.java.

Shadow replicas were removed with #23906.
2019-12-10 09:30:51 -05:00
Dimitris Athanasiou 8891f4db88
[7.x][ML] Introduce randomize_seed setting for regression and classification (#49990) (#50023)
This adds a new `randomize_seed` for regression and classification.
When not explicitly set, the seed is randomly generated. One can
reuse the seed in a similar job in order to ensure the same docs
are picked for training.

Backport of #49990
2019-12-10 15:29:19 +02:00
James Rodewig 33594380c7 [DOCS] Skip synced flush docs tests (#49986)
The current snippets in the synced flush docs can cause conflicts with
other background syncs, such as the global checkpoint sync or retention
lease sync, in the docs tests.

This skips tests for those snippets to avoid conflicts.
2019-12-09 13:17:38 -05:00
James Rodewig 322dabe3de [DOCS] Correct `for in` example in Painless docs (#49991)
Adds a needed `def` keyword to the `for in` example in the Painless docs.
2019-12-09 11:05:12 -05:00
Artur Carvalho d073bccaad [Docs] Fix typo in getting-started.asciidoc (#49985) 2019-12-09 16:24:30 +01:00
James Rodewig 1918a21baf [DOCS] Correct inline shape snippets in shape query docs (#49921)
In the shape query docs, the index mapping snippet uses the "geometry"
shape field mapping. However, the doc index snippet uses the "location"
property.

This changes the "location" property to "geometry". It also adds a
comment containing the search result snippet. This should prevent
similar issues in the future.
2019-12-09 08:47:59 -05:00
Ryan Ernst e66cfc4369 Fix incorrect use of multiline NOTE in rpm docs (#49962)
This was a copy/paste error from #49893. This commit converts the NOTE
to use inline style instead of one needing closing linebreak.
2019-12-06 17:43:51 -08:00
Ryan Ernst d29f04209b Disable repo configuration for rpm based systems (#49893)
This commit changes the recommended repository file for rpm based
systems to be disabled by default. This is a safer practice so upgrades
of the system do no accidentally upgrade elasticsearch itself.

closes #30660
2019-12-06 15:56:18 -08:00
Przemko Robakowski d7083a84f4
Allow list of IPs in geoip ingest processor (#49573) (#49947)
* Allow list of IPs in geoip ingest processor

This change lets you use array of IPs in addition to string in geoip processor source field.
It will set array containing geoip data for each element in source, unless first_only parameter
option is enabled, then only first found will be returned.

Closes #46193
2019-12-07 00:19:09 +01:00
István Zoltán Szabó 63d3933787 [DOCS] Fixes classification evaluation example response. (#49905) 2019-12-06 13:25:40 +01:00
István Zoltán Szabó bb91291273 [DOCS] Fixes attribute in transforms overview. (#49898) 2019-12-06 10:24:29 +01:00
Hendrik Muhs b17cfc93e3 [Transform][DOCS]rewrite client ip example to use continuous transform (#49822)
adapt the transform example for suspicious client ips to use continuous transform
2019-12-06 08:20:48 +01:00
Orhan Toy 1641fcd488 [DOCS] Minor typo fixes in reindex.asciidoc (#49863) 2019-12-05 20:25:11 +01:00
István Zoltán Szabó f4b3bb7d6b [DOCS] Adds an example of preprocessing actions to the PUT DFA API docs (#49831) 2019-12-05 14:16:38 +01:00
István Zoltán Szabó 04e99ff1ee [DOCS] Fixes typo in the ML anomaly detection time functions docs. (#49834) 2019-12-05 09:58:30 +01:00
James Rodewig 42f902977d [DOCS] Document `minimum_should_match` defaults for `bool` query (#48865)
Adds documentation for the `minimum_should_match` parameter to the `bool` query docs. Includes docs for the default values:

- `1` if the `bool` query includes at least one `should` clause and no `must` or `filter` clauses
- `0` otherwise
2019-12-04 12:45:38 -05:00
James Rodewig 87a73b6bdf [DOCS] Reformat length token filter docs (#49805)
* Adds a title abbreviation
* Updates the description and adds a Lucene link
* Reformats the parameters section
* Adds analyze, custom analyzer, and custom filter snippets

Relates to #44726.
2019-12-04 09:59:08 -05:00
Alexander Reelsen 6e751f5536 Docs: Fix & test more grok processor documentation (#49447)
The documentation contained a small error, as bytes and duration was not
properly converted to a number and thus remained a string.

The documentation is now also properly tested by providing a full blown
simulate pipeline example.
2019-12-03 11:55:49 +01:00
Colin Goodheart-Smithe 0592b3c726
Removes PR that was not in 7.5.0 release 2019-12-03 10:20:05 +00:00
cachedout c4cc90be1c
Recommend Metricbeat for 7.x (#49758)
* Recommend Metricbeat for 7.x

* Fix typo in link to configuring-metricbeat

* [DOCS] Fixes build error and some terminology

* Add to local exporter page per review feedback
2019-12-02 21:31:47 +00:00
James Rodewig f1fd41cb53 [DOCS] Document CCR compatibility requirements (#49776)
* Creates a prerequisites section in the cross-cluster replication (CCR)
  overview.
* Adds concise definitions for local and remote cluster in a CCR context.
* Documents that the ES version of the local cluster must be the same
  or a newer compatible version as the remote cluster.
2019-12-02 15:53:00 -05:00
Paul Sanwald ebc13ca498
re-categorize things that appeared in multiple area labels (#49777) 2019-12-02 15:03:38 -05:00
Hendrik Muhs a5dc6e062e Document issue 49730 in release notes for 7.5.0 (#49733)
document low severity issue about transform audit index potentially disappearing during rolling upgrade

See #49730 for details
2019-12-02 20:54:36 +01:00
lcawl 96f14fcfbd [DOCS] Removes coming tags 2019-12-02 08:17:10 -08:00
Jim Ferenczi e6dc5bf9c2 Add release highlights for 7.5.0 (#49320) 2019-12-02 07:45:12 -08:00
Andrei Stefan e2982b2110 SQL: handle NULL arithmetic operations with INTERVALs (#49633)
(cherry picked from commit ce727615c08cf5ae422feb77f69ea24fb53cd9d1)
2019-12-02 17:31:05 +02:00
James Rodewig ade72b97b7 [DOCS] Reformat keep types and keep words token filter docs (#49604)
* Adds title abbreviations
* Updates the descriptions and adds Lucene links
* Reformats parameter definitions
* Adds analyze and custom analyzer snippets
* Adds explanations of token types to keep types token filter and tokenizer docs
2019-12-02 09:40:50 -05:00
David Turner 86a40f6d8b Drop snapshot instructions for autobootstrap fix (#49755)
The "Restore any snapshots as required" step is a trap: it's somewhere between
tricky and impossible to restore multiple clusters into a single one.

Also add a note about configuring discovery during a rolling upgrade to
proscribe any rare cases where you might accidentally autobootstrap during the
upgrade.
2019-12-02 14:33:42 +00:00
James Rodewig 3d44c1163a [DOCS] Explicitly document enrich `target_field` includes `match_field` (#49407)
When the enrich processor appends enrich data to an incoming document,
it adds a `target_field` to contain the enrich data.

This `target_field` contains both the `match_field` AND `enrich_fields`
specified in the enrich policy.

Previously, this was reflected in the documented example but not
explicitly stated. This adds several explicit statements to the docs.
2019-12-02 09:13:24 -05:00
Henning Andersen 5adb33ec17
Deprecate sorting in reindex (#49458) (#49738)
Reindex sort never gave a guarantee about the order of documents being
indexed into the destination, though it could give a sense of locality
of source data.

It prevents us from doing resilient reindex and other optimizations and
it has therefore been deprecated.

Related to #47567
2019-12-01 19:24:27 +01:00
Henning Andersen 1d745f1e5c Revert "Deprecate sorting in reindex (#49458)"
This reverts commit 27d45c9f1f.
2019-11-29 22:08:19 +01:00
Mayya Sharipova 7cf170830c
Optimize sort on numeric long and date fields. (#49732)
This rewrites long sort as a `DistanceFeatureQuery`, which can
efficiently skip non-competitive blocks and segments of documents.
Depending on the dataset, the speedups can be 2 - 10 times.

The optimization can be disabled with setting the system property
`es.search.rewrite_sort` to `false`.

Optimization is skipped when an index has 50% or more data with
the same value.

Optimization is done through:
1. Rewriting sort as `DistanceFeatureQuery` which can
efficiently skip non-competitive blocks and segments of documents.

2. Sorting segments according to the primary numeric sort field(#44021)
This allows to skip non-competitive segments.

3. Using collector manager.
When we optimize sort, we sort segments by their min/max value.
As a collector expects to have segments in order,
we can not use a single collector for sorted segments.
We use collectorManager, where for every segment a dedicated collector
will be created.

4. Using Lucene's shared TopFieldCollector manager
This collector manager is able to exchange minimum competitive
score between collectors, which allows us to efficiently skip
the whole segments that don't contain competitive scores.

5. When index is force merged to a single segment, #48533 interleaving
old and new segments allows for this optimization as well,
as blocks with non-competitive docs can be skipped.

Backport for #48804


Co-authored-by: Jim Ferenczi <jim.ferenczi@elastic.co>
2019-11-29 15:37:40 -05:00
Henning Andersen 27d45c9f1f Deprecate sorting in reindex (#49458)
Reindex sort never gave a guarantee about the order of documents being
indexed into the destination, though it could give a sense of locality
of source data.

It prevents us from doing resilient reindex and other optimizations and
it has therefore been deprecated.

Related to #47567
2019-11-29 21:35:11 +01:00
Tugberk Ugurlu dcb9d5177c [Docs] Fix typo in templates.asciidoc (#49726) 2019-11-29 18:43:13 +01:00
Dimitris Athanasiou 4edb2e7bb6
[7.x][ML] Add optional source filtering during data frame reindexing (#49690) (#49718)
This adds a `_source` setting under the `source` setting of a data
frame analytics config. The new `_source` is reusing the structure
of a `FetchSourceContext` like `analyzed_fields` does. Specifying
includes and excludes for source allows selecting which fields
will get reindexed and will be available in the destination index.

Closes #49531

Backport of #49690
2019-11-29 16:10:44 +02:00
Tim Vernum e6f530c167
Improved diagnostics for TLS trust failures (#49669)
- Improves HTTP client hostname verification failure messages
- Adds "DiagnosticTrustManager" which logs certificate information
  when trust cannot be established (hostname failure, CA path failure,
  etc)

These diagnostic messages are designed so that many common TLS
problems can be diagnosed based solely (or primarily) on the
elasticsearch logs.

These diagnostics can be disabled by setting

     xpack.security.ssl.diagnose.trust: false

Backport of: #48911
2019-11-29 15:01:20 +11:00
Tim Vernum 31f13e839c
Correct the documentation for create_doc privilege (#49354)
The documentation was added in #47584 but those docs did not reflect the up-to-date behavior of the feature.

Backport of: #47784
2019-11-29 12:59:16 +11:00
Mayya Sharipova 2dafecc398
Upgrade lucene to 8.4.0-snapshot-e648d601efb (#49641) 2019-11-28 11:59:58 -05:00
Marios Trivyzas d5842aebab
[Docs] Enhance rolling upgrade guide (#49686)
Add a couple of pointers for the user to check the
overall cluster health and the version of ES running
on every node.

Fixes: #49670

(cherry picked from commit 8ca11f54cd839f41632c556601e94da67e91a3d1)
2019-11-28 17:02:36 +01:00
Ignacio Vera 326fe7566e
New Histogram field mapper that supports percentiles aggregations. (#48580) (#49683)
This commit adds  a new histogram field mapper that consists in a pre-aggregated format of numerical data to be used in percentiles aggregations.
2019-11-28 15:06:26 +01:00
Jim Ferenczi d6445fae4b Add a cluster setting to disallow loading fielddata on _id field (#49166)
This change adds a dynamic cluster setting named `indices.id_field_data.enabled`.
When set to `false` any attempt to load the fielddata for the `_id` field will fail
with an exception. The default value in this change is set to `false` in order to prevent
fielddata usage on this field for future versions but it will be set to `true` when backporting
to 7x. When the setting is set to true (manually or by default in 7x) the loading will also issue
a deprecation warning since we want to disallow fielddata entirely when https://github.com/elastic/elasticsearch/issues/26472
is implemented.

Closes #43599
2019-11-28 09:35:28 +01:00
Ryan Ernst f288696040 Remove legacy referene to file scripts (#49339)
This commit removes outdated documentation about a path setting for file
scripts which no longer exist.

closes #45827
2019-11-27 10:42:33 -08:00
Ryan Ernst 297efa8324 Add JAVA_HOME env override location to docs (#49565)
This commit clarifies how to override JAVA_HOME from the bundled jdk for
deb and rpm installs, which each have their own file that is sourced
upon service startup.

closes #49068
2019-11-27 10:40:30 -08:00
Martijn van Groningen 0a42395dfa
Backport: add templating support to pipeline processor (#49643)
Backport of #49030

This commit adds templating support to the pipeline processor's `name` option.

Closes #39955
2019-11-27 15:53:40 +01:00
Xiang Dai 15342c4dd2 [DOCS] Clarify how to update max memory size in bootstrap checks (#48975) 2019-11-27 09:40:00 -05:00
bellengao 2e0b079a16 [DOCS] Correct the request path for flush API docs (#49615) 2019-11-27 09:27:28 -05:00
Yannick Welsch 5ffb615b37 [DOCS] Correct request path for synced flush API docs (#49631)
Fixes an incorrect request path added with #46634
2019-11-27 08:44:51 -05:00
glerb d1ed2ae25b [Docs] Correct typo in log file name (#49620) 2019-11-27 14:37:47 +01:00
Dimitrios Liappis 4b6915ea41
Clarify gid used by docker image process and bind-mount method (#49632)
Fix reference about the uid:gid that Elasticsearch runs as inside
the Docker container and add a packaging test to ensure that bind
mounting a data dir with a random uid and gid:0 works as
expected.

Backport of #49529
Closes #47929
2019-11-27 13:42:54 +02:00
Martijn van Groningen 09c4269097
Add templating support to enrich processor (#49093)
Adds support for templating to `field` and `target_field` options.
2019-11-27 08:53:11 +01:00
Martijn van Groningen 90850f4ea0
Backport: Introduce on_failure_pipeline ingest metadata inside on_failure block (#49596)
Backport of #49076

In case an exception occurs inside a pipeline processor,
the pipeline stack is kept around as header in the exception.
Then in the on_failure processor the id of the pipeline the
exception occurred is made accessible via the `on_failure_pipeline`
ingest metadata.

Closes #44920
2019-11-27 07:52:08 +01:00
lcawl 777431265b [DOCS] Fixes typo in ML resources 2019-11-26 10:28:59 -08:00
Benjamin Trent b5d7c939f8
[7.x] [ML][Inference][HLRC] add GET _stats (#49562) (#49600)
* [ML][Inference][HLRC] add GET _stats (#49562)

* fixing for backport
2019-11-26 11:28:26 -05:00
lcawl a42003b95b [DOCS] Fixes data type formatting 2019-11-26 08:22:50 -08:00
Benjamin Trent 26a8ca00db
[7.x] [ML][Inference][HLRC] Delete trained model API (#49567) (#49585)
* [ML][Inference][HLRC] Delete trained model API (#49567)

* fixing for backport
2019-11-26 08:27:08 -05:00
Marios Trivyzas 3c69d4d0bd
SQL: Add TRUNC alias for TRUNCATE (#49571)
Add TRUNC as alias to already implemented TRUNCATE
numeric function which is the flavour supported by
Oracle and PostgreSQL.

Relates to: #41195

(cherry picked from commit f2aa7f0779bc5cce40cc0c1f5e5cf1a5bb7d84f0)
2019-11-26 12:32:54 +01:00
Christoph Büscher a4208e44f7 [Docs] Correct `max_doc_freq` default value (#49536)
The default is set to Integer.MAX_VALUE but is reported to be `0` in the docs.
With the current implementation a value of 0 would mean all terms are filtered
out, which is the opposite of "unbounded".

Closes #49520
2019-11-26 10:47:05 +01:00
Tim Vernum 9cb1ace1c2
Expand docs on TLSv1 breaking change (#49352)
The breaking changes cover the removal of TLSv1 from the default
protocols, but assume that users who need to retain TLSv1 support will
understand all the places where they may used it.

This has proven not to be true, as it is easy to be unaware that (for
example) an LDAP server is using TLSv1.

This change explicitly lists all the places where TLS protocols may
need to be configured.

Co-Authored-By: Lisa Cawley <lcawley@elastic.co>
Co-Authored-By: Pius <pius@elastic.co>
2019-11-26 16:34:55 +11:00
James Rodewig 2fd58bb845 [DOCS] Add missing "_type" to delimited payload token filter docs 2019-11-25 16:16:05 -05:00
Lisa Cawley 26beb486c7 [DOCS] Fixes security links (#49563) 2019-11-25 13:02:26 -08:00
James Rodewig c40449ac22 [DOCS] Reformat delimited payload token filter docs (#49380)
* Adds a title abbreviation
* Relocates the older name deprecation warning
* Updates the description and adds a Lucene link
* Adds a note to explain payloads and how to store them
* Adds analyze and custom analyzer snippets
* Adds a 'Return stored payloads' example
2019-11-25 15:40:05 -05:00
James Rodewig 99476db2d0 [DOCS] Remove individual task retrieval from cat/tasks API (#49550) 2019-11-25 10:32:39 -05:00
Kelly Campbell df5afa797e [DOCS] Correct GET path in cat tasks API docs (#49494)
Previously, the request example included `GET _cat/_tasks`. However, the resource should be `tasks`, not `_tasks`.
2019-11-25 09:37:59 -05:00
David Roberts 62811c2272 [ML] Add default categorization analyzer definition to ML info (#49545)
The categorization job wizard in the ML UI will use this
information when showing the effect of the chosen categorization
analyzer on a sample of input.
2019-11-25 13:39:16 +00:00
Dimitris Athanasiou d21df9eba9 [ML][DOCS] Anomaly detection job retention days settings do not require restart (#49546) 2019-11-25 14:19:10 +01:00
debadair 2ec047db04 [DOCS] Rename auditing topic. Closes #49012 (#49013)
* [DOCS] Rename auditing topic. Closes #49012

* Fixed file name, fixed settings link.

* Add link to settings
2019-11-22 14:16:58 -08:00
James Rodewig d06c71eb82 [DOCS] Fix edge n-gram tokenizer nav
Adds a missing float tag to the edge n-gram tokenizer docs. This tag
ensures the edge n-gram tokenizer docs display on the same page.
2019-11-22 15:54:07 -05:00
Dimitris Athanasiou 8eaee7cbdc
[7.x][ML] Explain data frame analytics API (#49455) (#49504)
This commit replaces the _estimate_memory_usage API with
a new API, the _explain API.

The API consolidates information that is useful before
creating a data frame analytics job.

It includes:

- memory estimation
- field selection explanation

Memory estimation is moved here from what was previously
calculated in the _estimate_memory_usage API.

Field selection is a new feature that explains to the user
whether each available field was selected to be included or
not in the analysis. In the case it was not included, it also
explains the reason why.

Backport of #49455
2019-11-22 22:06:10 +02:00
Jason Tedor 71bcfbf1e3
Replace required pipeline with final pipeline (#49470)
This commit enhances the required pipeline functionality by changing it
so that default/request pipelines can also be executed, but the required
pipeline is always executed last. This gives users the flexibility to
execute their own indexing pipelines, but also ensure that any required
pipelines are also executed. Since such pipelines are executed last, we
change the name of required pipelines to final pipelines.
2019-11-22 14:37:36 -05:00
Lisa Cawley ca895d3ad5 [DOCS] Merge rollup config details into API (#49412) 2019-11-22 08:39:49 -08:00
James Rodewig 562607d3f5 [DOCS] Reformat n-gram token filter docs (#49438)
Reformats the edge n-gram and n-gram token filter docs. Changes include:

* Adds title abbreviations
* Updates the descriptions and adds Lucene links
* Reformats parameter definitions
* Adds analyze and custom analyzer snippets
* Adds notes explaining differences between the edge n-gram and n-gram
  filters

Additional changes:
* Switches titles to use "n-gram" throughout.
* Fixes a typo in the edge n-gram tokenizer docs
* Adds an explicit anchor for the `index.max_ngram_diff` setting
2019-11-22 10:38:50 -05:00
Benjamin Trent ed787d06e8
[7.x] [ML][Inference][HLRC] GET trained models (#49464) (#49488)
* [ML][Inference][HLRC] GET trained models (#49464)

* fixing for backport
2019-11-22 09:24:06 -05:00
István Zoltán Szabó 56d97dcb6c
[DOCS] Replaces deprecated ScriptService.ScriptType.INLINE with supported script in Java update docs. (#49424) 2019-11-22 14:17:44 +01:00
István Zoltán Szabó 35cc0e0948 [DOCS] Removes the default size definition of thread pool types (#49442)
Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2019-11-22 11:20:11 +01:00
Florian Kelbert d444c334d7 Modify example for pinned query (#49481)
I do not see any reason to advertise phones of specific companies.
2019-11-22 11:03:04 +01:00
István Zoltán Szabó c13fce60a8 [DOCS] Removes data frame leftovers from transforms overview (#49434) 2019-11-22 10:20:15 +01:00
Przemyslaw Gomulka d42eac9cf3
[DOC] Modify the update example to change a document (#49228) (#49443)
Example at the moment is not changing the existing document. Update request should at least modify the existing document.
2019-11-22 09:54:34 +01:00
James Rodewig 0fa3b887b7 [DOCS] Document several missing thread pools (#48543)
Adds documentation for the following thread pools:
    - fetch_shard_started
    - fetch_shard_store
    - flush
    - force_merge
    - management

Closes #48524

Co-Authored-By: Jay Modi <jaymode@users.noreply.github.com>
2019-11-21 13:12:56 -05:00
Hendrik Muhs 779b4bd92b update the name of the audit index (#49432)
small update to the name of the audit index changed in 7.5
2019-11-21 16:15:53 +01:00
James Rodewig f264808a6a [DOCS] Replace cross-cluster search PNG images with SVGs (#49395) 2019-11-21 09:06:33 -05:00
Przemysław Witek c7ac2011eb
[7.x] Implement accuracy metric for multiclass classification (#47772) (#49430) 2019-11-21 15:01:18 +01:00
James Rodewig 03600e4e12 [DOCS] Document `script_score` float precision limit (#49402)
All document scores are positive 32-bit floating point numbers. However, this
wasn't previously documented.

This can result in surprising behavior, such as precision loss, for users when
customizing scores using the function score query.

This commit updates an existing admonition in the function score query docs to
document the 32-bits precision limit. It also updates the search API reference
docs to note that `_score` is a 32-bit float.
2019-11-21 08:54:49 -05:00
weizijun 3eb577f6c8
Document all shard allocation filtering attributes (#46992)
This commit adds coverage to the docs for some missing built-in shard
allocation attributes.
2019-11-21 08:30:30 -05:00
Peter Johnson 3221827a4b [Docs] Correct typo in match-query.asciidoc (#49082) 2019-11-21 11:31:01 +01:00
debadair d3bc9b7fb2 [DOCS] Clarify backport policy for important technical corrections. (#49131)
* [DOCS] Clarify backport policy for important technical corrections.

* Update docs/README.asciidoc

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2019-11-20 10:57:38 -08:00
Lisa Cawley 0f15736687 [DOCS] Reformat rollup API docs (#49397) 2019-11-20 10:46:16 -08:00
Bogdan Pintea 8c2ab8bb72 SQL:Docs: add the PIVOT clause to SELECT section (#49129)
The PR adds the documentation on the PIVOT clause.

(cherry picked from commit a55b36065e6496c44b6e3191296931d477a8e5f5)
2019-11-20 18:21:06 +01:00
Lisa Cawley a27e0fe10d [DOCS] Reformat ILM API docs (#49348) 2019-11-20 08:24:46 -08:00
Mayya Sharipova e3da60c23d Increase the number of vector dims to 2048 (#46895) 2019-11-20 07:47:33 -05:00
Christoph Büscher 4ffa050735 Allow custom characters in token_chars of ngram tokenizers (#49250)
Currently the `token_chars` setting in both `edgeNGram` and `ngram` tokenizers
only allows for a list of predefined character classes, which might not fit
every use case. For example, including underscore "_" in a token would currently
require the `punctuation` class which comes with a lot of other characters.
This change adds an additional "custom" option to the `token_chars` setting,
which requires an additional `custom_token_chars` setting to be present and
which will be interpreted as a set of characters to inlcude into a token.

Closes #25894
2019-11-20 10:37:12 +01:00
Mathew Davis 92a1faf545 Fixing a typo in the stop SLM api request header. 2019-11-19 23:06:49 -07:00
Lisa Cawley 2b9fb7ebe2 [DOCS] Merges security overview pages (#49342) 2019-11-19 16:19:02 -08:00
Benjamin Trent d068818b16
[ML][Inference] document new settings (#49309) (#49336)
* [ML][Inference] document new settings

* [DOCS] Minor edits
2019-11-19 16:43:19 -05:00
James Rodewig 62a3154d0e
[DOCS] [7.x] Add high-level docs for enrich processor and policies (#49194) (#49331) 2019-11-19 16:38:13 -05:00
Lisa Cawley 75f1f612c2 [DOCS] Merges duplicate pages for Active Directory realms (#49205) 2019-11-19 13:18:01 -08:00
Lisa Cawley c4c8a7a43c [DOCS] Merges duplicate pages for PKI realms (#49206) 2019-11-19 10:51:09 -08:00
Lisa Cawley 62bbe419d3 [DOCS] Removes Beats security page (#49276) 2019-11-19 09:15:30 -08:00
Lisa Cawley 97cdfd2848 [DOCS] Clarify ML job closure prerequisites (#49265) 2019-11-19 08:36:50 -08:00
James Rodewig a26916cc23 [DOCS] Reformat elision token filter docs (#49262) 2019-11-19 10:55:22 -05:00
James Rodewig 8639ddab5e [DOCS] Reformat fingerprint token filter docs (#49311) 2019-11-19 10:55:21 -05:00
jimczi cb5169ae37 update release notes for 7.5.0 after respin 2019-11-19 16:24:04 +01:00
Marios Trivyzas fd1bb4a33a SQL: Fix issue with mins & hours for DATEDIFF (#49252)
Previously, DATEDIFF for minutes and hours was doing a
rounding calculation using all the time fields (secs, msecs/micros/nanos).
Instead it should first truncate the 2 dates to the respective field (mins or hours)
zeroing out all the more detailed time fields and then make the subtraction.

(cherry picked from commit 124cd18e20429e19d52fd8dc383827ea5132d428)
2019-11-19 14:25:28 +01:00
Lisa Cawley abd4a70b10 [DOCS] Merges duplicate pages for Kerberos realms (#49207) 2019-11-18 15:23:06 -08:00
Lisa Cawley b4f82c9cdb [DOCS] Merges duplicate pages for LDAP realms (#49203) 2019-11-18 14:09:24 -08:00
Julie Tibshirani 81a9d98a47 Remove the 'experimental' marking from vector fields. (#49120)
We wrapped up the API changes we wanted to make, and vector fields can now be
considered GA.
2019-11-18 12:42:46 -08:00
Lisa Cawley b0054eecd6 [DOCS] Merges duplicate pages for file realms (#49200) 2019-11-18 12:02:18 -08:00
Lisa Cawley 48f53efd9a [DOCS] Merges duplicate pages for SAML realms (#49209) 2019-11-18 10:09:29 -08:00
Lisa Cawley b0b5fcc4f6
[DOCS] Removes closed security PRs from release notes (#49256) 2019-11-18 09:19:11 -08:00
gpaimla 7d20b50f45 Implement Lucene EstonianAnalyzer, Stemmer (#49149)
This PR adds a new analyzer and stemmer for the Estonian language.

Closes #48895
2019-11-18 17:24:21 +01:00
Antoine Garcia 288217e82b [Docs] Specify field types not supporting doc values (#49041)
The `string` type (with option `analyzed`) has been replaced by `text` after `6.0`, 
also the `annonated_text` field do not support doc values and should be mentioned.
2019-11-18 16:38:31 +01:00
Yannick Welsch af797a77a1 Auto-expand indices according to allocation filtering rules (#48974)
Honours allocation filtering rules when auto-expanding indices.
2019-11-18 12:01:56 +01:00
Rory Hunter e84e21174b
Support `_FILE` suffixed env vars in Docker entrypoint (#49182)
Backport of #47573.

Closes #43603. Allow environment variables to be passed to ES in a Docker
container via a file, by setting an environment variable with the `_FILE`
suffix that points to the file with the intended value of the env var.
2019-11-18 08:22:35 +00:00
Lisa Cawley 09a9ec4d23 [DOCS] Merges duplicate pages for native realms (#49198) 2019-11-15 15:35:53 -08:00
Lisa Cawley de8107e350
[DOCS] Adds ml-cpp PRs to release notes (#49185) 2019-11-15 09:36:39 -08:00
Lisa Cawley eca93fcc5f [DOCS] Adds machine learning node type and filters (#49121) 2019-11-15 08:31:59 -08:00
Christos Soulios d9f0245b10
[7.x] Implement stats aggregation for string terms (#49097)
Backport of #47468 to 7.x

This PR adds a new metric aggregation called string_stats that operates on string terms of a document and returns the following:

min_length: The length of the shortest term
max_length: The length of the longest term
avg_length: The average length of all terms
distribution: The probability distribution of all characters appearing in all terms
entropy: The total Shannon entropy value calculated for all terms

This aggregation has been implemented as an analytics plugin.
2019-11-15 14:36:21 +02:00
SylvainJuge e8f49cdee0 [DOCS] minor fix to documentation: http.host can't default to itself (#48135)
fix minor typos on http.host and transport.host default values.

7.x backport of https://github.com/elastic/elasticsearch/pull/48135
2019-11-14 18:16:38 +01:00
James Rodewig e1726fff56 [DOCS] Reformat update license API docs (#48967)
Makes a few changes to better align the update license API docs with
the [API reference template][0].

Changes:

* Replaces POST with PUT in several snippet examples.
  While both are valid, PUT is a bit more RESTful.

* Removes leading slashes (/) from all snippets.

* Relocates and retitles the 'Authorization' section to 'Prerequisites'.

* Replaces explicit titles with the appropriate API reference template
  attributes.

* Replaces unneeded `[float]` tags with explicit anchors.

Closes #35341

[0]: https://github.com/elastic/docs/blob/master/shared/api-ref-ex.asciidoc
2019-11-14 08:00:42 -05:00
Rory Hunter c46a0e8708
Apply 2-space indent to all gradle scripts (#49071)
Backport of #48849. Update `.editorconfig` to make the Java settings the
default for all files, and then apply a 2-space indent to all `*.gradle`
files. Then reformat all the files.
2019-11-14 11:01:23 +00:00
James Rodewig 095c34359f [DOCS] Note limitations of `max_gram` parm in `edge_ngram` tokenizer for index analyzers (#49007)
The `edge_ngram` tokenizer limits tokens to the `max_gram` character
length. Autocomplete searches for terms longer than this limit return
no results.

To prevent this, you can use the `truncate` token filter to truncate
tokens to the `max_gram` character length. However, this could return irrelevant results.

This commit adds some advisory text to make users aware of this limitation and outline the tradeoffs for each approach.

Closes #48956.
2019-11-13 14:28:12 -05:00
James Rodewig 838af15d29 [DOCS] Reformat compound word token filters (#49006)
* Separates the compound token filters doc pages into separate token
  filter pages:
  * Dictionary decompounder token filter
  * Hyphenation decompounder token filter

* Adds analyze API examples for each compound token filter

* Adds a redirect for the removed compound token filters page

Co-Authored-By: debadair <debadair@elastic.co>
2019-11-13 09:36:52 -05:00
István Zoltán Szabó b55022b59f [DOCS] Adds test clause to the code snippets in the cluster restart page (#49023) 2019-11-13 14:36:44 +01:00
Julie Tibshirani 37fa3fb4ff
Ensure parameters are updated when merging flattened mappings. (#48971) (#49014)
This PR makes the following two fixes around updating flattened fields:

* Make sure that the new value for ignore_above is immediately taken into
  affect. Previously we recorded the new value but did not use it when parsing
  documents.
* Allow depth_limit to be updated dynamically. It seems plausible that a user
  might want to tweak this setting as they encounter more data.
2019-11-12 21:50:39 -05:00
David Roberts 698ebd3d0a [TEST] Mute docs snippet test in close-job.asciidoc (#49000)
Due to https://github.com/elastic/elasticsearch/pull/48583#issuecomment-552991325
2019-11-12 17:34:27 +00:00
Michael Basnight bc23bc5146 Add delete alias to the HLRC (#48819)
The delete alias call is a rest only API call, but should still be added
to the rest client. This commit adds it as well as relevant tests.

Ref #47678
2019-11-12 11:02:53 -06:00
Orhan Toy 561351d2fc [Docs] Fix _count HTTP method (#48979) 2019-11-12 15:45:26 +01:00
István Zoltán Szabó fc145575c4 [DOCS] Creates a cluster restart documentation page (#48583)
Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2019-11-12 14:50:53 +01:00
James Rodewig 42e92616f6 [DOCS] Document indices response parameters for node stats API (#47525) 2019-11-12 08:35:35 -05:00
jimczi 0e82b5f59b add release notes for 7.5.0 2019-11-12 09:59:14 +01:00
Benjamin Trent 46ab1db54f
[7.x] [ML] Add new geo_results.(actual_point|typical_point) fields for `lat_long` results (#47050) (#48958)
* [ML] Add new geo_results.(actual_point|typical_point) fields for `lat_long` results (#47050)

[ML] Add new geo_results.(actual_point|typical_point) fields for `lat_long` results (#47050)

Related PR: https://github.com/elastic/ml-cpp/pull/809

* adjusting bwc version
2019-11-11 15:43:03 -05:00
István Zoltán Szabó c2f52015d3 [DOCS] Removes best practice about fields that are highly correlated to the dependent variable. (#48935) 2019-11-11 16:01:21 +01:00
István Zoltán Szabó 91888959e8 [DOCS] Extends analyzed_fields description in PUT DFA API docs. (#48307) 2019-11-11 15:55:12 +01:00
Patrick Maynard 4b85498617 [DOCS] Fix typo in search type docs (#48868) 2019-11-11 09:38:48 -05:00
James Rodewig dd92830801 [DOCS] Reformat condition token filter (#48775) 2019-11-11 08:49:44 -05:00
Arne Welzel f642baa9fb [DOCS] Remove extra "when" (#48926) 2019-11-11 10:11:02 +01:00
Yannick Welsch 87862868c6 Allow realtime get to read from translog (#48843)
The realtime GET API currently has erratic performance in case where a document is accessed
that has just been indexed but not refreshed yet, as the implementation will currently force an
internal refresh in that case. Refreshing can be an expensive operation, and also will block the
thread that executes the GET operation, blocking other GETs to be processed. In case of
frequent access of recently indexed documents, this can lead to a refresh storm and terrible
GET performance.

While older versions of Elasticsearch (2.x and older) did not trigger refreshes and instead opted
to read from the translog in case of realtime GET API or update API, this was removed in 5.0
(#20102) to avoid inconsistencies between values that were returned from the translog and
those returned by the index. This was partially reverted in 6.3 (#29264) to allow _update and
upsert to read from the translog again as it was easier to guarantee consistency for these, and
also brought back more predictable performance characteristics of this API. Calls to the realtime
GET API, however, would still always do a refresh if necessary to return consistent results. This
means that users that were calling realtime GET APIs to coordinate updates on client side
(realtime GET + CAS for conditional index of updated doc) would still see very erratic
performance.

This PR (together with #48707) resolves the inconsistencies between reading from translog and
index. In particular it fixes the inconsistencies that happen when requesting stored fields, which
were not available when reading from translog. In case where stored fields are requested, this
PR will reparse the _source from the translog and derive the stored fields to be returned. With
this, it changes the realtime GET API to allow reading from the translog again, avoid refresh
storms and blocking the GET threadpool, and provide overall much better and predictable
performance for this API.
2019-11-09 17:47:50 +01:00
Julian Simioni 5e4501eb3f [Docs] Consolidate single example into a single line (#48904)
The first example of splitting rules for the `word_delimiter` token filter was spread across two bullet points. This makes it look like they are two separate splitting rules.
2019-11-08 15:12:45 -05:00
Yannick Welsch af887be3e5 Hide orphaned tasks from follower stats (#48901)
CCR follower stats can return information for persistent tasks that are in the process of being cleaned up. This is problematic for tests where CCR follower indices have been deleted, but their persistent follower task is only cleaned up asynchronously afterwards. If one of the following tests then accesses the follower stats, it might still get the stats for that follower task.

In addition, some tests were not cleaning up their auto-follow patterns, leaving orphaned patterns behind. Other tests cleaned up their auto-follow patterns. As always the same name was used, it just depended on the test execution order whether this led to a failure or not. This commit fixes the offensive tests, and will also automatically remove auto-follow-patterns at the end of tests, like we do for many other features.

Closes #48700
2019-11-08 13:56:53 +01:00
bellengao bdc7057d58 [DOCS] Correct typo in split index API docs (#48894) 2019-11-07 15:27:27 -05:00
bellengao 293902c6a5 [DOCS] Fix shard type in CCR overview doc (#48882)
Closes #48875
2019-11-07 10:09:45 -05:00
Tanguy Leroux 552381d7f9 Add mention to Pause Auto-Follower API in Upgrade Clusters docs (#48764)
Relates #46665
2019-11-06 09:48:44 -05:00
István Zoltán Szabó 3c9bd13dca [DOCS] Adds classification type DFA API docs and ml-shared.asciidoc (#48241) 2019-11-06 07:41:38 -05:00
István Zoltán Szabó 70765dfb05 [DOCS] Adds classification type evaluation docs to the DFA evaluation API (#47657) 2019-11-06 07:38:33 -05:00
glerb baabc21a04 [DOCS] Correct typo in Discovery docs (#48494) 2019-11-05 08:48:43 -05:00
James Rodewig 700a316bb3 [DOCS] Reformat decimal digit token filter docs (#48722) 2019-11-01 12:38:14 -04:00
James Rodewig 680999f246 [DOCS] List `indices.lifecycle.poll_interval` as cluster-level (#48813)
Lists `indices.lifecycle.poll_interval` with other cluster-level ILM
settings.

Previously, it was included under index-level settings.
2019-11-01 11:54:46 -04:00
pulysak 9a0a7ab95a [DOCS] Fix typo in Index API reference docs (#48760) 2019-11-01 09:16:11 -04:00
Alexander Reelsen 80cde68af7 [DOCS] Remove unneeded // CONSOLE comments from snippets (#48763)
Updates the docs README file to remove outdated `// CONSOLE` instructions
2019-11-01 09:04:24 -04:00
debadair b9f4b32892 [DOCS] Fix cross-doc link. (#48783)
* [DOCS] Fix cross-doc link.

* Fixed xref
2019-10-31 18:59:17 -07:00
Lisa Cawley 40834c229f
[7.x][DOCS] Copies ESMS monitoring details to Elasticsearch Reference (#48780) 2019-10-31 18:22:08 -07:00
debadair 457379e74e [DOCS] Edited Docker install & tweaked Docker compose file. (#47715)
* [DOCS] Edited Docker install & tweaked Docker compose file.

* Synced with Docker GS in SO

* Incorporated review comments
2019-10-31 18:12:39 -07:00
Tal Levy 4be54402de
[7.x] Add ingest info to Cluster Stats (#48485) (#48661)
* Add ingest info to Cluster Stats (#48485)

This commit enhances the ClusterStatsNodes response to include global
processor usage stats on a per-processor basis.

example output:

```
...
    "processor_stats": {
      "gsub": {
        "count": 0,
        "failed": 0
        "current": 0
        "time_in_millis": 0
      },
      "script": {
        "count": 0,
        "failed": 0
        "current": 0,
        "time_in_millis": 0
      }
    }
...
```

The purpose for this enhancement is to make it easier to collect stats on how specific processors are being used across the cluster beyond the current per-node usage statistics that currently exist in node stats.

Closes #46146.

* fix BWC of ingest stats

The introduction of processor types into IngestStats had a bug.
It was set to `null` and set as the key to the map. This would
throw a NPE. This commit resolves this by setting all the processor
types from previous versions that are not serializing it out to
`_NOT_AVAILABLE`.
2019-10-31 14:36:54 -07:00
Deb Adair 6412d0f528 [DOCS] Remove coming tag from 7.4.2 RN backport. 2019-10-31 09:43:26 -07:00
Lisa Cawley b7559f23cc [DOCS] Fixes PR#48055 in release notes (#48726) 2019-10-31 07:37:44 -07:00
Peter Johnson 3f7aafa421 [DOCS] Fix typo in synonym token filter docs (#48691) 2019-10-31 09:12:24 -04:00
James Rodewig 3d5b1725a9 [DOCS] Remove unneeded filter from common grams analyze ex (#48748) 2019-10-31 09:08:14 -04:00
Brandon Morelli aa02174d53 [DOCS] Fix typo in ILM policy definition docs (#48723)
Removes an extra "by".
2019-10-31 08:30:54 -04:00
Andrei Dan ffe5d5417f
ILM Make the `check-rollover-ready` step retryable (#48256) (#48740)
This adds the infrastructure to be able to retry the execution of retryable
steps and makes the `check-rollover-ready` retryable as an initial step to
make the rollover action more resilient to transient errors.

(cherry picked from commit 454020ac8acb147eae97acb4ccd6fb470d1e5f48)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
2019-10-31 11:28:55 +00:00
debadair a876760848 [DOCS] Add placeholder for 7.4.2 release notes (#48724) 2019-10-30 16:09:29 -07:00
Jason Tedor 13043219ac
Fix specification for cluster.remote.connect (#48690)
The docs specify that cluster.remote.connect disables cross-cluster
search. This is correct, but not fully accurate as it disables any
functionality that relies on remote cluster connections: cross-cluster
search, remote data feeds, and cross-cluster replication. This commit
updates the docs to reflect this.
2019-10-30 11:26:15 -04:00
James Rodewig 0b062bbc82 [DOCS] Correct required file ext for user agent ingest processor (#48688)
For the user agent ingest processor, custom regex files must end
with the `.yml` file extension.

This corrects the docs which said the `.yaml` extension was required.
2019-10-30 11:11:29 -04:00
Dan Hermann dbc05cd808
Add option to split processor for preserving trailing empty fields (#48685) 2019-10-30 08:25:03 -05:00
James Rodewig 77acbc4fa9 [DOCS] Reformat common grams token filter (#48426) 2019-10-30 08:40:56 -04:00
Yannick Welsch 356066ce6a Revert "Mute get-ccr-stats doctest (#48375)"
This reverts commit f861927e8b8fc987949ce996a131a2d272b9646e.
2019-10-30 11:13:29 +01:00
Julie Tibshirani 89c65752dc
Update the signature of vector script functions. (#48653)
Previously the functions accepted a doc values reference, whereas they now
accept the name of the vector field. Here's an example of how a vector function
was called before and after the change.

```
Before: cosineSimilarity(params.query_vector, doc['field'])
After:  cosineSimilarity(params.query_vector, 'field')
```

This seems more intuitive, since we don't allow direct access to vector doc
values and the the meaning of `doc['field']` is unclear.

The PR makes the following changes (broken into distinct commits):
* Add new function signatures of the form `function(params.query_vector,
'field')` and deprecates the old ones. Because Painless doesn't allow two
methods with the same name and number of arguments, we allow a generic `Object`
to be passed in to the function and decide on the behavior through an
`instanceof` check.
* Refactor the class bindings so that the document field is passed to the
constructor instead of the instance method. This allows us to avoid retrieving
the vector doc values on every function invocation, which gives a tiny speed-up
in benchmarks.

Note that this PR adds new signatures for the sparse vector functions too, even
though sparse vectors are deprecated. It seemed simplest to understand (for both
us and users) to keep everything symmetric between dense and sparse vectors.
2019-10-29 15:46:05 -07:00
James Rodewig 7002ce1e9c [DOCS] Replace `_uid` refs in reindex slicing docs (#48649)
PR #25543 removed the `_uid` field in favor of the `_id` field,
including for use in slicing.

This removes an outdated reference to `_uid` in our reindex docs.
2019-10-29 16:41:53 -04:00
Christoph Büscher 1de49d8a70 Remove Ranking Evaluation API experimental status (#48603)
The API has been released long enough to remove the experimental status.
2019-10-29 20:57:39 +01:00
Lisa Cawley c6f4662038
[DOCS] Updates ML PRs in 7.4.1 release notes (#48600) 2019-10-29 09:35:11 -07:00
Daniel Andion d0cbbf9d58
SQL: [Docs] Typo in HAVING section (#48609)
`HAVING` section code states `GROUP BY`  instead of the appropriate keyword.

(cherry picked from commit 9d505dc3db51e250fdf1b44e4d952dcd97bf1bc1)
2019-10-29 16:37:39 +01:00
lgypro abddf51672 [Docs] Fix syntax error leading to wrong doc ID (#48554)
In order to index a document with id 2, the "&" should be replaced by "?"
2019-10-29 10:27:23 +01:00
Ian Danforth 82e25c4ac7 [Docs] Fix typo in suggesters search API doc (#48477) 2019-10-29 09:58:05 +01:00
Ian Danforth 4a076f5e92 [Doc] Fix typo in indices module docs (#48598) 2019-10-28 21:40:09 +01:00
Julie Tibshirani 605500df7e
Add sparse vector deprecation to 7.6 migration docs. (#48435)
This note was accidentally omitted from the deprecation PR.
2019-10-28 11:57:20 -07:00
Benjamin Trent 6ea59dd428
[ML][Transforms] add wait_for_checkpoint flag to stop (#47935) (#48591)
Adds `wait_for_checkpoint` for `_stop` API.
2019-10-28 13:02:57 -04:00
Lisa Cawley 13ce179706
[DOCS] Re-enable code snippet testing in close anomaly detection job API (#48259) (#48585) 2019-10-28 08:42:09 -07:00
Michael Basnight 1ba57dbe08 [Docs] add missing snapshot restore reference (#45256) 2019-10-28 09:55:10 -05:00
Shaunak Kashyap d27a307379 [DOCS] Remove extraneous comma in Enrich Stats API's JSON response (#48539) 2019-10-25 12:35:50 -04:00
James Rodewig e9c8e4f6d1 [DOCS] Fix note format in index suggestion docs (#48536) 2019-10-25 11:31:47 -04:00
Christoph Büscher 055a0800eb [Docs] Mention reserved completion suggestion characters (#48445)
We currently don't mention the three reserved characters anywhere. This change
adds a short note mentioning them

Closes #48341
2019-10-25 16:58:23 +02:00
Julie Tibshirani b2974e3816 Correct outdated information in _index docs. (#48436)
This PR makes the following updates:
* Update the supported query types to include `prefix` and `wildcard`.
* Specify that queries accept index aliases.
* Clarify that when querying on a remote index name, the separator `:` must be
  present.
2019-10-24 11:02:25 -07:00
Hendrik Muhs 5ecfcdb162 update warning about index names after transform rename (#48457)
update warning about index names after transform rename
2019-10-24 15:17:20 +02:00
Julie Tibshirani 4375316b9d Make sure to list the 7.5 migration docs. 2019-10-23 18:52:22 -07:00
Julie Tibshirani 2664cbd20b
Deprecate the sparse_vector field type. (#48368)
We have not seen much adoption of this experimental field type, and don't see a
clear use case as it's currently designed. This PR deprecates the field type in
7.x. It will be removed from 8.0 in a follow-up PR.
2019-10-23 16:35:03 -07:00
James Rodewig 06dc1fbd96 [DOCS] Reformat ASCII folding token filter docs (#48143) 2019-10-23 15:06:55 -05:00
Jim Ferenczi 96556d72cc
Add a known issue to the release notes of 7.4.0 (#48373)
A [bug](https://github.com/elastic/elasticsearch/issues/48358) in 7.4.0 prevents
the activation of the search slow log. This change adds an entry in the release
notes to warn users to not activate it in this version.

Relates #48358
2019-10-23 19:57:37 +02:00
James Rodewig 640d7416b1
[DOCS] Change prev version to 7.5 in upgrade docs (#48415) 2019-10-23 12:09:26 -05:00
James Rodewig 19afe3f84c [DOCS] Remove duplicate links for ingest processor overview (#48394) 2019-10-23 10:55:49 -05:00
James Rodewig 9c75f14a9f [DOCS] Reformat classic token filter docs (#48314) 2019-10-23 10:14:25 -05:00
James Rodewig 852622d970 [DOCS] Remove binary gendered language (#48362) 2019-10-23 09:37:12 -05:00
Alan Woodward 3be6135ceb Mute get-ccr-stats doctest (#48375)
This test is failing frequently, due to #47718
2019-10-23 15:20:35 +01:00
Tom Callahan f573cd6a2c [DOCS] Add 7.4.1 Release Notes (#48359) 2019-10-23 08:22:00 -05:00
James Rodewig e45b0cd7e3 [DOCS] Sort cluster API docs alphabetically (#48198) 2019-10-22 12:28:39 -05:00
Brandon DeVault 2f162a98a0 [DOCS] Added the voting node.role value (#48162)
* [DOCS] Added the voting node.role value 

With the elasticsearch.yml option "node.voting_only: true", if you perform a "_cat/nodes" there is a new option for `v` that will display if the node is a voting node.

* [DOCS] Added the voting node.role value

Updated "voting node" to read "voting-only node"
2019-10-22 15:26:05 +02:00
James Rodewig e931fcd331 [DOCS] Add placeholder for 7.4.1 release notes (#48316) 2019-10-22 07:53:28 -05:00
Alexandre Fonseca c41951c6b3 [Docs] Fix opType options in IndexRequest API example. (#48290) 2019-10-22 13:49:19 +02:00
Hendrik Muhs 0656207e9a [DOCS][Transform] document limitation regarding rolling upgrade with 7.2, 7.3 (#48118)
adds a limitation about rolling upgrade from 7.2 or 7.3. and fixes a problem with renamed preferences
2019-10-22 09:01:45 +02:00
Martijn van Groningen c09b62d5bf
Backport: also validate source index at put enrich policy time (#48311)
Backport of: #48254

This changes tests to create a valid
source index prior to creating the enrich policy.
2019-10-22 07:38:16 +02:00
James Rodewig f4fa61b2f2 [DOCS] Add 'Selecting gateway and seed nodes' section to CCS docs (#48297) 2019-10-21 11:14:23 -05:00
James Rodewig a66bb2c7ed [DOCS] Reformat CJK bigram and CJK width token filter docs (#48210) 2019-10-21 08:44:49 -05:00
Ignacio Vera b1224fca8c
upgrade to Lucene-8.3.0-snapshot-25968e3b75e (#48227) 2019-10-21 08:21:09 +02:00
Christoph Büscher 3ea666d694 Clarify mapping types that support ignore_malformed (#48206)
The `ignore_malformed` setting only works on selected mapping types, otherwise
we throw an mapper_parsing_exception. We should add a list of all the mapping
types that support it, since the number of types not supporting it seems larger.

Closes #47166
2019-10-18 20:39:38 +02:00
Guido Lena Cota 1beeb3a83e [DOCS] Fix typos in Painless execute API docs(#47518) 2019-10-18 10:31:04 -04:00
Alexander Reelsen 66581d8158
update ingest-user-agent regexes.yml (#47807)
This new regexes are from:
154eba17f5/regexes.yaml
2019-10-18 16:26:48 +02:00