Commit Graph

52769 Commits

Author SHA1 Message Date
Lee Hinman cb2ce3736a
[7.x] Make noop template updates be cluster state noops (#57851) (#57880)
Backports the following commits to 7.x:

    Make noop template updates be cluster state noops (#57851)
2020-06-09 09:26:06 -06:00
Dan Hermann b501b282f8
Change default backing index naming scheme 2020-06-09 09:31:34 -05:00
Nik Everett e7cc2448d2
Save memory when string terms are not on top (#57758) (#57876)
This reworks string flavored implementations of the `terms` aggregation
to save memory when it is under another bucket by dropping the usage of
`asMultiBucketAggregator`.
2020-06-09 10:26:29 -04:00
Yannick Welsch b5d3565214 Assert on request headers only (#57792)
Only assert that actual request headers are empty, as default headers might still be there when stashing the context.
2020-06-09 14:08:25 +02:00
Hossein Dehghan 2c6bd978d8 [Docs] Fix missing closing bracket for watcher webhook.asciidoc (#57803) 2020-06-09 13:59:51 +02:00
Dan Hermann 60db6c7c45
[7.x] Update rollover index API docs for data streams 2020-06-09 06:57:16 -05:00
Yannick Welsch 259be236cf Use clean thread context for transport and applier service (#57792)
Adds assertions to Netty to make sure that its threads are not polluted by thread contexts (and
also that thread contexts are not leaked). Moves the ClusterApplierService to use the system
context (same as we do for MasterService), which allows to remove a hack from
TemplateUgradeService and makes it clearer that applying CS updates is fully executing under
system context.
2020-06-09 12:32:28 +02:00
bellengao 19042ddf3b [DOCS] Fix typo in remote-clusters doc (#57820) 2020-06-09 11:28:30 +02:00
Rory Hunter 6bf06f5be7 Remove the last Perl scripts (#57767)
The Perl scripts in dev-tools have been replaced with tools in another
repository. It's time to say goodbye to Perl scripts in this repository.
2020-06-09 10:12:34 +01:00
Andrei Stefan 3cc8166946
SQL: handle MIN and MAX functions on dates in Painless scripts (#57605) (#57863)
* Convert to date/datetime the result of numeric aggregations (min, max)
in Painless scripts

(cherry picked from commit f1de99e2a6fbf3806c4f2b6b809738aa8faa2d75)
2020-06-09 10:09:01 +03:00
Tim Brooks 9eaee3da8d
Fix exception check in RecoveryRequestTrackerTests (#57493)
Currently we check that exceptions are the same in the recovery request
tracker test. This is inconsistent because the future wraps the
exception in a new instance. This commit fixes the test by comparing a
random exception message.

Fixes #57199
2020-06-08 15:42:48 -06:00
Lee Hinman fe2eaf0d03
[7.x] Throw exception on duplicate mappings metadata fields (#57839)
In #57701 we changed mappings merging so that duplicate fields specified in mappings caused an
exception during validation. This change makes the same exception thrown when metadata fields are
duplicated. This will allow us to be strict currently with plans to make the merging more
fine-grained in a later release.
2020-06-08 14:21:18 -06:00
Tim Brooks 952cf770ed
Reestablish peer recovery after network errors (#57827)
Currently a network disruption will fail a peer recovery. This commit
adds network errors as retryable actions for the source node.
Additionally, it adds sequence numbers to the recovery request to
ensure that the requests are idempotent.

Additionally it adds a reestablish recovery action. The target node
will attempt to reestablish an existing recovery after a network
failure. This is necessary to ensure that the retries occurring on the
source node provide value in bidirectional failures.
2020-06-08 14:17:52 -06:00
Benjamin Trent d5522c2747
[ML] add new circuit breaker for inference model caching (#57731) (#57830)
This adds new plugin level circuit breaker for the ML plugin.

`model_inference` is the circuit breaker qualified name.

Right now it simply adds to the breaker when the model is loaded (and possibly breaking) and removing from the breaker when the model is unloaded.
2020-06-08 16:02:48 -04:00
Lee Hinman 6e8cf0973f
[7.x] Disallow merging existing mapping field definitions in templates (#57701) (#57822)
Backports the following commits to 7.x:

    Disallow merging existing mapping field definitions in templates (#57701)
2020-06-08 12:56:09 -06:00
Benjamin Trent 16fcb64c99
Test mute (#57832) 2020-06-08 14:03:16 -04:00
Armin Braun 0987c0a5f3
Fix Broken Numeric Shard Generations in RepositoryData (#57813) (#57821)
Fix broken numeric shard generations when reading them from the wire
or physically from the physical repository.
This should be the cheapest way to clean up broken shard generations
in a BwC and safe-to-backport manner for now. We can potentially
further optimize this by also not doing the checks on the generations
based on the versions we see in the `RepositoryData` but I don't think
it matters much since we will read `RepositoryData` from cache in almost
all cases.

Closes #57798
2020-06-08 18:36:56 +02:00
Przemysław Witek 7a1300a09e
[7.x] Make ModelPlotConfig.annotations_enabled default to ModelPlotConfig.enabled if unset (#57808) (#57815) 2020-06-08 17:41:12 +02:00
James Rodewig 63e962c99a [DOCS] Fix typo in `html_strip` char filter docs 2020-06-08 10:37:35 -04:00
James Rodewig 2ef30217e0 [DOCS] Fix typo in data stream docs 2020-06-08 10:19:39 -04:00
Nik Everett ee0ce8ffaf
Fix a bug with missing fields in sig_terms (#57757)
When you run a `significant_terms` aggregation on a field and it *is*
mapped but there aren't any values for it then the count of the
documents that match the query on that shard still have to be added to
the overall doc count. I broke that in #57361. This fixes that.

Closes #57402
2020-06-08 10:07:14 -04:00
Mayya Sharipova 70e63a365a
Refactor how to determine if a field is metafield (#57378) (#57771)
Before to determine if a field is meta-field, a static method of MapperService
isMetadataField was used. This method was using an outdated static list
of meta-fields.

This PR instead changes this method to the instance method that
is also aware of meta-fields in all registered plugins.

Related #38373, #41656
Closes #24422
2020-06-08 09:16:18 -04:00
István Zoltán Szabó 536dadc4e0 [DOCS] Extends geo_shape query docs to explain behavior when data is indexed as array of shapes (#57806)
Co-authored-by: James Rodewig <james.rodewig@elastic.co>
2020-06-08 15:06:10 +02:00
David Turner 663702e609 Small improvements to resilience design docs (#57791)
A follow-up to #47233 to clarify a few points.
2020-06-08 13:50:41 +01:00
James Rodewig 2121eb528c
[DOCS] Reformat `html_strip` charfilter (#57764) (#57810)
Changes:

* Converts title to sentence case
* Adds a title abbreviation
* Adds Lucene link to description
* Reformat sections
2020-06-08 08:44:43 -04:00
István Zoltán Szabó 37293a9cf6 [DOCS] Adds security privileges to _scripts endpoint docs. (#57795) 2020-06-08 14:35:56 +02:00
Andrei Dan 1b84e93d83
[7.x] DataStream creation validation allows for prefixed indices (#57750) (#57799)
We want to validate the DataStreams on creation to make sure the future backing
indices would not clash with existing indices in the system (so we can
always rollover the data stream).
This changes the validation logic to allow for a DataStream to be created
with a backing index that has a prefix (eg. `shrink-foo-000001`) even if the
former backing index (`foo-000001`) exists in the system.
The new validation logic will look for potential index conflicts with indices
in the system that have the counter in the name greater than the data stream's
generation.

This ensures that the `DataStream`'s future rollovers are safe because for a
`DataStream` `foo` of generation 4, we will look for standalone indices in the
form of `foo-%06d` with the counter greater than 4 (ie. validation will fail if
`foo-000006` exists in the system), but will also allow replacing a
backing index with an index named by prefixing the backing index it replaces.

(cherry picked from commit 695b242d69f0dc017e732b63737625adb01fe595)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
2020-06-08 13:31:52 +01:00
David Kyle 08d1286de7
[7.x] Delete expired data by job (#57337) (#57796)
Deleting expired data can take a long time leading to timeouts if there
are many jobs. Often the problem is due to a few large jobs which 
prevent the regular maintenance of the remaining jobs. This change adds
a job_id parameter to the delete expired data endpoint to help clean up
those problematic jobs.
2020-06-08 13:00:23 +01:00
Armin Braun 004eb8bd7e
Fix Bug With RepositoryData Caching (#57785) (#57800)
* Fix Bug With RepositoryData Caching

This fixes a really subtle bug with caching `RepositoryData`
that can corrupt a repository.
We were caching `RepositoryData` serialized in the newest
metadata format. This lead to a confusing situation where
numeric shard generations would be cached in `ShardGenerations`
that were not written to the repository because the repository
or cluster did not yet support `ShardGenerations`.
In the case where shard generations are not actually supported yet,
these cached numeric generations are not safe and there's multiple
scenarios where they would be incorrect, leading to the repository
trying to read shard level metadata from index-N that don't exist.
This commit makes it so that cached metadata is always in the same
format as the metadata in the repository.

Relates #57798
2020-06-08 13:16:45 +02:00
Luca Cavanna 7a06a13d99 Add description to submit and get async search, as well as cancel tasks (#57745)
This makes it easier to debug where such tasks come from in case they are returned from the get tasks API.

Also renamed the last occurrence of waitForCompletion to waitForCompletionTimeout in get async search request.
2020-06-08 11:17:29 +02:00
Luca Cavanna 06ef3042c1 Specify reason whenever async search gets cancelled (#57761)
This allows to trace where the cancel tasks request came from given that it may be triggered for multiple reasons.
2020-06-08 10:25:31 +02:00
Armin Braun 619e4f8c02
Make BackgroundIndexer more Efficient (#57781) (#57789)
Improve efficiency of background indexer by allowing to add
an assertion for failures while they are produced to prevent
queuing them up.
Also, add non-blocking stop to the background indexer so that when
stopping multiple indexers we don't needlessly continue indexing
on some indexers while stopping another one.

Closes #57766
2020-06-08 10:18:47 +02:00
James Rodewig 6c93fed204
[DOCS] Document `doc_as_upsert` does not support ingest pipelines (#57649) (#57783)
Co-authored-by: James Rodewig <james.rodewig@elastic.co>

Co-authored-by: Asfaloth <asfalots@users.noreply.github.com>
2020-06-06 16:10:42 -04:00
David Roberts 1d64d55a86
[7.x][ML] Add per-partition categorization option (#57723)
This PR adds the initial Java side changes to enable
use of the per-partition categorization functionality
added in elastic/ml-cpp#1293.

There will be a followup change to complete the work,
as there cannot be any end-to-end integration tests
until elastic/ml-cpp#1293 is merged, and also
elastic/ml-cpp#1293 does not implement some of the
more peripheral functionality, like stop_on_warn and
per-partition stats documents.

The changes so far cover REST APIs, results object
formats, HLRC and docs.

Backport of #57683
2020-06-06 08:15:17 +01:00
debadair 100d2bd063
[DOCS] Editorial ILM cleanup (#57565) (#57776)
* [DOCS] Editorial cleanup

* Moved example of applying a template to multiple indices.

* Combine existing indices topics

* Fixed test

* Add skip rollover file.

* Revert rename.

* Update include.

* Revert rename

* Apply suggestions from code review

Co-authored-by: Adam Locke <adam.locke@elastic.co>
Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>

* Apply suggestions from code review

* Fixed callout

* Update docs/reference/ilm/ilm-with-existing-indices.asciidoc

Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>

* Update docs/reference/ilm/ilm-with-existing-indices.asciidoc

Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>

* Apply suggestions from code review

* Restored policy to template example.

* Fixed JSON parse error

Co-authored-by: Adam Locke <adam.locke@elastic.co>
Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>

Co-authored-by: Adam Locke <adam.locke@elastic.co>
Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>
2020-06-05 18:55:51 -07:00
James Rodewig 26d2b4f871
[DOCS] Fix typo in CCR overview docs (#57709) (#57773)
Co-authored-by: bellengao <gbl_long@163.com>
2020-06-05 17:42:49 -04:00
Nik Everett 3b1dfa3b5d
Remove deprecated wrapped from scripted_metric (backport of #57627) (#57763)
This removes the deprecated `asMultiBucketAggregator` wrapper from
`scripted_metric`. Unlike most other such removals, this isn't likely to
save much memory. But it does make the internals of the aggregator
slightly less twisted.

Relates to #56487
2020-06-05 16:14:28 -04:00
Benjamin Trent 9666a895f7
[ML] inference performance optimizations and refactor (#57674) (#57753)
This is a major refactor of the underlying inference logic.

The main refactor is now we are separating the model configuration and
the inference interfaces.

This has the following benefits:
 - we can store extra things with the model that are not
   necessary for inference (i.e. treenode split information gain)
 - we can optimize inference separate from model serialization and storage.
 - The user is oblivious to the optimizations (other than seeing the benefits).

A major part of this commit is removing all inference related methods from the
trained model configurations (ensemble, tree, etc.) and moving them to a new class.

This new class satisfies a new interface that is ONLY for inference.

The optimizations applied currently are:
- feature maps are flattened once
- feature extraction only happens once at the highest level
  (improves inference + feature importance through put)
- Only storing what we need for inference + feature importance on heap
2020-06-05 14:20:58 -04:00
Martijn van Groningen f170b52e64
Backing indices should use composable template matching with the corresponding data stream name (#57728)
Backport of #57640 to 7.x branch.

Composable templates with exact matches, can match with the data stream name, but not with the backing index name.
Also if the backing index naming scheme changes, then a composable template may never match with a backing index.

In that case mappings and settings may not get applied.
2020-06-05 18:38:22 +02:00
Dan Hermann 3fe93e24a6
[7.x] Prohibit closing the write index for a data stream (#57740) 2020-06-05 11:14:43 -05:00
James Rodewig bc921ea17c
[DOCS] Add docs for designing resilient clusters (#47233) (#57743)
Adds some guidance for designing clusters to be resilient to
failures, including example architectures.

Co-authored-by: James Rodewig <james.rodewig@elastic.co>

Co-authored-by: David Turner <david.turner@elastic.co>
2020-06-05 12:08:45 -04:00
Jake Landis 459ab9a0b2
[7.x] Ensure type exists for all monitoring configuration (#57399) (#57704)
#47711 and #47246 helped to validate that monitoring settings are
rejected at time of setting the monitoring settings. Else an invalid
monitoring setting can find it's way into the cluster state and result
in an exception thrown [1] on the cluster state application (there by
causing significant issues). Some additional monitoring settings have
been identified that can result in invalid cluster state that also
result in exceptions thrown on cluster state application.

All settings require a type of either http or local to be
applicable. When a setting is changed, the exporters are automatically
updated with the new settings. However, if the old or new settings lack
of a type setting an exception will be thrown (since exporters are
always of type 'http' or 'local'). Arguably we shouldn't blindly create
and destroy new exporters on each monitoring setting update, but the
lifecycle of the exporters is abit out the scope this PR is trying to
address.

This commit introduces a similar methodology to check for validity as
#47711 and #47246 but this time for ALL (including non-http) settings.
Monitoring settings are not useful unless there an exporter with a type
defined. The type is used as dependent setting, such that it must
exist to set the value. This ensures that when any monitoring settings
changes that they can only get added to cluster state if the type
exists. If the type exists (and the other validations pass) then the
exporters will get re-built and the cluster state remains valid.

Tests have been included to ensure that all dynamic monitoring settings
have the type as dependent settings.

[1]
org.elasticsearch.common.settings.SettingsException: missing exporter type for [found-user-defined] exporter
at org.elasticsearch.xpack.monitoring.exporter.Exporters.initExporters(Exporters.java:126) ~[?:?]
2020-06-05 10:47:11 -05:00
Adam Locke fe558f6373
[DOCS] Clarifying env variable substitution (#57370) (#57737)
* Clarifying environment variable substitution in the ES configuration YAML
* Update code snippet
* Remove extraneous quotes from string example
* Incorporating review feedback
2020-06-05 11:08:03 -04:00
Nik Everett 94b3eed6be Re-mute test
Tracked in #57402
2020-06-05 10:52:24 -04:00
Dimitris Athanasiou f49a14ce6f
[7.x][ML] Fix race condition when force stopping DF analytics job (#57680) (#57717)
When we force delete a DF analytics job, we currently first force
stop it and then we proceed with deleting the job config.
This may result in logging errors if the job config is deleted
before it is retrieved while the job is starting.

Instead of force stopping the job, it would make more sense to
try to stop the job gracefully first. So we now try that out first.
If normal stop fails, then we resort to force stopping the job to
ensure we can go through with the delete.

In addition, this commit introduces `timeout` for the delete action
and makes use of it in the child requests.

Backport of #57680
2020-06-05 17:50:01 +03:00
Martijn van Groningen c407b0f40d
[DOCS] Add data stream overview and intro (#57724)
Backporting #57596 to 7.x branch.

Added data streams overview page and
an introduction to data streams.

Relates to #53100

Co-authored-by: Dan Hermann danhermann@users.noreply.github.com
Co-authored-by: James Rodewig james.rodewig@elastic.co
2020-06-05 16:10:03 +02:00
Armin Braun 8805c1f112
Manually Craft CreateSnapshotRequest to fix BwC Test (#57661) (#57715)
We can't use the high level create snapshot request any longer
since we changed some of its default parameters in `8` and those
are not understood by older versions like `7.4`.

Closes #57650
2020-06-05 15:49:44 +02:00
Jun Ohtani c75c8b6e9d Expose discard_compound_token option to kuromoji_tokenizer (#57421)
This commit exposes the new Lucene option `discard_compound_token` to the Elasticsearch Kuromoji plugin.
2020-06-05 15:41:01 +02:00
Tanguy Leroux 0e57528d5d Remove more //NORELEASE (#57517)
We agreed on removing the following //NORELEASE tags.
2020-06-05 15:34:06 +02:00
James Rodewig b03a83a69d
[DOCS] Fix source filtering xrefs (#57720) (#57725) 2020-06-05 09:05:30 -04:00