Commit Graph

52918 Commits

Author SHA1 Message Date
Ryan Ernst 07aede135d
Set the systemd initial timeout to 75 seconds (#60345)
For systemd, while we are starting up, we notify the system every 15
seconds that we are still in the middle of starting up. However, if
initial startup before plugin initialization is slower than 15 seconds,
we won't ever get the chance to run the first timeout extension. This
commit sets the initial timeout to 75 seconds, up from the default 30
seconds used by systemd.

closes #60140
2020-07-28 16:57:36 -07:00
Benjamin Trent 54c8936508
[ML] do not summerize importance for custom features (#60198) (#60333)
If a feature is created via a custom pre-processor,
we should return the importance for that feature.

This means we will not return the importance for the
original document field for custom processed features.

closes https://github.com/elastic/elasticsearch/issues/59330
2020-07-28 15:58:20 -04:00
Mark Vieira 9776424062
Update IntelliJ compiler options to support version 2020.2 (#60322)
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-07-28 12:40:50 -07:00
Zachary Tong e3d85feecd Mute testForStringIPv6WithScopeIdInput test
Tracking issue: https://github.com/elastic/elasticsearch/issues/60332
2020-07-28 15:05:19 -04:00
James Rodewig 1a172afbb2
[DOCS] Add full-width char section to kuromoji analyzer docs (#60317) (#60324) 2020-07-28 14:17:23 -04:00
Igor Motov 0dd53b76bd
Add aggregation list to node info (#60074) (#60256)
Adds a full list of supported aggregations to the node info API. This list
will be used in transform tests and telemetry mapping tests that will be added
as follow-up PRs.

Fixes #59774
2020-07-28 14:06:12 -04:00
Julie Tibshirani c7bfb5de41
Add search `fields` parameter to support high-level field retrieval. (#60258)
This feature adds a new `fields` parameter to the search request, which
consults both the document `_source` and the mappings to fetch fields in a
consistent way. The PR merges the `field-retrieval` feature branch.

Addresses #49028 and #55363.
2020-07-28 10:58:20 -07:00
James Rodewig 025e7bee80
[DOCS] Fix allowed values for numeric sort types (#60176) (#60299)
Co-authored-by: Philippus Baalman <philippus@gmail.com>
2020-07-28 13:51:59 -04:00
James Rodewig 8edae3cd15
[DOCS] Update pre-existing data stream refs (#60289) (#60293) 2020-07-28 13:51:43 -04:00
Nhat Nguyen 416e51980c Relax ShardFollowTasksExecutor validation (#60054)
If a primary shard of a follower index is being relocated, then we
will fail to create a follow-task. This validation is too restricted.
We should ensure that all primaries of the follower index are active
instead.

Closes #59625
2020-07-28 13:46:49 -04:00
Nhat Nguyen 6ece629ec3 Set timeout of master requests on follower to unbounded (#60070)
Today, a follow task will fail if the master node of the follower
cluster is temporarily overloaded and unable to process master node
requests (such as update mapping, setting, or alias) from a follow-task
within the default timeout. This error is transient, and follow-tasks
should not abort. We can avoid this problem by setting the timeout of
master node requests on the follower cluster to unbounded.

Closes #56891
2020-07-28 13:46:49 -04:00
Howard 11b86b3f88
Remove unused clusterService instance in ActionModule. (#59826) 2020-07-28 10:36:04 -07:00
Zachary Tong 9f8ec3e3fb Mute SSLDriverTests#testCloseDuringHandshakePreJDK11
Tracking issue: https://github.com/elastic/elasticsearch/issues/59992
2020-07-28 13:20:53 -04:00
David Turner 9c62b5cb96 Mute tests for #60321 2020-07-28 18:12:54 +01:00
Zachary Tong 46f9c38c33 Mute tests while waiting on 58807
Bugurl: https://github.com/elastic/elasticsearch/issues/58807
2020-07-28 12:45:49 -04:00
jimczi 4e4ed6ee48 fix race condition in SearchPhaseControllerTests#consumerTestCase 2020-07-28 18:27:39 +02:00
markharwood e0286e9bd3
Search - remove allow-expensive-query checks from wildcard field. (#60273) (#60308)
Removing allow-expensive-query checks because we think this field type is fast enough.

Closes #60139
2020-07-28 17:12:33 +01:00
David Turner 9450ea08b4 Log and track open/close of transport connections (#60297)
Transport connections between nodes remain in place until one or other
node shuts down or the connection is disrupted by a flaky network.
Today it is very difficult to demonstrate that transient failures and
cluster instability are caused by the network even though this is often
the case. In particular, transport connections open and close without
logging anything, even at `DEBUG` level, making it very hard to quantify
the scale of the problem or to correlate the networking problems with
external events.

This commit adds the missing `DEBUG`-level logging when transport
connections open and close, and also tracks the total number of
transport connections a node has opened as a measure of the stability of
the underlying network.
2020-07-28 17:08:04 +01:00
Adam Locke ee18538fd7
[DOCS] Adds table with icons for version compatibility (#60159) (#60302)
* Adds table with icons for simplicity.

* Updating table for clarity.

* Changing table formatting and incorporating more feedback.

* Changing table alignment.
2020-07-28 11:08:58 -04:00
Armin Braun 9222070f22
Fix Test Failure in testCorrectCountsForDoneShards (#60254) (#60286)
* Fix Test Failure in testCorrectCountsForDoneShards

Fixing the freak edge case where the node shard status request returns before
the node was able to send the state update request to master and update the cluster state.
Without this change, the snapshot shard status would report as `DONE` once the data node
has finished updating the shard in the cluster state.
If the data node then drops out of the cluster before the state has been updated, then
the status will jump to "FAILURE" because the master updates the state once the data node
leaves the cluster.

Closes #60247
2020-07-28 15:46:18 +02:00
Jake Landis 92ce41cfaf
[7.x] Introduce javaRestTest source set/task and convert modules (#59939) (#60026)
Introduce a javaRestTest source set and task to compliment the yamlRestTest.
javaRestTest differs such that the code is sourced from Java and may have
different dependencies and setup requirements for the test clusters. This also
allows the tests to run in parallel in different cluster instances to prevent any
cross test contamination between the two types of tests.

Included in this PR is all :modules no longer use the integTest task. The tests
are now driven by test, yamlRestTest, javaRestTest, and internalClusterTest.
Since only :modules (and :rest-api-spec) have been converted to yamlRestTest
we can now disable the integTest task if either yamlRestTest or javaRestTest have
been applied. Once all projects are converted, we can delete the integTest task.

related: #56841
related: #59444
2020-07-28 08:39:11 -05:00
Dimitris Athanasiou ed7dcff7c4
[7.x][ML] Audit updates on data frame analytics jobs (#60126) (#60287)
Closes #59652

Backport of #60126
2020-07-28 16:33:35 +03:00
James Rodewig 4eee6d274d
[DOCS] Fix broken link to Lucene docs (#59365) (#60290)
Co-authored-by: Alexander Reelsen <alexander@reelsen.net>
2020-07-28 09:09:48 -04:00
Dimitris Athanasiou 16ffcfb9f6
[7.x][ML] Ensure bulk requests are not over memory limit (#60219) (#60283)
Data frame analytics jobs that work with very large datasets
may produce bulk requests that are over the memory limit
for indexing. This commit adds a helper class that bundles
index requests in bulk requests that steer away from the
memory limit. We then use this class both from the results
joiner and the inference runner ensuring data frame analytics
jobs do not generate bulk requests that are too large.

Note the limit was implemented in #58885.

Backport of #60219
2020-07-28 16:04:03 +03:00
David Turner b78caa5c00 Add more useful toString on cluster state observers (#60277)
Today if a cluster state observer's listener takes a long time to
process a notification then we log the following rather useless warning
message:

    [notifying listener [org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener@12345678]] took [34567ms]

This commit adds a handful of simple `toString()` implementations in
order to identify the owner of the listener in question.
2020-07-28 12:56:58 +01:00
Jim Ferenczi 1144534093
Executes incremental reduce in the search thread pool (#58461) (#60275)
This change forks the execution of partial
reduces in the coordinating node to the search thread pool.
It also ensures that partial reduces are executed sequentially
and asynchronously in order to limit the memory and cpu that a
single search request can use but also to avoid blocking a
network thread.
If a partial reduce fails with an exception, the search
request is cancelled and the reporting of the error is
delayed to the start of the fetch phase (when the final
reduce is performed). This ensures that we cleanup the
in-flight search requests before returning an error to
the user.

Closes #53411
Relates #51857
2020-07-28 13:40:47 +02:00
Armin Braun d39622e17e
Stop Serializing RepositoryData Twice when Writing (#60107) (#60269)
We can save one round of serializing `RepositoryData` on the write path.
This also leads to somewhat better compression because we compress larger chunks
in one go potentially when compared to serializing and compressing in one go.
Also, fixed the double wrapping of collections when copying the repository
data instance via the `withGenId`.
2020-07-28 11:42:14 +02:00
Yannick Welsch a55c869aab Properly document keepalive and other tcp options (#60216)
Keepalive options are not well-documented (only in transport section, although also available at http and network level).

Co-authored-by: David Turner <david.turner@elastic.co>
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
2020-07-28 11:10:04 +02:00
Yannick Welsch ffe114b890 Set specific keepalive options by default on supported platforms (#59278)
keepalives tell any intermediate devices that the connection remains alive, which helps with overzealous firewalls that are
killing idle connections. keepalives are enabled by default in Elasticsearch, but use system defaults for their
configuration, which often times do not have reasonable defaults (e.g. 7200s for TCP_KEEP_IDLE) in the context of
distributed systems such as Elasticsearch.

This PR sets the socket-level keep_alive options for network.tcp.{keep_idle,keep_interval} to 5 minutes on configurations
that support it (>= Java 11 & (MacOS || Linux)) and where the system defaults are set to something higher than 5
minutes. This helps keep the connections alive while not interfering with system defaults or user-specified settings
unless they are deemed to be set too high by providing better out-of-the-box defaults.
2020-07-28 11:10:04 +02:00
Dimitris Athanasiou 981e436d6c
[7.x][ML] Improve assertion on regression alias field test (#60221) (#60264)
Previously the test was asserting the prediction on each document
was close 10.0 from the expected. It turned out that was not enough
as we occasionally saw the test failing by little.

Instead of relaxing that assertion, this commit changes it to
assert the mean prediction error is less than 10.0. This should
reduce the chances of the test failing significantly.

Fixes #60212

Backport of #60221
2020-07-28 11:48:00 +03:00
Armin Braun fac5953d13
Let `isInetAddress` utility understand the scope ID on ipv6 (#60172) (#60263)
Make `isInetAddress` utility method understand the scope ID on ipv6.

Fixes #60115

Co-authored-by: Yang Cheng <chengyang2048@163.com>
2020-07-28 09:37:39 +02:00
James Rodewig aba785cb6e
[DOCS] Update my-index examples (#60132) (#60248)
Changes the following example index names to `my-index-000001` for consistency:

* `my-index`
* `my_index`
* `myindex`
2020-07-27 15:58:26 -04:00
James Rodewig 3bb58eb5c1
[DOCS] Fix `fuzzy_rewrite` ref in match query docs (#60237) (#60251) 2020-07-27 15:36:09 -04:00
James Rodewig cb4c21fa7b
[DOCS] Fix typo in adapt auto expand replica comments (#60187) (#60239)
Co-authored-by: Howard <danielhuang@tencent.com>
2020-07-27 14:18:53 -04:00
weizijun 5df043d0e0 Fix wait_for_no_initializing_shards params (#58379) 2020-07-27 14:03:26 -04:00
James Rodewig 1178f5c6db
[DOCS] Fix ingest processor docs for autogen doc IDs (#60147) (#60242)
If you autogen doc IDs, you cannot use the `{{_id}}` value in an ingest
processor.

This adds a related admonition to the ingest processor docs.
2020-07-27 13:55:21 -04:00
Adrien Grand f1f275c91b Add 6.8.12 and 7.8.2 version constants. 2020-07-27 19:26:22 +02:00
James Rodewig 95d7ce76ec
[DOCS] Fix `rewrite` => `fuzzy_rewrite` in multi match query docs (#60175) (#60233)
Co-authored-by: homersimpsons <guillaume.alabre@gmail.com>
2020-07-27 12:33:14 -04:00
James Rodewig 7a23c6b6ec
[DOCS] Fix formatting in simple query string query docs (#60226)
Co-authored-by: Ulas Keles <ulaskeles@users.noreply.github.com>
2020-07-27 12:20:02 -04:00
Tim Brooks df0f68da23
Identify the operation type in rejected exception (#60138)
Currently, we do not categorize the operation type in the rejection
exception messsage when we reject an indexing operation for indexing
memory limits. This commit fixes this to ensure that it is identified as
coordinating, primary, or replica.
2020-07-27 10:09:46 -06:00
James Rodewig dbd7e7793a
[DOCS] Fix default gap policy for moving fn, moving avg aggs (#60223) 2020-07-27 12:08:33 -04:00
Tim Brooks 47922c9e4a
Fix indexing pressure replica rejections logic (#60150)
Currently the logic to rejection replica rejections is evaluate before
adding the additional bytes of the current operation. This means that
the first replica operation which should be rejected will be allowed to
proceed. This commit fixes this logic and adds unit level test to ensure
indexing pressure behavior is correct.
2020-07-27 10:00:01 -06:00
Dan Hermann b98caf58ee
Mark data stream APIs as stable (#59860) (#60206) 2020-07-27 10:37:52 -05:00
lcawl a27d630bdf [DOCS] Removes coming tag 2020-07-27 07:55:18 -07:00
James Rodewig 747f8bfe79
[DOCS] Add Kibana screenshots to data stream docs (#60118) (#60217) 2020-07-27 10:39:32 -04:00
Benjamin Trent ea3c49979e
Test mute for issue 60212 (#60214) 2020-07-27 10:10:40 -04:00
James Rodewig 608a5b9e71
[DOCS] Clarify compatibility for upgrade via reindex (#60045) (#60209)
Co-authored-by: Inbar Shimshon <inbar.shimshon@elastic.co>
2020-07-27 09:38:39 -04:00
Nik Everett a451dd87aa
Reduce merge map memory overhead in the Variable Width Histogram Aggregation (#59366) (#60171)
When a document which is distant from existing buckets gets collected, the
`variable_width_histogram` will create a new bucket and then insert it
into the ordered list of buckets.

Currently, a new merge map array is created to move this bucket. This is
very expensive as there might be thousands of buckets.

This PR creates `mergeBuckets(UnaryOperator<Long> mergeMap)` methods in
`BucketsAggregator` and `MergingBucketsDefferingCollector`, and updates
the `variable_width_histogram`  to use them. This eliminates the need to
create an entire merge map array for each new bucket and reduces the
memory overhead of the algorithm.

Co-authored-by: James Dorfman <jamesdorfman@users.noreply.github.com>
2020-07-27 09:23:06 -04:00
Hendrik Muhs 95c99ca887 [Transform] Fix Regression: continuous transform can fail for (date) histogram group_by(#60196)
do not create change collector if group_by configuration does not support change detection

fixes #60125
2020-07-27 14:50:03 +02:00
James Rodewig 08e11814c3
[DOCS] Fix clarity of 7.6 derived key breaking change (#60154) 2020-07-27 08:36:48 -04:00