Commit Graph

54207 Commits

Author SHA1 Message Date
István Zoltán Szabó aeb955e07e
[DOCS] Adds note box about mappings to Transforms Painless examples (#64285) (#64363) 2020-10-29 17:35:38 +01:00
István Zoltán Szabó d007255e8f
[DOCS] Fixes typos in Transform Painless examples. (#64352) (#64358) 2020-10-29 16:58:19 +01:00
James Rodewig 68d72f1e01
[DOCS] Fix "the the" typos (#64344) (#64354) 2020-10-29 10:47:42 -04:00
James Rodewig 135a032fab
[DOCS] Fix whitespace in pattern replace token filter docs (#64345) (#64350) 2020-10-29 10:42:02 -04:00
Armin Braun fada4a1c78
Fix CachedBlobContainerIndexInputTests (#64239) (#64348)
Closing the input stream happens on a separate thread
now that the `CacheFile` is implemented in a lock-free
fashion. 

Closes #64215
2020-10-29 15:11:39 +01:00
James Rodewig a2b18e9ab9
[DOCS] Fix case for 'Boolean' (#64299) (#64342) 2020-10-29 10:05:57 -04:00
James Rodewig 9df7c943cb
[DOCS] Re-add sparse vector xref (#64307) (#64338) 2020-10-29 09:29:27 -04:00
Andrei Stefan a6d8319231
* Wrap a verification_exception in case there is no valid index available (#64267)
Wrap a verification_exception in case there is no valid index available in an index_not_found_exception providing also the original index pattern that may be lost in the chain of filters involving the Security one.

(cherry picked from commit 9c9da2f2f9a4ad12704f7d3a273f067e96cd2054)
2020-10-29 10:14:50 +02:00
debadair 536e100125
[DOCS] Add top-level Data management section. (#64185) (#64322)
* [DOCS] Add top-level Data management section.

* Edits

* Edits

* Fixed xrefs

* Apply suggestions from code review

Co-authored-by: Andrei Dan <andrei.dan@elastic.co>
Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>

* Update docs/reference/datatiers.asciidoc

* Update docs/reference/datatiers.asciidoc

Co-authored-by: Andrei Dan <andrei.dan@elastic.co>
Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>

Co-authored-by: Andrei Dan <andrei.dan@elastic.co>
Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>
2020-10-28 16:39:35 -07:00
debadair 9aabf3a50d
[DOCS] Add searchable snapshots topic. (#63040) (#64320)
* [DOCS] Add searchable snapshots topic. (#63040)

* [DOCS] Add searchable snapshots topic.

* [DOCS] Add definitions & remove fully-remote storage.

* [DOCS] Fixed duplicate anchor.

* Expand conceptual docs for searchable snapshots

* Rewordings

* Glossary tidy-up

* Beta

* Reword

* More performance idea to a TIP

* use -> manage

* red -> not green

* Missing space?

* Update docs/reference/glossary.asciidoc

* Fix beta label

* Use more attributes, fix link titles

* Apply suggestions from code review

Co-authored-by: debadair <debadair@elastic.co>

* Reformat

* Minor rewordings

* More minor rewordings

* Address Henning's comments

Co-authored-by: David Turner <david.turner@elastic.co>
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>

* Fixed glossary entries

Co-authored-by: David Turner <david.turner@elastic.co>
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
2020-10-28 16:18:08 -07:00
Jason Tedor 827dd39a12
Filter node.roles setting in transport client (#64276)
This commit filters out the node.roles setting from the transport
client, since the transport client does not take on these roles.
2020-10-28 16:24:14 -04:00
Mark Vieira 13fb6dbe4b Fix BWC distribution resolution errors (#64310)
(cherry picked from commit e6ffe9e7b73f7f4dd28ec1059bb9651a98d8f5f5)
2020-10-28 12:50:56 -07:00
James Rodewig e7f2beae05
[DOCS] Sort field data types in docs (#64288) (#64305)
Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
2020-10-28 13:54:12 -04:00
James Rodewig 0980b88e80
[DOCS] Fix `refresh` def in `update_by_query` docs (#64277) (#64296)
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>

Co-authored-by: Rudolf Meijering <skaapgif@gmail.com>
2020-10-28 13:52:35 -04:00
James Rodewig 41db9f99da
[DOCS] Fix glossary formatting (#64118) (#64293) 2020-10-28 11:21:00 -04:00
Jason Tedor 4b4c88496a
Fix checkstyle violation in ESTestCase
This commit fixes a line-length violation in ESTestCase.
2020-10-28 10:45:53 -04:00
James Rodewig 22e931ed72
[DOCS] EQL: Fix operator docs (#64286) (#64290) 2020-10-28 10:44:22 -04:00
Jason Tedor 5d42c2b06e
Deprecate the no-jdk distributions (#64275)
This commit adds logging to indicate that the no-jdk distributions are
deprecated and will be removed in a future release.
2020-10-28 10:35:23 -04:00
Jason Tedor 72009a9edd
Clarify preferred method for setting heap size (#64256)
This commit clarifies that the preferred method for setting the heap
size is via jvm.options.d and that using the ES_JAVA_OPTS environment
variable is discouraged for production deployments.
2020-10-28 08:50:49 -04:00
Armin Braun 6bd8f079a7
Enhance CacheFile#invariant Assertion (#64272) (#64280)
Follow up to #64180 tightening the assertion further.
2020-10-28 13:43:39 +01:00
Nik Everett 0c47d49784
Make sure non-collecting aggs include sub-aggs (backport of #64214) (#64247)
Now that we're consistently using `cat_match` to filter which shards we
run on we can get this confusing case:
1. You have a search with, say, a range and a sub-agg.
2. That search has a query that `can_match` can recognize will match no
   docs. On *any* shard.
3. So we dutifully run it on a single shard so it can produce the
   "empty" aggs.
4. The shard we pick happens to not have the target of the range mapped.
5. This kicks in the special range aggregator that doesn't collect any
   documents.
6. Before this commit, that range aggregator *also* never produced any
   sub-aggs.

So, without this change, it was quite possible for a search that
happened to match no documents to "throw away" the sub-aggs of a range
and a few other aggs.

We've had this problem for a long, long time but it is more confusing
now because `can_match` is really kicking in and causing us to see cases
where it looks like you are targeting a lot of shards but you really are
only targeting a couple. It used to be that to get the "no sub-aggs"
behavior you had to explicitly target only shards that didn't map the
target field of the `range` agg. And, like, in that case it isn't too
bad because you targeted a sort of degenerate shard. But now that
`can_match` is doing its thing you can end up with the confusing steps
above. It took me several hours to track down what what happening I know
how the individual pieces of all of this works. It took four hours to
figure out how they fit together in this case....

Anyway! This replaces all the aggregator implementations that throw out
the sub-aggregators with ones that keep them. I think this'll be less
confusing in the future.

Closes #64142
2020-10-28 08:38:05 -04:00
Jason Tedor 78c741ab32
Log whether or not we are using the bundled JDK (#64255)
This commit adds logging to indicate whether or not we are using the
bundled JDK. We distinguish between using a distribution that bundles
the JDK versus using a distribution that does not bundle the JDK.
2020-10-28 07:10:47 -04:00
Armin Braun 2983584ef6
Fix #invariant Assertion in CacheFile (#64180) (#64264)
Fix #invariant Assertion in CacheFile

closes #64141
2020-10-28 10:22:47 +01:00
Armin Braun a697d5edae
Don't Generate an Index Setting History UUID unless it's Supported (#64164) (#64213)
In 7.x we can't just by default generate this setting as it might not be
supported by data nodes that are assigned shards for an older version in mixed version
clusters.

Closes #64152
2020-10-28 09:03:09 +01:00
jimczi 2492f48375 Fix test compilation 2020-10-28 08:58:09 +01:00
Jim Ferenczi dcc433c971 Fix UOE when fetching flattened field (#64241)
The new fields option allows to fetch the value of all fields in the mapping.
However, internal fields that are used by some field mappers are also shown when
concrete fields retrieved through a pattern (`*` or `foo*`).
We have a [long term plan](https://github.com/elastic/elasticsearch/issues/63446) to hide these fields in field_caps and from pattern resolution
so this change is just a hot fix to ensure that they don't break the retrieval in the meantime.
The `flattened._keyed field will show up as an empty field when using a pattern that match the
flattened field.

Relates #63446
2020-10-28 08:49:03 +01:00
Jason Tedor dfc8ae48cc
Fix using bundled JDK detection on macOS (#64236)
This commit fixes an issue with the detection on macOS for whether or
not the bundled JDK is being used. The logic between macOS and non-macOS
is different because the JDK has a different directory structure on
macOS versus non-macOS. However, due to notarization issues, we changed
the top-level directory from jdk to jdk.app, yet never updated this
detection logic to account for that.

Ideally, we would have a packaging test that asserts that we have the
behavior here correct, and it maintains over time. Alas, we do not
currently have packaging tests on macOS.
2020-10-27 16:47:02 -04:00
Jason Tedor b46b6d5977
Fix compilation in DataTierTests.java
This commit fixes a compilation issue in DataTierTests.java that was
introduced due to language-level differences between 7.10/7.x and
master.
2020-10-27 13:04:55 -04:00
Jake Landis 0d0c42c746
[7.10] Update getting-started.asciidoc for Java version (#63106) (#64083)
Update client documentation to state "at least" Java 1.8

Co-authored-by: junmuz <mjunaidmuzammil@gmail.com>
2020-10-27 11:52:13 -05:00
Jason Tedor 04a9845a49
Adjust defaults for tiered data roles (#64015)
This commit adjusts the defaults for the tiered data roles so that they
are enabled by default, or if the node has the legacy data role. This
ensures that the default experience is that the tiered data roles are
enabled.

To fully specifiy the behavior for the tiered data roles then:
 - starting a new node with the defaults: enabled
 - starting a new node with node.roles configured: enabled if and only
   if the tiered data roles are explicitly configured, independently
   of the node having the data role
 - starting a new node with node.data enabled: enabled unless the
   tiered data roles are explicitly disabled
 - starting a new node with node.data disabled: disabled unless the
   tiered data roles are explicitly enabled
2020-10-27 12:48:31 -04:00
Costin Leau 6ca0b6ae6d EQL: Improve request logging (#64206)
Add logging to multi-search queries
Log response count

(cherry picked from commit ee9b9d58f68e2d545d5d841e2f683ec4e96f79e6)
(cherry picked from commit 02a4c6b83475cebe715311eeba123ad6fc8d6ba1)
2020-10-27 17:23:43 +02:00
Costin Leau 2363c4be4b EQL: Polish testing infra (#64205)
Add tie-breaker inside request creation
Add configurable timeout

(cherry picked from commit ff281d7b6fd7b4cd2f08bac49aa0b354b6812940)
(cherry picked from commit 34bd76fc2987b1ad0b6275ac4358e362a0ba7fb0)
2020-10-27 17:23:43 +02:00
Adam Locke ce4ec307a2
Fixing typo in file name. (#64230) 2020-10-27 11:22:44 -04:00
Henning Andersen 0cba23e08f XPack Usage should run on MANAGEMENT threads (#64160)
XPack usage starts out on management threads, but depending on the
implementation of the usage plugin, they could end up running on
transport threads instead. Fixed to always reschedule on a management
thread.
2020-10-27 16:03:26 +01:00
James Rodewig 50d806dc9f
[DOCS] Fix pattern replace token filter intro (#64189) (#64218)
Removes an incorrect statement about anchoring regex patterns on tokens.

Co-authored-by: Elasticsearch addict <amitmbm87@gmail.com>
2020-10-27 10:35:34 -04:00
Nhat Nguyen 566d1fd459 Return the same point in time in search response (#64188)
With this change, we will always return the same point in time in a
search response as its input until we implement the retry mechanism
for the point in times.
2020-10-27 10:17:44 -04:00
Armin Braun 6f1c8136a6
Fix CachedBlobContainerIndexInputTests Shutdown (#64181) (#64211)
Same problem as in #64100 we have to safely wait for all operations to go through to
not leak file handles potentially in this test.
2020-10-27 14:22:43 +01:00
David Roberts 2422f62bab
[ML] Copy ml-cpp release note into ES docs (#64198)
Copies the release note for elastic/ml-cpp#1546 to where
it needs to be to get published
2020-10-27 12:11:22 +00:00
Jim Ferenczi e34014eb6a Fix sorted query when date_nanos is used as the numeric_type (#64183)
The formatting of the global bottom value does not take the resolution of the provided
numeric_type into account. This change fixes this bug by providing the resolution
directly in the doc value format if the numeric_type is provided as `date_nanos`.

Closes #63719
2020-10-27 11:00:23 +01:00
Andrei Dan 89e3a3178f
Add indexing speed improvement to the 7.10 release highlights (#63621)
Add indexing speed improvement to the 7.10 release highlights

Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
2020-10-26 15:11:40 +00:00
Armin Braun e02561476e
Fix Broken Clone Snapshot CS Update (#64116) (#64159)
We must not remove the snapshot from the initializing set
in the `timeout` getter. This was a plain oversight/mistake
and went unnoticed. It can lead to the removal of a valid
snapshot clone from the cluster state in rare circumstances
(e.g. when a node concurrently joins the cluster or a routing
change happens as it did in the linked test failure).

Closes #64115
2020-10-26 14:32:42 +01:00
Armin Braun 93b52df8c1
Fix SearchableSnapshotDirectoryTests Leaking Listeners (#64150) (#64156)
We need to make sure we don't shut down the pool while ref count release
operations are enqueued but not yet executing.
2020-10-26 13:46:14 +01:00
David Kyle 4545779415 Mute MixedClusterClientYamlTestSuiteIT 'Create a snapshot and then restore it' (#64153)
For #64152
2020-10-26 11:58:17 +00:00
David Kyle 51f8b0b8d2 Mute MonitoringWithWatcherRestIT.testThatLocalExporterAddsWatche (#64143) 2020-10-26 10:19:48 +00:00
Armin Braun 8161513cb6
Fix Expected Exception Check in BlobstoreCacheService (#63474) (#64135)
The `NodeNotConnectedException` exception can be nested as well in the
fairly unlikley case of the disconnect occuring between the connected check
and actually sending the request in the transport service.

Closes #63233
2020-10-26 10:43:29 +01:00
Armin Braun 17843a40ef
Fix SearchableSnapshotDirectoryTests.testClearCache (#64100) (#64132)
There is a small chance that the file deletion will run
on the searchable snapshot thread pool and not on the test
thread now that the cache is non-blocking in which case
we fail the assertion unless we wait for that thread.
2020-10-26 10:27:08 +01:00
David Roberts adc5509eda
[ML] Support the unsigned_long type in data frame analytics (#64072)
Adds support for the unsigned_long type to data frame analytics.

This type is handled in the same way as the long type.  Values
sent to the ML native processes are converted to floats and
hence will lose accuracy when outside the range where a float
can uniquely represent long values.

Backport of #64066
2020-10-26 09:05:49 +00:00
Armin Braun 96407268a0
Fix Background Merge Breaking Snapshot Restore Test (#63579) (#64129)
If we run into a background merge between creating the snapshot and closing the index
then with compound files we could be in a situation where we get zero file reuse
on restore.
Force merging before the snapshot gives us a single segment that won't change down the line
so the restore always sees file reuse from the close index.

Closes #63476
2020-10-26 09:34:43 +01:00
Armin Braun bd07e44c9a
Make Searchable Snapshot's CacheFile Lock less (#63911) (#64125)
Replacing the mechanism for eviction and listener references via a read-write lock by
a reference counting implementation.
This fixes a bug that caused test failure #63586 in which concurrently trying to acquire or release
an eviction listener while doing a file operation would sometimes lead to throwing an exception
since the `tryLock` call on the read lock would fail in this case.
Also this removes the possibility of blocking cluster state updates as a result of them waiting
on the write-lock which might take a long time if a slow read operation executes concurrently.

Closes #63586
2020-10-26 09:30:22 +01:00
Armin Braun bdea16301d
Fix testMasterFailoverDuringCloneStep1 (#63580) (#64127)
Assuming the clone failed when the request failed is not sufficient.
There are failure modes where the request fails but the clone still works out
because the data node resent the requeest after the first clone had already been
failed and removed from the cluster state when master was restarted.

Closes #63473
2020-10-26 09:30:09 +01:00