Commit Graph

53150 Commits

Author SHA1 Message Date
Julie Tibshirani fc63f8224f
Simplify class hierarchy for ordinals field data. (#60606)
This PR simplifies the hierarchy for ordinals field data classes:
* Remove `AbstractIndexFieldData`, since only `AbstractIndexOrdinalsFieldData`
inherits directly from it.
* Make `SortedSetOrdinalsIndexFieldData` extend
`AbstractIndexOrdinalsFieldData`. This lets us remove some redundant code.
2020-08-03 09:58:29 -07:00
James Rodewig c5f4f91ac4 [DOCS] Clarify reindex does not require existing dest 2020-08-03 12:46:40 -04:00
James Rodewig 5be515f126 [DOCS] Unhide EQL search in data streams docs 2020-08-03 11:59:12 -04:00
Hendrik Muhs 1e01832b0c fix possible NPE introduced in #60591 2020-08-03 16:40:38 +02:00
James Rodewig cfab3bccab
[DOCS] Replace `twitter` dataset in cat API docs (#60588) (#60597) 2020-08-03 10:22:36 -04:00
Hendrik Muhs cd6492fc11 [Transform] fix regression of date histogram optimization (#60591)
fixes mix up of input and output field name for date histogram optimization.

minimal fix, more tests to be added with #60469

fixes #60590
2020-08-03 15:52:08 +02:00
James Rodewig bffe82449a
[DOCS] Update Amazon Linux AMI link in EC2 docs (#60589) (#60593) 2020-08-03 09:32:31 -04:00
István Zoltán Szabó 7c995aa3ff [DOCS] Fixes broken links in rest API spec. (#60582) 2020-08-03 15:31:25 +02:00
Yannick Welsch 3409e019d2 Ignore shutdown when retrying recoveries (#60586)
Avoids failures when shutting down a node.
2020-08-03 15:14:38 +02:00
Nik Everett 2cde43b799
Allows nanosecond resolution in search_after (backport of #60328) (#60426)
Allows nanosecond resolution in search_after (#60328)

This fixes `search_after` to properly parse string formatted dates that
have nanosecond resolution.

Closes #52424
2020-08-03 08:17:48 -04:00
Yannick Welsch b0d601fa63 Adjust searchable snapshot license (#60578)
No longer needs Platinum license for testing on staging.
2020-08-03 13:19:53 +02:00
Yannick Welsch 9e24a54382 Clean existing index folder when loading searchable snapshot (#60122)
Closing a regular index and mounting a snapshot-backed index into that existing index does not clean the existing index
folders of those preexisting shards.

This PR removes the existing Lucene / translog files once the searchable snapshot shard is starting up. Future PRs will
make reuse of the existing index files to populate the cache.
2020-08-03 13:19:11 +02:00
David Turner d2ddf8cd6a Improve deserialization failure logging (#60577)
Today when a node fails to properly deserialize a transport message with
a parent task we log the following relatively uninformative message:

    java.lang.IllegalStateException: Message not fully read (response) for requestId [9999], handler [org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler/org.elasticsearch.transport.TransportService$6@abcdefgh], error [false]; resetting

In particular, the wrapping of the listener in the `TransportService`
obscures all clues as to the source of the problem, e.g. the action name
or the identity of the underlying listener. This commit exposes the
inner listener to the logs.

Also if the listener is wrapped with `ContextPreservingActionListener`
then its identity is similarly hidden. This commit also exposes the
wrapped listener in this case.

Relates #38939
2020-08-03 11:51:01 +01:00
Yang Wang a76fc324d4
Fix get-license test failure by ensure cluster is ready (#60498) (#60569)
When a new cluster starts, the HTTP layer becomes ready to accept incoming 
requests while the basic license is still being populated in the background. 
When a get license request comes in before the license is ready, it can get 
404 error. This PR fixes it by either wrap the license check in assertBusy or 
ensure the license is ready before perform the check.

This is a backport for both #60498 and #60573
2020-08-03 19:40:03 +10:00
Armin Braun 3270cb3088
More Efficient Writes for Snapshot Shard Generations (#60458) (#60575)
Same as #59905 but for shard level metadata. Since we wnat to retain
the ability to do safe+atomic writes for non-uuid shard generations
this PR has to create two separate write paths for both kinds of
shard generations.
2020-08-03 11:11:36 +02:00
Armin Braun 204efe9387
Add Repository Setting to Disable Writing index.latest (#60448) (#60576)
Writing the `index.latest` blob is unnecessary unless the contents of the repository
are to be used as a URL-repository. Also, in some edge cases, the fact that `index.latest` is the only
blob in the repository that regularly gets overwritten was causing compatibility issues with
some backing blobstores (Azure no-overwrite policy, Hitachy S3 equivalent).
=> this commit changes behavior to make snapshots not fail if writing `index.latest` fails
and adds a setting to disable writing `index.latest`.
2020-08-03 11:11:24 +02:00
Martijn van Groningen e76447833a
adjusted bwcVersions file 2020-08-03 10:58:24 +02:00
Tim Vernum 1a373b0c21
Only call listener once (SP template registration) (#60567)
This fixes a bug in the IdP's template registration that would
sometimes call the listener twice.

Resolves: #54285
Resolves: #54423

Backport of: #60497
2020-08-03 13:45:16 +10:00
Andrei Dan ac258f10d6
Data streams: throw ResourceAlreadyExists exception (#60518) (#60536)
For consistency reasons (and reducing the overload of IllegalArgumentException)
this changes the exception thrown when trying to create a data stream
that already exists.

(cherry picked from commit ac2184c4614bba0f3ee377da49aea0daed98bab4)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
2020-08-01 16:31:09 +01:00
James Rodewig fcc53d9e0e
[DOCS] Note refresh requests are synchronous (#60540) (#60550) 2020-07-31 16:22:34 -04:00
James Rodewig 5a2c6f0d4f
[DOCS] http -> https, remove outdated plugin docs (#60380) (#60545)
Plugin discovery documentation contained information about installing
Elasticsearch 2.0 and installing an oracle JDK, both of which is no
longer valid.

While noticing that the instructions used cleartext HTTP to install
packages, this commit replaces HTTPs links instead of HTTP where possible.

In addition a few community links have been removed, as they do not seem
to exist anymore.

Co-authored-by: Alexander Reelsen <alexander@reelsen.net>
2020-07-31 16:16:31 -04:00
James Rodewig fb599dc343
[DOCS] Add rollups to `Tune for disk usage` (#60436) (#60542)
Co-authored-by: Leaf-Lin <39002973+Leaf-Lin@users.noreply.github.com>
2020-07-31 16:10:57 -04:00
James Rodewig 19c12cc896
[DOCS] Fix merge conflict in `CONTRIBUTING.md` (#60537) 2020-07-31 16:04:18 -04:00
Armin Braun 8c7eae15ba
Increase Timeout in testSnapshotRestore (#60532) (#60538)
It seems this test only fails with `FsRepository` and mostly just barely times out (takes just a little over 30s to go green).
I think just increasing the timeout should be fine as a fix here as it's a little interesting to check larger amounts of
data in this test generally speaking.

Closes #39299
2020-07-31 21:53:58 +02:00
Julie Tibshirani f1d4fd8c3e Correct name of IndexFieldData#loadGlobalDirect. (#60492)
It seems 'localGlobalDirect' was just a typo.
2020-07-31 10:53:21 -07:00
James Rodewig 771e9f142a
[DOCS] Move search pagination content to one page (#60515) (#60525) 2020-07-31 12:40:40 -04:00
James Rodewig 9eba7f39b0
[DOCS] Replace `twitter` dataset in docs APIs (#60521) (#60529) 2020-07-31 12:40:03 -04:00
Julie Tibshirani fe1d877b69 Avoid using string 'y' in fields REST tests. (#60471)
Some yaml parsers interpret 'y' and 'yes' as the boolean 'true'.
2020-07-31 09:12:15 -07:00
Jim Ferenczi 8db896d290 Fix race condition in SearchPhaseControllerTests#testPartialMergeFailure (#60488)
This change ensures that we call the listener for partial merge failure **before**
calling the completion listener in order to avoid race condition in tests.

Closes #60446
2020-07-31 16:29:20 +02:00
James Rodewig 4b12e69e8e
[DOCS] Replace `twitter` dataset in index API docs (#60473) (#60510) 2020-07-31 09:51:47 -04:00
Dan Hermann bd6b20a4ec
Fix failing test for resolve index API (#60306) (#60508) 2020-07-31 08:43:24 -05:00
Rene Groeschke ed4b70190b
Replace immediate task creations by using task avoidance api (#60071) (#60504)
- Replace immediate task creations by using task avoidance api
- One step closer to #56610
- Still many tasks are created during configuration phase. Tackled in separate steps
2020-07-31 13:09:04 +02:00
Hendrik Muhs a721d6d19b [Transform] use correct version in BWC serialization test (#60500)
use correct version in BWC serialization test

fixes #60464
2020-07-31 11:23:05 +02:00
Rene Groeschke 79ec0f2442
Split internal distribution handling into separate internal plugin (7.x backport) (#60270)
* Split internal distribution handling into separate internal plugin (#60295)
* Provide proper failure if unexpected non jdk bundled bwc version is requested
2020-07-31 09:30:21 +02:00
Julie Tibshirani 8ac81a3447 Remove IndexFieldData#clear since it is unused. (#60475)
This method was never called. It also seemed tricky that calling a method on
`IndexFieldData` could clear the contents of a shared cache.
2020-07-30 14:07:55 -07:00
James Rodewig 0022d316bb
[DOCS] Merge search topic and overview pages (#60459) (#60479) 2020-07-30 16:45:18 -04:00
James Rodewig 134b69d3aa
[DOCS] Fix `template` param in put index template API (#60474) (#60476) 2020-07-30 16:44:50 -04:00
Lisa Cawley 9425de8dd1
[DOCS] Clarify support for run as in OIDC realms (#60246) (#60485) 2020-07-30 13:39:41 -07:00
Julie Tibshirani dfd7f226f0
Clarify SourceLookup sharing across fetch subphases. (#60484)
The `SourceLookup` class provides access to the _source for a particular
document, specified through `SourceLookup#setSegmentAndDocument`. Previously
the search context contained a single `SourceLookup` that was shared between
different fetch subphases. It was hard to reason about its state: is
`SourceLookup` set to the expected document? Is the _source already loaded and
available?

Instead of using a global source lookup, the fetch hit context now provides
access to a lookup that is set to load from the hit document.

This refactor closes #31000, since the same `SourceLookup` is no longer shared
between the 'fetch _source phase' and script execution.
2020-07-30 13:22:31 -07:00
Dan Hermann da69644498
Ignore template warnings in IndexingIT (#59962) (#60482) 2020-07-30 14:54:30 -05:00
James Rodewig 2d2b74dd32
[DOCS] Note remote reindex is not fwd compatible (#60425) (#60454) 2020-07-30 09:23:55 -04:00
James Rodewig b17ae33b3a
[DOCS] Move field collapse content to separate page (#60424) (#60451) 2020-07-30 09:19:05 -04:00
James Rodewig b3da548dab [DOCS] Fix awkward wording in kuromoji analyzer docs 2020-07-30 08:56:59 -04:00
Mark Tozzi 970a0c8957
[7.x] Aggregation tests for Wildcard Field (#58507) (#60423) 2020-07-30 08:56:21 -04:00
Dan Hermann 5e5503ac28
Change severity of negative stats messages from WARN to DEBUG (#60375) (#60444) 2020-07-30 06:06:13 -05:00
Armin Braun 3bf4c01d8e
Don't Allocate Redundant Pages in BigArrays (#60201) (#60441)
The oversize algorithm was allocating more pages than necessary to accommodate `minTargetSize`.
An example would be that a 16k page size and 15k `minTargetSize` would result in a new size of 32k (2 pages).
The difference between the minimum number of necessary pages and the estimated size then keeps growing as sizes increase.

I don't think there is much value in preemptively allocating pages by over-sizing aggressively since the behavior of
the system is quite different from that of a single array where over-sizing avoids copying
once the minimum target size is more than a single page.

Relates #60173 which lead me to this when `BytesStreamOutput` would allocate a large number of never used
pages during serialization of repository metadata.
2020-07-30 11:09:58 +02:00
Armin Braun a2c49a4f02
Reduce Heap Use during Shard Snapshot (#60370) (#60440)
Instances of `BlobStoreIndexShardSnapshots` can be of non-trivial size. In case of snapshotting a larger
number of shards the previous execution order would lead to memory use proportional to the number of shards
for these objects. With this change, the number of these objects on heap is bounded by the size of the snapshot
pool (except for in the BwC format path).
This PR makes it so that they are written to the repository at the earliest possible point in time
so that they can be garbage collected.
If shard generations are used, we can safely write these right at the beginning of the shard snapshot.
If shard generations are not used we can only write them at the end of the shard snapshot after all
other blobs have been written.

Closes #60173
2020-07-30 10:45:00 +02:00
Przemysław Witek 9e27f7474c
Make MlDailyMaintenanceService delete jobs that are in deleting state anyway (#60121) (#60439) 2020-07-30 09:53:11 +02:00
Hendrik Muhs aaed6b59d6
[7.x][Transform] add support for missing bucket (#59591) (#60390)
add support for "missing_bucket" in group_by

fixes #42941
fixes #55102
backport #59591
2020-07-30 08:26:51 +02:00
Igor Motov 00a1949852
Streamline GeoJSON to map serialization (#60413) (#60429)
Optimizes GeoJSON to map serialization when retrieving spatial data through
fields.

Closes #60259
2020-07-29 17:56:56 -04:00