Commit Graph

53753 Commits

Author SHA1 Message Date
Rene Groeschke f08981c4da
Simplify distribution download configurations (7.x backport) (#61708)
We leverage artifact transforms now when downloading and unpacking elasticsearch distributions.

This has the benefit of

- handcrafted extract tasks on the root project are not required.
- The general tight coupling to the root project has been removed.
- The overall required configurations required to handle a distribution have been reduced
- ElasticsearchDistribution has been simplified by making Extracted an ordinary Configuration
downloaded and unpacked external distributions are reused in later builds by been cached
in the gradle user home.

DistributionDownloadPlugin functional tests have been extended and ported
to DistributionDownloadPluginFuncTest.

* Fix ElasticsearchNode#getDistributionFiles (#61219)

Fixes #61647
2020-08-31 14:36:42 +02:00
Rory Hunter ff6c071275
Implement deprecation logging using log4j (#61629)
Backport of #61474.

Part of #46106. Simplify the implementation of deprecation logging by
relying of log4j more completely, and implementing additional behaviour
through custom appenders and filters.
2020-08-31 12:42:04 +01:00
Henning Andersen 4c9fe31da8 Mute testTooLowConfiguredMemoryStillStarts (#61705)
Related to #61704
2020-08-31 11:19:53 +02:00
Rene Groeschke d1c196a1b2
Minor cleanup 2020-08-31 11:11:32 +02:00
Rene Groeschke 0d488cfbb9
Fix bwc testcluster setup against elastic < 6.3 2020-08-31 11:07:54 +02:00
Armin Braun 5c86b216e8
Fix Race in testGetSnapshotsRequest (#61694) (#61700)
The fact that the data node is already blocked on writing
data files did not guarantee that the cluster state that made
the data node start snapshotting is already applied on master.
This could lead to races where the get snapshots action still
runs based on a state without the snapshot in it, tripping the assertion.
Much safer to handle this by waiting on the non-blocking snapshot create
to return, which guarantees that the CS has been applied on master.

Closes #61541
2020-08-31 11:06:51 +02:00
Armin Braun 0da20579ca
Cleanly Handle S3 SDK Exceptions in Request Counting (#61686) (#61698)
It looks like it is possible for a request to throw an exception early
before any API interaciton has happened. This can lead to the request count
map containing a `null` for the request count key.
The assertion is not correct and we should not NPE here
(as that might also hide the original exception since we are running this code in
a `finally` block from within the S3 SDK).

Closes #61670
2020-08-31 11:05:59 +02:00
Armin Braun 22e4d759c3
Speed up Reading Enum Set from Stream (#61678) (#61687)
No need in adding enum values to a normal set and then copying, the `EnumSet` is directly mutable just fine.
2020-08-30 20:49:51 +02:00
Ioannis Kakavas c621d291d2
Call ActionListener.onResponse exactly once (#61584) (#61682)
Under specific circumstances we would call onResponse twice, which led to unexpected behavior.
2020-08-30 16:47:09 +03:00
Jake Landis d2e5f2f532
[7.x] Enhance the ingest node simulate verbose output (#60433) (#60678)
This commit enhances the verbose output for the
`_ingest/pipeline/_simulate?verbose` api. Specifically
this adds the following:
* the pipeline processor is now included in the output
* the conditional (if) and result is now included in the output iff it was defined
* a status field is always displayed. the possible values of status are
  * `success` - if the processor ran with out errors
  * `error` - if the processor ran but threw an error that was not ingored
  * `error_ignored` - if the processor ran but threw an error that was ingored
  * `skipped` - if the process did not run (currently only possible if the if condition evaluates to false)
  * `dropped` - if the the `drop` processor ran and dropped the document
* a `processor_type` field for the type of processor (e.g. set, rename, etc.)
* throw a better error if trying to simulate with a pipeline that does not exist

closes #56004
2020-08-27 16:53:09 -05:00
Lee Hinman 1bfebd54ea
[7.x] Allocate newly created indices on data_hot tier nodes (#61342) (#61650)
This commit adds the functionality to allocate newly created indices on nodes in the "hot" tier by
default when they are created.

This does not break existing behavior, as nodes with the `data` role are considered to be part of
the hot tier. Users that separate their deployments by using the `data_hot` (and `data_warm`,
`data_cold`, `data_frozen`) roles will have their data allocated on the hot tier nodes now by
default.

This change is a little more complicated than changing the default value for
`index.routing.allocation.include._tier` from null to "data_hot". Instead, this adds the ability to
have a plugin inject a setting into the builder for a newly created index. This has the benefit of
allowing this setting to be visible as part of the settings when retrieving the index, for example:

```
// Create an index
PUT /eggplant

// Get an index
GET /eggplant?flat_settings
```

Returns the default settings now of:

```json
{
  "eggplant" : {
    "aliases" : { },
    "mappings" : { },
    "settings" : {
      "index.creation_date" : "1597855465598",
      "index.number_of_replicas" : "1",
      "index.number_of_shards" : "1",
      "index.provided_name" : "eggplant",
      "index.routing.allocation.include._tier" : "data_hot",
      "index.uuid" : "6ySG78s9RWGystRipoBFCA",
      "index.version.created" : "8000099"
    }
  }
}
```

After the initial setting of this setting, it can be treated like any other index level setting.

This new setting is *not* set on a new index if any of the following is true:

- The index is created with an `index.routing.allocation.include.<anything>` setting
- The index is created with an `index.routing.allocation.exclude.<anything>` setting
- The index is created with an `index.routing.allocation.require.<anything>` setting
- The index is created with a null `index.routing.allocation.include._tier` value
- The index was created from an existing source metadata (shrink, clone, split, etc)

Relates to #60848
2020-08-27 13:41:12 -06:00
Albert Zaharovits 1cb97a2c4f
Relax the index access control check for scroll searches (#61446)
The check introduced by #60640 for scroll searches, in which we log
if the index access control before the query and fetch phases differs
from when the scroll context is created, is too strict, leading to spurious
warning log messages.
The check verifies instance equality but this assumes that the fetch
phase is executed in the same thread context as the scroll context
validation. However, this is not true if the scroll search is executed
cross-cluster, and even for local scroll searches it is an unfounded assumption.

The check is hence reduced to a null check for the index access.
The fact that the access control is suitable given the indices that
are actually accessed (by the scroll) will be done in a follow-up,
after we better regulate the creation of index access controls in general.
2020-08-27 21:16:01 +03:00
James Rodewig 0407f1d19b
[DOCS] Change 'data type' to 'field type' (#61633) (#61635) 2020-08-27 12:47:28 -04:00
Luca Cavanna f769821bc8
Pass SearchLookup supplier through to fielddataBuilder (#61430) (#61638)
Runtime fields need to have a SearchLookup available, when building their fielddata implementations, so that they can look up other fields, runtime or not.

To achieve that, we add a Supplier<SearchLookup> argument to the existing MappedFieldType#fielddataBuilder method.

As we introduce the ability to look up other fields while building fielddata for mapped fields, we implicitly add the ability for a field to require other fields. This requires some protection mechanism that detects dependency cycles to prevent stack overflow errors.

With this commit we also introduce detection for cycles, as well as a limit on the depth of the references for a runtime field. Note that we also plan on introducing cycles detection at compile time, so the runtime cycles detection is a last resort to prevent stack overflow errors but we hope that we can reject runtime fields from being registered in the mappings when they create a cycle in their definition.

Note that this commit does not introduce any production implementation of runtime fields, but is rather a pre-requisite to merge the runtime fields feature branch.

This is a breaking change for MapperPlugins that plug in a mapper, as the signature of MappedFieldType#fielddataBuilder changes from taking a single argument (the index name), to also accept a Supplier<SearchLookup>.

Relates to #59332

Co-authored-by: Nik Everett <nik9000@gmail.com>
2020-08-27 18:09:56 +02:00
Sylvain Wallez 05aaa2efdc
[client] Handle non-default port in Cloud-Id (#61634)
The domain part of a Cloud-Id can contain an optional custom port, e.g.
cloud.example.org:9443. This feature is used for Elastic Cloud
Enterprise installations that can't use the default port 443.

This change fixes RestClient.build() to correctly handle custom ports.
2020-08-27 17:51:12 +02:00
Nik Everett 5a83e89a2b
Migrate histogram field test (#61602) (#61632)
Replaces the superclass of the test for `HistogramFieldMapperTests` with
one that doesn't extend `ESSingleNodeTestCase` so we don't depend on the
entire world to test the field mapper.

Continues #61301.
2020-08-27 11:08:19 -04:00
David Turner c89fb8b9fa Avoid listener call under SparseFileTracker#mutex (#61626)
Today we sometimes notify a listener of completion while holding
`SparseFileTracker#mutex`. This commit move all such calls out from
under the mutex and adds assertions that the mutex is not held in the
listener.

Closes #61520
2020-08-27 15:39:38 +01:00
Alan Woodward b6cb590685 Log more information when mappings fail on index creation (#61577)
Errors from bad mappings at index creation are currently logged at DEBUG level, which
can make it difficult to work out what's going on if the index is being auto-created. This
commit ups the log level to INFO for auto-created indices, and includes some more
information in the log message.
2020-08-27 15:08:51 +01:00
Rene Groeschke 4540610b62
Fix jvm options handling in testclusters < 6.2 2020-08-27 16:03:57 +02:00
Rene Groeschke c4773a4949
Remove debug println from build logic 2020-08-27 15:47:14 +02:00
Rene Groeschke fa7f1cc979
Update gradle wrapper to 6.6.1 (#61571) (#61579) 2020-08-27 14:16:53 +02:00
Armin Braun 9be36865ef
Speed up XContent Collection Parsing (#61442) (#61617)
1. Get rid of the capturing lambda on the hot path that inlines very badly
2. Remove as many bounds checks as possible, thereby reducing method size and improving inlining
2020-08-27 14:15:46 +02:00
Rene Groeschke ae90ae343a
Fixing jvm options for testcluster for older versions (#61620)
* Fixing jvm options for testcluster for older versions
2020-08-27 14:06:14 +02:00
Rene Groeschke c4a0c603f0
Fix normalizing build output for windows (#61622) (#61625)
- handle windows paths better
Fixes #61612
2020-08-27 14:05:22 +02:00
David Kyle 49a5afc6c1
[ML] Increase wait for templates timeout in tests (#61623) (#61628) 2020-08-27 12:57:12 +01:00
David Turner 411965d392 Allow background cluster state update in tests (#61455)
Today the `CoordinatorTests` run the publication process as a single
atomic action; however in production it appears possible that another
master may be elected, publish its state, then fail, then we win another
election, all in between the time we sampled our previous cluster state
and started to publish the one we first thought of.

This violates the `assertClusterStateConsistency()` assertion that
verifies the cluster state update event matches the states we actually
published and applied.

This commit adjusts the tests to run the publication process more
asynchronously so as to allow time for this behaviour to occur. This
should eventually result in a reproduction of the failure in #61437 that
will let us analyse what's really going on there and help us fix it.
2020-08-27 11:22:58 +01:00
David Kyle 25e811ced7
Rewrite Inference yml tests for better clean up (#61180) (#61555)
Inference processors asynchronously usage write stats to the .ml-stats index after they used. 
In tests the write can leak into the next test causing failures depending on which test follows.
This change waits for the usage stats docs to be written at the end of the test
2020-08-27 11:16:26 +01:00
David Turner f6055dc9b2 Suppress noisy SSL exceptions (#61359)
If a TLS-protected connection closes unexpectedly then today we often
emit a `WARN` log, typically one of the following:

    io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)

    io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: Received close_notify during handshake

We typically only report unexpectedly-closed connections at `DEBUG`
level, but these two messages don't follow that rule and generate a lot
of noise as a result. This commit adjusts the logging to report these
two exceptions at `DEBUG` level only.
2020-08-27 10:59:39 +01:00
David Turner b866aaf81c Use int for number of parts in blob store (#61618)
Today we use `long` to represent the number of parts of a blob. There's
no need for this extra range, it forces us to do some casting elsewhere,
and indeed when snapshotting we iterate over the parts using an `int`
which would be an infinite loop in case of overflow anyway:

    for (int i = 0; i < fileInfo.numberOfParts(); i++) {

This commit changes the representation of the number of parts of a blob
to an `int`.
2020-08-27 10:54:03 +01:00
Ioannis Kakavas aac9eb6b64
Kerberos doc kibana link (#61466) (#61619)
Add a note in Kerberos documentation that Kibana requires a
configuration change too, and link to that documentation page.
2020-08-27 12:42:52 +03:00
Ioannis Kakavas 3640ff1ff2
Add SAML AuthN request signing tests (#61582)
- Add a unit test for our signing code
- Change SAML IT to use signed authentication requests for Shibboleth to consume

Backport of #48444
2020-08-27 10:41:56 +03:00
David Turner 5df74cc888 Replace Math.toIntExact with toIntBytes (#61604)
We convert longs to ints using `Math.toIntExact` in places where we're
sure there will be no overflow, but this doesn't explain the intent of
these conversions very well. This commit introduces a dedicated method
for these conversions, and adds an assertion that we never overflow.
2020-08-27 08:28:54 +01:00
David Turner e14d9c9514
Introduce cache index for searchable snapshots (#61595)
If a searchable snapshot shard fails (e.g. its node leaves the cluster)
we want to be able to start it up again on a different node as quickly
as possible to avoid unnecessarily blocking or failing searches. It
isn't feasible to fully restore such shards in an acceptably short time.
In particular we would like to be able to deal with the `can_match`
phase of a search ASAP so that we can skip unnecessary waiting on shards
that may still be warming up but which are not required for the search.

This commit solves this problem by introducing a system index that holds
much of the data required to start a shard. Today(*) this means it holds
the contents of every file with size <8kB, and the first 4kB of every
other file in the shard. This system index acts as a second-level cache,
behind the first-level node-local disk cache but in front of the blob
store itself. Reading chunks from the index is slower than reading them
directly from disk, but faster than reading them from the blob store,
and is also replicated and accessible to all nodes in the cluster.

(*) the exact heuristics for what we should put into the system index
are still under investigation and may change in future.

This second-level cache is populated when we attempt to read a chunk
which is missing from both levels of cache and must therefore be read
from the blob store.

We also introduce `SearchableSnapshotsBlobStoreCacheIntegTests` which
verify that we do not hit the blob store more than necessary when
starting up a shard that we've seen before, whether due to a node
restart or because a snapshot was mounted multiple times.

Backport of #60522

Co-authored-by: Tanguy Leroux <tlrx.dev@gmail.com>
2020-08-27 06:38:32 +01:00
Ryan Ernst e60c74240a
Add base precommit task to all java projects (#61439)
This commit adds java compilation to the base precommit task, and adds
that to the java plugin. This further reduces dependence on the build
plugin.
2020-08-26 17:21:00 -07:00
Lisa Cawley 6d6f5d4acc [DOCS] Per-partition categorization (#61506) 2020-08-26 17:10:01 -07:00
James Rodewig 580ef8eb0c
[DOCS] Document static field cache settings (#61424) (#61606) 2020-08-26 17:29:15 -04:00
Jason Tedor 9840fd1485
Add Lucene 8.6.0 memory leak as a known issue (#61603)
This commit adds a note to the known issues docs that Lucene 8.6.0
contains a memory leak that manifests in Elasticsearch as a slow memory
leak.
2020-08-26 15:45:14 -04:00
James Rodewig 462754e4e6
[DOCS] Reorg field data types page (#61117) (#61599) 2020-08-26 14:24:09 -04:00
James Rodewig 8a6ecd5bfc [DOCS] Fix EQL syntax admon 2020-08-26 13:39:42 -04:00
James Rodewig 20053bfd8c [DOCS] Remove dupe EQl fn/pipe TOC 2020-08-26 12:45:09 -04:00
Jay Modi 34c4fc3b91
Remove tasks module to define tasks system index (#61588)
This commit removes the tasks module that only existed to define the
tasks result index, `.tasks`,  as a system index. The definition for
the tasks results system index descriptor is moved to the
`SystemIndices` class with a check that no other plugin or module
attempts to define an entry with the same source.

Additionally, this change also makes the pattern for the tasks result
index a wildcard pattern since we will need this when the index is
upgraded (reindex to new name and then alias that to .tasks).

Backport of #61540
2020-08-26 09:48:23 -06:00
David Turner f2dc664228 Remove dead code in EsExecutors (#61574)
Removes a couple of unused methods.
2020-08-26 16:08:36 +01:00
Dimitris Athanasiou 3ed65eb418
[7.x][ML] Recover data frame extraction search from latest sort key (#61544) (#61572)
If a search failure occurs during data frame extraction we catch
the error and retry once. However, we retry another search that is
identical to the first one. This means we will re-fetch any docs
that were already processed. This may result either to training
a model using duplicate data or in the case of outlier detection to
an error message that the process received more records than it
expected.

This commit fixes this issue by tracking the latest doc's sort key
and then using that in a range query in case we restart the search
due to a failure.

Backport of #61544

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-08-26 17:54:00 +03:00
Benjamin Trent a6e7a3d65f
[7.x] [ML] write warning if configured memory limit is too low for analytics job (#61505) (#61528)
Backports the following commits to 7.x:

[ML] write warning if configured memory limit is too low for analytics job (#61505)

Having `_start` fail when the configured memory limit is too low can be frustrating. 

We should instead warn the user that their job might not run properly if their configured limit is too low. 

It might be that our estimate is too high, and their configured limit works just fine.
2020-08-26 10:35:38 -04:00
Przemyslaw Gomulka 9f566644af
Do not create two loggers for DeprecationLogger backport(#58435) (#61530)
DeprecationLogger's constructor should not create two loggers. It was
taking parent logger instance, changing its name with a .deprecation
prefix and creating a new logger.
Most of the time parent logger was not needed. It was causing Log4j to
unnecessarily cache the unused parent logger instance.

depends on #61515
backports #58435
2020-08-26 16:04:02 +02:00
Rene Groeschke 3a8cfdc1f5
Extract distribution archive checks into plugin (7.x backport) (#61567)
- Added test coverage
- Removes build script cluttering
- Splits archive building and archive checking logic
- only rely on boost for now for ML licenses(tbd)
- Use Gradle build-in untar and unzip support

* Handle dynamic versions in func tests assertions
2020-08-26 15:04:12 +02:00
Rene Groeschke fac66a7528
Rework test cluster distribution handling (#61407) (#61566)
Driven by this issue 
https://github.com/elastic/elasticsearch/pull/60969#issuecomment-674962158
we apply some rework on how we handle distributions in our test cluster setups:

- If no custom modules, plugins or extra jar files are declared we do not create a cluster
specific distro folder and use the origin distribution folder instead.
- If a custom distribution folder is required, we fallback to file copy when hard linking
is not supported
2020-08-26 15:03:52 +02:00
Ioannis Kakavas 283eaabc71
[7.x] Refactor SamlAuthenticationIT (#57162) (#61568)
Refactor the tests to not require a mock HTTP Server. This has been
the cause of flakiness and removing it doesn't affect the logical
coverage of this suite. The "fake UI" is now simulated by an
http client that makes the necessary requests to Elasticsearch APIs.
2020-08-26 15:34:56 +03:00
James Rodewig 4701832879
[DOCS] Add 7.9 breaking change for built-in templates (#61549) (#61558) 2020-08-26 08:10:59 -04:00
Przemysław Witek 11c2710e7f
[7.x] [ML] Do not mark the DFA job as FAILED when a failure occurs after the node is shutdown (#61331) (#61526) 2020-08-26 09:53:13 +02:00