Commit Graph

53624 Commits

Author SHA1 Message Date
Benjamin Trent 341eeae6e7
[ML] fixes testWatchdog test verifying matcher is interrupted on timeout (#62391) (#62447)
Constructing the timout checker FIRST and THEN registering the watcher allows the test to have a race condition.

The timeout value could be reached BEFORE the matcher is added. To prevent the matcher never being interrupted, a new timedOut value is added to the watcher thread entry. Then when a new matcher is registered, if the thread was previously timedout, we interrupt the matcher immediately.

closes #48861
2020-09-16 09:13:22 -04:00
Nik Everett a68e47ab1f
Check for runtime field loops in queries (backport of #61927) (#62420)
We were checking for loops in queries before, but we had an "off by one"
error where we wouldn't notice the "top level" runtime field when
detecting a loop. So the error message would be wrong.

I also caught a few bugs with query generation caused by missing
`@Override` annotations and fixed a few of them. There is a bug with
`regexp` queries with match options that I'm not fixing in this PR but
will get to later.

Relates to #59332
2020-09-16 09:03:49 -04:00
Lyudmila Fokina 167172a057
Update authc failure headers on license change (#61734) (#62442)
Backport of #61734
2020-09-16 14:37:03 +02:00
Benjamin Trent 8d89a28126
[ML] unmuting test for testTooManyPartitions memory check on windows (#62393) (#62405)
This commit unmutes the windows check for testTooManyPartitions test.

The assertion has since changed to include a soft_limit check.

This coupled with changes over the past years means the test should be enabled again.

related to: #32033
2020-09-16 07:03:10 -04:00
Christoph Büscher 6a016fb755 Muting LogstashSystemIndexIT.testPipelineCRUD 2020-09-16 11:04:41 +02:00
Armin Braun f6a8599cf8
Don't Start Redundant ConsistentSettingsService (#62283) (#62428)
The consistent settings service is only used in tests so far. No need to start it
unless it's actually used.
2020-09-16 09:43:04 +02:00
Ignacio Vera f3ed641fc7
Adds bucketOrd back to cardinality algorithms (#62389) (#62427) 2020-09-16 08:41:57 +02:00
Hendrik Muhs 8566e9e3e7 [Transform] Make pivot validation sub-agg aware (#62381)
With the addition of sub aggregations like filter, the validation could fail if 2 sub aggs use the
same output name. This change makes validation sub-agg aware.

fixes #57814
2020-09-16 07:55:58 +02:00
Yang Wang a11dfbe031
Oidc additional client auth types (#58708) (#62289)
The OpenID Connect specification defines a number of ways for a
client (RP) to authenticate itself to the OP when accessing the
Token Endpoint. We currently only support `client_secret_basic`.

This change introduces support for 2 additional authentication
methods, namely `client_secret_post` (where the client credentials
are passed in the body of the POST request to the OP) and
`client_secret_jwt` where the client constructs a JWT and signs
it using the the client secret as a key.

Support for the above, and especially `client_secret_jwt` in our
integration tests meant that the OP we use ( Connect2id server )
should be able to validate the JWT that we send it from the RP.
Since we run the OP in docker and it listens on an ephemeral port
we would have no way of knowing the port so that we can configure
the ES running via the testcluster to know the "correct" Token
Endpoint, and even if we did, this would not be the Token Endpoint
URL that the OP would think it listens on. To alleviate this, we
run an ES single node cluster in docker, alongside the OP so that
we can configured it with the correct hostname and port within
the docker network.

Co-authored-by: Ioannis Kakavas <ioannis@elastic.co>
2020-09-16 14:29:09 +10:00
Nik Everett 24a24d050a
Implement fields fetch for runtime fields (backport of #61995) (#62416)
This implements the `fields` API in `_search` for runtime fields using
doc values. Most of that implementation is stolen from the
`docvalue_fields` fetch sub-phase, just moved into the same API that the
`fields` API uses. At this point the `docvalue_fields` fetch phase looks
like a special case of the `fields` API.

While I was at it I moved the "which doc values sub-implementation
should I use for fetching?" question from a bunch of `instanceof`s to a
method on `LeafFieldData` so we can be much more flexible with what is
returned and we're not forced to extend certain classes just to make the
fetch phase happy.

Relates to #59332
2020-09-15 20:24:10 -04:00
James Rodewig f94ae7ae26
[DOCS] Add static/dynamic type to audit logging settings (#61235) (#62417) 2020-09-15 17:34:24 -04:00
debadair 139a2988a0
[DOCS] Make it more obvious that the Java API is deprecated. (#62398) 2020-09-15 14:24:45 -07:00
Nik Everett e5ad3a41f1
Check for runtime field loops in queries (backport of #61927) (#62421)
We were checking for loops in queries before, but we had an "off by one"
error where we wouldn't notice the "top level" runtime field when
detecting a loop. So the error message would be wrong.

I also caught a few bugs with query generation caused by missing
`@Override` annotations and fixed a few of them. There is a bug with
`regexp` queries with match options that I'm not fixing in this PR but
will get to later.

Relates to #59332
2020-09-15 17:24:19 -04:00
Nik Everett 0a7f335215
Speed up writeVInt (backport of #62345) (#62419)
This speeds up `StreamOutput#writeVInt` quite a bit which is nice
because it is *very* commonly called when serializing aggregations. Well,
when serializing anything. All "collections" serialize their size as a
vint. Anyway, I was examining the serialization speeds of `StringTerms`
and this saves about 30% of the write time for that. I expect it'll be
useful other places.
2020-09-15 17:14:08 -04:00
Lisa Cawley 6320967546 [DOCS] Minor typo in ML API (#62414) 2020-09-15 13:20:55 -07:00
Costin Leau b2e85d5639 SQL: Do not resolve self-referencing aliases (#62382)
Prevent the analyzer for trying to resolve aliases on expressions that
reference themselves (or fields within themselves) as that causes
infinite recursion.

Fix #62296

(cherry picked from commit 021d27815b03e92e02859bc9c0c8eec78f30c72e)
2020-09-15 20:53:28 +03:00
Armin Braun 9ac4ee9c44
Increase Flaky Timeout in testIlmHistoryIndexCanRollover (#62353) (#62402)
This busy assert easily takes about 5s on a very fast work station
so the default of 10s is not sufficient here at all.
2020-09-15 19:50:45 +02:00
Nik Everett 771a8893a6
Add more debugging information for cardinality agg (#62317) (#62397)
This adds two extra bits of info to the profiler:
1. Count of the number of different types of collectors. This lets us figure
   out if we're using the optimization for segment ordinals. It adds a few
   more similar counters just for good measure.
2. Profiles the `getLeafCollector` and `postCollection` methods. These are
   non-trivial for some aggregations, like cardinality.
2020-09-15 13:21:11 -04:00
James Rodewig 2a7de79a1b
[DOCS] Remove redundant index.blocks.read_only_allow_delete setting (#62392) (#62399) 2020-09-15 12:43:48 -04:00
William Brafford af64e46065
Add logstash system index APIs (#53350) (#62347)
We want Logstash indices to be system indices, but the logstash
service will still need to be able to manage its indices. This PR
adds special system index APIs to the logstash plugin so that
logstash can manage its pipelines without direct access to the
underlying indices.

* Add logstash module with dedicated logstash APIs
* merge with x-pack plugin
* add system index access allowance
* Break out serialization tests into distinct classes
* Log failures for partial multiget failure
* Move LogstashSystemIndexIT to javaRestTest task

Co-authored-by: William Brafford <william.brafford@elastic.co>

Co-authored-by: Jay Modi <jaymode@users.noreply.github.com>
2020-09-15 12:42:14 -04:00
Armin Braun ffbc64bd10
Log WARN on Response Deserialization Failure (#62368) (#62388)
We never see this exception in the logs even though it's pretty severe.
All we might see is an exception about a transport message not having been read fully
from the logic that follows this code.
Technically we should probably bubble up the exception but that's a bigger change
and needs some carefully reasoning, this change for the time being at least simplifies
tracking down deserialization issues in responses.
2020-09-15 18:27:39 +02:00
Armin Braun 98f525f8a7
Faster Azure Blob InputStream (#61812) (#62387)
Building our own that should perform better than the one in the SDK.
Also, as a result saving a HEAD call for each ranged read on Azure.
2020-09-15 18:27:22 +02:00
Adam Locke 5dc0de04fb
[DOCS] Clarifying remote clusters based on feedback from Support (#62335) (#62394)
* Clarifying remote clusters based on feedback from Support.

* Apply suggestions from code review

* Making additional editorial changes.
2020-09-15 12:02:43 -04:00
Adrien Grand 6db8afefc2
Upgrade to lucene-8.7.0-snapshot-cdfdc1e0851. (#62376)
Upgrade to a new Lucene snapshot that (at least partially) addresses the
indexing rate regression when index sorting is enabled.

Backport of #62334.
2020-09-15 17:48:07 +02:00
Alan Woodward f89fa421e2 Remove unnecessary IndexSearcher field on HitContext (#62378)
FastVectorHighlighter uses the top-level reader to rewrite queries against, which
it gets via an IndexSearcher field on HitContext. However, we can already access
this top-level reader via HitContext's existing LeafReaderContext field.

This commit removes the unnecessary field and constructor parameter, and
changes the implementation of topLevelReader to go via ReaderUtils and
the leaf reader context.
2020-09-15 15:46:14 +01:00
Fernando Briano 7dd073c243
Wraps timestamp values in quotes in runtime fields YAML tests. (#62155) 2020-09-15 15:24:57 +01:00
Christoph Büscher 0ca9829867 Muting CoordinatorTests#testLogsMessagesIfPublicationDelayed 2020-09-15 15:40:51 +02:00
Albert Zaharovits aeed1c05b0
Ensure authz operation overrides transient authz headers (#61621)
AuthorizationService#authorize uses the thread context to carry the result of the
authorisation as transient headers. The listener argument to the `authorize` method
must necessarily observe the header values. This PR makes it so that
the authorisation transient headers (`_indices_permissions` and `_authz_info`, but
NOT `_originating_action_name`) of the child action override the ones of the parent action.

Co-authored-by: Tim Vernum tim@adjective.org
2020-09-15 16:37:38 +03:00
Armin Braun 76f56c1264
Add Missing NamedWritable Registration for ExecuteEnrichPolicyStatus (#62364) (#62374)
This was missing and caused nodes to drop out of the cluster on serialization failures
when ever one tried to get an enrich policy task by name.
The test in here is a little dirty but I figured it would be nice to have an actual reproducer
for the issue and I couldn't find any infrastructure to nicely time the tasks so I put this on
top of existing test infra.
2020-09-15 15:24:15 +02:00
István Zoltán Szabó d0c7b0a3a8
[DOCS] Removes init_script line from example Painless aggregation. (#62367) (#62379) 2020-09-15 15:13:23 +02:00
Costin Leau 03d2395183 EQL: Use Point In Time inside sequences (#62276)
Use the newly introduced PIT API to have a consistent view of the data
while doing sequence matching, which involves multiple calls, aka
repeatable reads and thus avoid race conditions or any in-flight updates
on the data.

(cherry picked from commit daa72fc3c71fd36afb55278021ff6bbc591ef148)
2020-09-15 15:40:03 +03:00
Tanguy Leroux faf96c175e
Abort non-fully consumed S3 input stream (#62167) (#62370)
Today when an S3RetryingInputStream is closed the remaining bytes 
that were not consumed are drained right before closing the underlying 
stream. In some contexts it might be more efficient to not consume the 
remaining bytes and just drop the connection.

This is for example the case with snapshot backed indices prewarming, 
where there is not point in reading potentially large blobs if we know 
the cache file we want to write the content of the blob as already been 
evicted. Draining all bytes here takes a slot in the prewarming thread 
pool for nothing.
2020-09-15 14:33:37 +02:00
Jim Ferenczi 4eea602d2d
Add a snapshot test module to delay shard aggregations (#62082) (#62359)
This change adds an aggregation that can be used to delay the
query phase execution on shards with a configurable time:

{
    "aggs": {
        "delay": {
            "shard_delay": {
                "value": "30s"
            },
            "aggs": {
                "host": {
                    "terms": {
                        "field": "hostname"
                    }
                }
            }
        }
    }
}

This test module is built on top of #61954 so the aggregation will be available only within
snapshots since this module is not meant to be used in production.

Closes #54159
2020-09-15 13:52:38 +02:00
Martijn van Groningen 3ed60df59d
Re-enable resolve index multi cluster test (#62365)
Backport of #62361 to 7.x branch.

This test was fine and shouldn't have been muted.
The test case class should have preserved data streams as part of #62205

Closes #62210
2020-09-15 13:26:52 +02:00
David Kyle 717259a049 Revert "[ML] Add debug logging of notification messages to upgrade test (#62342)"
This reverts commit c50899dd8f.
2020-09-15 11:07:23 +01:00
Armin Braun eae6a3b18e
Fix testMappingVersionAfterDynamicMappingUpdate (#62352) (#62360)
There is a race in this test where the index request will return
once the dynamic mapping update has been observed by the cluster
state observer internally used by the indexing but not hit all
state appliers and thus isn't showing up as the applied state returned
by `clusterService.state()` yet.
2020-09-15 11:59:22 +02:00
Alan Woodward a68f7077c7 Rationalise fetch phase exceptions (#62230)
We have a special FetchPhaseExecutionException which contains some useful
information about which shard and doc a fetch phase has failed in. However, this
is not used in many places - currently only the ExplainPhase and the highlighters
throw one, and the FetchPhase itself catches IOExceptions and just passes them
to the ExceptionsHelper with no extra context.

This commit changes FetchPhase to throw FetchPhaseExecutionException if it
encounters problems in any of its subphases, and removes the special handling
from the explain and highlight phases. It also removes the need to pass shard ids
around when building HitContext objects.
2020-09-15 09:28:19 +01:00
Rory Hunter 84ac72dced Fix log4j config for :qa:os tests (#62234)
The log4j config in :qa:os was broken because it referenced an appender plugin that is not
on that project's classpath. Resolve this by adding a dedicated logging config and removing
the copy step.
2020-09-15 09:17:19 +01:00
Alan Woodward 8089210815 Some small cleanups in TermVectorsService (#62292)
We removed the use of aggregated stats from term vectors back in #16452, but there is
a bunch of dead code left here which can be stripped out.
2020-09-15 09:01:49 +01:00
David Kyle c50899dd8f
[ML] Add debug logging of notification messages to upgrade test (#62342)
For #61908
2020-09-15 08:24:13 +01:00
Ignacio Vera 3536f7f7c2
Initialize BitArray storage as number of bits (#62327) (#62354) 2020-09-15 08:34:22 +02:00
Armin Braun c81a076f5a
Improve Efficiency of ClusterApplierService Iteration (#62282) (#62350)
The complexity of removing a timeout listener was `O(n)` which
means that in case of many queued up CS update tasks (such as in the
case of an avalanche of dynamic mapping updates) we're dealing with
quadratic complexity for timing out N tasks which was observed to be
an issue in practice.

This PR makes the complexity of timing out a task `O(1)` and generally
simplifies the iteration logic of listeners and applies to be a little
more efficient and inline better.
2020-09-15 05:59:48 +02:00
Lee Hinman 6b2af30a62
[7.x] Add "synthetics-*-*" templates for synthetics fleet data (#62193) (#62346)
* Add "synthetics-*-*" templates for synthetics fleet data

For the Elastic Agent we currently have `logs` and `metrics`, however, synthetic data doesn't belong
with those and thus we should have a place for it to live. This would be data reported from
heartbeat and under the 'monitoring' category.

This commit adds a composable index template for `synthetics-*-*` indices similar to the work in
 #56709 and #57629.

Resolves #61665
2020-09-14 17:14:34 -06:00
Julie Tibshirani f56ce4f39b
Fix failure in InnerHitBuilderTests around 'fields' option. (#62344)
The case InnerHitBuilderTests#testEqualsAndHashcode creates a copy of the object
by serializing + deserializing it, then applies a modification. If the 'fields'
list is empty, then deserializing it results in Collections.emptyList. Because
this is immutable, then modifying it can throw an UnsupportedOperationException.

This PR takes the same approach as for docvalue_fields, where we create a new
list instead of trying to add to an empty one.
2020-09-14 15:39:03 -07:00
Julie Tibshirani 9332a9c74b Add the fields option to the search API docs. (#62260) 2020-09-14 13:44:44 -07:00
Julie Tibshirani 4a19bdb2ea
Support the 'fields' option in inner_hits and top_hits. (#62337)
This PR adds support for the 'fields' option in the following places:
* Anytime `inner_hits` is used, for both fetching nested/ child docs and field collapsing
* The `top_hits` aggregation

Addresses #61949.
2020-09-14 11:51:45 -07:00
David Roberts 3d5c13f559
[ML] Add an assertion on annotations mappings to upgrade test (#62331)
The annotations index is not covered by the comparison between
mappings and templates, as it does not use an index template.

This commit adds an assertion on annotations index mappings
that will fail if the mappings are not upgraded as expected.

Backport of #62325
2020-09-14 18:46:35 +01:00
James Rodewig ec335c7c34
[DOCS] Fix capitalization for several headings (#62324) (#62329) 2020-09-14 12:35:15 -04:00
David Turner 9acd2fd1fd Minor cleanups to BytesReferenceStreamInput (#62302)
Followup to #61681:

- reuse the current iterator in `reset()` if possible
- simply some integer-overflow-avoidance in `skip()`
- clarify some comments
- address some IntelliJ warnings
2020-09-14 17:02:27 +01:00
David Roberts e4275f3749 [ML] Use utility thread pool for memory estimation (#62314)
The job comms thread pool is intended for the long-running job
processes that do anomaly detection or data frame analytics and
count towards job count and memory limits.

This commit moves the short-lived memory estimation processes
to the ML utility thread pool.

Although this doesn't matter in most cases, at the limits of
scale it could mean that memory estimations would get in the way
of starting jobs, or would queue up for an excessive period of
time while waiting for jobs to finish.
2020-09-14 16:47:12 +01:00