Commit Graph

51498 Commits

Author SHA1 Message Date
István Zoltán Szabó eeb23e9e73 [DOCS] Adds description of analysis_stats object and its properties to GET DFA stats API docs (#53881)
Co-authored-by: Valeriy Khakhutskyy <1292899+valeriy42@users.noreply.github.com>
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
2020-03-31 13:30:06 +02:00
Alan Woodward 25a0addb17 Don't double-wrap values (#54432)
After commit #53661 converted the lang-expressions module to using
DoubleValuesSource, we've seen a performance regression for expressions
that use geopoints. Some investigation suggests that this may be due to
GeoLatitudeValueSource and GeoLongitudeValueSource wrapping their
per-document values in a DoubleValues.withDefault() class. Values exposed
via expressions already have a '0' default value, so this extra wrapping is
unnecessary, and is directly on the hot path. This commit removes the extra
wrapping.
2020-03-31 10:50:13 +01:00
Tim Vernum a0853628cd
Add wildcard service providers to IdP (#54477)
This adds the ability for the IdP to define wildcard service
providers in a JSON file within the ES node's config directory.

If a request is made for a service provider that has not been
registered, then the set of wildcard services is consulted. If the
SP entity-id and ACS match one of the wildcard patterns, then a
dynamic service provider is defined from the associated mustache
template.

Backport of: #54148
2020-03-31 16:53:13 +11:00
Tim Brooks 915435bbe4
Fix issue with pipeline releasing bytes early (#54474)
Currently there is an issue with the InboundPipeline releasing bytes
earlier than appropriate. This can lead to the bytes being reused before
the message is handled. This commit fixes that issue and adds a test to
detect when it is occurring.
2020-03-30 22:39:15 -06:00
Jason Tedor 5d760051a9
Clarify autoscaling feature flag registration (#54427)
This commit clarifies the autoscaling feature flag registration system
property. The intention is that this system property is:
 - unset in snapshot builds
 - unset, true, or false in release builds
 - in release builds, unset behaves the same as false
 - therefore, we only register the enabled flag if the build is a
   snapshot build, or the build is a release build and the system
   property is set to true

This commit clarifies that intention, and removed a confusion situation
where the AUTOSCALING_FEATURE_FLAG_REGISTERED field would be set to
false in a snapshot build, even though we were going to register the
setting.
2020-03-30 21:37:25 -04:00
Ross Wolf d11e977b1f
EQL: Use In from QL (#53244)
* EQL: Use In from QL
* EQL: Add more In tests
* EQL: Test In duplicates
* EQL: Add test for In mixed types
* EQL: Copy In translation to QL
* SQL: Use InComparisons from QL
* EQL: Remove boost checks from QueryFolderOkTests
* QL: Add TranslatorHandler.convert
2020-03-30 15:19:23 -06:00
James Rodewig ce7539ce6c
[DOCS] Fix broken deprecated macros (#54466)
Updates formatting for `deprecated` macros in the translog and
synced flush docs.

Previously, these macros were rendering literally.
2020-03-30 17:16:27 -04:00
Dimitris Athanasiou b4b54efa73
[7.x][ML] Hyperparameter names should match config (#54401) (#54435)
Java side of elastic/ml-cpp#1096

Backport of #54401
2020-03-30 23:32:40 +03:00
Lee Hinman a3d1945254
[7.x] Add warnings/errors when V2 templates would match same i… (#54449)
* Add warnings/errors when V2 templates would match same indices… (#54367)

* Add warnings/errors when V2 templates would match same indices as V1

With the introduction of V2 index templates, we want to warn users that templates they put in place
might not take precedence (because v2 templates are going to "win"). This adds this validation at
`PUT` time for both V1 and V2 templates with the following rules:

** When creating or updating a V2 template
- If the v2 template would match indices for an existing v1 template or templates, provide a
warning (through the deprecation logging so it shows up to the client) as well as logging the
warning

The v2 warning looks like:

```
index template [my-v2-template] has index patterns [foo-*] matching patterns from existing older
templates [old-v1-template,match-all-template] with patterns (old-v1-template =>
[foo*],match-all-template => [*]); this template [my-v2-template] will take
precedence during new index creation
```

** When creating a V1 template
- If the v1 template is for index patterns of `"*"` and a v2 template exists, warn that the v2
template may take precedence
- If the v1 template is for index patterns other than all indices, and a v2 template exists that
would match, throw an error preventing creation of the v1 template

** When updating a V1 template (without changing its existing `index_patterns`!)
- If the v1 template is for index patterns that would match an existing v2 template, warn that the
v2 template may take precedence.

The v1 warning looks like:

```
template [my-v1-template] has index patterns [*] matching patterns from existing index templates
[existing-v2-template] with patterns (existing-v2-template => [foo*]); this template [my-v1-template] may be ignored in favor of an index template at index creation time
```

And the v1 error looks like:

```
template [my-v1-template] has index patterns [foo*] matching patterns from existing index templates
[existing-v2-template] with patterns (existing-v2-template => [f*]), use index templates (/_index_template) instead
```

Relates to #53101

* Remove v2 index and component templates when cleaning up tests

* Finish half-finished comment sentence

* Guard template removal and ignore for earlier versions of ES

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

* Also ignore 500 errors when clearing index template v2 templates

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-03-30 13:25:50 -06:00
Ryan Ernst c9421594bf
Remove allowTrial flag in license checking (#54293)
The allowTrial flag is always true, since trial licenses act as though
everything is licensed. This commit removes the allowTrial flag in
license checking helper methods.
2020-03-30 12:22:38 -07:00
Mark Tozzi 529622d4f4
Unit tests for Range and DateRange aggs (#52380) (#54455) 2020-03-30 15:07:43 -04:00
Mark Tozzi 10e0e59561
Tests for agg missing values (#51068) (#54452) 2020-03-30 15:05:38 -04:00
Ryan Ernst 3a24fe9d37
Move keystore-cli to its own tools project (#40787) (#54294)
This commit moves the keystore cli into its own project, so that the
test dependencies can be isolated from the rest of server.
2020-03-30 11:20:07 -07:00
Lisa Cawley 0fa1060ca4
[DOCS] Collapses content in machine learning APIs (#54234) (#54453) 2020-03-30 11:06:33 -07:00
Nik Everett 56047f74be
Fix auto_date_histogram serialization bug (#54447)
This fixes a serialization bug in `auto_date_histogram` that comes up in
a cluster mixed between pre-7.3.0 and post-7.3.0.

Includes #54429 to keep 7.x looking like master for simpler backports.

Closes #54382
2020-03-30 13:49:38 -04:00
James Rodewig ed1edb4964 [DOCS] Add missing word to keyword marker token filter docs 2020-03-30 10:52:14 -04:00
Nik Everett e58ad9fed3
Clean up how pipeline aggs check for multi-bucket (backport of #54161) (#54379)
Pipeline aggregations like `stats_bucket`, `sum_bucket`, and
`percentiles_bucket` only operate on buckets that have multiple buckets.
This adds support for those aggregations to `geo_distance`, `ip_range`,
`auto_date_histogram`, and `rare_terms`.

This all happened because we used a marker interface to mark compatible
aggs, `MultiBucketAggregationBuilder` and it was fairly easy to forget
to implement the interface.

This replaces the marker interface with an abstract method in
`AggregationBuilder`, `bucketCardinality` which makes you return `NONE`,
`ONE`, or `MANY`. The `bucket` aggregations can check for `MANY`. At
this point `ONE` and `NONE` amount to about the same thing, but I
suspect that'll be a useful distinction when validating bucket sorts.

Closes #53215
2020-03-30 10:44:55 -04:00
Jason Tedor 39b3010578
Add node local storage deprecation check (#54383)
The node.local_storage setting has been deprecated and will be removed
in 8.0.0. This commit adds a deprecation check to 7.x.
2020-03-30 10:23:43 -04:00
Christoph Büscher 67b9b68c66 [Docs] Add HLRC Async Search API documentation (#54353)
Adds documentation and a corresponding test case containing typical API usage
for the Async Search API to the High Level Rest Client.
2020-03-30 15:37:22 +02:00
Andrei Dan d5320d9d29
Read the index.number_of_replicas from template so that wait_for_active_shards is interpreted correctly (#54231) (#54413)
This commit takes into account the index.number_of_replicas (defaults to
0 - no replicas- ) value when setting an index template. This change
enables the index.wait_for_active_shards value to be interpreted
correctly

(cherry picked from commit 07026ac3d56dc9fae69467adfda7eaed7ea3ca00)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>

Co-authored-by: tninokehoe <62655306+tninokehoe@users.noreply.github.com>
2020-03-30 14:34:49 +01:00
Przemysław Witek 3c604da7f6
[7.x] Create an annotation when a model snapshot is stored (#53783) (#54405) 2020-03-30 15:17:08 +02:00
Armin Braun 9392fca36a
Improve Snapshot Abort Behavior (#54256) (#54410)
This commit improves the behavior of aborting snapshots and by that fixes
some extremely rare test failures.

Improvements:
1. When aborting a snapshot while it is in the `INIT` stage we do not need
to ever delete anything from the repository because nothing is written to the
repo during INIT any more (in the past running deletes for these snapshots made
sense because we were writing `snap-` and `meta-` blobs during the `INIT` step).
2. Do not try to finalize snapshots that never moved past `INIT`. Same reason as
with the first step. If we never moved past `INIT` no data was written to the repo
so no need to now write a useless entry for the aborted snapshot to `index-N`.
This is especially true, since the reason the snapshot was aborted during `INIT` was
a delete call so the useless empty snapshot just added to `index-N` would be removed
by the subsequent delete that is still waiting anyway.
3. if after aborting a snapshot we wait for it to finish we should not try deleting it
if it failed. If the snapshot failed it means it did not become part of the most recent
`RepositoryData` so a delete for it will needlessly fail with a confusing message about
that snapshot being missing or concurrent repository modification. I moved to throw the snapshot missing exception here because that seems the most user friendly. This allows the user to simply ignore `404` returns from the delete API when using it to make sure a snapshot is aborted+deleted.

Marking this as a non-issue since it doesn't have any negative repercussions other than confusing exceptions on some snapshot aborts.

Closes #52843
2020-03-30 15:08:18 +02:00
James Rodewig 21f362a2a8
[DOCS] Add a lowercase email example to keyword tokenizer docs (#53257) 2020-03-30 09:06:04 -04:00
Benjamin Trent 374e76d7cd
[Transform] fixing naming in HLRC and _cat to match API content (#54300) (#54408)
Fixing the naming of the HLRC values to match the ToXContent field names (i.e. the field names returned from an API call).

Also fixes the names in the _cat API as well.

closes #53946
2020-03-30 08:57:02 -04:00
Jim Ferenczi 12cfdc24b0 Fixed rewrite of time zone without DST (#54398)
We try to rewrite time zones to fixed offsets in the date histogram aggregation
if the data in the shard is within a single transition.
However this optimization is not applied on time zones that don't apply daylight saving changes
but had some random transitions in the past (e.g. Australia/Brisbane or Asia/Katmandu).
This changes fixes the rewrite of such time zones to fixed offsets.
2020-03-30 13:18:57 +02:00
Martijn van Groningen 4b4fbc160d
Refactor AliasOrIndex abstraction. (#54394)
Backport of #53982

In order to prepare the `AliasOrIndex` abstraction for the introduction of data streams,
the abstraction needs to be made more flexible, because currently it really can be only
an alias or an index.

* Renamed `AliasOrIndex` to `IndexAbstraction`.
* Introduced a `IndexAbstraction.Type` enum to indicate what a `IndexAbstraction` instance is.
* Replaced the `isAlias()` method that returns a boolean with the `getType()` method that returns the new Type enum.
* Moved `getWriteIndex()` up from the `IndexAbstraction.Alias` to the `IndexAbstraction` interface.
* Moved `getAliasName()` up from the `IndexAbstraction.Alias` to the `IndexAbstraction` interface and renamed it to `getName()`.
* Removed unnecessary casting to `IndexAbstraction.Alias` by just checking the `getType()` method.

Relates to #53100
2020-03-30 10:12:16 +02:00
István Zoltán Szabó 00eaa0ebe5 [DOCS] Changes scripted metric to filter aggs in transforms example (#54167) 2020-03-30 09:51:07 +02:00
Nhat Nguyen 6e025c12f0 Add debug logging for testRunningTasksCount
Relates #53594
2020-03-29 18:34:41 -04:00
Jason Tedor d2aced810d
Add assertion for get autoscaling decision API test
This commit adds a match assertion to the get autoscaling decision REST
test.
2020-03-29 14:36:38 -04:00
Jason Tedor 686f4f0158
Adjust BWC version on node roles being sorted
Node roles are sorted now as of 7.8.0. This commit adjusts the BWC
version for tests.
2020-03-28 15:30:42 -04:00
Jason Tedor f0033783db
Deprecate node local storage setting (#54374)
This setting is not documented and has dubious value since it means
there can be nodes in the cluster (non-data and non-master nodes) that
do not have persistent node IDs. This does not have any use cases so
this commit removes the setting.
2020-03-28 14:36:41 -04:00
Jason Tedor 60437b474d
Fix line-length violation in DiscoveryNodeRole
This commit fixes a line-length checkstyle violation in
DiscoveryNodeRole.java.
2020-03-28 13:06:20 -04:00
Jason Tedor 03cab96b2d
Fix imports in discovery node classes
This commit fixes some imports that were leftover after resolving some
merge conflicts on a backport.
2020-03-28 12:56:22 -04:00
Jason Tedor c3be3206ce
Decouple environment from DiscoveryNode (#54373)
Today Environment is coupled to DiscoveryNode via the node.local_storage
setting. This commit decouples Environment from this setting.
2020-03-28 12:52:47 -04:00
Jason Tedor 37b59a357f
Ensure that the output of node roles are sorted (#54376)
This commit ensures that node roles are sorted by node role name, which
makes the output easier to consume, and also makes it easier to rely on
the behavior of the output in assertions.
2020-03-28 12:51:21 -04:00
Jason Tedor 512a318b4b
Do not stash environment in security (#54372)
Today the security plugin stashes a copy of the environment in its
constructor, and uses the stashed copy to construct its components even
though it is provided with an environment to create these
components. What is more, the environment it creates in its constructor
is not fully initialized, as it does not have the final copy of the
settings, but the environment passed in while creating components
does. This commit removes that stashed copy of the environment.
2020-03-28 12:47:16 -04:00
Jason Tedor cf68ac8a2c
Do not stash environment in machine learning (#54371)
Today the machine learning plugin stashes a copy of the environment in
its constructor, and uses the stashed copy to construct its components
even though it is provided with an environment to create these
components. What is more, the environment it creates in its constructor
is not fully initialized, as it does not have the final copy of the
settings, but the environment passed in while creating components
does. This commit removes that stashed copy of the environment.
2020-03-28 12:46:16 -04:00
Stuart Tettemer 30c56087fd
Docs: Use splitOnToken instead of custom function (#48408) (#54364)
Painless ingest example uses a custom split function but
new splitOnToken function was added in 7.2

Backport of: 0c52a92
2020-03-27 15:04:27 -06:00
Tim Brooks 2ccddbfa88
Move transport decoding and aggregation to server (#54360)
Currently all of our transport protocol decoding and aggregation occurs
in the individual transport modules. This means that each implementation
(test, netty, nio) must implement this logic. Additionally, it means
that the entire message has been read from the network before the server
package receives it.

This commit creates a pipeline in server which can be passed arbitrary
bytes to handle. Internally, the pipeline will decode, decompress, and
aggregate the messages. Additionally, this allows us to run many
megabytes of bytes through the pipeline in tests to ensure that the
logic works.

This work will enable future work:

Circuit breaking or backoff logic based on message type and byte
in the content aggregator.
Sharing bytes with the application layer using the ref counted
releasable network bytes.
Improved network monitoring based specifically on channels.
Finally, this fixes the bug where we do not circuit break on the correct
message size when compression is enabled.
2020-03-27 14:13:10 -06:00
Stuart Tettemer 1630de4a42
Scripting: stats per context in nodes stats (#54008) (#54357)
Adds script cache stats to `_node/stats`.
If using the general cache:
```
      "script_cache": {
        "sum": {
          "compilations": 12,
          "cache_evictions": 9,
          "compilation_limit_triggered": 5
        }
      }

```
If using context caches:
```
      "script_cache": {
        "sum": {
          "compilations": 13,
          "cache_evictions": 9,
          "compilation_limit_triggered": 5
        },
        "contexts": [
          {
            "context": "aggregation_selector",
            "compilations": 8,
            "cache_evictions": 6,
            "compilation_limit_triggered": 3
          },
          {
            "context": "aggs",
            "compilations": 5,
            "cache_evictions": 3,
            "compilation_limit_triggered": 2
          },
```
Backport of: 32f46f2
Refs: #50152
2020-03-27 12:26:00 -06:00
Tim Brooks f5b4020819
Remove netty BytesReference implementations (#54355)
Elasticsearch has a number of different BytesReference implementations.
These implementations can all implement the interface in different ways
with subtly different behavior and performance characteristics. On the
other-hand, the JVM only represents bytes as an array or a direct byte
buffer. This commit deletes the specialized Netty implementations and
moves to using a generic ByteBuffer reference type. This will allow us
to focus on standardizing performance and behave around a smaller number
of implementations that can be used by all components in Elasticsearch.
2020-03-27 11:01:33 -06:00
Lee Hinman f2cc2b1127
[7.x] Add REST APIs for IndexTemplateV2Metadata CRUD (#54039) (#54347)
* Add REST APIs for IndexTemplateV2Metadata CRUD (#54039)

* Add REST APIs for IndexTemplateV2Metadata CRUD

This commit adds the get/put/delete APIs for interacting with the now v2 versions of index
templates.

These APIs are behind the existing `es.itv2_feature_flag_registered` system property feature flag.

Relates to #53101

* Add exceptions for HLRC tests

* Add skips for 7.x versions

* Use index_template instead of template_v2 in action names

* Add test for MetaDataIndexTemplateService.addIndexTemplateV2

* Move removal to static method and add test

* Add unit tests for request classes (implement hashCode & equals)

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

* Fix compilation

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-03-27 10:47:22 -06:00
Dan Hermann 1690e78646
Validation for data stream creation 2020-03-27 10:07:46 -05:00
Mark Vieira 2f619ad7d0
Improve checkstyle performance and readability (#54308)
Drop a nasty regex in our checkstyle config that I wrote a long time ago
in favor of a checkstyle extension. This is better because:
* It is faster. It saves a little more than a minute across the entire
  build.
* It is easier to read. Who knew 100 lines of Java would be easier to
  read than a regex, but it is.
* It has tests.
2020-03-27 08:05:04 -07:00
Mark Vieira 2ca8850e13
Disable Gradle daemon when executing Windows packaging tests (#54316)
The powershell call operator (&) seems to wait on any child processes.
In the case of Gradle build invocation, the daemon causes the script
execution to hang until the daemon terminates (or 5 minutes elapses?).
2020-03-27 08:03:13 -07:00
Christoph Büscher 0d17295601 [Docs] Minor fix for SubmitAsyncSearchRequest.keepOnCompletion javadoc (#54325)
The semantics and the default value for this parameter have changed, adapting
the javadoc accordingly.
2020-03-27 16:02:03 +01:00
Gil Raphaelli 2984a54b7f [DOCS] Fix typos in top metrics agg docs (#54299) 2020-03-27 10:49:21 -04:00
Alan Woodward 461f1307d6 Add XContentHelper.childBytes() method (#54287)
We have a number of places where we want to read a fairly complex object from
XContent, but aren't interested in its contents; for example, mappings are often
serialized and deserialized between several objects before they are actually built
into a MappingMetaData object. This means that potentially large maps of maps
are constructed several times, only to immediately be re-serialized again.

This commit adds a new helper method to XContentHelper that reads the children
of an xcontent object directly to a BytesReference, serialized via the same xcontenttype
as the parent parser, avoiding the construction of intermediary maps or lists.
2020-03-27 14:21:56 +00:00
Armin Braun d9d11f6d16
Remove Unused Apache Http Dependency from GCS Repo Plugin (#54331) (#54342)
We are not using the Apache HTTP client backed http transport
with the GCS repo. Same as with the app engine type transport
we can save ourselves the dependency on the http client here
and ignore the missing classes.
2020-03-27 15:10:19 +01:00
Przemysław Witek 2eb079b67f
Add version guards around ML hidden indices settings (#54322) 2020-03-27 14:50:57 +01:00