Commit Graph

53121 Commits

Author SHA1 Message Date
James Rodewig a160daa5d9
[DOCS] Remove collapsible examples (#59820) (#59857)
Snippets are now visible without additional clicks.
2020-07-20 09:14:36 -04:00
Rene Groeschke e31ebc96f9
Enforce fail on deprecated gradle usage (7.x backport) (#59758)
* Enforce fail on deprecated gradle usage (#59598)
* Fix branch specific deprecated gradle api usages
* Fix archiveVersion property usage
2020-07-20 08:52:30 +02:00
Albert Zaharovits 3ffb20bdfc
Fix DLS/FLS permission for the submit async search action (#59693)
The submit async search action should not populate the thread context
DLS/FLS permission set, because it is not currently authorised as an "indices request"
and hence the permission set that it builds is incomplete and it overrides the
DLS/FLS permission set of the actual spawned search request (which is built correctly).
2020-07-20 09:37:26 +03:00
Nhat Nguyen 120fe96402 Log shard list in testIndexVersionPropagation
Relates #59494
2020-07-18 23:23:57 -04:00
Costin Leau 9cc80621c3 EQL: Fix matching of tail/desc queries (#59827)
When dealing with tail queries, data is returned descending for the base
criterion yet the rest of the queries are ascending. This caused a
problem during insertion since while in a page, the data is ASC, between
pages the blocks of data is DESC.
This caused incorrectly sorting inside a SequenceGroup which led to
incorrect results.

Further more in case of limit, since the data in a page is ASC, early
return is not possible neither is desc matching. Thus the page needs to
be consumed first before finding the final results.
A future improvement could be to keep only the top N results dropping
the rest during insertion time.

(cherry picked from commit 77c88da054a1ce662a264f72cde5986d4ce37e3a)
2020-07-19 00:49:16 +03:00
Lee Hinman 8c7d414a3b
[7.x] Fix retrieving data stream stats for a DS with multiple backing indices (#59806) (#59810)
Backports the following commits to 7.x:

    Fix retrieving data stream stats for a DS with multiple backing indices (#59806)
2020-07-17 16:56:07 -06:00
Ryan Ernst e963918830
Ensure precommit runs as part of check (#59476)
Precommit is setup to run as a dependency of the check task, but
unfortunately this wiring was only happening when the java plugin (but
not java-base plugin) was applied. This commit moves the wiring to occur
whenever the check task exists, which is with the lifecycle-base plugin.
2020-07-17 15:41:37 -07:00
Nik Everett 514b2f3414
Clean up a few of vwh's rough edges (#59341) (#59807)
This cleans up a few rough edged in the `variable_width_histogram`,
mostly found by @wwang500:
1. Setting its tuning parameters in an unexpected order could cause the
   request to fail.
2. We checked that the maximum number of buckets was both less than
   50000 and MAX_BUCKETS. This drops the 50000.
3. Fixes a divide by 0 that can occur of the `shard_size` is 1.
4. Fixes a divide by 0 that can occur if the `shard_size * 3` overflows
   a signed int.
5. Requires `shard_size * 3 / 4` to be at least `buckets`. If it is less
   than `buckets` we will very consistently return fewer buckets than
   requested. For the most part we expect folks to leave it at the
   default. If they change it, we expect it to be much bigger than
   `buckets`.
6. Allocate a smaller `mergeMap` in when initially bucketing requests
   that don't use the entire `shard_size * 3 / 4`. Its just a waste.
7. Default `shard_size` to `10 * buckets` rather than `100`. It *looks*
   like that was our intention the whole time. And it feels like it'd
   keep the algorithm humming along more smoothly.
8. Default the `initial_buffer` to `min(10 * shard_size, 50000)` like
   we've documented it rather than `5000`. Like the point above, this
   feels like the right thing to do to keep the algorithm happy.

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-07-17 15:16:09 -04:00
Lee Hinman f6b08a3115
[7.x] Allow simulating existing composable index template (#59733) (#59798)
Backports the following commits to 7.x:

    Allow simulating existing composable index template (#59733)
2020-07-17 13:10:07 -06:00
Adam Locke 29ff05cbac
[7.x] [DOCS] Update snapshot/restore docs to align with API changes (#59730) (#59803)
* [DOCS] Updating snapshot/restore pages to align with API changes (#59730)

* Updating snapshot/restore pages to align with API changes.

* Fixing texts in delete snapshot page.

* Removing duplicate code sample and making editorial changes.

* Change "deleted" to "delete"

* Incorporating review feedback and making minor editorial changes.

* Remove titleabbrev

* Add paragraph break

* Remove titleabbrev from restore page

* Remove titleabbrev from create page

* Change "Create" to lowercase

* Change API names to lowercase

* Remove extraneous delimiters

* Change "Delete" to lowercase

* Single-sourcing warning and clarifying warning text.

* Fixing tests and removing erroneous example.
2020-07-17 14:33:18 -04:00
Nik Everett 95e6e4a452
Small cleanup for IndexFieldData (#59724) (#59800)
This drops `IndexComponent` from `IndexFieldData` because it wasn't
doing anything other than forcing us to perform a bunch of ceremony to
build them.
2020-07-17 13:38:15 -04:00
Tal Levy c9ab7bb651
Fix bug in circuit-breaker check for geoshape grid aggregations (#57962) (#59741)
There was a bug in the geoshape circuit-breaker check where the
hash values array was being allocated before its new size was
accounted for by the circuit breaker.

Fixes #57847.
2020-07-17 09:26:00 -07:00
Dan Hermann 48df9b1a0e
Update regex file for es user agent node processor (#59697) (#59794) 2020-07-17 11:04:01 -05:00
James Rodewig 40f319a1f6
[DOCS] Reformat Painless snippets to use two-space indents (#59776) (#59791) 2020-07-17 11:31:37 -04:00
Christoph Büscher f4ff5fe93b
Add `zero_terms_query` support to `match_phrase_prefix` (#58822) (#59784)
Currently `match_phrase_prefix` doesn't support `zero_terms_query` like the
other match-type queries. This change adds this support.

Closes #58468
2020-07-17 17:23:23 +02:00
Adam Locke 6ccf3548e7
Fix Snapshot Status API Docs Test (#59775) (#59787)
Introduce a fix to tests by snapshotting a single index+shard in the snapshot that
we get the status for and verifying consistency instead of equality
for total file counts.

Co-authored-by: Armin Braun <me@obrown.io>
2020-07-17 11:11:23 -04:00
James Rodewig a672a2a2d4
[DOCS] Move highlighting docs to separate page (#59768) (#59781)
Moves the highlighting docs from the deprecated 'Request Body Search'
chapter to the new subpage of the 'Run a search chapter' section.

No substantive changes were made to the content.
2020-07-17 10:57:00 -04:00
Benjamin Trent b7f30fc929
[7.x] Adding new `require_alias` option to indexing requests (#58917) (#59769)
* Adding new `require_alias` option to indexing requests (#58917)

This commit adds the `require_alias` flag to requests that create new documents.

This flag, when `true` prevents the request from automatically creating an index. Instead, the destination of the request MUST be an alias.

When the flag is not set, or `false`, the behavior defaults to the `action.auto_create_index` settings.

This is useful when an alias is required instead of a concrete index.

closes https://github.com/elastic/elasticsearch/issues/55267
2020-07-17 10:24:58 -04:00
Alan Woodward 65f6fb8e94
Shortcut mapping update if the incoming mapping version is the same as the current mapping version (#59517) (#59772)
Currently, when we apply a cluster state change to a shard on a non-master node,
we check to see if the mappings need to be updated by comparing the decompressed
serialized mappings from the update against the serialized version of the shard's
existing mappings. However, we already have a much simpler way of checking this,
by comparing mapping versions on the index metadata of the old and new states.

This commit adds a shortcut to MapperService.updateMappings() that compares
these mapping versions, and ignores the merge if they are equal.
2020-07-17 14:53:09 +01:00
Alan Woodward b29d368b52
Convert DateFieldMapper to parametrized format (#59429) (#59759)
This commit makes DateFieldMapper extend ParametrizedFieldMapper,
declaring its parameters explicitly. As well as changes to DateFieldMapper
itself, there are some changes to dynamic mapping code to ensure that
dynamically detected date formats are passed through to new date mapper
builders.
2020-07-17 12:46:18 +01:00
Andrei Dan 301d61a98e
Tests: fix TimeSeriesDataStreamsIT.testShrinkActionInPolicyWithoutHotPhase (#59603) (#59689)
The ILM policy for the source and shrunk indices run separately (ie. they
are two separate managed indices). This fixes the test which exhibited some
flakiness by allowing some time for the ILM policy for the source index
to finish executing.

(cherry picked from commit c78d5e8499fc5ca2ca1314f97bcc6b55ba06e2e7)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
2020-07-17 11:26:06 +01:00
Rene Groeschke afbc4678ae
Remove deprecated compileOnly resolution (#58857) (#59709)
- Ideally this should not be required but current runtime classpath
seems to require this. We need to revisit the classpath declarations in detail
to be able to remove this type of (in theory) superflous compile only references

Related to #57920
2020-07-17 12:06:57 +02:00
Przemko Robakowski 790fbbcd87
[7.x] Fix handling of final pipelines when destination is changed (#59522) (#59746)
* Fix handling of final pipelines when destination is changed (#59522)

This change fixes final pipelines if destination index is changed during pipeline run:
-final pipelines can't change destination anymore, exception is thrown if they try to
-if request/default pipeline changes destination final pipeline from old index won't be executed
-if request/default pipeline changes destination and new index has final pipeline it will be executed
-default pipeline from new index won't be executed
Additionally TransportBulkAction.resolvePipelines was moved to IngestService as it's needed for resolving pipelines from new index. Tests were moved accordingly.

Closes #57968
2020-07-17 11:13:48 +02:00
Andrei Stefan d513e1090f
Do not create the index, if it's already there (#59745) (#59747)
(cherry picked from commit d097447d257efdf0a36b1157e1f177aed86ecca1)
2020-07-17 11:38:30 +03:00
Rene Groeschke 11b6bba984
Fix all deprecations in resolveAllDependencies (#59751)
This fixes #59743
2020-07-17 10:16:45 +02:00
Tanguy Leroux 4827fec1cf
Revert "Mute AzureSearchableSnapshotsIT (#58775)" (#59749)
This reverts commit 74a78b3a7b.
2020-07-17 10:02:46 +02:00
Stuart Tettemer 9aa476196f
Scripting: Update ctx docs with sha augment #59719 (#59736)
Backport: 352a90c
2020-07-16 18:33:08 -05:00
Tim Brooks b6e6a8c090
Fix replication operation transient retry test (#58205)
After the work to retry transient replication failures, the local and
global checkpoint test metadata can be incremented on a different thread
than the test thread. This appears to introduce an extremely rare
scenario where this data is not visible for later test assertions. This
commit fixes the issue by using synchronized maps.
2020-07-16 16:01:47 -06:00
Martijn van Groningen 74c9402912
Re-enable data stream bwc tests (#59734)
after backporting #59503
Backport of #59732 yo 7.x
2020-07-16 23:59:52 +02:00
Martijn van Groningen 0096238df1
Replaced _data_stream_timestamp meta field's 'path' option with 'enabled' option (#59727)
Backport #59503 to 7.x

and adjusted exception messages.

Relates to #59076
2020-07-16 22:29:40 +02:00
Stuart Tettemer 8fdaed0642
Scripting: Augment String with sha1 and sha256 (#59671) (#59723)
Only available in the ingest context for use in ingest pipelines.

Digests are computed on the UTF-8 encoding of the String and are
returned as hex strings.

sha1() return hex strings of length 40, sha256() returns length 64

Fixes: #59647
Backport: 3c85272
2020-07-16 15:17:32 -05:00
Stuart Tettemer c491212dc1
Scripting: fix generateContextDoc path and url #59676 (#59722)
* Add doc runtime class path
* Use getAllHttpSocketURI.get(0) instead of getAllHttpSocketURI to get a single
test cluster URL rather than a list

Backport: 3057e0f
2020-07-16 15:03:36 -05:00
James Rodewig fa2167af0a
[7.x] [DOCS] Update upgrade docs and release highlights for 7.9 (#59674) 2020-07-16 15:58:40 -04:00
Igor Motov 2408803fad
Adds hard_bounds to histogram aggregations (#59175) (#59656)
Adds a hard_bounds parameter to explicitly limit the buckets that a histogram
can generate. This is especially useful in case of open ended ranges that can
produce a very large number of buckets.
2020-07-16 15:31:53 -04:00
Alan Woodward 10be10c99b
Migrate CompletionFieldMapper to parametrized format (#59691)
This adds a number of new optional parameters to Parameter, including:

* custom serialization (to handle analyzers)
* deprecated parameter names
* parameter validation
* allowing default values to be based on the values of other parameters

We preserve the previous serialization format of CompletionFieldMapper,
always emitting most fields, in order to meet mapping checks in mixed
version clusters, where the mapper service will check that mappings have
been correctly parsed and updated by checking their serialized outputs.
2020-07-16 19:15:00 +01:00
Martijn van Groningen 4089cbd767
Ignore multiple matching templates warning in specific tests. (#59692) (#59715)
Closes #59679
2020-07-16 20:07:38 +02:00
James Rodewig da85a40e7e
[DOCS] Reformat `predicate_token_filter` tokenfilter (#57705) (#59714) 2020-07-16 13:35:09 -04:00
lcawl f2b530dbdb [DOCS] Re-adds coming macro in release notes 2020-07-16 09:12:39 -07:00
István Zoltán Szabó 35512a9284 [DOCS] Adds security privilege info to inference bucket aggregation (#59604) 2020-07-16 18:03:19 +02:00
Marios Trivyzas c7efbc1b83
SQL: Implement DATE_PARSE function for parsing strings into DATE values (#57391) (#59699)
Implement DATE_PARSE(<date_str>, <pattern_str>) function
which allows to parse a date string according to the specified
pattern into a date object. The patterns allowed are those of
java.time.format.DateTimeFormatter.

Closes #54962

Co-authored-by: Marios Trivyzas <matriv@users.noreply.github.com>
Co-authored-by: Patrick Jiang(白泽) <dreamlike.sky@foxmail.com>

(cherry picked from commit 647a413d9b21bd3938f1716bb19f8407e1334125)
2020-07-16 17:24:30 +02:00
Adam Locke 305b46c7cd
[DOCS] Adding get snapshot status API docs (#59355) (#59670)
* Adding get snapshot status API docs.

* Adding more fields and a link to the new page.

* Adding missing spaces in TESTRESPONSES

* Adding more parameters and making some edits.

* Marking snapshot as optional

* Marking repository as optional

* Add data type for stats

* Add data type for shard_stats

* Incorporating review feedback.

* Lots of review feedback incorporated.

* Fixing tests to unbreak CI builds.

* Changing indices to index.
2020-07-16 11:21:17 -04:00
Benjamin Trent a28547c4b4
[7.x] [ML] add new `custom` field to trained model processors (#59542) (#59700)
* [ML] add new `custom` field to trained model processors (#59542)

This commit adds the new configurable field `custom`.

`custom` indicates if the preprocessor was submitted by a user or automatically created by the analytics job.

Eventually, this field will be used in calculating feature importance. When `custom` is true, the feature importance for
the processed fields is calculated. When `false` the current behavior is the same (we calculate the importance for the originating field/feature).

This also adds new required methods to the preprocessor interface. If users are to supply their own preprocessors
in the analytics job configuration, we need to know the input and output field names.
2020-07-16 10:57:38 -04:00
Rene Groeschke 3a228906a9
Fix license header checks on buildSrc (#58457) (#59705) 2020-07-16 16:33:30 +02:00
Howard c0d429863c
remove unused cluster name in environment. (backport of #59605) (#59681)
removes an unused variable
2020-07-16 09:25:55 -04:00
Nik Everett 343053c0a7 Fix compilation in Eclipse (backport #59675)
Eclipse was confused by #59583. It can't see a the public inner
interface within the superclass. This time. Usually that is fine, but
the Eclipse gods don't like this particular code, I guess.
2020-07-16 08:25:12 -04:00
István Zoltán Szabó 76fbe0a6d9 [DOCS] Sorts agg and grouping names alphabetically in PUT Transforms API docs. (#59688) 2020-07-16 12:45:29 +02:00
Alan Woodward 27067de699 Make MappedFieldType#meta final (#59383)
The MappedFieldType#updateMeta method was used for testing equality checks, but we
no longer need these after #59212 , so we can remove this method and make meta final.
2020-07-16 09:45:55 +01:00
David Kyle c349fdcb89 Mute RegressionIT testWithDataStream (#59687)
For #59664
2020-07-16 09:45:29 +01:00
Przemysław Witek df4fea79cb
Add a "verbose" option to the data frame analytics stats endpoint (#59589) (#59621) 2020-07-16 09:51:31 +02:00
Armin Braun 6db481f49e
Fix ConcurrentSnapshotsIT.testEquivalentDeletesAreDeduplicated (#59611) (#59653)
Trying to queue up snapshot deletes by blocking the delete of the latest
index-N doesn't work here. The first delete will block on the delete operation
but only do so after having already written the updated repository data.
Since that repository data will contain no snapshots, the subsequent deletes for
`*` will just fall through and complete instead of queue up.
=> Fixed by simply waiting on all files on master so that we block before updating
the repository data and get to test the queueing of equivalent operations

closes #59608
2020-07-16 09:28:36 +02:00