Commit Graph

48552 Commits

Author SHA1 Message Date
James Baiera 73263c654a Add basic task support for executing enrich policies (#47523)
Changes the execution logic to create a new task using the execute request,
and attaches the new task to the policy runner to be updated. Also, a new
response is now returned from the execute api, which contains either the task
id of the execution, or the completed status of the run. The fields are mutually
exclusive to make it easier to discern what type of response it is.
2019-10-11 13:32:06 -04:00
James Rodewig 256102042b
[DOCS] Reformat get snapshot lifecycle policy API docs (#47827) (#47930) 2019-10-11 13:00:48 -04:00
Christoph Büscher 2ef12c37f5 Add builder for distance_feature to QueryBuilders (#47846)
The QueryBuilders convenience class is currently missing a shortcut to construct
a DistanceFeatureQueryBuilder, which is added here.

Closes #47767
2019-10-11 18:20:01 +02:00
Alpar Torok 5cbc96ccb6
Fix dependency info tasks (#47857)
We fixed warnings related to task input and outputs in #45098.
This particular input was not considered, a warning was present for it
and Gradle didn't use it as part of task inputs.
As soon as we fixed it Gradle started considering it an input and
enforced that it exists.

With this change we make it optional as the task can work both wih and
without this directory.
2019-10-11 11:11:52 -05:00
Alan Woodward ec9198d0e2
Adjust Version.V_6_8_4 to refer to Lucene 7.7.2 (#47926)
6.8.4 will ship with Lucene 7.7.2, so we need to change our version settings to
reflect this.

Relates #47901
2019-10-11 17:01:42 +01:00
James Rodewig aa5689f5b3
[DOCS] Reformat put snapshot lifecycle policy API docs (#47811) (#47925) 2019-10-11 11:28:38 -04:00
Hendrik Muhs 0ca53bd80e
add BWC alias for internal index
create an alias for old nodes to retrieve new documents in the internal index as they do not know the new index pattern
2019-10-11 17:15:01 +02:00
David Turner ba62eb3dce Allow truncation of clean translog (#47866)
Today the `elasticsearch-shard remove-corrupted-data` tool will only truncate a
translog it determines to be corrupt. However there may be other cases in which
it is desirable to truncate the translog, for instance if an operation in the
translog cannot be replayed for some reason other than corruption. This commit
adds a `--truncate-clean-translog` option to skip the corruption check on the
translog and blindly truncate it.
2019-10-11 15:48:12 +01:00
James Rodewig 58289a857d
[DOCS] Reformat reload search analyzers API docs (#47824) (#47907) 2019-10-11 09:40:49 -04:00
Ioannis Kakavas 33705c4b95
Document SAML APIs (#45105) (#47909)
This change adds documentation for the SAML APIs in Elasticsearch
and adds simple instructions on how these APIs can be used to
authenticate a user with SAML by a custom web application other
than Kibana.

Resolves: #40352
2019-10-11 16:34:11 +03:00
Przemysław Witek c62fe8c344
Require that the dependent variable column has at most 2 distinct values in classfication analysis. (#47858) (#47906) 2019-10-11 14:57:08 +02:00
Henning Andersen a0d0866f59 Shrink should not touch max_retries (#47719)
Shrink would set `max_retries=1` in order to avoid retrying. This
however sticks to the shrunk index afterwards, causing issues when a
shard copy later fails to allocate just once.

Avoiding a retry of a shrink makes sense since there is no new node
to allocate to and a retry will likely fail again. However, the downside of
having max_retries=1 afterwards outweigh the benefit of not retrying
the failed shrink a few times. This change ensures shrink no longer
sets max_retries and also makes all resize operations (shrink, clone,
split) leave the setting at default value rather than copy it from source.
2019-10-11 14:22:56 +02:00
Hendrik Muhs 3da91d5f7a
[Transform] Rename internal indexes for transform plugin (#47788) (#47900)
rename internal indexes of transform plugin

- rename audit index and create an alias for accessing it, BWC: add an alias for old indexes to
  keep them working, kibana UI will switch to use the read alias
- rename config index and provide BWC to read from old and new ones
2019-10-11 14:16:17 +02:00
Hendrik Muhs 5dd6bd6f49
do not assert on state in mixed cluster due to endpoint differences (#47898)
do not assert on state in mixed cluster due to endpoint differences between 7.3 and 7.4

regression #46452
fixes #47693
2019-10-11 12:27:54 +02:00
Przemyslaw Gomulka 0c439fe495
[7.x] Allow partial parsing dates (#47872) backport(#46814)
Enable partial parsing of date part. This is making the behaviour in java.time implementation the same as with joda.
2018, 2018-01 and 2018-01-01 are all valid dates for date_optional_time or strict_date_optional_time
closes #45284
closes #47473
2019-10-11 11:17:19 +02:00
Hendrik Muhs fd1c4c198a [Transform] fixes tests which might fail due to auto-stop (#47867)
Batch transforms automatically stop after all data has processed, therefore tests can not reliable test the state. This change rewrites tests to remove the unreliable tests or use continuous transforms instead as they do not auto-stop.

fixes #47441
2019-10-11 11:10:38 +02:00
Przemysław Witek d210bfa888
[7.x] Add MlClientDocumentationIT tests for classification. (#47569) (#47896) 2019-10-11 10:19:55 +02:00
Alexander Reelsen e60221d2bd Update jakarta mail dependency to 1.6.4 (#47810)
This one contains a few small bugfixes, see https://eclipse-ee4j.github.io/mail/docs/CHANGES.txt
2019-10-11 09:24:11 +02:00
Alpar Torok ec91fa5df1 Make the run task honor tests.es properties (#47860)
* Make the run task honor tests.es properties

Closes #47797

With this PR we now again honor tests.es and tests.heap.size

* remove debuging
2019-10-11 04:38:32 +03:00
Matt Braymer-Hayes d315726c23 [DOCS] Fix gcloud command typo in discovery-gce.asciidoc. (#47312) 2019-10-10 14:21:46 -04:00
Armin Braun 48823b1112
Fix SLMSnapshotBlockingIntegTests (#47841) (#47863)
One of the tests in this suit stops a master node,
plus we're doing other node starts in this suit.
=> the internal test cluster should be TEST and not `SUITE`
scoped to avoid random failures like the one in #47834

Closes #47834
2019-10-10 18:41:57 +02:00
Zachary Tong 2de3411c9c Make sibling pipeline agg ctor's protected (#42808)
SiblingPipelineAggregator is a public interfaces,
but the ctor was package-private.  These should be protected so that
plugin authors can extend and implement their own sibling pipeline agg.
2019-10-10 12:31:14 -04:00
Marios Trivyzas 59b3294bc9 SQL: Implement DATEADD function (#47747)
Implement DATEADD/TIMESTAMPADD function as per the MS-SQL spec:
https://docs.microsoft.com/en-us/sql/t-sql/functions/dateadd-transact-sql?view=sql-server-2017
which allows a user to add/subtract specified number of specified units
to/from a date/datetime field/expression.

Closes: #47746
(cherry picked from commit e624bc281bebb4bbe0b0c2e0a8cbc712e50097a8)
2019-10-10 16:22:13 +02:00
Igor Motov b5afa95fd8 Fix Mute RunDataFrameAnalyticsIT.testOutlierDetectionStopAndRestart
Tracked by #47612
2019-10-10 18:17:01 +04:00
Igor Motov 17433e79d8 Mute RunDataFrameAnalyticsIT.testOutlierDetectionStopAndRestart
Tracked by #47612
2019-10-10 17:56:23 +04:00
James Rodewig a8e027551a
[DOCS] Document `bytes` and `time` params in cat API docs (#47672) (#47854) 2019-10-10 09:48:40 -04:00
Costin Leau dc6f0f9dc7 SQL: Re-enable muted test
Close #47080

(cherry picked from commit 63a0aa7b392f565ea01ac478fec1dd91a80202e5)
2019-10-10 15:47:47 +03:00
Martijn van Groningen 102016d571
Merge remote-tracking branch 'es/7.x' into enrich-7.x 2019-10-10 14:44:05 +02:00
Christoph Büscher f07de06cdd
Ensure random timestamps are within search boundary (#38753) (#47787)
The random timestamps were landing too close to the current time,
so an unlucky rollup interval would round such that the doc wasn't
included in the search range (and thus not "rolled up") which
would then fail the test.

The fix is to make sure the timestamp of all docs is sufficiently behind
'now' that the possible rounding intervals will always include them.

Backport of #38753 to 7.x where the test was still muted.
2019-10-10 14:38:01 +02:00
Marios Trivyzas c1f30e34ff SQL: Refactor binary date time functions (#47786)
Refactor DateTrunc and DatePart to use separate Pipe classes which
allows the removal of the BinaryDateOperation enum.

(cherry picked from commit a6075e7718dff94a90dbc0795dd924dcb7641092)
2019-10-10 13:52:41 +02:00
Andrei Stefan 6a4bf5de2c SQL: make date/datetime and interval types compatible in conditional functions (#47595)
(cherry picked from commit 6ff953e6396d7cc90640419aee5d036954e2eae3)
2019-10-10 13:58:35 +03:00
István Zoltán Szabó 9eac8bf2a8 [DOCS] Adds supported fields section to the PUT DFA API description (#47842) 2019-10-10 12:42:54 +02:00
David Turner ecb20ebc6c More bootstrap docs tweaks (#47809)
Clarifies not to set `cluster.initial_master_nodes` on nodes that are joining
an existing cluster.

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2019-10-10 09:55:30 +01:00
Jim Ferenczi dc39196ea4 Fix tag in the search request timeout option docs (#47776)
and add missing parentheses `search_timeout` param
2019-10-10 10:35:44 +02:00
Jim Ferenczi bd6e2592a7 Remove the SearchContext from the highlighter context (#47733)
Today built-in highlighter and plugins have access to the SearchContext through the
highlighter context. However most of the information exposed in the SearchContext are not needed and a QueryShardContext
would be enough to perform highlighting. This change replaces the SearchContext by the informations that are absolutely
required by highlighter: a QueryShardContext and the SearchContextHighlight. This change allows to reduce the exposure of the
complex SearchContext and remove the needs to clone it in the percolator sub phase.

Relates #47198
Relates #46523
2019-10-10 10:34:10 +02:00
Hendrik Muhs 0e7869128a
[7.5][Transform] introduce new roles and deprecate old ones (#47780) (#47819)
deprecate data_frame_transforms_{user,admin} roles and introduce transform_{user,admin} roles as replacement
2019-10-10 10:31:24 +02:00
Alpar Torok fbbe04b801 Add a verifyVersions to the test FW (#47192)
The test FW has a method to check that it's implementation of getting
index and wire compatible versions as well as reasoning about which
version is released or not produces the same rezults as the simillar
implementation in the build.

This PR adds the `verifyVersions` task to the test FW so we have one
task to check everything related to versions.
2019-10-10 11:23:56 +03:00
Jim Ferenczi 3d334a262b Ensure that we don't call listener twice when detecting a partial failure in _search (#47694)
This change fixes a bug that can occur when a shard failure is detected while we build
the search response and accept partial failures in set to false. In this case we currently
call onFailure on the provided listener but also continue the search as if the failure didn't
occur. This can lead to a listener called twice, once with onFailure and once with onSuccess
which is forbidden by design.
2019-10-10 09:59:49 +02:00
Martijn van Groningen aace42d38d
Add HLRC support for enrich stats API (#47306)
This PR also includes HLRC docs for the enrich stats api.

Relates to #32789
2019-10-10 09:08:29 +02:00
Martijn van Groningen 19393fc5a7
match processor should handler values other than string properly (#47419)
Currently if the document being ingested contains another field value
than a string then the processor fails with an error.

This commit changes the match processor to handle number values
and array values correctly.

If a json array is detected then the `terms` query is used instead
of the `term` query.
2019-10-10 08:49:17 +02:00
Mark Vieira 5825d2df83
Resolve more Gradle task validation warnings (#47825)
(cherry picked from commit 82e3d5ea31101f4c27e69162684c3aa4fb2193a4)

# Conflicts:
#	buildSrc/src/main/groovy/org/elasticsearch/gradle/doc/RestTestsFromSnippetsTask.groovy
2019-10-09 15:42:49 -07:00
Mark Vieira 0360a18f61
Mute test SLMSnapshotBlockingIntegTests.testRetentionWhileSnapshotInProgress
Signed-off-by: Mark Vieira <portugee@gmail.com>
(cherry picked from commit a8a7477c396554926f260d210364f009d85ae5f2)
2019-10-09 15:38:24 -07:00
dengweisysu dc4224fbdf Sync translog without lock before trim unreferenced readers (#47790)
This commit is similar to the optimization made in #45765. With this
change, we fsync most of the data of the current generation without
holding writeLock when trimming unreferenced readers.

Relates #45765
2019-10-09 17:56:30 -04:00
Armin Braun 302e09decf
Simplify some Common ActionRunnable Uses (#47799) (#47828)
Especially in the snapshot code there's a lot
of logic chaining `ActionRunnables` in tricky
ways now and the code is getting hard to follow.
This change introduces two convinience methods that
make it clear that a wrapped listener is invoked with
certainty in some trickier spots and shortens the code a bit.
2019-10-09 23:29:50 +02:00
Ryan Ernst 14b979a7bc Make internal project flag overrideable by tests (#47757)
In order to work with external elasticsearch plugins, some parts of
build-tools need to know when the current build is part of the
elasticsearch project or an external build. We identify these "internal"
builds by a marker in our buildSrc classpath. However, some build-tools
integ tests need to override this flag to test both the external and
internal behavior.

This commit moves the storage of the flag indicating whether we are
running in an internal build to global build info. This will allow
testkit projects to override the flag.
2019-10-09 12:01:30 -07:00
Gordon Brown 9b3790d4f2 Mute "Test All Indexes Lifecycle Explain" (#47317) 2019-10-09 21:32:58 +04:00
Jack Conradson 076d3073b5 Move binding member field generation to Painless semantic pass (#47739)
This adds an SField node that operates similarly to SFunction as a top level 
node meant only for use in an SClass node. Member fields are generated 
for both class bindings and instance bindings using the new SField node 
during the semantic pass, and information is no longer passed through 
Globals for this during the write pass.
2019-10-09 10:24:53 -07:00
Tanguy Leroux 8f86469d3f
Do not auto-follow closed indices (#47721) (#47800)
Backport of (#47721) for 7.x.

Similarly to #47582, Auto-follow patterns creates following
indices as long as the remote index matches the pattern and
the remote primary shards are all started. But since 7.2 closed
indices are also replicated, and it does not play well with CCR
auto-follow patterns as they create following indices for closed
leader indices too.

This commit changes the getLeaderIndicesToFollow() so that closed
indices are excluded from auto-follow patterns.
2019-10-09 19:16:23 +02:00
Ryan Ernst 813f44cec8 Move test seed setup to global build info (#47763)
The global build info plugin prints high level information about the
build, including the test seed. However, BuildPlugin sets up the test
seed, which creates an odd implicit dependency on it. This commit moves
the intialization of the testSeed property into the global build info.
2019-10-09 09:59:19 -07:00
Igor Motov 12e4e7ef54 Geo: implement proper handling of out of bounds geo points (#47734)
This is the first iteration in improving of handling of out of
bounds geopoints with a latitude outside of the -90 - +90 range
and a longitude outside of the -180 - +180 range.

Relates to #43916
2019-10-09 20:30:59 +04:00