Commit Graph

44091 Commits

Author SHA1 Message Date
Nhat Nguyen 76fb573569
Do not allow put mapping on follower (#37675)
Today, the mapping on the follower is managed and replicated from its
leader index by the ShardFollowTask. Thus, we should prevent users
from modifying the mapping on the follower indices.

Relates #30086
2019-01-24 12:13:00 -05:00
David Turner 187b233571 Read m_m_n from cluster states from 6.7
This completes the BWC serialisation changes required for a 6.7 master to
inform other nodes of the node-level value of the `minimum_master_nodes`
setting.

Relates #37701, #37811
2019-01-24 17:05:49 +00:00
David Roberts 0e36adc35f Mute SimpleClusterStateIT testMetadataVersion
Due to https://github.com/elastic/elasticsearch/issues/37820
2019-01-24 16:50:55 +00:00
Marios Trivyzas 74b6f308e9
SQL: Fix issue with complex expression as args of PERCENTILE/_RANK (#37102)
When the arguements of PERCENTILE and PERCENTILE_RANK can be folded,
the `ConstantFolding` rule kicks in and calls the `replaceChildren()`
method on `InnerAggregate` which is created from the aggregation rules
of the `Optimizerz. `InnerAggregate` in turn, cannot implement the method
as the logic of creating a new `InnerAggregate` instance from a list of
`Expression`s resides in the Optimizer. So, instead, `ConstantFolding`
should be applied before any of the aggregations related rules.

Fixes: #37099
2019-01-24 18:40:20 +02:00
Alpar Torok f6b6f927ec
Fail randomized testing tasks if no tests ran (#37764)
Reverts #36259 in part to make randomized test fail if no tests are ran.
This is useful when filtering tests as it's easy to make a typo and
think the test ran trough successfully.
2019-01-24 17:33:08 +02:00
Alpar Torok 37768b7eac
Testing conventions now checks for tests in main (#37321)
* Testing conventions now checks for tests in main

This is the last outstanding feature of the old NamingConventionsTask,
so time to remove it.

* PR review
2019-01-24 17:30:50 +02:00
Alpar Torok e7aa7e909a
Switch testclusters to use nio (#37365) 2019-01-24 17:30:24 +02:00
niloct ddee1926f9 Update update-by-query.asciidoc (#37555)
Similar fix as #37370.
2019-01-24 16:27:42 +01:00
Alpar Torok 4e08cca6bc
Ground work to start up the docker image in the build (#37754)
This change adds a docker compose configuration that's used with
the `elasticsearch.test.fixtures` plugin to start up the image
and check that the TCP ports are up.

We can build on this to add other checks for culster health,
run REST tests, etc.

We can add multiple containers and configurations to the compose
file (e.x. test different env vars) and form clusters.
2019-01-24 17:26:42 +02:00
David Roberts bd02ca4b7b Mute NoMasterNodeIT testNoMasterActionsWriteMasterBlock
Due to https://github.com/elastic/elasticsearch/issues/37823
2019-01-24 15:17:13 +00:00
Ioannis Kakavas 265710e658
Better msg on unmapped principal attribute (#37805)
When we can't map the principal attribute from the configured SAML
attribute in the realm settings, we can't complete the
authentication. We return an error to the user indicating this and
we present them with a list of attributes we did get from the SAML
response to point out that the expected one was not part of that
list. This list will never contain the NameIDs though as they are
not part of the SAMLAttribute list. So we might have a NameID but
just with a different format.
2019-01-24 17:05:01 +02:00
Nhat Nguyen a6abb28abf
Fix InternalEngineTests#assertOpsOnPrimary (#37746)
The assertion `assertOpsOnPrimary` does not store seq_no and primary
term of successful deletes to the `lastOpSeqNo` and `lastOpTerm`. This
leads to failures of the subsequence CAS deletes or indexes with seq_no
and term. Moreover, this assertion trips a translog assertion because it
bumps the primary term of some operations but not the primary term of
the engine.

Relates #36467
Closes #37684
2019-01-24 10:02:48 -05:00
David Roberts a81931bb2a Mute DynamicMappingIT testMappingsPropagatedToMasterNodeImmediately
Due to https://github.com/elastic/elasticsearch/issues/37816
2019-01-24 14:32:44 +00:00
Andrei Stefan 163a27b93c
SQL: Fix BasicFormatter NPE (#37804) 2019-01-24 15:40:51 +02:00
Jason Tedor 7517e3a7bd
Optimize warning header de-duplication (#37725)
Now that warning headers no longer contain a timestamp of when the
warning was generated, we no longer need to extract the warning value
from the warning to determine whether or not the warning value is
duplicated. Instead, we can compare strings directly.

Further, when de-duplicating warning headers, are constantly rebuilding
sets. Instead of doing that, we can carry about the set with us and
rebuild it if we find a new warning value.

This commit applies both of these optimizations.
2019-01-24 08:39:24 -05:00
Yannick Welsch feab59df03
Bubble exceptions up in ClusterApplierService (#37729)
Exceptions thrown by the cluster applier service's settings and cluster appliers are bubbled up, and
block the state from being applied instead of silently being ignored. In combination with the cluster
state publishing lag detector, this will throw a node out of the cluster that can't properly apply
cluster state updates.
2019-01-24 14:09:03 +01:00
Marios Trivyzas 9357929309
SQL: Improve handling of invalid args for PERCENTILE/PERCENTILE_RANK (#37803)
Improve the Exception and the error message returned when 2nd argument
of PERCENTILE and PERCENTILE_RANK is not a constant.
2019-01-24 15:03:49 +02:00
Simon Willnauer c7b16162ae
Remove unused ThreadBarrier class (#37666)
This class is pretty complex and only used in a test where we can simply
fail the test with an assertion error.
2019-01-24 13:52:22 +01:00
Yulong 20533c5990
Add built-in user and role for code plugin (#37030)
* Add built-in roles for code plugin

* Fix rest-client get-roles test count

* Fix broken test
2019-01-24 20:12:32 +08:00
Alpar Torok bc20142791
Consolidate testclusters tests into a single project (#37362) 2019-01-24 13:47:21 +02:00
Yannick Welsch 2bf269e628 Fix docs for MappingUpdatedAction
Follow-up to #31140
2019-01-24 12:44:36 +01:00
Marios Trivyzas f707fa9e0a
SQL: Introduce SQL DATE data type (#37693)
* SQL: Introduce SQL DATE data type

Support ANSI SQL's DATE type by introducing a runtime-only
ES SQL date type.

Closes: #37340
2019-01-24 13:41:58 +02:00
Boaz Leskes b6317ed70b disabling bwc test while backporting https://github.com/elastic/elasticsearch/pull/37639 2019-01-24 12:01:32 +01:00
David Roberts bcf5a4ca47 Mute ClusterDisruptionIT testAckedIndexing
Due to https://github.com/elastic/elasticsearch/issues/37810
2019-01-24 10:58:02 +00:00
Yannick Welsch 64adb5ad5b
Set acking timeout to 0 on dynamic mapping update (#31140)
As acking can fail for any reason (unrelated node being too slow, node disconnecting), it should not
be required for acking to succeed in order for index requests with dynamic mapping updates to
successfully complete.

Relates to #30672 and Closes #30844
2019-01-24 11:39:46 +01:00
Albert Zaharovits b6936e3c1e
Remove index audit output type (#37707)
This commit removes the Index Audit Output type, following its deprecation
in 6.7 by 8765a31d4e6770. It also adds the migration notice (settings notice).

In general, the problem with the index audit output is that event indexing
can be slower than the rate with which audit events are generated,
especially during the daily rollovers or the rolling cluster upgrades.
In this situation audit events will be lost which is a terrible failure situation
for an audit system.
Besides of the settings under the `xpack.security.audit.index` namespace, the
`xpack.security.audit.outputs` setting has also been deprecated and will be
removed in 7. Although explicitly configuring the logfile output does not touch
any deprecation bits, this setting is made redundant in 7 so this PR deprecates
it as well.

Relates #29881
2019-01-24 12:36:10 +02:00
David Roberts f12bfb4684 Mute FollowerFailOverIT testReadRequestsReturnsLatestMappingVersion
Due to https://github.com/elastic/elasticsearch/issues/37807
2019-01-24 09:58:50 +00:00
David Kyle e1226f69b7
[ML] Increase close job timeout and lower the max number (#37770) 2019-01-24 09:18:48 +00:00
Armin Braun 36889e8a2f
Remove Custom Listeners from SnapshotsService (#37629)
* Remove Custom Listeners from SnapshotsService

Motivations:
    * Shorten the code some more
    * Use ActionListener#wrap to get easy to reason about behavior in failure scenarios
    * Remove duplication in the logic of handling snapshot completion listeners (listeners removing themselves and comparing snapshots to their targets)
        * Also here, move all listener handling into `SnapshotsService` and remove custom listener class by putting listeners in a map
2019-01-24 10:11:18 +01:00
David Turner bdef2ab8c0
Use m_m_nodes from Zen1 master for Zen2 bootstrap (#37701)
Today we support a smooth rolling upgrade from Zen1 to Zen2 by automatically
bootstrapping the cluster once all the Zen1 nodes have left, as long as the
`minimum_master_nodes` count is satisfied. However this means that Zen2 nodes
also require the `minimum_master_nodes` setting for this one specific and
transient situation.

Since nodes only perform this automatic bootstrapping if they previously
belonged to a Zen1 cluster, they can keep track of the `minimum_master_nodes`
setting from the previous master instead of requiring it to be set on the Zen2
node.
2019-01-24 08:57:40 +00:00
Martijn van Groningen 2908ca1b35
Fix index filtering in follow info api. (#37752)
The filtering by follower index was completely broken.
Also the wrong persistent tasks were selected, causing the
wrong status to be reported.

Closes #37738
2019-01-24 08:50:23 +01:00
Ryan Ernst d9d13f3414
Use project dependency instead of substitutions for distributions (#37730)
Currently integration tests which use either bwc snapshot versions or
the current version of elasticsearch depend on project substitutions to
link to the build of those artifacts. Likewise, vagrant tests use
dependency substitutions to get to bwc snapshots of rpm and debs.
This commit changes those to depend on the relevant project/configuration
and removes the dependency substitutions for distributions we do not
publish.
2019-01-23 23:41:21 -08:00
Michael Basnight 04c64147bd
Update authenticate to allow unknown fields (#37713)
AuthenticateResponse did not allow unknown fields. This commit fixes the
test and ConstructingObjectParser such that it does now allow unknown
fields.

Relates #36938
2019-01-23 22:14:01 -06:00
Michael Basnight 944972a249
Deprecate HLRC EmptyResponse used by security (#37540)
The EmptyResponse is essentially the same as returning a boolean, which
is done in other places. This commit deprecates all the existing
EmptyResponse methods and creates new boolean methods that have method
params reordered so they can exist with the deprecated methods. A
followup PR in master will remove the existing deprecated methods, fix
the parameter ordering and deprecate the incorrectly ordered parameter
methods.

Relates #36938
2019-01-23 22:13:16 -06:00
Ryan Ernst 49b8b07758
Use explicit version for build-tools in example plugin integ tests (#37792)
The example plugins are currently built within the build-tools integ
tests as a means to ensure the gradle plugin works for external plugin
builds. These tests generate a dummy build.gradle, and a dummy local
maven repository to find the local builds dependencies in. Currently
that build-tools dependency uses "+" as the version. However, this
allows gradle to find the "latest" version, and unfortunately gradle has
its own plugin repository which is apparently connected to jcenter. This
recently triggered a flood of CI failures when jcenter suddenly pulled
alpha2, and all builds started trying to use that instead of the locally
built build-tools. This commit uses the explicit version of build-tools
that was build locally, which will cause resolution to stop when the
local repo is first checked.
2019-01-23 17:39:13 -08:00
Mayya Sharipova fdb66039d4
Change `rational` to `saturation` in script_score (#37766)
This change of the function name is necessary for conformity
with feature queries.

Closes #37714
2019-01-23 14:28:20 -05:00
Mayya Sharipova c8565fe692
Deprecate types in get field mapping API (#37667)
- Add deprecation warning to RestGetFieldMappingAction
- Add two new java HRLC classes GetFieldMappingsRequest and
GetFieldMappingsResponse. These classes use new typeless forms
of a request and response, and differ in that from the server
versions.

Relates to #35190
2019-01-23 14:24:35 -05:00
Tim Brooks f45b5fedb5
Add ability to listen to group of affix settings (#37679)
Currently we have the ability to listen for setting changes to two group
affix settings. However, it is possible that we might have the need to
listen to more than two. This commit adds a method that allows consumer
to listen to a list of affix settings for changes.
2019-01-23 12:05:39 -07:00
Nhat Nguyen 0096f1b2e4
Ensure changes requests return the latest mapping version (#37633)
Today we keep the mapping on the follower in sync with the leader's
using the mapping version from changes requests. There are two rare
cases where the mapping on the follower is not synced properly:

1. The returned mapping version (from ClusterService) is outdated than
the actual mapping. This happens because we expose the latest cluster
state in ClusterService after applying it to IndexService.

2. It's possible for the FollowTask to receive an outdated mapping than
the min_required_mapping. In that case, it should fetch the mapping
again; otherwise, the follower won't have the right mapping.

Relates to #31140
2019-01-23 13:41:13 -05:00
Armin Braun d7fe4e57fe
Make Minio Setup more Reliable (#37747)
* Retry starting Minio five times in case we run into a race between finding the free port and starting it up
* Closes #37680
2019-01-23 19:05:25 +01:00
Jason Tedor 169cb38778
Liberalize StreamOutput#writeStringList (#37768)
In some cases we only have a string collection instead of a string list
that we want to serialize out. We have a convenience method for writing
a list of strings, but no such method for writing a collection of
strings. Yet, a list of strings is a collection of strings, so we can
simply liberalize StreamOutput#writeStringList to be more generous in
the collections that it accepts and write out collections of strings
too. On the other side, we do not have a convenience method for reading
a list of strings. This commit addresses both of these issues.
2019-01-23 12:52:17 -05:00
Benjamin Trent 1c2ae9185c
Add PersistentTasksClusterService::unassignPersistentTask method (#37576)
* Add PersistentTasksClusterService::unassignPersistentTask method

* adding cancellation test

* Adding integration test for unallocating tasks from a node

* Addressing review comments

* adressing minor PR comments
2019-01-23 11:48:32 -06:00
Igor Motov e3672aa551
Tests: disable testRandomGeoCollectionQuery on tiny polygons (#37579)
Due to https://issues.apache.org/jira/browse/LUCENE-8634 this test
may fail if a really tiny polygon is generated. This commit checks for
tiny polygons and skips the final check, which is expected to fail
until the lucene bug is fixed and new version of lucene is released.
2019-01-23 12:25:54 -05:00
Lee Hinman 427bc7f940
Use ILM for Watcher history deletion (#37443)
* Use ILM for Watcher history deletion

This commit adds an index lifecycle policy for the `.watch-history-*` indices.
This policy is automatically used for all new watch history indices.

This does not yet remove the automatic cleanup that the monitoring plugin does
for the .watch-history indices, and it does not touch the
`xpack.watcher.history.cleaner_service.enabled` setting.

Relates to #32041
2019-01-23 10:18:08 -07:00
Julie Tibshirani f0fc6e8003
Make sure PutMappingRequest accepts content types other than JSON. (#37720) 2019-01-23 08:51:05 -08:00
Lee Hinman 647e225698
Retry ILM steps that fail due to SnapshotInProgressException (#37624)
Some steps, such as steps that delete, close, or freeze an index, may fail due to a currently running snapshot of the index. In those cases, rather than move to the ERROR step, we should retry the step when the snapshot has completed.

This change adds an abstract step (`AsyncRetryDuringSnapshotActionStep`) that certain steps (like the ones I mentioned above) can extend that will automatically handle a situation where a snapshot is taking place. When a `SnapshotInProgressException` is received by the listener wrapper, a `ClusterStateObserver` listener is registered to wait until the snapshot has completed, re-running the ILM action when no snapshot is occurring.

This also adds integration tests for these scenarios (thanks to @talevy in #37552).

Resolves #37541
2019-01-23 09:46:31 -07:00
David Kyle d193ca8aae
Use disassociate in preference to deassociate (#37704) 2019-01-23 16:06:25 +00:00
Armin Braun 2439f68745
Delete Redundant RoutingServiceTests (#37750)
* This test compleletly overrode the `reroute` method and hence did nothing put test the override itself
   * Removed the test since it tests nothing and simplified `reroute` accordingly
2019-01-23 16:39:02 +01:00
Nhat Nguyen 6a9838359c
Always return metadata version if metadata is requested (#37674)
If the indices of a ClusterStateRequest are specified, we fail to
include the cluster state metadata version in the response.

Relates  #37633
2019-01-23 10:24:51 -05:00
David Roberts 6a5d9d942a [TEST] Mute MlMappingsUpgradeIT testMappingsUpgrade
Due to https://github.com/elastic/elasticsearch/issues/37763
2019-01-23 13:50:31 +00:00