Commit Graph

44179 Commits

Author SHA1 Message Date
Jake Landis 99b75a9bdf
deprecate types for watcher (#37594)
This commit adds deprecation warnings for index actions
and search actions when executed via watcher. Unit and 
integration tests updated accordingly. 

relates #35190
2019-01-28 13:46:43 -06:00
Nhat Nguyen 9ceb218d85 Adjust bwc version for put mapping requests
Relates #37675
2019-01-28 10:57:11 -05:00
Luca Cavanna 0a850f032b Handle deprecation warnings in a permissive manner
Relates to #37290
2019-01-28 16:36:39 +01:00
Benjamin Trent 7e4c0e6991
ML: Adds set_upgrade_mode API endpoint (#37837)
* ML: Add MlMetadata.upgrade_mode and API

* Adding tests

* Adding wait conditionals for the upgrade_mode call to return

* Adding tests

* adjusting format and tests

* Adjusting wait conditions for api return and msgs

* adjusting doc tests

* adding upgrade mode tests to black list
2019-01-28 09:07:30 -06:00
Martijn van Groningen e401ab1724
Handle deprecation warnings in a permissive manner.
Closes #37920
2019-01-28 15:02:50 +01:00
Armin Braun 0d109396fa
Increase Timeout in #testSnapshotCanceled (#37890)
* The test failure reported in the issue looks like a mere timeout. Logging suggestst hat the snapshot completes/aborts correctly but the busy
loop polling the snapshot state times out too early.
* Closes #37888
2019-01-28 14:13:02 +01:00
Luca Cavanna 519423b8f3 Mute failing full-cluster-restart tests
Relates to #37920
2019-01-28 14:07:19 +01:00
Martijn van Groningen 445db97867
each full cluster restart round should use its own repository,
otherwise snapshots from e.g. 6.5.5 to current and 6.6.0 to current
full cluster restart round collides.
2019-01-28 13:56:02 +01:00
Luca Cavanna a9adc16922 Mute failing SearchQueryIT test
Relates to #37814
2019-01-28 13:41:13 +01:00
Alpar Torok 64b98db973
Add an alias for :server:integTest so it runs as part of internalClusterTest (#37910) 2019-01-28 14:26:22 +02:00
Jason Tedor 194cdfe208
Sync retention leases on expiration (#37902)
This commit introduces a sync of retention leases when a retention lease
expires. As expiration of retention leases is lazy, their expiration is
managed only when getting the current retention leases from the
replication tracker. At this point, we callback to our full retention
lease sync to sync and flush these on all shard copies. With this
change, replicas do not locally manage expiration of retention leases;
instead, that is done only on the primary.
2019-01-28 07:11:51 -05:00
Tanguy Leroux 758eb9d451 Track accurate total hits in CloseIndexIT
The test was not using the TRACK_TOTAL_HITS_ACCURATE and thus
encountered a different issue tracked in #37907. In the meanwhile
we can adapt the test to not fail anymore.

Closes #37897
2019-01-28 11:30:20 +01:00
David Kyle c0409fb9f0
[ML] Marginal gains in slow multi node QA tests (#37825)
Move 2 tests that are simple rest tests and out of the QA suite and cut the number
of post data calls in ForecastIT
2019-01-28 10:00:59 +00:00
David Turner 2a610abef2
Document that auto_create_index is dynamic (#37903)
We changed the `action.auto_create_index` setting to be a dynamic cluster-level
setting in #20274 but today the reference manual indicates that it is still a
static node-level setting. This commit addresses this, and clarifies the
semantics of patterns that may both permit and forbid the creation of certain
indices.

Relates #7513
2019-01-28 09:45:55 +00:00
David Roberts 57d321ed5f
[ML] Tighten up use of aliases rather than concrete indices (#37874)
We have read and write aliases for the ML results indices.  However,
the job still had methods that purported to reliably return the name
of the concrete results index being used by the job.  After reindexing
prior to upgrade to 7.x this will be wrong, so the method has been
renamed and the comments made more explicit to say the returned index
name may not be the actual concrete index name for the lifetime of the
job.  Additionally, the selection of indices when deleting the job
has been changed so that it works regardless of concrete index names.

All these changes are nice-to-have for 6.7 and 7.0, but will become
critical if we add rolling results indices in the 7.x release stream
as 6.7 and 7.0 nodes may have to operate in a mixed version cluster
that includes a version that can roll results indices.
2019-01-28 09:38:46 +00:00
David Turner a5f578f7ea
Fix duplicate note x-refs in mapping.asciidoc (#37904)
The docs silently accept duplicate note markers (such as `<3>` here) but
formats them in an unexpected way. This change removes this duplication so that
the rendered documentation looks as intended.
2019-01-28 09:29:07 +00:00
Martijn van Groningen 4e1a779773
Prepare ShardFollowNodeTask to bootstrap when it fall behind leader shard (#37562)
* Changed `LuceneSnapshot` to throw an `OperationsMissingException` if the requested ops are missing.
* Changed the shard changes api to handle the `OperationsMissingException` and wrap the exception into `ResourceNotFound` exception and include metadata to indicate the requested range can no longer be retrieved.
* Changed `ShardFollowNodeTask` to handle this `ResourceNotFound` exception with the included metdata header.

Relates to #35975
2019-01-28 09:30:04 +01:00
Jim Ferenczi a056804831 Track total hits in tests that index more than 10,000 docs
This change sets track_total_hits to true on a test that requires
to check the total hits of a query that can return more than 10,000 docs.

Closes #37895
2019-01-28 09:24:32 +01:00
Dimitrios Liappis 290c6637c2
Refactor into appropriate uses of scheduleUnlessShuttingDown (#37709)
Replace `threadPool().schedule()` / catch
`EsRejectedExecutionException` pattern with direct calls to
`ThreadPool#scheduleUnlessShuttingDown()`.

Closes #36318
2019-01-28 10:01:26 +02:00
Julie Tibshirani b1735aa93b
Support both typed and typeless 'get mapping' requests in the HLRC. (#37796)
From previous PRs, we've already added support for include_type_name to
the get mapping API. We had also taken an approach to the HLRC where the
server-side `GetMappingResponse#fromXContent` could only handle typeless
input.

This PR updates the HLRC for 'get mapping' to be in line with our new approach:

* Add a typeless 'get mappings' method to the Java HLRC, that accepts new
client-side request and response objects. This new response only handles
typeless mapping definitions.
* Switch the old version of `GetMappingResponse` back to expecting typed
mappings, and deprecate the corresponding method on the HLRC.

Finally, the PR also does some small, related clean-up around 'get field mappings'.
2019-01-27 16:02:22 -08:00
Albert Zaharovits 66ddd8d2f7
Create snapshot role (#35820)
This commit introduces the `create_snapshot` cluster privilege and
the `snapshot_user` role.
This role is to be used by "cronable" tools that call the snapshot API
periodically without recurring to the `manage` cluster privilege. The
`create_snapshot` cluster privilege is much more limited compared to
the `manage` privilege.

The `snapshot_user` role grants the privileges to view the metadata of
all indices (including restricted ones, i.e. .security). It obviously grants the
create snapshot privilege but the repository has to be created using another
role. In addition, it grants the privileges to (only) GET repositories and
snapshots, but not create and delete them.

The role does not allow to create repositories. This distinction is important
because snapshotting equates to the `read` index privilege if the user has
control of the snapshot destination, but this is not the case in this instance,
because the role does not grant control over repository configuration.
2019-01-27 23:07:32 +02:00
Jason Tedor f24dce1122
Fix newlines in retention lease sync action tests
There is a method invocation here spanning multiple lines. This commit
breaks it up into a line per parameter as this is friendlier to future
changes and diffs.
2019-01-27 08:16:14 -05:00
Jason Tedor 3801925cf0
Copy retention leases under lock
When adding a retention lease, we make a reference copy of the retention
leases under lock and then make a copy of that collection outside of the
lock. However, since we merely copied a reference to the retention
leases, after leaving a lock the underlying collection could change on
us. Rather, we want to copy these under lock. This commit adds a
dedicated method for doing this, asserts that we hold a lock when we use
this method, and changes adding a retention lease to use this method.

This commit was intended to be included with #37398 but was pushed to
the wrong branch.
2019-01-27 08:13:47 -05:00
Jason Tedor 5fddb631a2
Introduce retention lease syncing (#37398)
This commit introduces retention lease syncing from the primary to its
replicas when a new retention lease is added. A follow-up commit will
add a background sync of the retention leases as well so that renewed
retention leases are synced to replicas.
2019-01-27 07:49:56 -05:00
David Roberts cb134470c1
[TEST] Fix MlMappingsUpgradeIT testMappingsUpgrade (#37769)
Made the test tolerant to index upgrade being run
in between the old/mixed/upgraded portions.  This
can occur because the rolling upgrade tests all
share the same indices.

Fixes #37763
2019-01-27 08:27:40 +00:00
Nhat Nguyen 780b4c72fe
Make ChannelActionListener a top-level class (#37797)
We start using this class more often. Let's make it a top-level class.
2019-01-26 22:01:30 -05:00
David Roberts f2c0c26d15
[ML] Adjust structure finder for Joda to Java time migration (#37306)
The ML file structure finder has always reported both Joda
and Java time format strings.  This change makes the Java time
format strings the ones that are incorporated into mappings
and ingest pipeline definitions.

The BWC syntax of prepending "8" to these formats is used.
This will need to be removed once Java time format strings
become the default in Elasticsearch.

This commit also removes direct imports of Joda classes in the
structure finder unit tests.  Instead the core Joda BWC class
is used.
2019-01-26 20:19:57 +00:00
Julie Tibshirani afc60bb0e5 Mute DynamicMappingIT#testConflictingDynamicMappings
Tracked in #37898.
2019-01-25 18:09:34 -08:00
Tal Levy eb973a4744 fix GeoHashGridTests precision parsing error
Previously, a hardcoded precision value of 4 was
used by these tests resulting in no approximation
errors. Now that the precision is between 1-12,
precision values of 1 and 2 result in potential
bucketing errors.

This commit adjusts the range to be 4-12.

Fixes #37892.
2019-01-25 17:29:04 -08:00
Julie Tibshirani 58301ead6d Mute IndexShardIT#testMaybeFlush
Tracked in #37896.
2019-01-25 17:12:16 -08:00
Julie Tibshirani 23b0d9b3ed Mute RecoveryWhileUnderLoadIT#testRecoverWhileUnderLoadAllocateReplicasRelocatePrimariesTest
Tracked in #37895.
2019-01-25 16:50:39 -08:00
Julie Tibshirani e41ccdc1a0 Mute GeoWKTShapeParserTests#testParseGeometryCollection
Tracked in #37894.
2019-01-25 16:15:16 -08:00
Julie Tibshirani 24ff23812e Mute TestClustersPluginIT and BuildExamplePluginsIT
Tracked in #37889.
2019-01-25 15:41:56 -08:00
Julie Tibshirani 827ed12146 Mute TasksIT#testTransportBulkTasks
Tracked in #37893.
2019-01-25 15:29:24 -08:00
Sivagurunathan Velayutham a35701e437 Fix potential IllegalCapacityException in LLRC when selecting nodes (#37821) 2019-01-25 15:57:50 -07:00
Julie Tibshirani 7c130d235a Mute CcrRepositoryIT#testFollowerMappingIsUpdated
Tracked in #37887.
2019-01-25 14:55:47 -08:00
Armin Braun 899dfc38bc
Fix S3 Repository ITs When Docker is not Available (#37878)
* Disable Minio fixture and tests that require it when fixtures are disabled or Docker is not available
* Relates #37852
2019-01-25 22:55:29 +01:00
Ryan Ernst 980c13d85c
Pass distribution type through to docs tests (#37885)
This commit fixes the distribution flavor passed to the docs tests to be
the same as the distribution. These two values are now in sync (either
oss or default) for the docs tests.
2019-01-25 13:49:16 -08:00
Julie Tibshirani a4020f4587 Mute SharedClusterSnapshotRestoreIT#testSnapshotCanceledOnRemovedShard
Tracked in #37888.
2019-01-25 13:40:29 -08:00
Marios Trivyzas d1ff450edc
SQL: Fix casting from date to numeric type to use millis (#37869)
Previously casting from a DATE[TIME] type to a numeric (DOUBLE, LONG,
INT, etc. used seconds instead of the epoch millis.

Fixes: #37655
2019-01-25 23:29:10 +02:00
Like eb7bf16427 Migrate o.e.i.r.RecoveryState to Writeable (#37380)
Relates to #34389
2019-01-25 15:52:04 -05:00
Benjamin Trent 9e932f4869
ML: removing unnecessary upgrade code (#37879) 2019-01-25 13:57:41 -06:00
Nhat Nguyen 5cd4dfb0e4
Relax cluster metadata version check (#37834)
If the in_sync_allocations of index-1 or index-2 is changed, the
metadata version will be increased. This leads to the failure in
the metadata version checks. We need to relax them.

Closes #37820
2019-01-25 14:54:13 -05:00
Julie Tibshirani 455f223c3a Mute TransformIntegrationTests#testSearchTransform
Tracked in #37882.
2019-01-25 11:12:45 -08:00
Yuri Astrakhan f1e71be8b2
Refactored GeoHashGrid unit tests (#37832)
* Refactored GeoHashGrid unit tests

This change allows other grid aggregations to reuse the same tests.

The change mostly just moves code to the base classes, trying to
keep changes to a bare minimum.

* rename createInternalGeoHashGridBucket to createInternalGeoGridBucket

* indentation
2019-01-25 13:37:24 -05:00
Zachary Tong afd4618851 Fixes for a few randomized agg tests that fail hasValue() checks
Closes #37743
Closes #37873
2019-01-25 12:39:42 -05:00
Igor Motov 68149b6058
Geo: replace intermediate geo objects with libs/geo (#37721)
Replaces intermediate geo objects built by ShapeBuilders with
objects from the libs/geo hierarchy. This should allow us to build
all geo functionality around a single hierarchy.

Follow up for #35320
2019-01-25 11:37:27 -05:00
Ryan Ernst e88ae99340
Remove NOREPLACE for /etc/elasticsearch in rpm and deb (#37839)
The /etc/elasticsearch directory is currently configured as a config
file with noreplace. However, the directory itself is not config, and
can lead to an entire /etc/elasticsearch.rpmsave directory in some
situations. This commit fixes the ospackage config to not specify those
file bits for the directory itself, but only the files underneath it.
2019-01-25 08:11:48 -08:00
Ryan Ernst acc3cae40c
Remove "reinstall" packaging tests (#37851)
The packaging tests currently have a test which installs elasticsearch,
removes it, modifies ownership of /etc/elasticsearch, and
reinstalls. It then checks that the /etc/elasticsearch directory has
ownership that the package expects. But the recursive change touches
files not owned by the package. In the past this worked because we did a
recursive ownership change within the package postinst. However, that
was recently removed, and thus this test no longer makes sense.
2019-01-25 08:07:49 -08:00
Tanguy Leroux a644bc095c
Add unit tests for ShardStateAction's ShardStartedClusterStateTaskExecutor (#37756) 2019-01-25 16:51:53 +01:00