Commit Graph

46861 Commits

Author SHA1 Message Date
Armin Braun f1ebb82031
Update the gcs chunk_size documentation. (#38749) (#44098)
Remove `1g` from the examples, as the GCS repository chunk_size can be at most 100m.
2019-07-09 09:18:03 +02:00
Armin Braun dc8f8e40eb
Fix DedicatedClusterSnapshotRestoreIT testSnapshotWithStuckNode (#43537) (#44082)
* Fix DedicatedClusterSnapshotRestoreIT testSnapshotWithStuckNode

* See comment in the test: The problem is that when the snapshot delete works out partially on master failover and the retry fails on `SnapshotMissingException` no repository cleanup is run => we still failed even with repo cleanup logic in the delete path now
   * Fixed the test by rerunning a create snapshot and delete loop to clean up the repo before verifying file counts
* Closes #39852
2019-07-09 06:32:08 +02:00
Lisa Cawley 94578a8b47 [DOCS] Defines data frame transform resources (#43996)
Co-Authored-By: István Zoltán Szabó <istvan.szabo@elastic.co>
2019-07-08 17:53:00 -07:00
Ryan Ernst 2b1cd58648 Remove ActionResponse uses from HLRC (#44091)
The rest client does not communicate over the transport protocol.
However, in the move to make all apis supported in the HLRC, some
response classes were copied with extending ActionResponse, which is
meant strictly for the transport protocol. This commit removes uses of
that base class from HLRC.
2019-07-08 17:27:29 -07:00
lcawl cd4021274a [DOCS] Enables testing for create job ML API (#44022) 2019-07-08 11:43:18 -07:00
Lisa Cawley 117f14e0ed
[DOCS] Updates 7.x version in data frame analytics API (#44026) 2019-07-08 11:20:57 -07:00
Lisa Cawley efddbcc1d1 [DOCS] Fixes earliest_record_timestamp data type (#44030) 2019-07-08 10:16:07 -07:00
David Turner 6dce458ecc Randomise retention lease expiry time (#44067)
In today's test suite indices mostly use the default value of `12h` for the
`index.soft_deletes.retention_lease.period` setting, which in the context of
the test suite essentially means "never expires". In fact, the tests should all
behave correctly even if the lease period is much shorter; tests that rely on
leases not expiring should configure their indices appropriately.

This commit randomises the lease expiry time for those indices created during
tests which do not set a specific value for this setting.
2019-07-08 18:29:27 +02:00
Lisa Cawley 4b3f1003b0 [DOCS] Reformat freeze unfreeze APis to use new API format (#43948) 2019-07-08 09:01:06 -07:00
Christoph Büscher 8e8d7667cb [Tests] Fix type inference issue (#44063) 2019-07-08 17:34:35 +02:00
Armin Braun 03332b5aeb
Don't Consistency Check Broken Repository in Test (#43499) (#44071)
* Missed this one in #42189 and it randomly runs into a situation where the broken mock repo is broken such that we can't get to a consistent end state via a delete
* Closes #43498
2019-07-08 17:21:40 +02:00
Tanguy Leroux 251287f89d Check again on-going snapshots/restores of indices before closing (#43873)
Today we prevent any index that is actively snapshotted or restored to be closed. 
This verification is done during the execution of the first phase of index closing 
(ie before blocking the indices).

We should also do this verification again in the last phase of index closing 
(ie after the shard sanity checks and right before actually changing the index 
state and the routing table) because a snapshot/restore could sneak in while
 the shards are verified-before-close.
2019-07-08 17:07:04 +02:00
Alpar Torok 0c8294e633 Make sure the clean task doesn't break test fixtures (#43641)
Use a dedicated fixture dir.
2019-07-08 17:58:27 +03:00
Mark Tozzi 299a52c17d
Enable validating user-supplied missing values on unmapped fields (#43718) (#43940)
Provides a hook for aggregations to introspect the `ValuesSourceType` for a user supplied Missing value on an unmapped field, when the type would otherwise be `ANY`.  Mapped field behavior is unchanged, and still applies the `ValuesSourceType` of the field.  This PR just provides the hook for doing this, no existing aggregations have their behavior changed.
2019-07-08 10:46:23 -04:00
James Rodewig 4390d4a8af [DOCS] Clarify array is not a field datatype (#43931) 2019-07-08 08:58:10 -04:00
Armin Braun 2918363e90
Simplify BlobStoreRepository (Flatten Nested Classes) (#42833) (#44060)
* In the current codebase it is hardly obvious what code operates on a shard and is run by a datanode what code operates on the global metadata and is run on master
   * Fixed by adjusting the method names accordingly
* The nested context classes don't add much if any value, they simply spread out the parameters that go into a shard snapshot create or delete all over the place since their
constructors can be inlined in all spots
   * Fixed by flattening the nested classes into BlobStoreRepository
* Also:
  * Inlined the other single use inner classes
2019-07-08 14:57:27 +02:00
Armin Braun afe81fd625
Some Cleanup in Test Framework (#44039) (#44059)
* Remove some obvious dead code
* Move assert methods that were only used in a single test class to the child they belong to
* Inline some redundant methods
2019-07-08 14:15:31 +02:00
David Kyle 5fc12917c3 Data frame task failure does not make a 500 response (#44058)
Data frame task responses had logic to return a HTTP 500 status code if there was 
any node or task failures even if other tasks in the same request reported correctly. 
This is different to how other task responses are handled where a 200 is always 
returned leaving the client should check for failures. Returning a 500 also breaks
the high level rest client so always return a 200

Closes #44011
2019-07-08 11:53:11 +01:00
David Turner 3f3bcb23c2 AwaitsFix testForceStaleReplicaToBePromotedToPrimary
Relates #44049
2019-07-08 11:26:57 +01:00
David Turner 3129f5b42e Do not copy initial recovery filter during split (#44053)
If an index is the result of a shrink then it will have a value set for
`index.routing.allocation.initial_recovery._id`. If this index is subsequently
split then this value will be copied over, forcing the initial allocation of
the split shards to occur on the node on which the shrink took place. Moreover
if this node no longer exists then the split will fail.  This commit suppresses
the copying of this setting when splitting an index.

Fixes #43955
2019-07-08 10:32:05 +01:00
Armin Braun af9b98e81c
Recursively Delete Unreferenced Index Directories (#42189) (#44051)
* Use ability to list child "folders" in the blob store to implement recursive delete on all stale index folders when cleaning up instead of using the diff between two `RepositoryData` instances to cover aborted deletes
* Runs after ever delete operation
* Relates  #13159 (fixing most of this issues caused by unreferenced indices, leaving some meta files to be cleaned up only)
2019-07-08 10:55:39 +02:00
Przemyslaw Gomulka 247f2dabad
Fix decimal point parsing for date_optional_time backport(#43859) #44050
Joda allowed for date_optional_time and strict_date_optional_time a decimal point to be . dot or , comma
For our java.time implementation we should also extend this for strict_date_optional_time-nanos
the approach to fix this is the same as in iso8601 parser
closes #43730
2019-07-08 09:56:01 +02:00
Armin Braun 2176d09c37
Provide an Option to Use Path-Style-Access with S3 Repo (#41966) (#44046)
* Provide an Option to Use Path-Style-Access with S3 Repo

* As discussed, added the option to use path style access back again and
deprecated it.
* Defaulted to `false`
* Added warning to docs

* Closes #41816
2019-07-08 08:10:01 +02:00
Ioannis Kakavas 9beb51fc44 Revert "Mute testEnableDisableBehaviour (#42929)"
This reverts commit 6ee578c6eb.
2019-07-08 08:52:21 +03:00
Armin Braun f6efc55556
Fix SnapshotResiliencyTest (#44015) (#44041)
* Closes #43989
2019-07-07 19:59:16 +02:00
Armin Braun 990ac4ca83
Some Cleanup in BlobStoreRepository (#43323) (#44043)
* Some Cleanup in BlobStoreRepository

* Extracted from #42833:
  * Dry up index and shard path handling
  * Shorten XContent handling
2019-07-07 19:50:46 +02:00
Nhat Nguyen 9089820d8f Enable indexing optimization using sequence numbers on replicas (#43616)
This PR enables the indexing optimization using sequence numbers on
replicas. With this optimization, indexing on replicas should be faster
and use less memory as it can forgo the version lookup when possible.
This change also deactivates the append-only optimization on replicas.

Relates #34099
2019-07-05 22:12:08 -04:00
Dimitris Athanasiou d3ddedf9fc
[7.x][ML] Add missing doc links to df-analytics rest spec and HLRC javadocs (#44025) (#44033) 2019-07-06 02:03:29 +03:00
Mayya Sharipova 37e1ad7062 Forbid empty doc values on vector functions (#43944)
Currently when a document misses a vector value, vector function
returns 0 as a score for this document. We think this is incorrect
behaviour.
With this change, an error will be thrown if vector functions are
used with docs that are missing vector doc values.
Also VectorScriptDocValues is modified to allow size() function,
which can be used to check if a document has a value for the
vector field.
2019-07-05 18:09:06 -04:00
Dimitris Athanasiou a1a62fded3
[7.x][ML] Stop df-analytics action request should filter tasks (#44016) (#44023)
As a `BaseTasksRequest`, `StopDataFrameAnalyticsAction.Request` should
implement a `match` method that makes sure only df-analytics tasks
are applied.
2019-07-05 23:10:45 +03:00
Yannick Welsch 504a43d43a Move ConnectionManager to async APIs (#42636)
This commit converts the ConnectionManager's openConnection and connectToNode methods to
async-style. This will allow us to not block threads anymore when opening connections. This PR also
adapts the cluster coordination subsystem to make use of the new async APIs, allowing to remove
some hacks in the test infrastructure that had to account for the previous synchronous nature of the
connection APIs.
2019-07-05 20:40:22 +02:00
Nhat Nguyen 8bfe18477e Clarify consequence of translog async setting (#44020)
Relates #43915
2019-07-05 13:56:42 -04:00
lcawl a831d4707c [DOCS] Temporarily disables data frame API testing 2019-07-05 10:56:09 -07:00
Yannick Welsch 88783927d1 Weaken assertion in PublicationTransportHandler (#44014)
These assertions do not hold true when a master fails during publication and quickly becomes
master again, publishing a new cluster state in a higher term which races against the previous
cluster state publication to self (which does not matter anyway).

Relates #43994

Closes #44012
2019-07-05 18:27:42 +02:00
István Zoltán Szabó 5aeb736801 Merge branch '7.x' of github.com:elastic/elasticsearch into 7.x 2019-07-05 14:26:47 +02:00
István Zoltán Szabó 7242267f5d [DOCS] Adds data frame analytics APIs to the ML APIs (#43875)
This PR adds the reference documentation pages of the data frame analytics APIs (PUT, START, STOP, GET, GET stats, DELETE, Evaluate) to the ML APIs pool.
2019-07-05 14:25:54 +02:00
Akshesh Doshi 01b982fd31 Draw attention to transport layer in remote cluster docs (#43883)
Closes #43858
2019-07-05 13:44:36 +02:00
Yannick Welsch 1220ff5b6d Publish to self through transport (#43994)
This commit ensures that cluster state publications to self also go through the transport layer. This
allows voting-only nodes to intercept the publication to self.

Fixes an issue discovered by a test failure where a voting-only node, which was the only
bootstrapped node, would not step down as master after state transfer because publishing to self
would succeed.

Closes #43631
2019-07-05 13:00:52 +02:00
Yannick Welsch 5cdf3ff3fa Revert "[TEST] Mute RemoteClusterServiceTests.testCollectNodes"
This reverts commit d8a2970fa4.
2019-07-05 11:02:42 +02:00
Yannick Welsch d090fa514f Use unique ports per test worker (#43983)
* Use unique ports per test worker

* Add test for system property

* check presence of tests.gradle

* Revert "check presence of tests.gradle"

This reverts commit 2fee7512a28f95c94c5bf7a3312e808f918a9510.
2019-07-05 11:02:28 +02:00
David Turner 06df0c0a4c Improve RetentionLease(Bgrd)SyncAction#toString() (#43987)
Today `RetentionLeaseSyncAction.Request` and
`RetentionLeaseBackgroundSyncAction.Request` both describe themselves as
`Request{...}` in the value returned from their respective `toString()`
methods. This commit adds the name of the owning class to both so we have
something a bit easier to search for and so we can distinguish foreground from
background syncs in logs and test failures and so on.
2019-07-05 09:58:35 +01:00
David Turner 435a83f3fd Add more logging to testOnlyBlocksOnConnectionsToNewNodes (#43979)
Some more output from this occasionally-failing test tracked in #40170.
2019-07-05 09:54:48 +01:00
István Zoltán Szabó 4c3e71b61a [DOCS] Adds description to the preview data frame transform API (#43745) 2019-07-05 09:53:24 +02:00
Dimitris Athanasiou 30b20920b9
[7.x][ML] Report correct count for df-analytics get-stats API (#43969) (#43981)
The count should match the number of all df-analytics that
matched the id in the request. However, we set the count
to the number of df-analytics returned which was bound to the
`size` parameter. This commit fixes this by setting the count
to the count of the `get` response.
2019-07-05 10:28:57 +03:00
Jim Ferenczi cdf55cb5c5 Refactor index engines to manage readers instead of searchers (#43860)
This commit changes the way we manage refreshes in the index engines.
Instead of relying on a SearcherManager, this change uses a ReaderManager that
creates ElasticsearchDirectoryReader when needed. Searchers are now created on-demand
(when acquireSearcher is called) from the current ElasticsearchDirectoryReader.
It also slightly changes the Engine.Searcher to extend IndexSearcher in order
to simplify the usage in the consumer.
2019-07-04 22:49:43 +02:00
Hendrik Muhs 4128b9b4f7 audit message missing for autostop
call onStop when auto stopping (#43984)

fixes #43977
2019-07-04 21:40:42 +02:00
lcawl 688bf1b388 [DOCS] Fixes broken link 2019-07-04 09:13:56 -07:00
Lisa Cawley a030e3e513 [DOCS] Reformat CCR APIs to use new API format (#43952) 2019-07-04 08:29:54 -07:00
Christoph Büscher aeb3c1fd1b Prevent types deprecation warning for indices.exists requests (#43963)
Currently we log a deprecation warning to the types removal in
RestGetIndicesAction even if the REST method is HEAD, which is used by the
indices.exists API. Since the body is empty in this case we should not need to
show the deprecation warning.

Closes #43905
2019-07-04 17:20:43 +02:00
Tanguy Leroux b037aeaa6e
Fix IndexShardIT.testIndexCanChangeCustomDataPath() (#43978)
The test IndexShardIT.testIndexCanChangeCustomDataPath() fails
 on 7.x and 7.3 because the translog cannot be recovered.

While I can't reproduce the issue, I think it has been introduced in #43752 
which changed ReadOnlyEngine so that it opens the translog in its 
constructor in order to load the translog stats. This opening writes a 
new checkpoint file, but because 7.x/7.3 does not wait for shards to be 
started after being closed, the test immediately starts to copy shard files
 to a new directory and possibly does not copy all the required translog files.

By waiting for the shards to be started after being closed, we ensure 
that the shards (and engines) have been correctly initialized and that
 the translog checkpoint file is not currently being written.

closes #43964
2019-07-04 17:06:37 +02:00