Commit Graph

43725 Commits

Author SHA1 Message Date
Jun Ohtani 38b698d455
[Analysis] Deprecate Standard Html Strip Analyzer in master (#26719)
* [Analysis] Deprecate Standard Html Strip Analyzer

Deprecate only Standard Html Strip Analyzer
If user create index with the analyzer since 7.0, es throws an exception.
If an index was created before 7.0, es issue deprecation log
We will remove it in 8.0

Related #4704
2019-01-09 12:42:00 +09:00
Mayya Sharipova ec32e66088 Deprecate reference to _type in lookup queries (#37016)
Relates to #35190
2019-01-08 18:46:41 -08:00
Marios Trivyzas 5f2fbedd8c
SQL: Replace String.format() with LoggerMessageFormat.format() (#37216)
Fixes: #36532
2019-01-08 23:56:00 +02:00
Nhat Nguyen 87ac3103f4
Do not mutate RecoveryResponse (#37204)
Today we create a global instance of RecoveryResponse then mutate it
when executing each recovery step. This is okay for the current
sequential recovery flow but  not suitable for an asynchronous recovery
which we are targeting. With this commit, we return the result of each
step separately, then construct a RecoveryResponse at the end.

Relates #37174
2019-01-08 16:12:18 -05:00
Ryan Ernst 86b71930f6
Remove more checkstyle suppressions (#36735)
This commit removes several more line length violations from checkstyle
suppression.

relates #34884
2019-01-08 11:04:54 -08:00
Martijn van Groningen d6608caf55
Muted rolling upgrade tests.
Relates to #37231
2019-01-08 16:52:22 +01:00
Jay Modi 1514bbcdde
Security: propagate auth result to listeners (#36900)
After #30794, our caching realms limit each principal to a single auth
attempt at a time. This prevents hammering of external servers but can
cause a significant performance hit when requests need to go through a
realm that takes a long time to attempt to authenticate in order to get
to the realm that actually authenticates. In order to address this,
this change will propagate failed results to listeners if they use the
same set of credentials that the authentication attempt used. This does
prevent these stalled requests from retrying the authentication attempt
but the implementation does allow for new requests to retry the
attempt.
2019-01-08 08:52:12 -07:00
Michael Basnight dd69553d4d
HLRC: Use nonblocking entity for requests (#32249)
Previously the HLRC used a blocking ByteArrayEntity, but the Request
class also allows to set a NByteArrayEntity, and defaults to nonblocking
when calling the createJsonEntity method. This commit cleans up all the
uses of ByteArrayEntity in the RequestConverters to use the nonblocking
entity.
2019-01-08 09:11:58 -06:00
Jim Ferenczi 054c3bb04f
Fix topDocs.totalHits assignment on scroll queries (#37180)
This change fixes an unreleased bug that assigns the wrong totalHits to scroll
queries.

Closes #37179
2019-01-08 13:31:53 +01:00
Alpar Torok 20c2c439e7
Upgrade to Gradle 5.1 (#36767) 2019-01-08 13:43:10 +02:00
Alpar Torok 6344e9a3ce
Testing conventions: add support for checking base classes (#36650) 2019-01-08 13:39:03 +02:00
Alpar Torok 0c68eead51 Mute failing test clusters test
Tracking issue: #37218
2019-01-08 13:00:55 +02:00
Tanguy Leroux 6e852dfa7c Merge branch 'master' into close-index-api-refactoring 2019-01-08 11:28:51 +01:00
Martijn van Groningen c980cc12df
Added CCR rolling upgrade tests (#36648)
Added CCR rolling upgrade tests.
2019-01-08 11:05:18 +01:00
Andrew Banchich 1d371a427f [Docs] Fix wrong math in overview.asciidoc (#37209) 2019-01-08 10:25:02 +01:00
Tanguy Leroux 6c78aff830 Merge branch 'master' into close-index-api-refactoring 2019-01-08 10:00:47 +01:00
Luca Cavanna 67f97e3686 Re-enable bwc tests
Relates to #37187
2019-01-08 09:53:58 +01:00
Tanguy Leroux d70ebfd1d6 Merge branch 'master' into close-index-api-refactoring 2019-01-08 09:17:48 +01:00
Andrei Stefan 3fad9d25f6
SQL: fix COUNT DISTINCT filtering (#37176)
* Use `_count` aggregation value only for not-DISTINCT COUNT function calls
* COUNT DISTINCT will use the _exact_ version of a field (the `keyword` sub-field for example), if there is one
2019-01-08 08:47:35 +02:00
Tim Brooks b0665963e8
Make remote ping and compress settings dynamic (#37200)
Traditionally remote clusters can be configured dynamically. However,
the compress and ping settings are not currently set to be configured
dynamically. This commit changes that.
2019-01-07 23:35:22 -07:00
Jason Tedor c8c596cead
Introduce retention lease expiration (#37195)
This commit implements a straightforward approach to retention lease
expiration. Namely, we inspect which leases are expired when obtaining
the current leases through the replication tracker. At that moment, we
clean the map that persists the retention leases in memory.
2019-01-07 22:03:52 -08:00
Julie Tibshirani c5aac4705d Revert "Stop automatically nesting mappings in index creation requests. (#36924)"
This reverts commit ac1c6940d2.
2019-01-07 17:56:40 -08:00
Ryan Ernst 55d3ca3aa8
Core: Rework epoch time parsing for java time (#36914)
This commit converts the epoch time parsing implementation which uses
the java time api to create DateTimeFormatters instead of DateFormatter
implementations. This will allow multi formats for java time to be
implemented in a single DateTimeFormatter in a future change.
2019-01-07 16:15:30 -08:00
Christoph Büscher 56e472bfbc
Removing unused methods in Numbers (#37186)
Remove several unused helper methods. Most of them are one-liners and should 
be easier to be used from the corresponding primitive wrapper classes. 
The bytes array conversion methods are unused as well, it should be easy to 
re-create them if needed.
2019-01-08 00:14:50 +01:00
Jason Tedor 3b48b99861
Fix setting by time unit (#37192)
This commit fixes an issue with a settings builder method that allows
setting a duration by time unit. In particular, this method can suffer
from a loss of precision. For example, if the input duration is 1500
microseconds then internally we are converting this to "1ms",
demonstrating the loss of precision. Instead, we should internally
convert this to a TimeValue that correctly represents the input
duration, and then convert this to a string using a method that does not
lose the unit. That is what this commit does.
2019-01-07 14:59:24 -08:00
lcawl 382e4d39ef [DOCS] Cleans up xpackml attributes 2019-01-07 14:33:10 -08:00
Benjamin Trent 6b376a1ff4
ML: fix delayed data annotations on secured cluster (#37193)
* changing executing context for writing annotation

* adjusting user

* removing unused import
2019-01-07 15:18:38 -06:00
Luca Cavanna e34658edba Update version in SearchRequest and related test
Version needs to be updated after backporting #36997 & #37142
where we added support for providing and serializing localClusterAlias
as well ass absoluteStartMillis.

Relates to #36997 & #37142
2019-01-07 18:47:17 +01:00
Tanguy Leroux 97bf4d7176 Merge branch 'master' into close-index-api-refactoring 2019-01-07 18:38:27 +01:00
Lisa Cawley f307847f29
[DOCS] Adds overview and API ref for cluster voting configurations (#36954) 2019-01-07 09:11:14 -08:00
Benjamin Trent 1780ced82d
ML: changing JobResultsProvider.getForecastRequestStats to support > 1 index (#37157)
* ML: changing JobResultsProvider.getForecastRequestStats to support more than one index

* moving to use idsQuery()
2019-01-07 10:58:55 -06:00
Christophe Bismuth 9602d794c6 Separate out validation of groups of settings (#34184)
Today, a setting can declare that its validity depends on the values of other
related settings. However, the validity of a setting is not always checked
against the correct values of its dependent settings because those settings'
correct values may not be available when the validator runs.

This commit separates the validation of a settings updates into two phases,
with separate methods on the `Setting.Validator` interface. In the first phase
the setting's validity is checked in isolation, and in the second phase it is
checked again against the values of its related settings. Most settings only
use the first phase, and only the few settings with dependencies make use of
the second phase.
2019-01-07 16:12:58 +00:00
David Turner 9d0e0eb0f3
[Zen2] Remove initial master node count setting (#37150)
The `cluster.unsafe_initial_master_node_count` setting was introduced as a
temporary measure while the design of `cluster.initial_master_nodes` was being
finalised. This commit removes this temporary setting, replacing it with usages
of `cluster.initial_master_nodes` where appropriate.
2019-01-07 16:05:00 +00:00
Tanguy Leroux e149b0852e
[Close Index API] Add unique UUID to ClusterBlock (#36775)
This commit adds a unique id to cluster blocks, so that they can be uniquely 
identified if needed. This is important for the Close Index API where multiple 
concurrent closing requests can be executed at the same time. By adding a 
UUID to the cluster block, we can generate unique "closing block" that can 
later be verified on shards and then checked again from the cluster state 
before closing the index. When the verification on shard is done, the closing 
block is replaced by the regular INDEX_CLOSED_BLOCK instance.

If something goes wrong, calling the Open Index API will remove the block.

Related to #33888
2019-01-07 16:44:59 +01:00
Jason Tedor c0f8c89172
Introduce shard history retention leases (#37167)
This commit is the first in a series which will culminate with
fully-functional shard history retention leases.

Shard history retention leases are aimed at preventing shard history
consumers from having to fallback to expensive file copy operations if
shard history is not available from a certain point. These consumers
include following indices in cross-cluster replication, and local shard
recoveries. A future consumer will be the changes API.

Further, index lifecycle management requires coordinating with some of
these consumers otherwise it could remove the source before all
consumers have finished reading all operations. The notion of shard
history retention leases that we are introducing here will also be used
to address this problem.

Shard history retention leases are a property of the replication group
managed under the authority of the primary. A shard history retention
lease is a combination of an identifier, a retaining sequence number, a
timestamp indicating when the lease was acquired or renewed, and a
string indicating the source of the lease. Being leases they have a
limited lifespan that will expire if not renewed. The idea of these
leases is that all operations above the minimum of all retaining
sequence numbers will be retained during merges (which would otherwise
clear away operations that are soft deleted). These leases will be
periodically persisted to Lucene and restored during recovery, and
broadcast to replicas under certain circumstances.

This commit is merely putting the basics in place. This first commit
only introduces the concept and integrates their use with the soft
delete retention policy. We add some tests to demonstrate the basic
management is correct, and that the soft delete policy is correctly
influenced by the existence of any retention leases. We make no effort
in this commit to implement any of the following:
 - timestamps
 - expiration
 - persistence to and recovery from Lucene
 - handoff during primary relocation
 - sharing retention leases with replicas
 - exposing leases in shard-level statistics
 - integration with cross-cluster replication

These will occur individually in follow-up commits.
2019-01-07 07:43:57 -08:00
Alpar Torok a7c3d5842a
Split third party audit exclusions by type (#36763) 2019-01-07 17:24:19 +02:00
Jason Tedor a233db7367
Fix handling of fractional byte size value settings (#37172)
This commit addresses an issue when setting a byte size value setting
using a value that has a fractional component when converted to its
string representation. For example, trying to set a byte size value
setting to a value of 1536 bytes is problematic because internally this
is converted to the string "1.5k". When we go to get this setting, we
try to parse "1.5k" back to a byte size value, which does not support
fractional values. The problem is that internally we are relying on a
method which loses the unit when doing the string conversion. Instead,
we are going to use a method that does not lose the unit and therefore
we can roundtrip from the byte size value to the string and back to the
byte size value.
2019-01-07 07:13:50 -08:00
Luca Cavanna ad4e4fd9b8
Disable bwc tests until #36997 & #37142 backports are complete (#37187) 2019-01-07 15:59:26 +01:00
Armin Braun 82b1f10eb1
SNAPSHOT+TESTS: Rem. Mock Atomic Writes Randomness (#37011)
* Randomly doing non-atomic writes causes rare 0 byte reads from `index-N` files in tests
* Removing this randomness fixes these random failures and is valid because it does not reproduce a real-world failure-mode:
  * Cloud-based Blob stores (S3, GCS, and Azure) do not have inconsistent partial reads of a blob, either you read a complete blob or nothing on them
  * For file system based blob stores the atomic move we do (to atomically write a file) by setting `java.nio.file.StandardCopyOption#ATOMIC_MOVE` would throw if the file system does not provide for atomic moves
* Closes #37005
2019-01-07 15:24:23 +01:00
Josh Soref edb48321ba [DOCS] Various spelling corrections (#37046) 2019-01-07 14:44:12 +01:00
Simon Willnauer ac2e09b25a
Fix suite scope random initializaation (#37163)
The initialization of a suite scope cluster had some sideffects on
subsequent runs which causes issues when tests must be reproduced.
This moves the suite scope initialization to a privte random context.

Closes #36202
2019-01-07 14:20:17 +01:00
Alpar Torok 21996f7690 Expand on #37058
Use the wrapper
2019-01-07 14:06:06 +02:00
Tanguy Leroux f5af79b9cd Merge branch 'master' into close-index-api-refactoring 2019-01-07 12:43:03 +01:00
Armin Braun 617e294133
SNAPSHOT: Make Atomic Blob Writes Mandatory (#37168)
* With #37066 introducing atomic writes to HDFS repository we can enforce atomic write capabilities on this interface
* The overrides on the other three cloud implementations are ok because:
   * https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPUT.html states that "Amazon S3 never adds partial objects; if you receive a success response, Amazon S3 added the entire object to the bucket."
   * https://cloud.google.com/storage/docs/consistency states that GCS has strong read-after-write consistency
   * https://docs.microsoft.com/en-us/rest/api/storageservices/put-block#remarks Azure has the concept of committing blobs, so there's no partial content here either
* Relates #37011
2019-01-07 12:11:19 +01:00
David Kyle 7cc749dced Mute SearchStatsIT.testOpenContexts 2019-01-07 09:47:58 +00:00
Christoph Büscher 60742b12e9
Remove deprecated CardinalityAggregationBuilder#rehash method (#37154)
It has been deprecated since at least 6.0, is a no-op and unused in the rest of
our code.
2019-01-07 10:42:07 +01:00
Christoph Büscher d18c3d651d
Introduce an `include_type_name` constant (#37155)
I started referring to this parameter name from various places in #37149 so I
think it's a good idea to simplify things by referring to a common constant.
2019-01-07 10:41:40 +01:00
Christoph Büscher 12a105e5ef
Remove deprecated PutIndexTemplateRequestBuilder#setTemplate (#37151)
The method has been removed since 6.0, there is a direct replacement and it is
only used in tests still.
2019-01-07 10:41:04 +01:00
Luca Cavanna 2f4dafa69f
Add support for providing absolute start time to SearchRequest (#37142)
We have recently added support for providing a local cluster alias to a
SearchRequest through a package protected constructor. When executing
cross-cluster search requests with local reduction on each cluster, the
CCS coordinating node will have to provide such cluster alias to each
remote cluster, as well as the absolute start time of the search action
in milliseconds from the time epoch, to be used when evaluating date
math expressions both while executing queries / scripts as well as when
resolving index names.

This commit adds support for providing the start time together with the
cluster alias. It is a final member in the search request, which will
only be set when using cross-cluster search with local reduction (also
known as alternate execution mode). When not provided, the coordinating
node will determine the current time and pass it through (by calling
`System.currentTimeMillis`).

Relates to #32125
2019-01-07 10:28:31 +01:00
Tanguy Leroux 19593884ee
[Close Index API] Propagate tasks ids between Freeze, Close and Verify Shard actions (#36630)
This pull request changes the Freeze Index and Close Index actions so 
that these actions always requires a Task. The task's id is then propagated 
from the Freeze action to the Close action, and then to the Verify shard action. 
This way it is possible to track which Freeze task initiates the closing of an index, 
and which consecutive verifiy shard are executed for the index closing.
2019-01-07 09:43:50 +01:00