Commit Graph

43875 Commits

Author SHA1 Message Date
Yannick Welsch d9fa4e4ada
Fix testRelocateWhileContinuouslyIndexingAndWaitingForRefresh (#37560)
This test failed because the refresh at the end of the test is not guaranteed to run before the
indexing is completed, and therefore there's no guarantee that the refresh will free all operations.
This triggers an assertion failure in the test clean-up, which asserts that there are no more pending
operations.
2019-01-17 13:59:09 +01:00
Martijn van Groningen 99b09845da
Moved ccr integration to the package with other ccr integration tests. 2019-01-17 13:57:56 +01:00
Yannick Welsch 6fe2d6da03 Mute TransportClientNodesServiceTests#testListenerFailures
Relates to #37567
2019-01-17 13:54:48 +01:00
Martijn van Groningen da799306a8
Decreased time out in test
Relates to #37378
2019-01-17 11:51:17 +01:00
Torgeir Thoresen 676e1b1a13 Fix erroneous docstrings for abstract bulk by scroll request (#37517) 2019-01-17 10:22:49 +01:00
Marios Trivyzas 1686c32ba9
SQL: Rename SQL type DATE to DATETIME (#37395)
* SQL: Rename SQL data type DATE to DATETIME

SQL data type DATE has only the date part (e.g.: 2019-01-14)
without any time information. Previously the SQL type DATE was
referring to the ES DATE which contains also the time part along
with TZ information. To conform with SQL data types the data type
`DATE` is renamed to `DATETIME`, since it includes also the time,
as a new runtime SQL `DATE` data type will be introduced down the road,
which only contains the date part and meets the SQL standard.

Closes: #36440

* Address comments
2019-01-17 10:17:58 +02:00
Przemyslaw Gomulka b6e5ccaf8a
Remove the AbstracLifecycleComponent constructor with Settings (#37523)
Adding the migration guide and removing the deprecated in 6.x
constructor

relates #35560
relates #34488
2019-01-17 09:10:09 +01:00
Jason Tedor 18a3e48a4a
Change file descriptor limit to 65535 (#37537)
Some systems default to a nofile ulimit of 65535. To reduce the pain of
deploying Elasticsearch to such systems, this commit lowers the required
limit from 65536 to 65535.
2019-01-16 17:19:12 -05:00
Nhat Nguyen 655103de58 Increase timeout for testAddNewReplicas
We flush quite often in testAddNewReplicas to create the safe index
commit with gaps in sequence numbers. This test is failing recently
because CI is too slow to complete 5 small flushes in 10 seconds.

This commit increases timeout for this test and also ensures to always
terminate the background indexing. The latter is to eliminate unrelated
failures if this test fails again.

Closes #37183
2019-01-16 13:17:10 -05:00
Marios Trivyzas ecf0de30ba
SQL: Lowercase the datatypes in validation error msgs (#37524)
To follow the ES convention display the datatypes in lowercase
in error messages thrown during validation if `IN` and conditional
functions.
2019-01-16 18:41:10 +02:00
Andrey Ershov 4e72f3c5c6
DedicatedClusterSnapshotRestoreIT to Zen2 (#37489)
All tests except testRestorePersistentSettings (renamed to
testExceptionWhenRestoringPersistentSettings) worked fine.
testExceptionWhenRestoringPersistentSettings re-written to use a custom
setting, because "minimum master node" setting is no longer available
in Zen2. It turns out there is no good replacement for "minimum master
node" setting for this test, that's why the custom setting is
introduced.

Unfortunately, there is #37485 bug and currently
RestoreService does not perform setting validation. That's why the
test is annotated with @AwaitsFix, the idea is to merge this commit and
then fix the issue and enable the test. (The test passes with a simple
fix, that adds a single line to RestoreService).
2019-01-16 11:14:16 -05:00
Marios Trivyzas 2cf4b1863f
SQL: Lowercase es data type (mapping) returned from SQL Commands (#37531)
To follow the ES convention, convert the es data type, returned as
column `mapping` from SQL Commands, to lowercase.

Fixes: #37521
2019-01-16 18:08:33 +02:00
Costin Leau 19d2c29ca6 Remove SYS CATALOGS leftover
Relates #37506
2019-01-16 17:28:06 +02:00
Costin Leau 1f76b5fc31
SQL: Describe aliases as views (#37496)
When reporting metadata, several clients have issues with the 'ALIAS'
type. To improve compatibility and be consistent with the ANSI SQL
expectations and because they are similar, aliases targets are now
reported as views.

Close #37422
2019-01-16 17:26:00 +02:00
Jack Conradson 3d8c04659c
Deprecate _type from LeafDocLookup (#37491)
* Deprecate _type from LeafDocLookup

* Response to PR comments.

* Response to PR comments.
2019-01-16 07:05:09 -08:00
Tim Brooks 0b5af276a8
Allow system privilege to execute proxied actions (#37508)
Currently all proxied actions are denied for the `SystemPrivilege`.
Unfortunately, there are use cases (CCR) where we would like to proxy
actions to a remote node that are normally performed by the
system context. This commit allows the system context to perform
proxy actions if they are actions that the system context is normally
allowed to execute.
2019-01-16 07:52:38 -07:00
Michael Basnight 86697f23a3
Update Put Watch to allow unknown fields (#37494)
PutWatchResponse did not allow unknown fields. This commit fixes the
test and ConstructingObjectParser such that it does now allow unknown
fields.

Relates #36938
2019-01-16 08:50:03 -06:00
Nhat Nguyen 0160ba2539 AwaitsFix testAddNewReplicas
Tracked at #37183
2019-01-16 09:48:35 -05:00
Andrei Stefan 659326fdd6
SQL: Add protocol tests and remove jdbc_type from drivers response (#37516) 2019-01-16 16:28:46 +02:00
Marios Trivyzas 8932750efb
SQL: [Docs] Add an ES-SQL column for data types (#37529)
In order to distinguish the ES-SQL type from the standard SQL type
add a new ES-SQL column that will make clear this distingstion,
e.g.: datetime vs TIMSTAMP

Fixes: #37519
2019-01-16 16:16:21 +02:00
Adrien Grand 9d8afe68a5
IndexMetaData#mappingOrDefault doesn't need to take a type argument. (#37480)
Currently it takes a type, but this isn't really needed now that indices can
have at most one type. The only downside is that we might return a different
error when trying to index into a type that doesnt't exist yet.
2019-01-16 14:01:09 +01:00
Armin Braun 21a88d5505
Simplify + Cleanup Dead Code in Settings (#37341)
* Remove dead code
* Simplify some overly complex code, this class is long enough already
2019-01-16 13:57:16 +01:00
Jason Tedor 687978b7d1
Reject all requests that have an unconsumed body (#37504)
This commit removes some leniency from REST handling where we move to
reject all requests that have a body where the body is not used during
the course of handling the request. For example,

DELETE /index
{
  "query" : {
    "term" :  {
      "field" : "value"
    }
  }
}

is now rejected.
2019-01-16 07:29:25 -05:00
David Kyle 75410dc632 [Ml] Prevent config snapshot failure blocking migration (#37493) 2019-01-16 11:51:15 +00:00
Dimitrios Liappis 347cbaf0ed
Fix line length for aliases and remove suppression (#37455)
Relates #34884
2019-01-16 13:06:29 +02:00
Tim Vernum 6d99e790b3
Add SSL Configuration Library (#37287)
This introduces a new ssl-config library that can parse
and validate SSL/TLS settings and files.

It supports the standard configuration settings as used in the
Elastic Stack such as "ssl.verification_mode" and
"ssl.certificate_authorities" as well as all file formats used
in other parts of Elasticsearch security (such as PEM, JKS,
PKCS#12, PKCS#8, et al).
2019-01-16 21:52:17 +11:00
Costin Leau 023bb2f1e4
SQL: Remove slightly used meta commands (#37506)
Remove SYS CATALOGS and SYS TABLE TYPES as they are a subset of SYS
TABLES (and thus somewhat redundant) and used only by JDBC.

Close #37409
2019-01-16 12:36:35 +02:00
Armin Braun 5a5e44d1de
Simplify Snapshot Create Request Handling (#37464)
* The internal create request is absolutely redundant, the only difference to the transport request is that we resolved the snapshot
name when moving from the transport to the internal version
  * Removed it and passed the transport request into the snapshot service instead
* nicer way of resolve snapshot name in callback
2019-01-16 11:08:48 +01:00
Przemyslaw Gomulka 5e94f384c4
Remove the use of AbstracLifecycleComponent constructor #37488 (#37488)
The AbstracLifecycleComponent used to extend AbstractComponent, so it had to pass settings to the constractor of its supper class.
It no longer extends the AbstractComponent so there is no need for this constructor
There is also no need for AbstracLifecycleComponent subclasses to have Settings in their constructors if they were only passing it over to super constructor.
This is part 1. which will be backported to 6.x with a migration guide/deprecation log.
part 2 will have this constructor removed in 7
relates #35560

relates #34488
2019-01-16 09:05:30 +01:00
Hendrik Muhs 15d1b904a1
[ML] log minimum diskspace setting if forecast fails due to insufficient d… (#37486)
log minimum disk space setting if forecast fails due to insufficient disk space
2019-01-16 08:10:13 +01:00
Jay Modi 987576b013
Consistently use loopback address for ssl profile (#37487)
This change fixes failures in the SslMultiPortTests where we attempt to
connect to a profile on a port it is listening on but the connection
fails. The failure is due to the profile being bound to multiple
addresses and randomization will pick one of these addresses to
determine the listening port. However, the address we get the port for
may not be the address we are actually connecting to. In order to
resolve this, the test now sets the bind host for profiles to the
loopback address and uses the same address for connecting.

Closes #37481
2019-01-15 14:03:21 -07:00
Julie Tibshirani 0a3bff2ca9
Only log one types warning per bulk search request. (#37446) 2019-01-15 12:38:32 -08:00
Alpar Torok e7511753a9
Upgrade to Gradle 5.1.1 (#37410)
Fixes mem leak in the daemon.
2019-01-15 21:20:19 +02:00
Lisa Cawley 17b0aa3f31
[DOCS] Intro for adding nodes (#37202) 2019-01-15 11:11:01 -08:00
Lisa Cawley 03cacdf16e
[DOCS] Adds CCR screenshots (#37439) 2019-01-15 10:53:29 -08:00
Andrey Ershov 42fd68ed38
Use GatewayMetaState in CoordinatorTests rarely (#36897)
This commit adds one more underlying implementation of MockPersistedState.
Previously only InMemoryPersistentState was used, not GatewayMetaState
is used rarely.
When adding GatewayMetaState support the main question was: do we want to
emulate exceptions as we do today in MockPersistedState before
delegating to GatewayMetaState or do we want these exceptions to
propagate from the lower level, i.e. file system exceptions?
On the one hand, lower level exception propagation is already tested in
GatewayMetaStateTests, so this won't improve the coverage.
On the other hand, the benefit of low-level exceptions is to see how all these
components work in conjunction. Finally, we abandoned the idea of low-level
exceptions because we don't have a way to deal with IOError today in
CoordinatorTests, but hacking GatewayMetaState not to throw
IOError seems unnatural.
So MockPersistedState rarely throws an exception before delegating to
GatewayMetaState, which is not supposed to throw the exception.

This commit required two changes:

Move GatewayMetaStateUT to upper-level from
GatewayMetaStatePersistedStateTests, because otherwise, it's not easy
to construct GatewayMetaState instance in CoordinatorTests.
Move addition of STATE_NOT_RECOVERED_BLOCK from GatewayMetaState
constructor to GatewayMetaState.applyClusterUpdaters, because
CoordinatorTests class assumes that there is no such block and most of
them fail.
2019-01-15 13:33:25 -05:00
Jim Ferenczi f8d80dff7c
Fix duplicate removal when merging completion suggestions (#36996)
The completion suggester ignores the original weight of the suggestion when duplicates are removed. This change fixes this bug and keeps the best weighted suggestion among the duplicates. It also removes the custom implementation of the top docs suggest collector now that https://issues.apache.org/jira/browse/LUCENE-8529 is committed in Lucene.

Closes #35836
2019-01-15 19:27:31 +01:00
David Kyle bea46f7b52
[ML] Migrate unallocated jobs and datafeeds (#37430)
Migrate ml job and datafeed config of open jobs and update
the parameters of the persistent tasks as they become unallocated
during a rolling upgrade. Block allocation of ml persistent tasks
until the configs are migrated.
2019-01-15 18:21:39 +00:00
Ryan Ernst 2cf7a8016f
Packaging: Remove permission editing in postinst (#37242)
This commit removes permission editing commands from the postinst
scriptlet. Instead, we now fully configure the owner/group (as well as
sticky bit) for these files and directories.

closes #37143
2019-01-15 10:19:32 -08:00
Nhat Nguyen 6647122f1c
Prepare to make send translog of recovery non-blocking (#37458)
This commit prepares the required infra to make send a translog snapshot
of the recovery source non-blocking. I'll make a follow-up to make the send
snapshot method non-blocking.

Relates #37291
2019-01-15 13:17:25 -05:00
Andrey Ershov 02d4d8b409
MinimumMasterNodesIT changed for Zen2 (#37428)
There were 5 tests in MinimumMasterNodesIT. 2 of them removed, 3 of
them changed and renamed.
1) testSimpleMinimumMasterNodes -> testTwoNodesNoMasterBlock. The
flow of this test is left intact but in order to make it work on
Zen2, additional work for the cluster bootstrapping and voting
exclusions is needed. 
2) testDynamicUpdateMinimumMasterNodes -> removed, there is nothing
that corresponds to the dynamic change of the minimum master nodes
setting.
3) testCanNotBringClusterDown -> removed, it also plays with changing
minimum master nodes dynamically.
4) testMultipleNodesShutdownNonMasterNodes ->
testThreeNodesNoMasterBlock. Previously this test was checking that
there would be no master block, if min_master_nodes=3 and 4 nodes are
started, then 2 nodes are brought down. Zen2 dynamically accommodates
to the number of nodes in the cluster, so it's possible that there
still will be a master in 2 nodes cluster. For Zen2, we start up 3
nodes. And shut down 2 of them (w/o voting exclusions), which results
in no master block.
5) testCanNotPublishWithoutMinMastNodes ->
testCanNotCommitStateThreeNodes. Test flow is not changed. But
previously there was no check that nodes in the bigger part of
network partition will elect the master, before healing the network
partition. For Zen2 it does not work, because persistent setting
addition is accepted on the old master and if it's elected new master
again, this setting will appear in the cluster state.

Also, I have a feeling that we need to remove this class, but could not
come up with a good name.
2019-01-15 13:09:48 -05:00
Dimitrios Liappis 63793499bd
Fix line length for `node` and remove suppresion (#37454)
Relates #34884
2019-01-15 19:57:24 +02:00
David Turner a2a40c50a0
Report terms and version if cluster does not form (#37473)
Adds the node's current term and the term and version of the the last-accepted
cluster state to the message reported by the `ClusterFormationFailureHelper`,
since these values may be of importance when tracking down a cluster formation
failure.
2019-01-15 17:32:08 +00:00
David Kyle 7c11b05c28
[ML] Remove unused code from the JIndex project (#37477) 2019-01-15 17:19:58 +00:00
Lisa Cawley 44e83f30e2
[DOCS] Edits rollup API description (#37444) 2019-01-15 08:46:36 -08:00
Marios Trivyzas 6129e9d9dd Revert "[TEST] Muted TokenBackwardsCompatibilityIT.*"
This reverts commit 65e42ab63b.

The test is only failing in 6.x not master.
2019-01-15 18:25:53 +02:00
Nhat Nguyen 68e2d36fa3 Adjust bwc version for max_concurrent_file_chunks
Relates #36981
2019-01-15 11:18:55 -05:00
Marios Trivyzas 65e42ab63b [TEST] Muted TokenBackwardsCompatibilityIT.*
Relates to #37379
2019-01-15 18:05:20 +02:00
Igor Motov 6f91f06d86
Geo: Adds a set of no dependency geo classes for JDBC driver (#36477)
Adds a set of geo classes to represent geo data in the JDBC driver and 
to be used as an intermediate format to pass geo shapes for indexing 
and query generation in #35320.

Relates to #35767 and #35320
2019-01-15 10:52:46 -05:00
Luca Cavanna 0b396a0c5e Restore assertion on discount overlaps in SimilarityTests
This assertion was commented out as the getDiscountOverlaps getter was
missing from LegacyBm25Similarity. That has been fixed in lucene.
2019-01-15 16:43:09 +01:00