Commit Graph

44324 Commits

Author SHA1 Message Date
Martijn van Groningen 6d81e7c3e7
[CCR] FollowingEngine should fail with 403 if operation has no seqno assigned (#37213)
Fail with a 403 when indexing a document directly into a follower index.

In order to test this change, I had to move specific assertions into a dedicated class and
disable assertions for that class in the rest qa module. I think that is the right trade off.
2019-01-10 15:54:34 +01:00
Przemyslaw Gomulka c812e6aea6
Fix line length in org.elasticsearch.routing (#37253)
Remove the line length suppression for this package and fix offending
lines

relates: #34884
2019-01-10 15:23:34 +01:00
Martijn van Groningen df488720e0
[CCR] Make shard follow tasks more resilient for restarts (#37239)
If a running shard follow task needs to be restarted and
the remote connection seeds have changed then
a shard follow task currently fails with a fatal error.

The change creates the remote client lazily and adjusts
the errors a shard follow task should retry.

This issue was found in test failures in the recently added
ccr rolling upgrade tests. The reason why this issue occurs
more frequently in the rolling upgrade test is because ccr
is setup in local mode (so remote connection seed will become stale) and
all nodes are restarted, which forces the shard follow tasks to get
restarted at some point during the test. Note that these tests
cannot be enabled yet, because this change will need to be backported
to 6.x first. (otherwise the issue still occurs on non upgraded nodes)

I also changed the RestartIndexFollowingIT to setup remote cluster
via persistent settings and to also restart the leader cluster. This
way what happens during the ccr rolling upgrade qa tests, also happens
in this test.

Relates to #37231
2019-01-10 15:02:30 +01:00
Armin Braun 26cb7466ef
SNAPSHOT+TESTS: Stabilize SnapshotDisruptionIT (#37289)
* Ensure retry by busy assert on SnapshotMissingException
* Closes #36739
2019-01-10 14:13:20 +01:00
Martijn van Groningen 44acb016a6
[HLRC] Ignore unknown fields in responses of CCR APIs (#36821)
Otherwise hlrc fails if new fields are introduced in ccr api responses
in future versions.
2019-01-10 14:04:55 +01:00
Luca Cavanna 61b54196c4 [TEST] Fixed compile issue in SnapshotsServiceTests
Relates to #37203
2019-01-10 13:38:39 +01:00
Alexander Reelsen 71287b0759
Remove unused EpochMillisDateFormatter (#37293)
This class has been superceded by a custom java time epoch millis date
parser.
2019-01-10 13:20:24 +01:00
Yannick Welsch d499233068
Zen2: Add join validation (#37203)
Adds join validation to Zen2, which prevents a node from joining a cluster when the node does not
have the right ES version or does not satisfy any other of the join validation constraints.
2019-01-10 12:57:50 +01:00
Christoph Büscher cd608848e7
Remove deprecated QUERY_AND_FETCH SearchType (#37257)
This SearchType was deprecated since at least 6.0 and according to the
documentation is only kept around for pre-5.3 requests. Removing and leaving a
comment as placeholder so we don't reuse the byte value associated with it
without further consideration.
2019-01-10 12:50:07 +01:00
Alpar Torok 3d66764660 Mute watcher SingleNodeTests
Tracking:  #36782
2019-01-10 12:23:29 +02:00
Martijn van Groningen 1a41d84536
[CCR] Resume follow Api should not require a request body (#37217)
Closes #37022
2019-01-10 09:48:26 +01:00
Alexander Reelsen eb12de550a
Java Time: Fix timezone parsing (#37262)
* Java Time: Fix timezone parsing

An independent test uncovered an issue when parsing a timezone
containing a colon like `01:00` - some formats did not properly support
this.

This commit adds test for all formats in the dueling tests and fixes a
few issues with existing date formatters.

* fix tests, so they run under java8
2019-01-10 09:26:01 +01:00
Alexander Reelsen b2e8437424
Tests: Add ElasticsearchAssertions.awaitLatch method (#36777)
* Tests: Add ElasticsearchAssertions.awaitLatch method

Some tests are using assertTrue(latch.await(...)) in their code. This
leads to an assertion error without any error message. This adds a
method which has a nicer error message and can be used in tests.

* fix forbidden apis

* fix spaces
2019-01-10 09:25:36 +01:00
Andrei Stefan 4a92de214a
SQL: Proper handling of COUNT(field_name) and COUNT(DISTINCT field_name) (#37254)
* provide overriden `hashCode` and toString methods to account for `DISTINCT`
* change the analyzer for scenarios where `COUNT <field_name>` and `COUNT DISTINCT` have different paths
* defined a new `filter` aggregation encapsulating an `exists` query to filter out null or missing values
2019-01-10 09:51:51 +02:00
Michael Basnight d625b79df2
Add getZone to JodaCompatibleZonedDateTime (#37084)
The ZonedDateTime#getZone() was not accessible via the Joda shim. This
commit adds getZone() and exposes it through painless.
2019-01-09 22:09:34 -06:00
Jim Ferenczi 586093ec5e Handle TopFieldDocs copy in TopDocsCollectorContext
This commit fixes the clone of TopFieldDocs.

Relates #37179
Relates #37266
2019-01-10 00:26:55 +01:00
Simon Willnauer 234059d2c0
Enable Bulk-Merge if all source remains (#37269)
Today we still wrap recovery source readers on merge even if we
keep all documents recovery source. This basically disables bulk
merging for stored fields. This change skips wrapping if all docs
sources are kept anyway.
2019-01-09 23:46:31 +01:00
Jim Ferenczi 95479f1766 Ensure that a non static top docs is created during the search phase
This change fixes an unreleased bug that trips an assertion because a static instance
shared among threads is modified during the search. This commit copies the static
instance in order to ensure that each thread can modify the value without modifying
the other instances.

Closes #37179
Closes #37266
2019-01-09 22:57:34 +01:00
Jake Landis 195873002b
ingest: compile mustache template only if field includes '{{'' (#37207)
* ingest: compile mustache template only if field includes '{{''

Prior to this change, any field in an ingest node processor that supports
script templates would be compiled as mustache template regardless if they
contain a template or not. Compiling normal text as mustache templates is
harmless. However, each compilation counts against the script compilation
circuit breaker. A large number of processors without any templates or scripts
could un-intuitively trip the too many script compilations circuit breaker.

This change simple checks for '{{' in the text before it attempts to compile.

fixes #37120
2019-01-09 14:47:47 -06:00
Benjamin Trent df3b58cb04
ML: add migrate anomalies assistant (#36643)
* ML: add migrate anomalies assistant

* adjusting failure handling for reindex

* Fixing request and tests

* Adding tests to blacklist

* adjusting test

* test fix: posting data directly to the job instead of relying on datafeed

* adjusting API usage

* adding Todos and adjusting endpoint

* Adding types to reindexRequest

* removing unreliable "live" data test

* adding index refresh to test

* adding index refresh to test

* adding index refresh to yaml test

* fixing bad exists call

* removing todo

* Addressing remove comments

* Adjusting rest endpoint name

* making service have its own logger

* adjusting validity check for newindex names

* fixing typos

* fixing renaming
2019-01-09 14:25:35 -06:00
jaymode c71060fa01
Test: fix race in auth result propagation test
This commit fixes a race condition in a test introduced by #36900 that
verifies concurrent authentications get a result propagated from the
first thread that attempts to authenticate. Previously, a thread may
be in a state where it had not attempted to authenticate when the first
thread that authenticates finishes the authentication, which would
cause the test to fail as there would be an additional authentication
attempt. This change adds additional latches to ensure all threads have
attempted to authenticate before a result gets returned in the
thread that is performing authentication.
2019-01-09 12:17:43 -07:00
Ryan Ernst 722b850efd
Build: Make assemble a noop in bwc projects (#37245)
This commit makes the assemble tasks in the bwc projects noops by
setting the dependsOn directly. While we can not remove things from
dependsOn, we can still completely override the dependencies.

closes #33581
2019-01-09 10:57:26 -08:00
Ryan Ernst 29c895b55c
Build: Quiet naming convention logging (#37244)
This commit moves log statements related to classification of naming
convention checks for tests to debug level. At info level they emit an
enormous amount of output in CI, while these are not generally useful
for debugging normal build failures.
2019-01-09 10:56:30 -08:00
Jack Conradson 95eef77ad4
[Style] Fix line length violations for threadpool, indexing, and script packages (#37205) 2019-01-09 10:55:52 -08:00
Tim Brooks cfa58a51af
Add TLS/SSL channel close timeouts (#37246)
Closing a channel using TLS/SSL requires reading and writing a
CLOSE_NOTIFY message (for pre-1.3 TLS versions). Many implementations do
not actually send the CLOSE_NOTIFY message, which means we are depending
on the TCP close from the other side to ensure channels are closed. In
case there is an issue with this, we need a timeout. This commit adds a
timeout to the channel close process for TLS secured channels.

As part of this change, we need a timer service. We could use the
generic Elasticsearch timeout threadpool. However, it would be nice to
have a local to the nio event loop timer service dedicated to network needs. In
the future this service could support read timeouts, connect timeouts,
request timeouts, etc. This commit adds a basic priority queue backed
service. Since our timeout volume (channel closes) is very low, this
should be fine. However, this can be updated to something more efficient
in the future if needed (timer wheel). Everything being local to the event loop
thread makes the logic simple as no locking or synchronization is necessary.
2019-01-09 11:46:24 -07:00
Morris Schreibman 13b8bad2b8 [DOCS] Clarify client settings (#31469) 2019-01-09 10:00:00 -08:00
Alpar Torok 6a5f3f05f4 Fix build on Fips
testing convetions need to be disabled if the test task is for fips.
2019-01-09 19:27:01 +02:00
Alpar Torok d68d6aa00c
Compatability mode for forbidden APIs for Java 12 (#37199) 2019-01-09 18:26:08 +02:00
Albert Zaharovits 4efbda1bc0
[DOCS] Rolling upgrade with old internal indices (#37184)
Upgrading the Elastic Stack perfectly documents the process to
upgrade ES from 5 to 6 when internal indices are present. However,
the rolling upgrade docs do not mention anything about internal indices.

This adds a warning in the rolling upgrade procedure, highlighting that
internal indices should be upgraded before the rolling upgrade procedure
can be started.
2019-01-09 18:14:22 +02:00
Evangelos Chatzikalymnios 85a603ee61 Use List instead of priority queue for stable sorting in bucket sort aggregator (#36748)
Update BucketSortPipelineAggregator to use a List and Collections.sort() for sorting instead of a priority queue. This preserves the order for equal values. Closes #36322.
2019-01-09 18:01:39 +02:00
Armin Braun eacc63b032
TESTS: Real Coordinator in SnapshotServiceTests (#37162)
* TESTS: Real Coordinator in SnapshotServiceTests

* Introduce real coordinator in SnapshotServiceTests to be able to test network disruptions realistically
  * Make adjustments to cluster applier service so that we can pass a mocked single threaded executor for tests
2019-01-09 16:53:49 +01:00
Alpar Torok ae086ebcc4 Muting SnapshotDisruptionIT
Tracked in #36779
2019-01-09 16:55:11 +02:00
Alpar Torok 7de4d2cb0f Mute failing test ChildQuerySearchIT
Tracked in #37266
2019-01-09 16:48:49 +02:00
Christoph Büscher c149bb8cc2
Support 'include_type_name' in RestGetIndicesAction (#37149)
This change adds support for the 'include_type_name' parameter for the
indices.get API. This parameter, which defaults to `false` starting in 7.0,
changes the response to not include the indices type names any longer.

If the parameter is set in the request, we additionally emit a deprecation
warning since using the parameter should be only temporarily necessary while
adapting to the new response format and we will remove it with the next major
version.
2019-01-09 14:17:17 +01:00
Jim Ferenczi 84d520b0e5 Throw an ISE rather than an hard assertion in SearchPhaseController#getTotalHits
This change turns an assertion into an IllegalStateException in SearchPhaseController#getTotalHits.
The goal is to help identify the cause of the failures in https://github.com/elastic/elasticsearch/issues/37179
which seems to fail only in CI.
The assertion will be restored when the issue is solved (NORELEASE).
2019-01-09 13:36:48 +01:00
Tanguy Leroux dc371ef593 [Tests] Fix ReopenWhileClosingIT with correct min num shards
The test intercepts TransportVerifyShardBeforeCloseAction shard
requests, so it needs a minimum of 2 primary shards on 2 different
 nodes to correctly intercepts requests.
2019-01-09 13:34:38 +01:00
Alpar Torok 7ff846ffb2 mute failing test: org.elasticsearch.index.shard.RefreshListenersTests.testConcurrentRefresh 2019-01-09 13:43:21 +02:00
Martijn van Groningen 9122585359
[CCR] Added more logging. 2019-01-09 12:17:47 +01:00
Tanguy Leroux f1f5d834c3 Merge branch 'close-index-api-refactoring' 2019-01-09 11:48:57 +01:00
David Roberts e0ce73713f
[ML] Stop datafeeds running when their jobs are stale (#37227)
We already had logic to stop datafeeds running against
jobs that were OPENING, but a job that relocates from
one node to another while OPENED stays OPENED, and this
could cause the datafeed to fail when it sent data to
the OPENED job on its new node before it had a
corresponding autodetect process.

This change extends the check to stop datafeeds running
when their job is OPENING _or_ stale (i.e. has not had
its status reset since relocating to a different node).

Relates #36810
2019-01-09 10:42:47 +00:00
Tanguy Leroux 096a83183e Merge branch 'master' into close-index-api-refactoring 2019-01-09 10:52:46 +01:00
David Roberts f14cff2102
[TEST] Ensure interrupted flag reset after test that sets it (#37230)
Test fix to stop a problem in one test leaking into a different
test and causing that other test to spuriously fail.
2019-01-09 08:51:00 +00:00
Armin Braun 224475e6aa
MINOR: Some Cleanups in Snapshots (#37175)
* Removed pre v6 Bwc logic + comments
* Removed some dead code
2019-01-09 09:45:37 +01:00
Yannick Welsch 9040a96daf
Zen2: Add test that checks that CoordinatorTests are reproducible (#37225)
Ensures reproducibility of the tests in CoordinatorTests.
2019-01-09 09:32:24 +01:00
Tanguy Leroux 7f6fe14b66 Merge branch 'master' into close-index-api-refactoring 2019-01-09 09:26:05 +01:00
Tanguy Leroux 0a40564068
[Tests] Change cluster scope in CorruptedFileIT and FlushIT (#37229)
These tests failed on CI multiple times in the past weeks because they use a 
test cluster with a SUITE scope that recreates nodes between tests. With such 
a scope, nodes can be recreated in between test executions and can inherit a 
node id from a previous test execution, while they are assigned a random data 
path. With the successive node recreations it is possible that a newly recreated 
node shares the same node id (but different data path) as a non recreated node.

This commit changes the cluster scope of the CorruptedFileIT and FlushIT 
tests which often fail.

The failure is reproducable with :

./gradlew :server:integTest -Dtests.seed=EF3A50C225CF377 
-Dtests.class=org.elasticsearch.index.store.CorruptedFileIT 
-Dtests.security.manager=true -Dtests.locale=th-TH-u-nu-thai-x-lvariant-TH -Dtests.timezone=America/Rio_Branco 
-Dcompiler.java=11 -Druntime.java=8
2019-01-09 09:24:56 +01:00
David Turner d9e2ebca67
Add more detail to recovery bandwidth limit docs (#37156) 2019-01-09 08:18:25 +00:00
Ioannis Kakavas 9049263c2c
Handle malformed license signatures (#37137)
This commit adds a more user friendly error message when a license
signature is malformed/truncated in a way that it cannot be
meaningfully parsed.
2019-01-09 07:29:22 +02:00
Ioannis Kakavas 2a79c468f8 Ensure that ActionListener is called exactly once
This bug was introduced in #36893 and had the effect that
execution would continue after calling onFailure on the the
listener in checkIfTokenIsValid in the case that the token is
expired. In a case of many consecutive requests this could lead to
the unwelcome side effect of an expired access token producing a
successful authentication response.
2019-01-09 07:23:35 +02:00
Jun Ohtani 38b698d455
[Analysis] Deprecate Standard Html Strip Analyzer in master (#26719)
* [Analysis] Deprecate Standard Html Strip Analyzer

Deprecate only Standard Html Strip Analyzer
If user create index with the analyzer since 7.0, es throws an exception.
If an index was created before 7.0, es issue deprecation log
We will remove it in 8.0

Related #4704
2019-01-09 12:42:00 +09:00