Commit Graph

44888 Commits

Author SHA1 Message Date
David Turner 6e4f304f88 Synchronize pendingOutgoingJoins (#39900)
Today we use a ConcurrentHashSet to track the in-flight outgoing joins in the
`JoinHelper`. This is fine for adding and removing elements but not for the
emptiness test in `isJoinPending()` which might return false if one join
finishes just after another one starts, even though joins were pending
throughout.

As used today this is ok: it means the node was trying to join a master but
this join attempt just finished unsuccessfully, and causes it to (rightfully)
reject a `FollowerCheck` from the failed master. However this kind of API
inconsistency is trappy and there is no need to be clever here, so this change
replaces the set with a `synchronizedSet()`.
2019-03-11 12:13:21 +00:00
Costin Leau a079b9fd6d SQL: ConstantProcessor can now handle NamedWriteable (#39876)
Enhance ConstantProcessor to properly serialize complex objects
(Intervals) that have their own custom serialization/deserialization
mechanism

Fix #39875

(cherry picked from commit ed8a1f9340673e69a44ea7a89679cadb4762e43d)
2019-03-11 12:49:23 +02:00
Ankit Jain 471aa6a16a Fixing 503 Service Unavailable errors during fetch phase (#39086)
When ESRejectedExecutionException gets thrown on the coordinating node while trying to fetch hits, the resulting exception will hold no shard failures, hence `503` is used as the response status code. In that case, `429` should be returned instead. Also, the status code should be taken from the cause if available whenever there are no shard failures instead of blindly returning `503` like we currently do.

Closes #38586
2019-03-11 10:13:55 +01:00
Martijn van Groningen 8925a2c6c2
Further tweak AutoFollowIT#testAutoFollowManyIndices:
* reduce the number of leader indices to be auto followed
* also check the number of follower indices being created
* also check the whether leader indices are marked as auto followed

Relates to #36761
2019-03-11 10:01:56 +01:00
Daniel Mitterdorfer 1bc31aca03
Mute CcrRetentionLeaseIT#testRetentionLeaseRenewalIsCancelledWhenFollowingIsPaused (#39897)
Relates #39509
2019-03-11 08:47:51 +01:00
Adrien Grand b841de2e38
Don't emit deprecation warnings on calls to the monitoring bulk API. (#39805) (#39838)
The monitoring bulk API accepts the same format as the bulk API, yet its concept
of types is different from "mapping types" and the deprecation warning is only
emitted as a side-effect of this API reusing the parsing logic of bulk requests.

This commit extracts the parsing logic from `_bulk` into its own class with a
new flag that allows to configure whether usage of `_type` should emit a warning
or not. Support for payloads has been removed for simplicity since they were
unused.

@jakelandis has a separate change that removes this notion of type from the
monitoring bulk API that we are considering bringing to 8.0.
2019-03-11 07:58:28 +01:00
Adrien Grand 2bbef67770
Propagate exceptions in o.e.common.io.Streams. (#39042) (#39848)
This commit propagates some exceptions that were previously swallowed and also
makes sure that exceptions closing streams are either propagated if the try
block succeeded or added as suppressed exceptions otherwise.
2019-03-11 07:58:01 +01:00
Benjamin Trent 4da04616c9
[ML] refactoring lazy query and agg parsing (#39776) (#39881)
* [ML] refactoring lazy query and agg parsing

* Clean up and addressing PR comments

* removing unnecessary try/catch block

* removing bad call to logger

* removing unused import

* fixing bwc test failure due to serialization and config migrator test

* fixing style issues

* Adjusting DafafeedUpdate class serialization

* Adding todo for refactor in v8

* Making query non-optional so it does not write a boolean byte
2019-03-10 14:54:02 -05:00
David Emanuel Buchmann b5ed039160
plugins/repository-gcs: Update google-cloud-storage/core to 1.59.0 (#39748)
* plugins/repository-gcs: Update google-cloud-storage /
google-cloud-core to 1.59.0

* plugins: Update sha1 for google-cloud-core & google-cloud-storage
2019-03-10 11:04:52 -04:00
Benjamin Trent 6c6549fc51
[Data-Frame] make the config be strictly parsed on _preview (#39713) (#39873)
* [Data-Frame] make the config be strictly parsed on _preview

* adding test to verify strictly parsing

* adjusting test after master merge
2019-03-09 14:03:57 -06:00
Julie Tibshirani 8454cfc1b2 Move validation from FieldTypeLookup to MapperMergeValidator. (#39814)
This commit consolidates more mapping validation logic into the same class.
`FieldTypeLookup` is now a bit simpler, and has the sole responsibility of quickly
resolving field names to their types.

I have a broader refactor planned around mapping merge validation, but this
change should at least be a step in the right direction.
2019-03-08 18:05:21 -08:00
Nhat Nguyen 993182e426 Combine overriddenOps and skippedOps in translog (#39771)
These two stats are not important enough to be distinguishable.
This change combines them into a single stat.

Closes #33317
2019-03-08 16:28:50 -05:00
Mayya Sharipova 671a209ed9 Correct errors in min_hash filter documentation
Related to #39671
2019-03-08 16:21:24 -05:00
Julie Tibshirani be9c37fc76 Small simplifications to mapping validation. (#39777)
These simplifications to `MapperMergeValidator` are possible now that there is
always a single mapping definition.

* Remove the type argument in `validateMapperStructure`.
* Remove unnecessary checks against existing mappers.
2019-03-08 12:34:09 -08:00
Nhat Nguyen a0a91f74ff Treat TransportService stopped error as node is closing (#39800)
If TransportService is stopped before a shard-failure request is sent
but after the request is registered, TransportService will notify
ReplicationOperation a TransportException with an error message:
"transport stop, action: internal:cluster/shard/failure".

Relates #39584
2019-03-08 15:15:56 -05:00
Tal Levy 6c52da54c8
fix index refresh in test within 20_mix_typeless_typeful (#39198) (#39804)
the test "Implicitly create a typeless ... typed template"
fails occasionally because the index operation hasn't
propogated to update the index mapping in time for the
following assertion about a dynamically mapped field "bar".

error failed with:

```
field [test-1.mappings.my_type.properties.bar] doesn't have a true value
Expected: not null
     but: was null
```

refreshing the index should resolve this timing issue.
2019-03-08 12:15:32 -08:00
Tim Brooks 5612ed97ca
Add log warnings for long running event handling (#39729)
Recently we have had a number of test issues related to blocking
activity occuring on the io thread. This commit adds a log warning for
when handling event takes a >150 milliseconds. This is implemented
for the MockNioTransport which is the transport used in
ESIntegTestCase.
2019-03-08 13:07:24 -07:00
Jason Tedor 73a672b8dd
Fix Watcher stats class cast exception (#39821)
The watcher stats implementation tries to look at all queued watches
before preparing the result. We want to cast these to a
WatchExecutionTask to extract the context to prepare the stats for
queued watches. The problem is that not all tasks on the watcher queue
were WatchExecutionTask. This is because a manually executed watch was
not even at all wrapped in a WatchExecutionTask. Moreover, we were using
ExecutorService#submit(Runnable) which would wrap the Runnable in a
FutureTask<?>. This commit addresses this by using a WatchExecutionTask,
and also using ExecutorService#execute(Runnable) so that no wrapping
occurs. This will let us continue with the assumption that all queued
tasks are WatchExecutionTasks.
2019-03-08 14:52:10 -05:00
lcawl 3246d3e355 [DOCS] Adds tagged region for notable breaking changes 2019-03-08 11:14:27 -08:00
Ryan Ernst 35dd52461f Make integ tests always use runtime java (#39732)
This commit ensures cluster formation always uses runtime java, never
attempting to use the bundled jdk, since the integ test zip does not
contain it.
2019-03-08 11:08:26 -08:00
David Roberts 89a1f155d3 Give jspawnhelper execute permissions in bundled JDK (#39787)
The posix_spawn method of launching a process from Java
goes via an intermediate process called jspawnhelper
which lives in the lib directory rather than the bin
directory and hence got missed by the original chmod
loop.  This change adds jspawnhelper as a special case.
It's the only program that's in the lib directory in a
macOS JDK 11.
2019-03-08 11:05:13 -08:00
Ryan Ernst 465343f12a
Bundle java in distributions (#38013)
* Bundle java in distributions

Setting up a jdk is currently a required external step when installing
elasticsearch. This is particularly problematic for the rpm/deb packages
as installing a jdk in the same package installation command does not
guarantee any order, so must be done in separate steps. Additionally,
JAVA_HOME must be set and often causes problems in selecting a correct
jdk when, for example, the system java is an older unsupported version.

This commit bundles platform specific openjdks into each distribution.
In addition to eliminating the issues above, it also presents future
possible improvements like using jlink to build jdk images only
containing modules that elasticsearch uses.

closes #31845
2019-03-08 11:04:18 -08:00
Jake Landis e0abc3ce96
Remove the index type from internal watcher indexes (#39761) (#39853)
This commit removes the "doc" type from watcher internal indexes.
The template still carries the "_doc" type since that is needed for
the internal representation.

This impacts the .watches, .triggered-watches, and .watch-history indexes.

External consumers do not need any changes since all external calls
go through the _watcher API, and should not interact with the the .index directly.

Relates #38637
2019-03-08 12:46:36 -06:00
Albert Zaharovits 3c7fafd0cc Fix token invalidation when retries exhausted (#39799)
Fixes an error about missing to call the index invalidation listener
when retry count is exhausted but there are still tokens to be retried.
2019-03-08 20:18:59 +02:00
Jason Tedor 6675bafc49
Simplify CcrRetentionLeaseIT#testForgetFollower
This test was more complicated than necessary, where we were capturing
requests to prevent removal of retention leases, so that our forget
follower request could remove the retention leases instead. Instead, a
pause is enough to ensure that the retention leases are not re-added
after we remove them by the forget follower request. This commit
simplifies this test, and should remove some spurious failures.

Relates #39850
2019-03-08 12:33:17 -05:00
Przemyslaw Gomulka c23ff0803e
Fix JsonLoggerTests.testJsonInStacktraceMessageIsSplitted backport#39830 #39834
a test was splitting a json log message with a wrong line separator.
Using System.lineSeparator consistently prevents this problem
closes #39829
backport#39830
2019-03-08 16:34:17 +01:00
Gordon Brown e6b9262a31 Mute testOpenCloseApiWildcards (#39578) (#39579) 2019-03-08 15:18:16 +00:00
Jake Landis a8530c5531
Update logstash-management.json to use typeless template (#38653) (#39819)
This commit changes the type from "doc" to "_doc" for the
.logstash-management template. Since this is an internally
managed template it does not always go through the REST
layer for it's internal representation.  The internal
representation requires the default "_doc" type, which for
external templates is added in the REST layer.

Related #38637
2019-03-08 08:23:30 -06:00
David Roberts aec2db78ea Mute RareClusterStateIT.testDelayedMappingPropagationOnReplica
Due to https://github.com/elastic/elasticsearch/issues/36813
2019-03-08 13:28:27 +00:00
David Kyle 6c2e831e94
[ML-Dataframe] Data frame config HLRC objects (#39825) 2019-03-08 12:18:55 +00:00
Martijn van Groningen 8666aa1ed2
unmuted and tweaked test
Relates to #36761
2019-03-08 12:43:23 +01:00
David Roberts 366eef99a1 Mute SharedClusterSnapshotRestoreIT.testCloseOrDeleteIndexDuringSnapshot
Due to https://github.com/elastic/elasticsearch/issues/39828
2019-03-08 11:42:13 +00:00
Hendrik Muhs 50d742320d store the doc type in the internal index (#39824)
store the doc type in the internal data frame index
2019-03-08 12:17:23 +01:00
David Turner 5d68143b18 Reformat elasticsearch-node messages (#39811)
Flows the warning messages emitted by the `elasticsearch-node` tool to a width
of 72 characters and tweaks the wording slightly.
2019-03-08 10:01:29 +00:00
CJ Cenizal 1520ddc8b6
Update distribution build instructions to reflect file names with OS/architecture classifiers. (#39762) (#39812) 2019-03-07 14:08:47 -08:00
Jake Landis 797d6b8a66
Execute ingest node pipeline before creating the index (#39607) (#39796)
Prior to this commit (and after 6.5.0), if an ingest node changes
the _index in a pipeline, the original target index would be created.
For daily indexes this could create an extra, empty index per day.

This commit changes the TransportBulkAction to execute the ingest node
pipeline before attempting to create the index. This ensures that the 
only index created is the original or one set by the ingest node pipeline. 
This was the execution order prior to 6.5.0 (#32786). 

The execution order was changed in 6.5 to better support default pipelines. 
Specifically the execution order was changed to be able to read the settings
from the index meta data. This commit also includes a change in logic such 
that if the target index does not exist when ingest node pipeline runs, it 
will now pull the default pipeline (if one exists) from the settings of the 
best matched of the index template. 

Relates #32786
Relates #32758 
Closes #36545
2019-03-07 13:31:41 -06:00
Lee Hinman 8ec456b5df Maintain step order for ILM trace logging (#39522)
When trace logging is enabled we log the computed steps for a policy. This
commit makes sure that the steps that are logged are in the same order they will
be run when the policy executes. This makes it much easier to reason about the
policy if the move-to-step API is ever required in the future.
2019-03-07 11:37:58 -07:00
Hendrik Muhs 4d41310be5 [ML-DataFrame] fix wire serialization issues in data frame response objects (#39790)
fix wire serialization issues in data frame response objects
2019-03-07 19:28:44 +01:00
Martijn Laarman af4e740500 Document scroll param on reindex.json (#38615)
The Reindex API also exposes `scroll` as a querystring parameter.
2019-03-07 18:14:29 +01:00
Martijn van Groningen 3dfd62cf51
unmute test #39355 2019-03-07 17:55:42 +01:00
Tim Brooks 8043fefcf6
Log close_notify during handshake at debug level (#39715)
A TLS handshake requires exchanging multiple messages to initiate a
session. If one side decides to close during the handshake, it is
supposed to send a close_notify alert (similar to closing during
application data exchange). The java SSLEngine engine throws an
exception when this happens. We currently log this at the warn level if
trace logging is not enabled. This level is too high for a valid
scenario. Additionally it happens all the time in tests (quickly closing
and opened transports). This commit changes this to be logged at the
debug level if trace is not enabled. Additionally, it extracts the
transport security exception handling to a common class.
2019-03-07 09:52:18 -07:00
Jason Tedor b9586f62cc
Fix CCR HLRC docs
This commit fixes the CCR HLRC docs by including the forget follower API
docs in the HLRC docs.
2019-03-07 11:44:35 -05:00
Jason Tedor 0250d554b6
Introduce forget follower API (#39718)
This commit introduces the forget follower API. This API is needed in cases that
unfollowing a following index fails to remove the shard history retention leases
on the leader index. This can happen explicitly through user action, or
implicitly through an index managed by ILM. When this occurs, history will be
retained longer than necessary. While the retention lease will eventually
expire, it can be expensive to allow history to persist for that long, and also
prevent ILM from performing actions like shrink on the leader index. As such, we
introduce an API to allow for manual removal of the shard history retention
leases in this case.
2019-03-07 11:08:45 -05:00
Alpar Torok 6c75a2f2b0 Testclusters: start using it for testing some plugins (#39693)
* enable testclusters for some plugins
2019-03-07 17:52:50 +02:00
Alpar Torok 7dcc191aa8 Fix verify versions (#39624)
closes #38708
2019-03-07 17:40:25 +02:00
Alpar Torok 0f89427eb6
Back port build changes from same version bwc tests (#39744)
* Back port build changes from #39102

This back-ports how versions are determined and bwc test are set up from
 #39102 without enabling the bwc from current version tests so it's
 easier/possible to backmerge future buld changes.
It's expected that the tets are lacking many of the required fixes in
this version to enable them.
2019-03-07 17:25:09 +02:00
Mayya Sharipova 54d41afac1 Add documentation for min_hash filter (#39671)
Closes #20757
2019-03-07 08:49:48 -05:00
Akshesh Doshi 4df26a2a90 Implement ConcatFilesTask from Groovy to Java issues#34459 (#37497)
relates  #344597
2019-03-07 15:30:54 +02:00
Ioannis Kakavas 6c19d872a0 Fix testRefreshingMultipleTimesWithinWindowSucceeds (#39701)
Previously all the threads were writing the received tokens to a
HashSet. In cases with many threads, sometimes (1 every ~25 tests)
calling size() on the HashSet returned 2 even though it seemed to
contain only one String and there was no evidence from logging that
threadSecurityClient.refreshToken() ever returned a different
access or refresh token.

This commit changes the test to use a ConcurrentHashMap instead,
checking that we only received one pair of access token/refresh token
eventually. It also adds a check so that we won't take into consideration
tokens that are returned after 30s, hence not in the concurrent refresh
time window.
2019-03-07 13:13:50 +02:00
Przemyslaw Gomulka 95bed81198
Change licence expiration date pattern Backport(#39681) #39781
Due to migration from joda to java.time licence expiration 'full date' format
has to use 4-char pattern (MMMM). Also since jdk9 the date with ROOT
locale will still return abbreviated days and month names.

closes #39136
backport #39681
2019-03-07 12:06:18 +01:00