Commit Graph

25104 Commits

Author SHA1 Message Date
Yannick Welsch 6f69f54216 Use pre-JDK9 style FilePermissions on JDK9 when running from start scripts 2016-11-14 19:52:35 +01:00
Yannick Welsch ea65a01789 Use pre-JDK9 style FilePermissions on JDK9
JDK9 removed pathname canonicalization when constructing FilePermission objects, which breaks some of the FilePermissions added by
Elasticsearch. This commit adds the system property jdk.io.permissionsUseCanonicalPath which makes JDK9 behave like JDK8 w.r.t. FilePermissions (see
https://github.com/elastic/elasticsearch/issues/21534).
2016-11-14 14:13:23 +01:00
Daniel Mitterdorfer 9d3d6c5409 [docs] clients need to add jackson-databind (#21527)
With ES 5.0 we do not include Jackson
Databind anymore with ES core. This commit
updates our docs to state that users need
to add this artifact now in their projects.
2016-11-14 10:07:07 +01:00
Clinton Gormley 30d342c87c Update significantterms-aggregation.asciidoc
Fix scripted significant terms example to use `params.` prefix for painless
2016-11-14 09:40:04 +01:00
Adrien Grand 1fd5c47e7f Upgrade to lucene-6.3.0. (#21464) 2016-11-14 09:36:45 +01:00
Jason Tedor 19decd7552 Hack around cluster service and logging race
When a cluster update task executes, there can be log messages after the
update task has finished processing and the new cluster state becomes
visible. The visibility of the cluster state allows the test thread in
UpdateSettingsIT#testUpdateAutoThrottleSettings and
UpdateSettingsiT#testUpdateMergeMaxThreadCount to proceed. The test
thread will remove and stop a mock appender setup at the beginning of
the test. The log messages in the cluster state update task that occur
after processing has finished can race with the removal of the
appender. Log4j will grab a reference to the appenders when processing
these log messages, and this races with the removal and stopping of the
appenders. If Log4j grabs a reference to the appenders before the mock
appender has been removed, and the test thread subsequently removes and
stops the appender before Log4j has appended the log message, Log4j will
get angry that we are appending to a stopped appender, causing the test
to fail. This commit addresses this race by waiting for the cluster
state update task to have finished processing before freeing the test
thread to make its assertions and finally remove and stop the
appender. Yes, this is a hack.

Relates #21518
2016-11-13 18:06:12 -05:00
Jason Tedor d273419d00 Do not prematurely shutdown Log4j
When a node closes, we shutdown logging as the last statement. This
statement must be last lest any subsequent attempts to log will blow up
by running into security permissions. Yet, in the case of a tribe node
this isn't enough. The first internal tribe node to close will shutdown
logging, and subsequent node closes will blow up with the aforementioned
problem. This commit migrate the Log4j shutdown to occur as part of the
shutdown hook that closes the node, after all nodes have
closed. Consequently, we can remove a hack in the test infrastructure to
prevent Log4j shutdowns when internal test nodes close and instead just
register a single shutdown hook that runs when the test JVM exits.

Relates #21519
2016-11-13 17:27:30 -05:00
Nik Everett 2a328034ef Support decimal constants with trailing [dD] in painless (#21412)
This adds support to painless for decimal constants with trailing `d` or
`D` to make it compatible with Java. It already supported integer
constants with a trailing `d` or `D` but this adds tests for it.

Closes #21116
2016-11-12 11:08:39 -05:00
Nik Everett a26b5a113c In painless suggest a long constant if int won't do (#21415)
In painless we prefer explicit types over implicit ones whereas
groovy is the other way around. Take this groovy code:

```
> 86400000.class
java.lang.Integer
> 864000000000.class
java.lang.Long
```

Painless accepts `86400000` just fine because that is a valid `int`
in the jvm. It rejects `864000000000` as an invlid `int` constant
because, in painless as in java, `long` constants always end in `L`
or `l`.

To ease the transition from groovy to painless, this changes the
compilation error returned from these invalid constants from:

```
Invalid int constant [864000000000].
```

to

```
Invalid int constant [864000000000]. If you want a long constant then change it to [864000000000L].
```

Inspired by #21313
2016-11-12 11:08:18 -05:00
Jason Tedor 404b9afeca Account for different paths for sysctl utilities
On some systems these utilities are in /usr/lib/systemd/systemd-sysctl
and /usr/sbin/sysctl, and on others the /usr is dropped. This commit
accounts for that fact.
2016-11-12 07:54:17 -05:00
Ali Beyad 38023fb58d [TEST] testRebalancePossible() may not have an assigned node id 2016-11-11 23:10:34 -05:00
Igor Motov ca639e8c86 Tests: Disable merge in SearchCancellationTests
We have to have at least 2 segments for the test to work and sometimes random merge policy merges them into one.
2016-11-11 18:22:28 -05:00
Igor Motov 058b6e019c Tests: clean search scroll at the end of SearchCancellationIT
Under some rare conditions search cancellation response might not fully clean scroll context. For now this commit adds the cleaning operation to the test, and we will address the root cause in https://github.com/elastic/elasticsearch/issues/21511
2016-11-11 18:22:15 -05:00
Jason Tedor 1fde26347a Set vm.max_map_count on systemd package install
Our docs claim that we set vm.max_map_count automatically. This is not
quite the case. The story is that on SysV init we set vm.max_map_count
each time the service starts, which is good. On systemd, we create a
sysctl.d conf file that sets vm.map_max_count, but this is only
meaningful if the system is rebooted after package install. This commit
modifies the post-install script so that we run systemd-sysctl so that
the vm.max_map_count change occurs after package install without a
reboot.

Relates #21507
2016-11-11 16:41:54 -05:00
Ali Beyad 5f1d108704 [TEST] reduce the number of snapshotted shards to 1 in testSnapshotSucceedsAfterSnapshotFailure()
so that we are more likely to trigger I/O exceptions on writing the control files during the
finalize phase of snapshotting (with the aim of triggering an I/O failure when writing pending-index-*).
2016-11-11 16:22:11 -05:00
David Pilato 5336e72fbb Add documentation for Logger with Transport Client
Backport of #21477 in master branch
2016-11-11 21:23:23 +01:00
Jason Tedor 9352d16602 Enable appender exceptions in UpdateSettingsIT
This commit sets the mock appender in UpdateSettingsIT to not ignore
exceptions. This means that when an exception is hit, we will see an
actual stack trace that could be useful in debugging a non-reproducible
test failure.

Relates #21461
2016-11-11 12:41:20 -05:00
Ali Beyad c9c3992f94 [TEST] remove AwaitsFix from testSnapshotSucceedsAfterSnapshotFailure,
turns out the issue is specific to Java 9 v143
2016-11-11 12:37:04 -05:00
Jason Tedor 79076334ae Cleanup formatting in UpdateSettingsIT.java
This commit cleans up some code formatting in UpdateSettingsIT.java and
removes this from from the checkstyle line-length supressions.
2016-11-11 12:10:32 -05:00
Ali Beyad 8f85e388da [TEST] mute the testSnapshotSucceedsAfterSnapshotFailure() test
until its clear what is going wrong.

Relates #21496
2016-11-11 11:50:23 -05:00
Jason Tedor 372480a16a Mark SearchQueryIT test as awaits fix
This commit marks the test SearchQueryIT#testRangeQueryWithTimeZone as
awaits fix.

Relates #21501
2016-11-11 11:33:17 -05:00
Ali Beyad a5ccd02e76 Makes snapshot throttling test go much faster (#21485)
[TEST] Makes the snapshot throttling test go much faster.  Before, 
the snapshot throttling test would throttle at a rate of 0.5 kb per
second, even though it would snapshot/restore about 25 kb of data.
This commit increases the throttling rate to 10kb per second, so
we still test the throttling mechanism while speeding up the test from
taking 30 plus seconds down to 2 seconds or less.
2016-11-11 10:52:26 -05:00
Nik Everett 7c3886769f Breaking changes docs for template index_patterns
0219a211d3 added support for templates
to have multiple patterns and renamed `template` to `index_patterns`.
This adds the breaking changes docs for that.
2016-11-11 10:13:19 -05:00
Ali Beyad adb7aaded4 [TEST] adds randomness between atomic and non-atomic move
operations in MockRepository
2016-11-11 09:07:28 -05:00
Yannick Welsch 2d3a52c0f2 Cache successful shard deletion checks (#21438)
Each node checks on every cluster state update if there are shards that it can possibly delete from its disk. It decides this by doing a file-system lookup for each shard id that is fully allocated in the cluster. With lots of shards, this amounts to lots of Files.exists() checks, considerably slowing down cluster state updates. This commit adds a caching layer so that the Files.exists() checks can be skipped if not needed.
2016-11-11 10:06:15 +01:00
Igor Motov df965fc9b3 Task cancellation command should wait for all child nodes to receive cancellation request before returning
Currently the task cancellation command returns as soon as the top-level parent child is marked as cancelled. This create race conditions in tests where child tasks on other nodes may continue to run for some time after the main task is cancelled. This commit fixes this situation making task cancellation command to wait until it got propagated to all nodes that have child tasks.

Closes #21126
2016-11-10 22:43:43 -05:00
Igor Motov 06a50fa31e ShardActiveResponseHandler shouldn't hold to an entire cluster state
ShardActiveResponseHandler doesn't need to hold to an entire cluster state since it only needs to know the cluster state version. It seems that on overloaded systems where nodes are unresponsive holding onto a lot of different cluster states can make the situation worse.

Closes #21394
2016-11-10 22:28:49 -05:00
Ali Beyad 3001b636db Ensures cleanup of temporary index-* generational blobs during snapshotting (#21469)
Ensures pending index-* blobs are deleted when snapshotting.  The
index-* blobs are generational files that maintain the snapshots
in the repository.  To write these atomically, we first write a
`pending-index-*` blob, then move it to `index-*`, which also deletes
`pending-index-*` in case its not a file-system level move (e.g.
S3 repositories) .  For example, to write the 5th generation of the
index blob for the repository, we would first write the bytes to
`pending-index-5` and then move `pending-index-5` to `index-5`.  It is
possible that we fail after writing `pending-index-5`, but before
moving it to `index-5` or deleting `pending-index-5`.  In this case,
we will have a dangling `pending-index-5` blob laying around.  Since
snapshot #5 would have failed, the next snapshot assumes a generation
number of 5, so it tries to write to `index-5`, which first tries to
write to `pending-index-5` before moving the blob to `index-5`.  Since
`pending-index-5` is leftover from the previous failure, the snapshot
fails as it cannot overwrite this blob.

This commit solves the problem by first, adding a UUID to the
`pending-index-*` blobs, and secondly, strengthen the logic around
failure to write the `index-*` generational blob to ensure pending
files are deleted on cleanup.

Closes #21462
2016-11-10 21:45:02 -05:00
Ryan Ernst 48bfb142b9 Remove (again) test uses of onModule (#21414)
This change was reverted after it caused random test failures. This was
due to a copy/paste error in the original PR which caused the mock
version of ClusterInfoService to be used whenever the mock *ZenPing* was
used, and the real ClusterInfoService to be used when MockZenPing was
not used.
2016-11-10 16:06:14 -08:00
Areek Zillur 7ed195fe93 [TEST] Add assertBusy when checking for pending operation counter after tests
Currently, pending operations can complete after tests with disruption scheme
completes. This commit waits for the pending operation counter to complete
after the tests are run
2016-11-10 18:35:52 -05:00
Areek Zillur 5b4c3fb1ac Revert "Add trace logging when aquiring and releasing operation locks for replication requests"
This reverts commit 4e996ca9f5.
2016-11-10 18:35:25 -05:00
Alexander Lin 0219a211d3 Allows multiple patterns to be specified for index templates (#21009)
* Allows for an array of index template patterns to be provided to an
index template, and rename the field from 'template' to 'index_pattern'.

Closes #20690
2016-11-10 18:00:30 -05:00
Ali Beyad 5c4392e58a [TEST] fixes rebalance single shard check as it isn't guaranteed that a
rebalance makes sense and the method only tests if rebalance is allowed
2016-11-10 17:13:39 -05:00
Nik Everett eeb6602c98 Document _reindex with random_score
You can use `_reindex` and `random_score` to extract a random
subset of an index but you have to be careful to sort by `_score`
or it won't work.

Closes #21432
2016-11-10 16:14:30 -05:00
Jason Tedor 179dd885e2 Avoid angering Log4j in TransportNodesActionTests
When logging a mock exception, Log4j attempts to render the stack
trace. On a mock exception, this will be null and Log4j will hit a
NullPointerException. This NullPointerException will get recorded in the
status logger buffer that we use to ensure that we do not having any
misuses of Log4j in production code. This commit replaces the use of a
mock exception with an actual exception to avoid angering the Log4j
assertions in ESTestCase.
2016-11-10 16:08:08 -05:00
Areek Zillur 4e996ca9f5 Add trace logging when aquiring and releasing operation locks for replication requests 2016-11-10 15:13:42 -05:00
Jason Tedor 0a06a0c2b3 Fix handler name on message not fully read
Today when a message is not fully read on a response, we log (among
other details) the handler name. Unfortunately, if the handler is a
wrapper, all that we see is

o.e.t.TransportService$ContextRestoreResponseHandler@7446ba18

completely losing the offending handler. This commit adds an override
for TransportService$ContextRestoreResponseHandler#toString so that the
underlying offender can be discovered.

Relates #21478
2016-11-10 14:56:48 -05:00
Jack Conradson 834976823a Remove accidental import. 2016-11-10 11:46:14 -08:00
Jason Tedor fdbe336104 Improve log message in TransportNodesAction
Today when handling responses from nodes in TransportNodesAction, if a
node timeouts or some other failure occurs and the action is not
accumulating exceptions, we log a confusing message:

org.elasticsearch.action.admin.cluster.stats.TransportClusterStatsAction]
ignoring unexpected response [null] of type [null], expected
[ClusterStatsNodeResponse] or [FailedNodeException]

Moreover, the original exception is completely lost. Since this log
message is confusing and unhelpful, we can drop it. Instead, we hold
onto the exception and log it at the warn level before dropping it from
the response.

Relates #21476
2016-11-10 14:32:14 -05:00
Jack Conradson aeb97ff412 Clean up of Script.
Closes #21321
2016-11-10 09:59:13 -08:00
Tanguy Leroux 2e531902ff Update Joda Time to version 2.9.5 (#21468)
This commit updates JodaTime to version 2.9.5 that contain a fix for a bug when parsing time zones (see https://github.com/JodaOrg/joda-time/pull/332, https://github.com/JodaOrg/joda-time/issues/386 and https://github.com/JodaOrg/joda-time/issues/373).

It also remove the joda-convert dependency that seems to be unused.
    
closes #20911

Here is the changelog for 2.9.5:
```
Changes in 2.9.5
----------------
 - Add Norwegian period translations [#378]

 - Add Duration.dividedBy(long,RoundingMode) [#69, #379]

 - DateTimeZone data updated to version 2016i

 - Fixed bug where clock read twice when comparing two nulls in DateTimeComparator [#404]

 - Fixed minor issues with historic time-zone data [#373]

 - Fix bug in time-zone binary search [#332, #386]
  The fix in v2.9.2 caused problems when the time-zone being parsed
  was not the last element in the input string. New approach uses a
  different approach to the problem.

 - Update tests for JDK 9 [#394]

 - Close buffered reader correctly in zone info compiler [#396]

 - Handle locale correctly zone info compiler [#397]
```
2016-11-10 17:32:46 +01:00
Luca Cavanna 10a4288a4c Remove unused ClusterService dependency from SearchPhaseController (#21421) 2016-11-10 17:32:19 +01:00
Luca Cavanna 06aabd9ecd Remove max_local_storage_nodes from elasticsearch.yml (#21467)
Given that the default is now 1, the comment in the config file was outdated. Also considering that the default value is production ready, we shouldn't list it among the values that need attention when going to production.

Relates to #19964
2016-11-10 16:52:38 +01:00
Nik Everett 4db21db0aa Wait for all reindex subtasks before rethrottling
In the test for reindex and friend's rethrottling feature we were waiting
only for a single reindex sub task to start before rethrottling. This
mostly worked because starting tasks is fast. But it didn't *always work
and CI found that for us. This fixes the test to wait for all subtasks
to start before rethrottling.

I reproduced this locally semi-consistently with some fairly creative
`Thread.sleep` calls and this test fix fixes the issue even with the
sleeps so I'm fairly sure this will work consistently.

Closes #21446
2016-11-10 10:49:25 -05:00
Sandeep Kanabar 70bb51edfe Correcting a typo-Maan to Man-in README.textile (#21466) 2016-11-10 10:17:19 -05:00
Luca Cavanna bd23921a3a Fix InternalSearchHit#hasSource to return the proper boolean value (#21441)
The method used to be called `isSourceEmpty`, and was renamed to `hasSource`, but the return value never changed. Updated tests and users accordingly.

Closes #21419
2016-11-10 13:13:38 +01:00
Clinton Gormley 860efb2ad2 Replace all index date-math examples with the URI encoded form
Closes #21454
2016-11-10 12:23:19 +01:00
lslxdx e560c5be43 Fix typos (#21456)
Delete repeated words " between nodes and".
2016-11-10 11:55:04 +01:00
Tanguy Leroux 3f7f8e4b97 Adapt ES_JVM_OPTIONS packaging test to ubuntu-1204
This commit adapts the "[INIT.D] start Elasticsearch with custom JVM options" packaging test so that it works on ubuntu-1204.

Related to #21445
2016-11-10 11:11:47 +01:00
Nguyễn Thanh Tiến 27a7b30349 Add null check in InternalSearchHit#sourceRef to prevent NPE (#21431)
Add null check in InternalSearchHit#sourceRef to prevent NPE

Closes #19279
2016-11-10 10:54:43 +01:00