Commit Graph

27366 Commits

Author SHA1 Message Date
Ryan Ernst 17d01550c2 S3 Repository: Add back repository level credentials (#24609)
Specifying s3 access and secret keys inside repository settings are not
secure. However, until there is a way to dynamically update secure
settings, this is the only way to dynamically add repositories with
credentials that are not known at node startup time. This commit adds
back `access_key` and `secret_key` s3 repository settings, but protects
it with a required system property `allow_insecure_settings`.
2017-05-11 12:14:23 -07:00
Simon Willnauer 1155615536 Move DeleteByQuery and Reindex requests into core (#24578)
This allows other plugins to use a client to call the functionality
that is in the core modules without duplicating the logic.
Plugins can now safely send the request and response classes via the
client even if the requests are executed locally. All relevant classes
are loaded by the core classloader such that plugins can share them.

This is re-adds this commit that was revered in 952feb58e4
2017-05-11 20:22:30 +02:00
Jason Tedor 657686cefb Fix runtime class cast exception in TRA
If we fail to acquire the shard lock, need to retry and wait for the new
cluster state, we were sending the wrong kind of request for the replica
action. This commit fixes this issue.
2017-05-11 13:58:03 -04:00
qwerty4030 e7d352b489 Compound order for histogram aggregations. (#22343)
This commit adds support for histogram and date_histogram agg compound order by refactoring and reusing terms agg order code. The major change is that the Terms.Order and Histogram.Order classes have been replaced/refactored into a new class BucketOrder. This is a breaking change for the Java Transport API. For backward compatibility with previous ES versions the (date)histogram compound order will use the first order. Also the _term and _time aggregation order keys have been deprecated; replaced by _key.

Relates to #20003: now that all these aggregations use the same order code, it should be easier to move validation to parse time (as a follow up PR).

Relates to #14771: histogram and date_histogram aggregation order will now be validated at reduce time.

Closes #23613: if a single BucketOrder that is not a tie-breaker is added with the Java Transport API, it will be converted into a CompoundOrder with a tie-breaker.
2017-05-11 18:06:26 +01:00
Simon Willnauer 952feb58e4 Revert "Move DeleteByQuery and Reindex requests into core (#24578)"
This reverts commit 6ea2ae32b8.
2017-05-11 18:26:40 +02:00
Lee Hinman 57fddce8c4 [TEST] Use at least 1ms for FunctionScoreQueryBuilderTests
Previously micros or nanoseconds could be used, which was reduced to 0
milliseconds and `scale` must be higher than 0.
2017-05-11 10:10:55 -06:00
Uwe Schindler f7c50f5f71 Painless: Optimize instance creation in LambdaBootstrap (#24618)
Optimize instance creation in LambdaBootstrap to allow Hotspot's escape analysis, preventing us from creating many instances stressing GC
2017-05-11 09:10:27 -07:00
Jason Tedor d9cac191a2 Fix missing active IDs prevent advance test
This commit addresses an issue in the missing active IDs prevent advance
test from the global checkpoint tracker. The assumptions this test was
making about reality were violated when global checkpoints were inlined
(specifically, the component of that change where the tracker's
knowledge of the global checkpoint was updated inline with updates to
the tracker's knowledge of local checkpoints for an allocatio ID). The
point of the test was to ensure that a lagging shard prevents the global
checkpoint from advancing, so this commit rewrites the test with that in
mind.
2017-05-11 11:39:33 -04:00
Nik Everett 9953a96143 Build: move dependency so it works in 5.x
Relates to #24561
2017-05-11 10:41:56 -04:00
Simon Willnauer 6ea2ae32b8 Move DeleteByQuery and Reindex requests into core (#24578)
This allows other plugins to use a client to call the functionality
that is in the core modules without duplicating the logic.
Plugins can now safely send the request and response classes via the
client even if the requests are executed locally. All relevant classes
are loaded by the core classloader such that plugins can share them.
2017-05-11 16:20:40 +02:00
Nik Everett 8188569fd1 Add qa module that tests reindex-from-remote against pre-5.0 versions of Elasticsearch (#24561)
Adds tests for reindex-from-remote for the latest 2.4, 1.7, and
0.90 releases. 2.4 and 1.7 are fairly popular versions but 0.90
is a point of pride.

This fixes any issues those tests revealed.

Closes #23828
Closes #24520
2017-05-11 10:06:20 -04:00
Simon Willnauer 8f798f1231 Add up-to-date javadocs on Translog#syncNeeded 2017-05-11 15:28:39 +02:00
Simon Willnauer 64d0d9184d Fix IndexShardIT#testDurableFlagHasEffect to only test if operations have been synced
With global checkpoints we also take into account if a global checkpoint must be fsynced.
Yet, with recent addition of inlining global checkpoints into indexing operations from a
test perspective unnecessary fsyncs might be reported if `Translog#syncNeeded` is checked.
Now the test only check if the last write location triggers an fsync instead.

Closes #24600
2017-05-11 15:13:09 +02:00
Martijn van Groningen 840da4aebf
Removed deprecated template query.
Relates to #19390
2017-05-11 14:56:45 +02:00
Nik Everett e49ebd04fd Get more information when reindex test fails
It looks like it leaks contexts but it isn't clear why so this
adds a little more logging. This is the failure:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+5.4+multijob-intake/94/console
2017-05-11 08:36:54 -04:00
Jason Tedor b6cb9a4e8a Fix checkpoints advance test
This commit fixes an issue in the checkpoints advance test. Namely, when
there zero documents indexed, after the global checkpoint is synced, the
global checkpoint will have advanced to the no ops performed. There is a
larger conceptual problem here, namely that the primary does not update
its knowledge of its own local checkpoint upon recovery which causes the
global checkpoint to initially be unassigned and then advance to no ops
performed, but this will be addressed in a follow-up.
2017-05-11 07:25:11 -04:00
Martijn van Groningen e5b42bed50
inner_hits: Reuse inner hit query weight
Previously query weight was created for each search hit that needed to compute inner hits,
 with this change the weight of the inner hit query is computed once for all search hits.

Closes #23917
2017-05-11 11:48:13 +02:00
Jim Ferenczi e13db1b269 Validates updated settings on closed indices (#24487)
We allow non-dynamic settings to be updated on closed indices but we don't
check if the updated settings can be used to open/create the index.
This can lead to unrecoverable state where the settings are updated but the index
cannot be reopened since the settings are not valid. Trying to update the invalid settings
is also not possible since the update will fail to validate the current settings.
This change adds the validation of the updated settings for closed indices and make sure that the new settings
do not prevent the reopen of the index.

Fixes #23787
2017-05-11 11:06:55 +02:00
propulkit 77feabb3d5 Machine Permission update on Google Compute
Machine permission can be updated without deleting instances.

Backport of #24607 in master branch
2017-05-11 07:52:34 +02:00
Ryan Ernst 0789a74055 S3 Repository: Remove deprecated settings (#24445)
These settings are deprecated in 5.5. This change removes them for 6.0.
2017-05-10 20:12:17 -07:00
Ali Beyad bc223b6aef Removes completed snapshot from cluster state on master change (#24605)
Previously, if a master node updated the cluster state to reflect that a
snapshot is completed, but subsequently failed before processing a
cluster state to remove the snapshot from the cluster state, then the
newly elected master would not know that it needed to clean up the
leftover cluster state.

This commit ensures that the newly elected master sees if there is a
snapshot in the cluster state that is in the completed state but has not
yet been removed from the cluster state.

Closes #24452
2017-05-10 23:08:59 -04:00
Nik Everett 65f2717ab7 Make PreConfiguredTokenFilter harder to misuse (#24572)
There are now three public static method to build instances of
PreConfiguredTokenFilter and the ctor is private. I chose static
methods instead of constructors because those allow us to change
out the implementation returned if we so desire.

Relates to #23658
2017-05-10 22:39:43 -04:00
Chris Ivens d447b79e16 Docs: Tiny typo to Painless dispatch justification (#24588) 2017-05-10 22:05:19 -04:00
Jason Tedor 0b089ac084 Refresh at end of concurrent relocate test
We have to do something to force the global checkpoint to be
synchronized to the replicas or the assertions at the end of the test
that they are in sync will trip. Since the last write operation to hit a
replica shard will only carry the penultimate global checkpoint (it will
advance when the replicas respond with their local checkpoint), and a
background sync will not happen until the primary shard falls idle, we
force a sync through a refresh action.
2017-05-10 18:54:22 -04:00
Jack Conradson 349d1bd23e Fixes ScriptServiceTests.testDefaultBehaviourFineGrainedSettings due to
an error with the fine grained settings deprecation.
2017-05-10 15:00:29 -07:00
James Baiera 6a113ae499 Introduce Kerberos Test Fixture for Repository HDFS Security Tests (#24493)
This PR introduces a subproject in test/fixtures that contains a Vagrantfile used for standing up a 
KRB5 KDC (Kerberos). The PR also includes helper scripts for provisioning principals, a few 
changes to the HDFS Fixture to allow it to interface with the KDC, as well as a new suite of 
integration tests for the HDFS Repository plugin.

The HDFS Repository plugin senses if the local environment can support the HDFS Fixture 
(Windows is generally a restricted environment). If it can use the regular fixture, it then tests if 
Vagrant is installed with a compatible version to determine if the secure test fixtures should be 
enabled. If the secure tests are enabled, then we create a Kerberos KDC fixture, tasks for adding 
the required principals, and an HDFS fixture configured for security. A new integration test task is 
also configured to use the KDC and secure HDFS fixture and to run a testing suite that uses 
authentication. At the end of the secure integration test the fixtures are torn down.
2017-05-10 17:42:20 -04:00
Ali Beyad 63a408a128 Updates version for get snapshot API with verbose option to 5.5 2017-05-10 17:38:23 -04:00
Jack Conradson 6ac8a1eb85 Deprecate Fine Grain Settings for Scripts (#24573) 2017-05-10 13:09:31 -07:00
Ali Beyad 743217a430 Enhances get snapshots API to allow retrieving repository index only (#24477)
Currently, the get snapshots API (e.g. /_snapshot/{repositoryName}/_all)
provides information about snapshots in the repository, including the
snapshot state, number of shards snapshotted, failures, etc.  In order
to provide information about each snapshot in the repository, the call
must read the snapshot metadata blob (`snap-{snapshot_uuid}.dat`) for
every snapshot.  In cloud-based repositories, this can be expensive,
both from a cost and performance perspective.  Sometimes, all the user
wants is to retrieve all the names/uuids of each snapshot, and the
indices that went into each snapshot, without any of the other status
information about the snapshot.  This minimal information can be
retrieved from the repository index blob (`index-N`) without needing to
read each snapshot metadata blob.

This commit enhances the get snapshots API with an optional `verbose`
parameter.  If `verbose` is set to false on the request, then the get
snapshots API will only retrieve the minimal information about each
snapshot (the name, uuid, and indices in the snapshot), and only read
this information from the repository index blob, thereby giving users
the option to retrieve the snapshots in a repository in a more
cost-effective and efficient manner.

Closes #24288
2017-05-10 15:48:40 -04:00
Jason Tedor fbf532a626 Revert "Add global checkpoint assertion in index shard"
This reverts commit 89b305c09e.
2017-05-10 14:28:07 -04:00
Lee Hinman d2c69701e6 Fix SocketPermission in test framework for alpha2 bump
It was using the wrong version, which can cause errors like

```
  1> java.security.AccessControlException: access denied ("java.net.SocketPermission" "[0:0:0:0:0:0:0:1]:34221" "connect,resolve")
  1> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:1.8.0_111]
  1> 	at java.security.AccessController.checkPermission(AccessController.java:884) ~[?:1.8.0_111]
  1> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549) ~[?:1.8.0_111]
  1> 	at java.lang.SecurityManager.checkConnect(SecurityManager.java:1051) ~[?:1.8.0_111]
  1> 	at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:625) ~[?:?]
  1> 	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processSessionRequests(DefaultConnectingIOReactor.java:273) ~[httpcore-nio-4.4.5.jar:4.4.5]
  1> 	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:139) ~[httpcore-nio-4.4.5.jar:4.4.5]
  1> 	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348) ~[httpcore-nio-4.4.5.jar:4.4.5]
  1> 	at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:192) ~[httpasyncclient-4.1.2.jar:4.1.2]
1> at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64) ~[httpasyncclient-4.1.2.jar:4.1.2]
```

When running tests
2017-05-10 11:27:57 -06:00
Christoph Büscher ae5919e5df Tests: Fix VersionTests after version bump 2017-05-10 18:18:22 +02:00
Christoph Büscher fbc8345db5 Tests: Fix VersionUtilsTests after version bump 2017-05-10 17:36:12 +02:00
Christoph Büscher 64ecf4f4bf Bumping version to v6.0.0-alpha2 2017-05-10 17:12:56 +02:00
David Roberts d611ab4855 Avoid race when shutting down controller processes (#24579)
This commit terminates any controller processes plugins might have after
the node has been closed.  This gives the plugins a chance to shut down their
controllers gracefully.

Previously there was a race condition where controller processes could be shut
down gracefully and terminated by two threads running in parallel, leading to
non-deterministic outcomes.

Additionally, controller processes that failed to shut down gracefully were
not forcibly terminated when running as a Windows service; there was a reliance
on the plugin to shut down its controller gracefully in this situation.
This commit also fixes this problem.
2017-05-10 14:59:14 +01:00
Clinton Gormley 2486086980 Deprecate the Tribe node
The Tribe node is deprecated in favour of Cross Cluster Search and will be removed in 7.0.
2017-05-10 14:05:12 +02:00
Tanguy Leroux 3e4406f044 [Test] Fix wrong import 2017-05-10 12:31:28 +02:00
Tanguy Leroux 2fe53be0db [Test] Add unit tests for Range aggregations (#24569)
Related to #22278
2017-05-10 12:22:52 +02:00
Martijn van Groningen 0ff5933a55
Rewrite multi search template api to delegate to multi search api instead of to search template api.
The max concurrent searches logic is complex and we shouldn't duplicate that in multi search template api,
so we should template each individual template search request and then delegate to multi search api.
2017-05-10 11:47:53 +02:00
Martijn van Groningen 760e5fce77
Rewrite multi search template api to delegate to multi search api instead of to search template api.
The max concurrent searches logic is complex and we shouldn't duplicate that in multi search template api,
so we should template each individual template search request and then delegate to multi search api.
2017-05-10 11:23:24 +02:00
Isabel Drost-Fromm bd559d96d4
This adds max_concurrent_searches to multi-search-template endpoint.
Closes #20912
2017-05-10 11:23:24 +02:00
Martijn van Groningen 51c74ce547
Added unit tests for InternalMatrixStats.
Also moved InternalAggregationTestCase to test-framework module in order to make use of it from other modules than core.

Relates to #22278
2017-05-10 11:06:18 +02:00
Matt Weber b24326271e Add ICUCollationFieldMapper (#24126)
Adds a new "icu_collation" field type that exposes lucene's
ICUCollationDocValuesField.  ICUCollationDocValuesField is the replacement
for ICUCollationKeyFilter which has been deprecated since Lucene 5.
2017-05-10 10:35:11 +02:00
Ryan Ernst 3f1ef488cd fix checkstyle after script engine rename 2017-05-10 01:04:02 -07:00
Ryan Ernst 9ca7d28552 Scripting: Remove "service" from ScriptEngine interface name (#24574)
This commit renames ScriptEngineService to ScriptEngine.  It is often
confusing because we have the ScriptService, and then
ScriptEngineService implementations, but the latter are not services as
we see in other places in elasticsearch.
2017-05-10 00:47:33 -07:00
Ryan Ernst 5242d3aeb0 Fix ids query builder test to use the same logic for matchnodocs as the
builder itself
2017-05-09 16:53:19 -07:00
Ryan Ernst ebd3e5f73f Scripting: Deprecate file script settings (#24555)
File scripts have 2 related settings: the path of file scripts, and
whether they can be dynamically reloaded. This commit deprecates those
settings.

relates #21798
2017-05-09 16:14:57 -07:00
Tal Levy 945b3cd4ca fix ingest pipeline timestamp timezone format check to be even more generic 2017-05-09 15:31:32 -07:00
Ryan Ernst 53f6d9412e Fix ids query test when none or ALL type is used
See #24460
2017-05-09 14:52:28 -07:00
Tal Levy d2416a3a40 modify ingest timestamp format check to remove timezone 2017-05-09 14:33:43 -07:00