Commit Graph

149 Commits

Author SHA1 Message Date
Simon Willnauer 0404db65e3 Merge branch 'master' into convert_some_settings 2016-01-27 11:31:00 +01:00
Simon Willnauer 9cb9bdb736 Merge pull request #16232 from s1monw/use_version_id_in_settings
Use Version#id to parse version from settings.
2016-01-27 11:10:44 +01:00
Jim Ferenczi aea7660e37 Add search_after parameter in the Search API.
The search_after parameter provides a way to efficiently paginate from one page to the next. This parameter accepts an array of sort values, those values are then used by the searcher to sort the top hits from the first document that is greater to the sort values.
This parameter must be used in conjunction with the sort parameter, it must contain exactly the same number of values than the number of fields to sort on.

NOTE: A field with one unique value per document should be used as the last element of the sort specification. Otherwise the sort order for documents that have the same sort values would be undefined. The recommended way is to use the field `_uuid` which is certain to contain one unique value for each document.

Fixes #8192
2016-01-27 09:42:58 +01:00
Simon Willnauer daadf8e04f Convert processors to new settings infrastructure 2016-01-26 17:06:35 +01:00
javanna 9cefeabd9e [TEST] make sure node.mode is used in TransportClientFactory 2016-01-26 15:38:16 +01:00
Simon Willnauer cf358e426c Use Version#id to parse version from settings.
When a Version is passed to `Settings#put(String, Version)` it's id is used as
an integer which should also be used to deserialize it on the consumer end.
Today AssertinLocalTransport expects Version#toString() to be used which can lead
to subtile bugs in tests.
2016-01-26 14:53:06 +01:00
javanna 61630c2b27 migrate node.local and node.mode to new Setting infra 2016-01-26 14:40:46 +01:00
javanna 80b74a4885 migrate node.client, node.data and node.master to new Setting infra 2016-01-26 14:40:13 +01:00
javanna 36d98478bf Merge branch 'master' into feature/ingest 2016-01-25 18:01:09 +01:00
Ryan Ernst b349746bbe Fix repro line for tests to use language tag
See https://issues.apache.org/jira/browse/LUCENE-6978

Also fixed date histogram test to use its version constant as a string
setting.
2016-01-22 22:54:49 -08:00
Boaz Leskes ec31feca93 Settings: migrate network service to the new infra
This commit migrates all the settings under network service to the new settings infra.

It also adds some chaining utils to make fall back settings slightly less verbose.

Breaking (but I think acceptable)  - network.tcp.no_delay and network.tcp.keep_alive used to accept the value `default` which make us not set them at all on netty. Our default was true so we weren't using this feature. I removed it and now we only accept a true boolean.
2016-01-22 16:59:07 +01:00
markharwood d371ef35f4 Settings - change over to o.e.common.settings.Setting for http settings:
http.cors.allow-credentials
http.cors.enabled
http.detailed_errors.enabled
http.enabled
http.pipelining
http.cors.max-age
2016-01-22 15:13:11 +00:00
Simon Willnauer 9b3559f473 Remove `test.cluster.node.seed` and special case `tests.portsfile`
* `test.cluster.node.seed` was only used in one place where it wasn't adding value and was now replaced with a constant
 * `tests.portsfile` is now an official setting and has been renamed to `node.portsfile`

this commit also convert `search.default_keep_alive` and `search.keep_alive_interval` to the new settings infrastrucutre
2016-01-22 15:47:32 +01:00
Yannick Welsch 296b48b9d1 Move discovery.* settings to new setting infrastructure
Closes #16182
2016-01-22 15:35:00 +01:00
Daniel Mitterdorfer 9a1fdb6987 Merge remote-tracking branch 'danielmitterdorfer/convert-path-settings' 2016-01-22 15:18:18 +01:00
Daniel Mitterdorfer e9bb3d31a3 Convert "path.*" and "pidfile" to new settings infra 2016-01-22 15:14:13 +01:00
Jim Ferenczi 9c286b601e Convert "indices.*" settings to new infra. 2016-01-22 13:51:06 +01:00
Simon Willnauer 20c3cbf98d [TEST] only pass valid random settings 2016-01-22 11:42:18 +01:00
Simon Willnauer 1fb2f22f32 Convert to the new settings infra 2016-01-22 11:25:26 +01:00
Ryan Ernst df24019261 Merge pull request #16038 from rjernst/remove_site_plugin
Plugins: Remove site plugins
2016-01-21 12:32:22 -08:00
Martijn van Groningen 44465c94f8 Merge remote-tracking branch 'es/master' into feature/ingest 2016-01-21 11:46:27 +01:00
Simon Willnauer 5374bd8e34 Merge branch 'master' into trash_context_and_headers 2016-01-20 09:40:03 +01:00
Martijn van Groningen 602a0f183e Merge remote-tracking branch 'es/master' into feature/ingest 2016-01-19 22:01:38 +01:00
Simon Willnauer 40783b5d32 delete dead code 2016-01-19 21:06:23 +01:00
Simon Willnauer 3d0cedbabb Merge branch 'master' into trash_context_and_headers 2016-01-19 20:47:43 +01:00
Simon Willnauer eb498282dd Add infrastructure to wrap/filter all clients exposed by the test clusters.
This commit allows an integ test to wrap all clients that are exposed by the
InternalTestCluster which is useful for request interception or to add general headers
to request or to intercept client to server call even if the client calls are done from within
the test framework.
2016-01-19 20:12:47 +01:00
Nik Everett 0d6d77328d Merge branch 'master' into feature/reindex 2016-01-19 13:27:24 -05:00
Simon Willnauer fa0e1488aa Merge branch 'master' into new_index_settings 2016-01-19 10:50:11 +01:00
Jason Tedor 296ab3ec16 Fix transport exceptions for shard state actions
This commit fixes an issue in the handling of TransportExceptions in
ShardStateAction. There were two cases not being handled correctly.
 - when the local node is shutting down, handlers will be notified with
   a TransportException with a message starting "transport stopped"
 - when the remote node disconnects, handlers will be notified with a
   NodeDisconnectedException

In both of these cases, the cause of the exception will be null and this
was incorrectly being handled. The first case can passed to the listener
like any other critical non-channel failure, and the second case can be
handled by modifying the logic for detecting master channel exceptions.

There was a third case of NodeNotConnectedException that was not being
treated as a master channel exception but should be.

This commit adds an integration test that simulates the handling of a
shard failure request during a network partition. By isolating the
master from the cluster while a shard failed request is in flight, this
test simulates that we wait until a new master is elected and then retry
sending that shard failed request to the newly elected master.

This commit adds methods to CapturingTransport to separate local and
remote transport exceptions. The motivation for this change is that
local transport exceptions are delivered to listeners (usually, but not
always) wrapped in SendRequestTransportException while remote transport
exceptions are delivered to listeners wrapped in
RemoteTransportException. By making this distinction clear in the
CapturingTransport, this makes it less likely that tests will make
incorrect assumptions about the exceptions coming out of the transport
layer to listeners.

Closes #16057
2016-01-19 10:43:50 +01:00
Simon Willnauer fbfa9f4925 Merge branch 'master' into new_index_settings 2016-01-19 10:13:48 +01:00
Simon Willnauer b73c737608 apply feedback from @nik9000 2016-01-19 09:54:00 +01:00
Ryan Ernst ef4f0a8699 Test: Make rest test framework accept http directly for the test cluster
The rest test framework, because it used to be tightly integrated with
ESIntegTestCase, currently expects the addresses for the test cluster to
be passed using the transport protocol port. However, it only uses this
to then find the http address.

This change makes ESRestTestCase extend from ESTestCase instead of
ESIntegTestCase, and changes the sysprop used to tests.rest.cluster,
which now takes the http address.

closes #15459
2016-01-18 16:44:14 -08:00
Nik Everett 093600cd63 Cleanup ContextAndHeaderTransportTests
We have two similar tests with the same name, ContextAndHeaderTransportTests.
They shared lots of common code so I extracted much of it into
ActionRecordingPlugin, a plugin which records all action requests for later
inspection.

I also removed all the warnings from both tests. That made lang-mustache
compile cleanly without any custom -Xlint so I removed those. To remove
the warnings I had to add type parameters to ActionFilter which seemed
like a good idea anyway.
2016-01-18 16:14:58 -05:00
Simon Willnauer c2e8ba8970 Register index.version_created on unittest that set it 2016-01-18 15:04:35 +01:00
Simon Willnauer dce29d29c0 Normalize template index settings before validating 2016-01-18 14:39:21 +01:00
Martijn van Groningen 62089539db s/processorId/processorTag 2016-01-18 13:11:46 +01:00
Simon Willnauer 9562fb76bc expose default settings via rest API 2016-01-18 12:48:47 +01:00
Simon Willnauer 7925e2ef84 convert IndexModule settings 2016-01-18 09:23:35 +01:00
Simon Willnauer 79f4697f3e Register MockFSDirectoryService settings 2016-01-18 09:23:34 +01:00
Simon Willnauer a8eedd0457 fix all kinds of crazy test failures 2016-01-18 09:23:34 +01:00
Simon Willnauer 8a4fca2947 make all unittests pass 2016-01-18 09:23:33 +01:00
Simon Willnauer 04045a072f First steps of integrating converted settings into the cluster/index settings infrastructure
The old infa has been removed in this commit such that nothing uses `DynamicSettings` anymore
and all index-scoped settings require to be registered before the node has fully started up.
2016-01-18 09:23:33 +01:00
Simon Willnauer 211f80fcad first cut at integrating new settings infra 2016-01-18 09:23:33 +01:00
Simon Willnauer abfef7aa73 convert index allocation filtering 2016-01-18 09:23:31 +01:00
Britta Weber a9ff6bc405 convert compound_format 2016-01-18 09:23:31 +01:00
Britta Weber c4a019d93d convert translog.flush_threshold_size 2016-01-18 09:23:31 +01:00
Simon Willnauer 2e1bc1ea44 cut over index.routing.rebalance.enable and index.routing.allocation.enable 2016-01-18 09:23:30 +01:00
Simon Willnauer 0bef73269a cut over index.unassigned.node_left.delayed_timeout 2016-01-18 09:23:30 +01:00
Simon Willnauer b7b104a585 convert index.translog.durability 2016-01-18 09:23:29 +01:00
Simon Willnauer da0424c75f Convert index level setting to the new setting infrastrucutre
this is an initial commit of cutting over simple string key based settings
to a more contained scoped settings infrastructure.
2016-01-18 09:23:29 +01:00
Ryan Ernst 3b78267c71 Plugins: Remove site plugins
Site plugins used to be used for things like kibana and marvel, but
there is no longer a need since kibana (and marvel as a kibana plugin)
uses node.js. This change removes site plugins, as well as the flag for
jvm plugins. Now all plugins are jvm plugins.
2016-01-16 22:45:37 -08:00
Tal Levy 9f48df9736 Add on_failure support for verbose _simulate execution and introduce optional processor_tag to Processors 2016-01-15 14:56:20 -08:00
Nik Everett 645b053061 Merge branch 'master' into feature/reindex 2016-01-15 14:04:37 -05:00
javanna 9c06736dbd Merge branch 'master' into feature/ingest 2016-01-15 10:11:56 +01:00
Simon Willnauer aaf7e55c5f Remove index.flush_on_close entirely
This undocumented setting was mainly used for testing and a safety net for
a new flushOnClose feature back in 1.x. We can now safely remove this setting.
The test usage now uses a mock plugin setting to achive the same.
2016-01-14 10:34:58 +01:00
Simon Willnauer 4526f117a2 Randomize which MockPlugins are used for IT 2016-01-13 23:00:59 +01:00
Simon Willnauer 437d7c179a Move all dynamic settings and their config classes to the index level
Today we maintain a lot of settings on the shard level which are all index level settings.
In order to cut over to the new settings API where we register update listener we have to move
all of them on to the index level otherwise we need a way to un-register listeners which is error-prone
and requires additional handling when shards are closed. It's simpler and also more accurate to handle all of
them on the index level where we can trash the entire registry for update listener once the index goes out of scope.
2016-01-13 14:25:37 +01:00
Simon Willnauer 574d1b35b3 Replace ContextAndHeaders with a ThreadPool based ThreadLocal implementation
ContextAndHeaders has a massive impact on the core infrastructure since it has to
be manually passed on to all relevant places across threads/network calls etc. For the same reason
it's also very error prone and easily forgotten on potentially relevant APIs.

The new ThreadContext is associated with a ThreadPool (node or transport client) and ensures that
headers and context registered on a current thread are inherited to new threads spawned, send across
the network to be deserialized on the receiver end as well as restored on the response handling thread
once the response is received.
2016-01-13 11:53:32 +01:00
javanna ea8065aa3d Merge branch 'master' into feature/ingest 2016-01-12 18:28:42 +01:00
Nik Everett 78dc1c32de Merge pull request #15899 from nik9000/xlint_serial
Ban Serializable
2016-01-12 10:04:50 -05:00
Jason Tedor ed723fbe5a Merge pull request #15929 from jasontedor/capturing-transport-clear
Add convenience method for capturing and clearing requests

Closes #15897
2016-01-12 09:44:23 -05:00
Jason Tedor c19981c67b Add convenience method for capturing and clearing requests
This commit adds convenience methods to o.e.t.t.CapturingTransport
that enables capturing requests and clearing the captured requests
with a single method. This is to simplify a common pattern in tests of
capturing requests, and then clearing the captured requests.
2016-01-12 09:20:54 -05:00
Martijn van Groningen 7bdd2583aa Merge remote-tracking branch 'es/master' into feature/ingest 2016-01-12 01:01:30 +01:00
Nik Everett 01ce49e94e Ban Serializable
1. Uses forbidden patterns to prevent things from referencing
java.io.Serializable or from mentioning serialVersionUID.
2. Uses -Xlint:-serial so we don't have to hear from javac that we aren't
declaring serialVersionUID on any classes that we make that happen to extend
Serializable.
3. Remove Serializable and serialVersionUID declarations.

I didn't use forbidden apis because it doesn't look like it has a way to ban
explicitly implementing Serializable. If you try to ban Serializable with
forbidden apis you end up banning all Exceptions and all Strings.

Closes #15847
2016-01-11 16:57:31 -05:00
Nik Everett b29c416b7c Merge pull request #15848 from nik9000/xlint3
Remove -Xlint:-deprecation from all but core
2016-01-11 16:55:13 -05:00
Jason Tedor abaf816d00 Remove and forbid use of IndexWriter#isLocked
This commit removes and now forbids use of
org.apache.lucene.index.IndexWriter#isLocked as this method was
deprecated in LUCENE-6508. The deprecation is due to the fact that
checking if a lock is held before acquiring that lock is subject to a
time-of-check-to-time-of-use race condition. There were three uses of
IndexWriter#isLocked in the code base:
 - a logging statement in o.e.i.e.InternalEngine where we are already in
   an exceptional condition that the lock was held; in this case,
   logging whether or not the directory is locked is superfluous
 - in o.e.c.l.u.VersionsTests where we were verifying that a write lock
   is released upon closing an IndexWriter; in this case, the check is
   not needed as successfully closing an IndexWriter releases its
   write lock
 - in o.e.t.s.MockFSDirectoryService where we were verifying that a
   directory is not write-locked before (implicitly) trying to obtain
   such a write lock in org.apache.lucene.index.CheckIndex#<init> (this
   is the exact type of a situation that is subject to a race
   condition); in this case we can proceed by just (implicitly) trying
   to obtain the write lock and failing if we encounter a
   LockObtainFailedException
2016-01-10 08:28:44 -05:00
Nik Everett 00edc8bc87 Remove -Xlint:-deprecated from test/framwork
There were a few uses of deprecation I couldn't clear up quickly so I filed
issues for them and suppressed the warnings.
2016-01-07 21:22:20 -05:00
javanna 1ea690e814 Merge branch 'feature/ingest' into enhancement/move_to_core 2016-01-07 15:13:31 +01:00
Nik Everett 20e7fa97db Remove Xlint:-override,-fallthrough,-static
Adds `@SuppressWarnings("fallthrough")` in two places where the fallthrough
is used to implement well known hashing algorithms.
2016-01-06 22:27:14 -05:00
javanna 9079a7e891 wip: move all the ingest infra to core 2016-01-06 19:10:44 +01:00
javanna 456351de2e move common test classes to test module 2016-01-06 17:31:39 +01:00
Martijn van Groningen 2d6adf6428 Percolator refactoring:
* Added percolator field mapper that extracts the query terms and indexes these terms with the percolator query.
* At percolate time these extracted terms are used to query percolator queries that are like to be evaluated. This can significantly cut down the time it takes to percolate. Whereas before all percolator queries were evaluated if they matches with the document being percolated.
* Changes made to percolator queries are no longer immediately visible, a refresh needs to happen before the changes are visible.
* By default the percolate api only returns upto 10 matches instead of returning all matching percolator queries.
* Made percolate more modular, so that it is easier to add unit tests.
* Added unit tests for the percolator.

Closes #12664
Closes #13646
2016-01-06 16:08:10 +01:00
Igor Motov a89dba27c2 Task Management: Add framework for registering and communicating with tasks
Adds task manager class and enables all activities to register with the task manager. Currently, the immutable Transport*Activity class represents activity itself shared across all requests. This PR adds and an additional structure Task that keeps track of currently running requests and can be used to communicate with these requests using TransportTaskAction.

Related to #15117
2016-01-05 12:24:43 -05:00
Jason Tedor 419f3976c2 Fix NPE in TestClusterService when waiting indefinitely
When waiting indefinitely for a new cluster state in a test,
TestClusterService#add will throw a NullPointerException if the timeout
is null. Instead, TestClusterService#add should guard against a null
timeout and not even attempt to add a notification for the timeout
expiring. Note that the usage of null is the agreed upon contract for
specifying an indefinite wait from ClusterStateObserver.
2016-01-05 09:52:29 -05:00
Robert Muir 25914ae879 Merge pull request #15688 from rmuir/thirdPartyAudit3
Improve thirdPartyAudit check, round 3
2015-12-29 09:24:51 -05:00
Simon Willnauer 39cec9f2ff [TEST] Improve test speed
DedicatedClusterSnapshotRestoreIT#testRestoreIndexWithMissingShards took ~1.5 min to finish
due to timeouts that are applied if not all shards are allocated. Now that the index that has
unallocated shareds is not refreshed the test is more reasonable and runs in 15 sec
2015-12-29 14:53:15 +01:00
Robert Muir 180ab2493e Improve thirdPartyAudit check, round 3 2015-12-28 22:38:55 -05:00
Simon Willnauer a20b07d581 Merge branch 'master' into trash_recovery_threads 2015-12-28 12:45:14 +01:00
Robert Muir d144ba24a5 Merge pull request #15588 from rmuir/hdfs2-only
merge current hdfs improvements to master
2015-12-23 18:17:22 -05:00
Adrien Grand d8d8666877 Remove `index_name` back compat.
Since 2.0 we enforce that fields have the same full and index names. So in 3.x
we can remove the ability to have different names on the same field.
2015-12-23 14:55:26 +01:00
Simon Willnauer f5e4cd4616 Remove recovery threadpools and throttle outgoing recoveries on the master
Today we throttle recoveries only for incoming recoveries. Nodes that have a lot
of primaries can get overloaded due to too many recoveries. To still keep that at bay
we limit the number of threads that are sending files to the target to overcome this problem.

The right solution here is to also throttle the outgoing recoveries that are today unbounded on
the master and don't start the recovery until we have enough resources on both source and target nodes.

The concurrency aspects of the recovery source also added a lot of complexity and additional threadpools
that are hard to configure. This commit removes the concurrent streamns notion completely and sends files
in the thread that drives the recovery simplifying the recovery code considerably.
Outgoing recoveries are not throttled on the master via a allocation decider.
2015-12-22 14:59:43 +01:00
Robert Muir 010d1a89c5 Merge branch 'master' into hdfs2-only 2015-12-22 00:40:54 -05:00
Ryan Ernst af7d6b629c Change hdfs unit tests to be a single node test instead of integ test 2015-12-21 18:32:28 -08:00
Simon Willnauer 87494dd7db fix spelling and remove wildcard imports 2015-12-21 22:06:51 +01:00
Simon Willnauer c3f901400c Move async translog sync logic into IndexService
Today the logic to async - commit the translog is in every translog instance
itself. While the setting is a per index setting we manageing it per shard. This
polluts the translog code and can more easily be managed in IndexService.
2015-12-21 21:36:02 +01:00
Simon Willnauer fcfd98e9e8 Drop support for simple translog and hard-wire buffer to 8kb
Today we have two variants of translogs for indexing. We only recommend the buffered
one which also has a 20% advantage in indexing speed. This commit removes the option and defaults
to the buffered case. It also hard-wires the translog buffer to 8kb instead of 64kb. We used to
adjust that buffer based on if the shard is active or not, this code has also been removed and
instead we just keep an 8kb buffer arround.
2015-12-21 16:44:35 +01:00
Simon Willnauer afc1cc19af Simplify translog-based flush settings
This commit removes `index.translog.flush_threshold_ops` and `index.translog.disable_flush`
in favor of `index.translog.flush_threshold_size`. The number of operations is meaningless by itself and
can easily be turned into a size value with knowledge of the data. Disabling the flush is only useful in
tests and we can set the size value to a really high value. If users really need to do this they can
also apply a very high value like `1PB`.
2015-12-21 15:15:00 +01:00
Robert Muir f67390e0c8 in the plugin: guard against HADOOP_HOME in environment on any platform.
hdfs fixture: minihdfs works on windows now, if things are properly set
but our test fixture still cannot launch this on windows.
2015-12-21 02:21:53 -05:00
Robert Muir e93c491dbe simplify hdfs fixture 2015-12-20 23:50:27 -05:00
Robert Muir 99f2cde225 Fail fast if HDFS cluster shuts itself down 2015-12-20 22:30:41 -05:00
Robert Muir f4f8b6e3fe Merge branch 'master' of github.com:elastic/elasticsearch into hdfs2-only 2015-12-20 21:59:02 -05:00
Robert Muir 08d1d2f192 add simple javaexec for now until test fixtures can be debugged when they do not start 2015-12-20 19:04:28 -05:00
Robert Muir 12a8428dfb Add MiniHDFS test fixture, started before integTest and shut down after.
Currently uses a hardcoded port (9999), need to apply MavenFilteringHack after it starts.
2015-12-20 16:00:37 -05:00
Simon Willnauer a9e9dc7d88 Disable tests for example-fixture 2015-12-20 15:13:14 +01:00
Ryan Ernst 9cb4c82c58 Build: Add fixture capabilities to integ tests
This change adds a Fixture class for use by gradle. A Fixture is an
external process that integration tests will use. It can be added as a
dependsOn for integTest, and will automatically be shutdown upon success
or failure, as well as relevant information dumped on failure. There is
also an example fixture in this change.
2015-12-19 15:46:21 -08:00
Ryan Ernst 4ea19995cf Remove wildcard imports 2015-12-18 12:43:47 -08:00
Areek Zillur 9d9b557cea Don't allow nodes with missing custom data types to join cluster 2015-12-18 11:33:29 -05:00
Simon Willnauer eca2435838 Merge branch 'master' into settings_prototype 2015-12-18 09:15:58 +01:00
Ryan Ernst 6a99796b02 Build: Move test framework under a "test" top level dir
This allows adding more test projects, eg integ test fixtures that will
be coming soon.
2015-12-17 21:25:06 -08:00