Commit Graph

8897 Commits

Author SHA1 Message Date
Simon Willnauer a302f3a9dd [TEST] Randomize Preference in search requests
Closes #6863
2014-07-18 12:30:53 +02:00
Simon Willnauer 6b519633ab [TEST] Randomize FS Translog implementation
Closes #6658
2014-07-18 11:51:42 +02:00
Simon Willnauer d79717c341 [CLIENT] Unknown node version should be a lower bound
Today when we start a `TransportClient` we use the given transport
addresses and create a `DiscoveryNode` from it without knowing the
actual nodes version. We just use the `Version.CURRENT` which is an
upper bound. Yet, the other node might be a version less than the
currently running and serialisation of the nodes info might break. We
should rather use a lower bound here which is the version of the first
release with the same major version as `Version.CURRENT` since this is
what we officially support.

This commit moves to use the minimum major version or an RC / Snapshot
if the current version is a snapshot.

Closes #6894
2014-07-18 11:31:15 +02:00
Alex Ksikes f22f3db30f Term Vectors API: Computes term vectors on the fly if not stored in the index.
Adds the ability to the Term Vector API to generate term vectors for some
chosen fields, even though they haven't been explicitely stored in the index.

Relates to #5184
Closes #6567
2014-07-17 23:29:05 +02:00
Peter Kim 6a25d9b7b5 [DOCS] Fixed typos 2014-07-17 15:25:34 -04:00
Simon Willnauer 6fc5acd760 Return null info if service is not started
Closes #6906
2014-07-17 16:47:46 +02:00
Simon Willnauer 2bec59ec3e [Bulk] Release semaphore again after close
The bulk processor tries to acquire all leases for the semaphore to wait
for all pending requests. Yet, we should release them afterwards again to
ensure we don't ever deadlock if there is a bug in the processor.
This commit also adds a testcase for this method
2014-07-17 16:24:43 +02:00
matt-preston 42b71a004a [Bulk] Add blocking close method to BulkProcessor
Blocks until all bulk requests have completed.

Closes #4158
Closes #6314
2014-07-17 15:57:45 +02:00
uboness cc8f7ddb9a Added three frequency levels for resource watching
It's now possible to register watchers along with a specified check frequency. There are three frequencies: low, medium, high. Each one is associated with a check interval that determines how frequent the watchers will check for changes and notify listeners if needed. By default, the intervals are 5s, 30s and 60s respectively, but they can also be customized in the settings. also:

  - Added the WatcherHandle construct by which one can stop it (remove it) and resume it (re add it). Also provices access to the watchers itself and the frequency by which it's checked
  - Change the default frequency to 30 seconds interval (used to be 60 seconds). The only watcher that is currently effected by this is the script watcher (now auto-loading scripts will auto-load every 30 seconds if changed)
2014-07-17 14:57:53 +02:00
Boaz Leskes bdbe86dd2d [Store] delete unallocated shards under a cluster state task
This is to prevent a rare racing condition where the very same shard gets allocated to the node after our sanity check that the cluster state didn't check and the actual deletion of the files.

Closes #6902
2014-07-17 14:51:07 +02:00
Alexander Reelsen bb421d7ea3 Revert "Netty: Refactoring to make MessageChannelHandler extensible"
This results in unstable tests, most likely due to Channels being mixed
up by wrongly creating the pipelines. Needs investigation and a test.

This reverts commit db7f0d36af.
2014-07-17 14:49:15 +02:00
Boaz Leskes 38d8e3ccc2 [Infra] remove indicesLifecycle.Listener from IndexingMemoryController
The IndexingMemoryController determines the amount of indexing buffer size and translog buffer size each shard should have. It takes memory from inactive shards (indexing wise) and assigns it to other shards. To do so it needs to know about the addition and closing of shards. The current implementation hooks into the indicesService.indicesLifecycle() mechanism to receive call backs, such shard entered the POST_RECOVERY state. Those call backs are typically run on the thread that actually made the change. A mutex was used to synchronize those callbacks with IndexingMemoryController's background thread, which updates the internal engines memory usage on a regular interval. This introduced a dependency between those threads and the locks of the internal engines hosted on the node. In a *very* rare situation (two tests runs locally) this can cause recovery time outs where two nodes are recovering replicas from each other.

 This commit introduces a a lock free approach that updates the internal data structures during iterations in the background thread.

Closes #6892
2014-07-17 14:31:38 +02:00
Adrien Grand 9714dd55c2 Refactoring: Rename FieldMapper.termsFilter to fieldDataTermsFilter.
FieldMapper has two methods
`Filter termsFilter(List values, @Nullable QueryParseContext)` which is supposed
to work on the inverted index and
`Filter termsFilter(QueryParseContext, List, QueryParseContext)` which is
supposed to work on field data. Let's rename the second one to
`fieldDataTermsFilter` and remove the unused `QueryParseContext`.

Close #6888
2014-07-17 12:42:18 +02:00
Colin Goodheart-Smithe 7223e9959d [TEST] Fixed intermittent failure due to lack of mapping 2014-07-17 09:05:58 +01:00
Martijn van Groningen 9037365adf [TEST] Use array* helper assert methods instead of verifying the array directly 2014-07-17 09:39:51 +02:00
David Pilato 26bac39e0e bin/plugin removes itself
If you call `bin/plugin --remove es-plugin` the plugin got removed but the file `bin/plugin` itself was also deleted.

We now don't allow the following plugin names:

* elasticsearch
* plugin
* elasticsearch.bat
* plugin.bat
* elasticsearch.in.sh
* service.bat

Closes #6745
2014-07-17 08:57:01 +02:00
Alexander Reelsen db7f0d36af Netty: Refactoring to make MessageChannelHandler extensible
Small refactorings to make the MessageChannelHandler more extensible.
Also allowed access to the different netty pipelines

Closes #6889
2014-07-17 08:28:55 +02:00
Boaz Leskes 6fe1d9860e [Scripts] GroovyScriptEngineService don't create ScoreAccess if lookup is null 2014-07-17 08:05:10 +02:00
Ryan Ernst 7139f44241 Scripts: Make ScoreAccessor utility available for other script engines.
With the removal of setNextScore in #6864, script engines must use
the Scorer to find the score of a document.  The DocLookup is updated
appropriately to do this, but most script engines require a Number to be
bound for numeric variables.  Groovy already had an encapsulation for
this funtionality, and this moves it out to be shared with other script
engines.

closes #6898
2014-07-16 16:33:04 -07:00
Ryan Ernst 770447ce1a Scripting: Remove setNextScore in SearchScript.
While it would be nice to do this all the way up the chain (into
score functions), this at least removes the weird dual
setNextScore/setScorer for SearchScripts.

closes #6864
2014-07-16 15:10:01 -07:00
Martijn van Groningen ca7fa4f9ec Query: When parent doc is found stop iterating over segments and continue with the next top matching child doc.
The parent doc has been found and there is no need to check subsequent segments.

Closes #6895
Closes #1843
2014-07-17 00:01:06 +02:00
Martijn van Groningen 473d1717be Search: Added support for named filters/queries inside nested filters and queries.
Closes #6293
2014-07-16 23:43:18 +02:00
Bill Hwang bdf77fec65 [BUILD] Randomization script refactoring
1) Refactored randomization logic to the top
2) Add logics to support window platforms
2014-07-16 14:03:59 -07:00
Alexander Reelsen 70bad405c3 Serialization: Fix bwc issue by falling back to old threadpool serialization
This fixes an issue introduced by the serialization changes in #6486
which are not needed at all. Node that the serialization itself is not broken
but the TransportClient uses its own version on initial connect and getting
the NodeInfos.
2014-07-16 17:58:32 +02:00
Clinton Gormley cccc8867b9 REST: Fixed indexed script/template tests
The clients don't have access to the error message
for matching purposes.  Rewritten tests to work
with the clients
2014-07-16 17:31:31 +02:00
Clinton Gormley 65b85566a0 REST: Renamed indexed_script and indexed_template specs
The file name of the REST specs should be the same as
the endpoint which it documents.
2014-07-16 17:31:26 +02:00
Simon Willnauer 90ea4610c8 [FIELDDATA] Use KeyedLock in IndexFieldDataService
Today we synchronize when updating the IndexFieldDataService
datastructures. This might unnecessarily block progress if multiple
request need different fielddata instance for different fields.

This commit also fixes clear calls to actually consistently clear
the caches in the case of an exception.

Closes #6855
2014-07-16 15:49:17 +02:00
Simon Willnauer e30176cc69 Add read/write lock semantics to KeyedLock 2014-07-16 15:49:00 +02:00
Alexander Reelsen b0c0ff8ac0 Threadpool Info: Allow to serialize negative thread pool sizes
As a SizeValue is used for serializing the thread pool size, a negative number
resulted in throwing an exception when deserializing (using -ea an assertionerror
was thrown).

This fixes a check for changing the serialization logic, so that negative numbers are read correctly, by adding an internal UNBOUNDED value.

Closes #6325
Closes #5357
2014-07-16 15:25:53 +02:00
Simon Willnauer f9a9348508 [DOCS] Move benchmark API to 1.4 2014-07-16 15:02:20 +02:00
Martijn van Groningen 62da59f76f [TEST] Verify if clear cache request went to all shards. 2014-07-16 13:48:39 +02:00
Colin Goodheart-Smithe 8260138e59 Aggregations: Fixed Histogram key_as_string bug
The key as string field in the response for the histogram aggregation will now only show if format is specified on the request.

Closes #6655
2014-07-16 11:36:48 +01:00
Brian Murphy d6cd2c2b73 [DOCS][FIX] Fix reference check in indexed scripts/templates doc. 2014-07-16 11:24:18 +01:00
Brian Murphy bc570919ee [DOCS][FIX] Fix doc parsing, broken closing block 2014-07-16 11:18:21 +01:00
Brian Murphy cbd2a97abd [DOCS] : Indexed scripts/templates
These are the docs for the indexed scripts/templates feature.
Also moved the namespace for the REST endpoints.

Closes #6851
2014-07-16 10:49:02 +01:00
Adrien Grand 7e44fe6017 [TESTS] Stabilize DisabledFieldDataFormatTests by setting number_of_replicas to 0. 2014-07-16 10:37:25 +02:00
Boaz Leskes 84ad29ddfa [Recovery] don't start a gateway recovery if source node is not found
Due to change introduced in #6825, we now start a local gateway recovery for replicas, if the source node can not be found. The recovery then fails because we never recover replicas from disk.

Closes #6879
2014-07-16 10:01:01 +02:00
Boaz Leskes d869163b66 [Infra] re-send failed shard messages when receiving a cluster state that still refers to them
In rare cases we may fail to send a shard failure event to the master, or there is no known master when the shard has failed (ex. a couple of node leave the cluster canceling recoveries and causing a master to step down at the same time). When that happens and a cluster state arrives from the (new) master we should resend the shard failure in order for the master to remove the shard from this node.

Closes #6881
2014-07-16 09:59:49 +02:00
Martijn van Groningen f1c2cdb9c8 Store: Only send shard exists requests if shards exist locally on disk and are not allocated on that node according to the cluster state.
Closes #6870
2014-07-16 08:23:32 +01:00
Ryan Ernst 35037e31b2 [AGGS] Pass current docid being processed to scripts.
Scripts may internally cache based on docid (as expressions do). This
change makes numeric aggregations using scripts pass the docid when
it changes.
2014-07-15 14:40:12 -07:00
Shay Banon 323210729e Releasable XContentBuilder
make the builder releasable (auto closeable), and use it in shards state
also make XContentParser releasable (AutoCloseable) and not closeable since it doesn't throw an IOException
closes #6869
2014-07-16 04:28:03 +09:00
Shay Banon 9345194a65 Share numeric data analyzer instances between mappings
use similar mechanism that shares numeric analyzers for long/double/... for dates as well. This has nice memory save properties with many date fields mapping case, as well as analysis saves (thread local resources)
closes #6843
2014-07-16 04:24:52 +09:00
Ryan Ernst e764c5f08a Fix transient testScore failure by making DF consistent for query. 2014-07-15 11:18:13 -07:00
Brian Murphy 86dfad24fe [FIX] normalize serialization of ScriptType in UpdateRequest 2014-07-15 18:14:48 +01:00
Nik Everett da5fb34163 Mappings: Add transform to document before index.
Closes #6566
2014-07-15 18:40:46 +02:00
mikemccand 63cab559e3 Docs: explain that SerialMergeScheduler just maps to CMS for back compat
Closes #6878
2014-07-15 11:38:43 -04:00
Adrien Grand 56cfa8e47e [TEST] Enforce that the node from ElasticsearchSingleNodeTest is alone in its cluster.
In particular, this removes the ability to create new node and adds the ability
to get a reference to the single node that the cluster is made of.
2014-07-15 17:26:52 +02:00
mikemccand ba443a4819 Test: make test less evil so it doesn't take so long 2014-07-15 11:19:18 -04:00
Simon Willnauer 05d92b9b68 [TEST] Enforce provided settings in test base classes
Enviorment variables might override the tests settings even if
they are explicitly set. Other base classes like InternalTestCluster
also specify `config.ignore_system_properties: true` to ensure `what
we set is what we get`
2014-07-15 17:06:20 +02:00
Ryan Ernst 64ab22816c Scripting: Add script engine for lucene expressions.
These are javascript expressions, which can only access numeric
fielddata, parameters, and _score. They can only be used for searches (not document updates).

closes #6818
2014-07-15 07:49:01 -07:00