Commit Graph

3286 Commits

Author SHA1 Message Date
Nik Everett b7cb04cbf4 Merge pull request #15809 from nik9000/unraw
Remove some raw from ActionRequest subclasses
2016-01-13 09:43:57 -05:00
Nik Everett df44754605 Renames Self to Request in ActionRequests 2016-01-13 09:40:32 -05:00
Simon Willnauer 437d7c179a Move all dynamic settings and their config classes to the index level
Today we maintain a lot of settings on the shard level which are all index level settings.
In order to cut over to the new settings API where we register update listener we have to move
all of them on to the index level otherwise we need a way to un-register listeners which is error-prone
and requires additional handling when shards are closed. It's simpler and also more accurate to handle all of
them on the index level where we can trash the entire registry for update listener once the index goes out of scope.
2016-01-13 14:25:37 +01:00
Britta Weber 2c2264d8d0 Avoid circular reference in exception
Don't set the suppressed Exception in Translog.closeOnTragicEvent(Exception ex) if it is an
AlreadyClosedException. ACE is thrown by the TranslogWriter and as cause might
contain the Exception that we add the suppressed ACE to. We then end up with a
circular reference where Exception A has a suppressed Exception B that has as cause A.
This would cause a stackoverflow when we try to serialize it.
For a more detailed description see #15941

closes #15941
2016-01-13 13:56:04 +01:00
Simon Willnauer 4d152b1474 Merge branch 'master' into move_refresh_into_index_service 2016-01-13 11:52:24 +01:00
Simon Willnauer 91289b8560 Merge branch 'master' into move_refresh_into_index_service 2016-01-13 11:41:30 +01:00
Michael McCandless 148557a61c remove the settings_require_units setting (effectively hardwire it to true) 2016-01-13 04:46:18 -05:00
Simon Willnauer 2c978941f5 fix test to wipe lenient index first 2016-01-13 10:03:20 +01:00
Martijn van Groningen a2796b555f mappings: Search filter should wrap the types filters in a separate boolean as should clauses
So that a document must either match with one of the types and the non nested clause.

Closes #15757
2016-01-13 09:44:01 +01:00
Simon Willnauer 050afe9171 add yet another test 2016-01-12 21:17:14 +01:00
Simon Willnauer 5204440471 add more tests and apply feedback from @mikemccand 2016-01-12 21:09:26 +01:00
Michael McCandless ce32b959fd fix possible concurrency bug in IMC when indexing threads are faster in writing bytes than the status checker is in checking all shards 2016-01-12 14:40:47 -05:00
Christoph Büscher d7d3e5d441 Fixing javadoc problem 2016-01-12 20:07:20 +01:00
Christoph Büscher ab456f6174 Merge pull request #15551 from cbuescher/shapes-add-validation
Geo: Add validation of shapes to ShapeBuilders
2016-01-12 19:15:26 +01:00
Christoph Büscher b9af22ccdc Merge pull request #15379 from cbuescher/fix-geoDistanceQueryBuilderTest
Geo: Fix small error in distance normalization in test
2016-01-12 19:02:04 +01:00
Jason Tedor 2839e351af Add load averages to OS stats on FreeBSD
This commit adds load averages to the OS stats on FreeBSD. For these
stats to be available, linprocfs must be available and mounted at
/compat/linux/proc.
2016-01-12 12:08:29 -05:00
Simon Willnauer 59211927b6 Move RefreshTask into IndexService and use since task per index
`refresh_interval` is a per index setting but we interpret and maintain it per shard. This
change moves the refresh task outside of IndexShard to the IndexService where it logically belongs
and reuses scheduling infrastructure used for translog fsync (async commit).

This change will use the same task for all shards of an index while previously we used on thread/task
per shard to refresh. This will also prevent too many concurrent refreshes if there are many indices and
shards allocated on a single node.
2016-01-12 17:46:40 +01:00
Ali Beyad 7846e5ae99 Merge pull request #15904 from abeyad/fix_15822
Throw exception if content type could not be determined in Update API
2016-01-12 11:13:54 -05:00
Nik Everett da63c87c7f Merge pull request #15793 from nik9000/highlight_typename
Don't override originalQuery with request filters
2016-01-12 11:08:31 -05:00
Nik Everett d3a4a9f871 Don't override originalQuery with request filters
These filters leak into highlighting and probably other places and cause
things like the type name to be highlighted when using
requireFieldMatch=false. We could have special hacks to keep them out of
highlighting but it feals better to keep them out of any variable named
"originalQuery".

Closes #15689
2016-01-12 11:03:15 -05:00
Nik Everett 78dc1c32de Merge pull request #15899 from nik9000/xlint_serial
Ban Serializable
2016-01-12 10:04:50 -05:00
Nik Everett a8ceb37d07 Merge pull request #15913 from nik9000/deprecate_collect_payloads
Deprecate span_near's collect_payloads
2016-01-12 09:56:19 -05:00
Nik Everett db3447fb03 Properly deprecate the field 2016-01-12 09:55:52 -05:00
Nik Everett 5afcb4305e Merge pull request #15911 from nik9000/dismax_constructor
Stop using deprecated constructors for queries
2016-01-12 09:47:33 -05:00
Jason Tedor ed723fbe5a Merge pull request #15929 from jasontedor/capturing-transport-clear
Add convenience method for capturing and clearing requests

Closes #15897
2016-01-12 09:44:23 -05:00
Jason Tedor c19981c67b Add convenience method for capturing and clearing requests
This commit adds convenience methods to o.e.t.t.CapturingTransport
that enables capturing requests and clearing the captured requests
with a single method. This is to simplify a common pattern in tests of
capturing requests, and then clearing the captured requests.
2016-01-12 09:20:54 -05:00
Christoph Büscher ff2593e74a Rename PointCollectionBuilder to CoordinatesBuilder
Also renaming internal methods to reflect that they are dealing with
jts coordinates. Also renamed the list() to build() method for creating
the coordinates lists and adding constructors to PolygonBuilder that
take CoordinatesBuilders and implicitely call build() on them.
2016-01-12 13:30:35 +01:00
Michael McCandless b4a095d430 Merge pull request #14121 from mikemccand/fair_indexing_buffers
The indexing buffer on a node (default: 10% of the JVM heap) is now a "shared pool" across all shards on that node.  This way, shards doing intense indexing can use much more than other shards doing only light indexing, and only once the sum of all indexing buffers across all shards exceeds the node's indexing buffer will we ask shards to move recently indexed documents to segments on disk.
2016-01-12 04:59:10 -05:00
Jason Tedor d9fd6e2fe3 Merge pull request #15907 from jasontedor/load-average
Reintroduce five-minute and fifteen-minute load averages on Linux

Relates #12049, relates #14741
2016-01-11 23:43:40 -05:00
Jason Tedor 1de2081ed3 Reintroduce five-minute and fifteen-minute load averages on Linux
This commit reintroduces the five-minute and fifteen-minute load stats
on Linux, and changes the format of the load_average field back to an
array.
2016-01-11 23:42:47 -05:00
Nik Everett ae0bf1b32f Deprecate span_near's collect_payloads 2016-01-11 22:11:38 -05:00
Nik Everett 5ded9ac9eb Stop using deprecated constructors for queries
DisjunctionMaxQuery and BooleanQuery
2016-01-11 21:36:16 -05:00
Ali Beyad 42134cca4d Fixes an issue where, if the content type of the request body could not be
determined, the UpdateRequest would still try to parse the content instead
of throwing the standard ElasticsearchParseException.  This manifests when
passing illegal JSON in the request body that does not begin with a '{'.
By trying to parse the content from an unknown request body content type,
the UpdateRequest was throwing a null pointer exception.  This has been
fixed to throw an ElasticsearchParseException, to be consistent with the
behavior of all other requests in the face of undecipherable request
content types.

Closes #15822
2016-01-11 17:43:36 -05:00
Michael McCandless 07e8370322 feedback 2016-01-11 17:16:28 -05:00
Nik Everett 01ce49e94e Ban Serializable
1. Uses forbidden patterns to prevent things from referencing
java.io.Serializable or from mentioning serialVersionUID.
2. Uses -Xlint:-serial so we don't have to hear from javac that we aren't
declaring serialVersionUID on any classes that we make that happen to extend
Serializable.
3. Remove Serializable and serialVersionUID declarations.

I didn't use forbidden apis because it doesn't look like it has a way to ban
explicitly implementing Serializable. If you try to ban Serializable with
forbidden apis you end up banning all Exceptions and all Strings.

Closes #15847
2016-01-11 16:57:31 -05:00
Nik Everett dfe77879b9 Merge pull request #15850 from nik9000/more_deprecation
Handle some deprecation warnings
2016-01-11 14:23:48 -05:00
Christoph Büscher 57c579e7b7 Geo: Add validation of shapes to shape builders
So far the validation of geo shapes was only taking place in the
parse methods in ShapeBuilder. With the recent refactoring we no
longer can rely on shapes being parsed from json, so the same kind
of validation should take place when just using the java api.

A lot of validation concerns the number of points a shape needs to
have in order to be valid. Since this is not possible with current
builders where points can be added one by one, the builder constructors
are changed to require the mandatory parameters and validate those
already at construction time. To help with constructing longer lists
of points, a new utility PointsListBuilder is instroduces which can
produce list of coordinates accepted by most of the other shape builder
constructors.

Also adding tests for invalid shape exceptions to the already existing
shape builder tests.
2016-01-11 19:05:27 +01:00
Nik Everett d8af49eb91 Revert "Fix blended terms for non-strings"
This reverts commit 6bb01984b6.

It caused test failures.
2016-01-11 10:05:48 -05:00
Christoph Büscher bd1d935c26 Fix small error in distance normalization in test 2016-01-11 14:35:58 +01:00
Nik Everett c2259a2edd Merge pull request #15869 from nik9000/cross_fields_numeric
Fix blended terms for non-strings
2016-01-11 08:19:14 -05:00
Martijn van Groningen ace1b33c3c test: simplify percolator recovery test by only testing what is needed
Running requests via the percolate or mpercolate api is irrelevant.
What is relevant is that when nodes come back that they report the expected number of matches.
2016-01-11 13:38:50 +01:00
Simon Willnauer d32d16f117 Check lenient_expand_open after aliases have been resolved
We fail today with ClusterBlockExceptions if an alias expands to a closed index
during search since we miss to check the index option down the road after we expanded
aliases.

Closes #13278
2016-01-11 13:31:14 +01:00
Michael McCandless 5e7144f5c5 IMC is now just another IndexingOperationListener 2016-01-11 06:07:19 -05:00
Michael McCandless f3de7783d2 merge master 2016-01-11 05:36:16 -05:00
Michael McCandless db832cce06 improve logging messages a bit 2016-01-11 04:52:26 -05:00
Adrien Grand 77dbfbc988 Merge pull request #15864 from jpountz/fix/initial_sizing_of_BytesStreamOutput
Fix initial sizing of BytesStreamOutput.
2016-01-11 09:15:09 +01:00
Simon Willnauer 2f5e2c09c8 add @Override annotation 2016-01-10 21:09:26 +01:00
Simon Willnauer 91e8d156f5 Make IndexingMemoryController private to IndicesService
This commit detaches the IndexingMemoryController from guice and moves
it's creation and closing into IndicesService where it logically belongs.
2016-01-10 20:51:31 +01:00
Simon Willnauer 37f6b5ced7 Merge pull request #15875 from s1monw/cleanup_indexing_operation_listeners
Cleanup IndexingOperationListeners infrastructure
2016-01-10 20:22:42 +01:00
Simon Willnauer 80ef201514 Remove dead code and add missing @Override annotations 2016-01-10 20:04:25 +01:00
Jason Tedor abaf816d00 Remove and forbid use of IndexWriter#isLocked
This commit removes and now forbids use of
org.apache.lucene.index.IndexWriter#isLocked as this method was
deprecated in LUCENE-6508. The deprecation is due to the fact that
checking if a lock is held before acquiring that lock is subject to a
time-of-check-to-time-of-use race condition. There were three uses of
IndexWriter#isLocked in the code base:
 - a logging statement in o.e.i.e.InternalEngine where we are already in
   an exceptional condition that the lock was held; in this case,
   logging whether or not the directory is locked is superfluous
 - in o.e.c.l.u.VersionsTests where we were verifying that a write lock
   is released upon closing an IndexWriter; in this case, the check is
   not needed as successfully closing an IndexWriter releases its
   write lock
 - in o.e.t.s.MockFSDirectoryService where we were verifying that a
   directory is not write-locked before (implicitly) trying to obtain
   such a write lock in org.apache.lucene.index.CheckIndex#<init> (this
   is the exact type of a situation that is subject to a race
   condition); in this case we can proceed by just (implicitly) trying
   to obtain the write lock and failing if we encounter a
   LockObtainFailedException
2016-01-10 08:28:44 -05:00
Simon Willnauer 54d1e35d84 Cleanup IndexingOperationListeners infrastructure
This commit reduces the former ShardIndexinService to a simple stats/metrics
class, moves IndexingSlowLog to the IndexService level since it can be shared
across shards of an index and is now hidden behind IndexingOperationListener.

IndexingOperationListener is now a first class citizen in IndexShard and is passed
in from IndexService.
2016-01-09 21:50:17 +01:00
Jason Tedor 37880040a2 Remove unused imports from o/e/i/e/InternalEngine.java 2016-01-09 12:16:09 -05:00
Nik Everett 4772b1fe9d Remove tests for backwards compatibility we don't support any more 2016-01-08 17:25:26 -05:00
Nik Everett d8526f4d00 Handle some deprecation warnings
Suppress lots of them with comments. Fix a few. Removes lots of pre-built
stuff used to support pre-2.0 indexes.
2016-01-08 17:17:33 -05:00
Nik Everett 6bb01984b6 Fix blended terms for non-strings
It had some funky errors, like lenient:true not working and queries with
two integer fields blowing up if there was no analyzer defined on the
query. This throws a bunch more tests at it and rejiggers how non-strings
are handled so they don't wander off into scary QueryBuilder-land unless
they have a nice strong analyzer to protect them.

Closes #15860
2016-01-08 16:53:40 -05:00
Michael McCandless d3d41fc22b Merge pull request #15585 from mikemccand/max_indexing_thread_pool_size
Limit the max size of bulk and index thread pools to bounded number of processors
2016-01-08 14:36:21 -05:00
Adrien Grand 8568480a74 Fix initial sizing of BytesStreamOutput.
It currently tries to align to the page size (16KB) by default. However, this
might waste a significant memory (if many BytesStreamOutputs are allocated)
and is also useless given that BytesStreamOutput does not recycle (on the
contrary to ReleasableBytesStreamOutput). So the initial size has been changed
to 0.

Closes #15789
2016-01-08 20:08:31 +01:00
Jason Tedor 871d1b4885 Remove and forbid use of j.u.c.ThreadLocalRandom
This commit removes and now forbids all uses of
java.util.concurrent.ThreadLocalRandom across the codebase. The
underlying issue with ThreadLocalRandom is that it can not be
seeded. This means that if ThreadLocalRandom is used in production code,
then tests that cover any code path containing ThreadLocalRandom will be
prevented from being reproducible by use of ThreadLocalRandom. Instead,
using org.elasticsearch.common.random.Randomness#get will give
reproducible sources of random when running under tests and otherwise
still give an instance of ThreadLocalRandom when running as production
code.
2016-01-08 12:23:48 -05:00
Jason Tedor 21f5b0ff35 Remove dead o.e.c.m.UnboxedMathUtils
This commit removes the dead UnboxedMathUtils from the codebase.
2016-01-08 11:58:39 -05:00
Simon Willnauer 71796e2319 [TEST] Close failable translog in a controlled way otherwise assertions are off in the test 2016-01-08 13:10:09 +01:00
Adrien Grand 581fd49dac Merge pull request #15836 from jpountz/fix/bitset_cache_duplicates
BitSetFilterCache duplicates its content.
2016-01-08 09:59:28 +01:00
Igor Motov 8fbb3686cd Improve stability of the testBatchingShardUpdateTask test
On slow machines when this test randomly picks a large number of shards it can occasionally take more than 32.5 seconds to snapshot all shards. That is causing the test to miss the second to last assert in awaitsBusy at 32.5 seconds and then timeout in BlockingClusterStateListener at 60 seconds. Due to the timeout, the pending task queue is cleaned before the last awaitsBusy assert at 65 seconds and as a result the last assert runs on a completely empty queue and fails with a very confusing assert error.

This commit makes the timeout in BlockingClusterStateListener to occur after the last assert in assertBusyPendingTasks and therefore allows assertBusyPendingTasks to perform the last assert before cleaning the pending tasks queue takes place.

 This commit also reduces the maximum number of shards used in the test to 10 in order to speed up this test.
2016-01-07 19:33:50 -05:00
Adrien Grand 3ef9ec25f8 BitSetFilterCache duplicates its content.
We have a bug that makes all per-index bitset caches store bitsets for all
indices. In the case that you have many indices, which is fairly common with
time-based data, this could translate to a lot of wasted memory.

Closes #15820
2016-01-07 18:50:14 +01:00
Britta Weber f93b4cb215 sync translog to disk after recovery from primary
Otherwise if that node is shutdown and restarted it might will have lost all operations
that were in the translog.
2016-01-07 16:27:40 +01:00
Adrien Grand 8bd54dbf5a Merge pull request #15828 from jpountz/enhancement/stricter_metadata_parsing
Make MetaData parsing less lenient.
2016-01-07 15:20:51 +01:00
Adrien Grand 6ce7a972bc Make MetaData parsing less lenient.
Today this simply ignores everything that is not recognized.
2016-01-07 15:20:16 +01:00
Nik Everett 52f28888d5 Merge pull request #15813 from nik9000/xlint1
Remove Xlint:-override,-fallthrough,-static
2016-01-07 08:34:40 -05:00
Boaz Leskes d5e6eb58a8 Log uncaught exceptions from scheduled once tasks
`ScheduledThreadPoolExecutor` allows you to schedule tasks to run once or periodically at the future. If such a task throws an exception, that exception is caught and reported in the future that `ScheduledThreadPoolExecutor#schedule` returns. However, we typically do not capture the future / do not test it for errors. This results in exception being swallowed and not reported. To mitigate this we now wrap any command in a LoggingRunnable  (already used for periodic tasks).  Also, RunnableCommand is changed not to swallow exception but percolate them further for reporting by the future.

Closes #15824
2016-01-07 14:04:35 +01:00
Simon Willnauer e7f9d685f1 [TEST] Test that translog can recover after random IOException
This commit adds a new test that can throw an IOException at any point in time
and ensures that all previously synced documents can be successfully recovered after hitting
an excepiton.

Relates to #15788
2016-01-07 10:17:31 +01:00
Adrien Grand 67d233cecd Remove warmers and the warmer API.
Warmers are now barely useful and will be removed in 3.0. Note that this only
removes the warmer API and query-based warmers. We still have warmers internally
for eg. global ordinals.

Close #15607
2016-01-07 09:57:07 +01:00
Martijn van Groningen 604d59a95e muted test 2016-01-07 09:54:59 +01:00
Nik Everett 20e7fa97db Remove Xlint:-override,-fallthrough,-static
Adds `@SuppressWarnings("fallthrough")` in two places where the fallthrough
is used to implement well known hashing algorithms.
2016-01-06 22:27:14 -05:00
Nik Everett 74c132afc6 Standardize some methods on varargs
Right now we define the same sort of methods as taking String arrays and
string varargs. We should standardize on one and varargs is easier to
call so lets use varargs!
2016-01-06 21:01:58 -05:00
Jason Tedor a583edb2df Merge pull request #15801 from jasontedor/cyclic-barriers-for-boaz
Use CyclicBarriers for sychronizing driver and test threads
2016-01-06 20:09:28 -05:00
Jason Tedor c147fe5691 Do not lose CacheTest failure stack traces 2016-01-06 20:00:11 -05:00
Nik Everett 91464c7687 Remove some raw from ActionRequest subclasses
Renames "T" used in many subclasses of ActionRequest to "Self" and tightens
the type bounds.
2016-01-06 19:01:58 -05:00
Nik Everett d54f1a8f20 Merge pull request #15796 from nik9000/boundary_chars
Add test for boundary chars
2016-01-06 18:26:38 -05:00
Nik Everett 9935ae921e Version.LATEST instead of Lucene.VERSION
There was a TODO for it.
2016-01-06 17:36:10 -05:00
Jason Tedor 4c0f5bda47 Use CyclicBarriers for sychronizing driver and test threads
This commit modifies some tests to use CyclicBarriers to correctly and
simply sychronize driver and test threads.
2016-01-06 15:07:05 -05:00
Jason Tedor 22abf14812 Visible failures in cluster state update task execution ordering test 2016-01-06 14:43:24 -05:00
Jason Tedor 557b11cc2b Sychronize threads in cluster state update task execution ordering test
This commit uses a CyclicBarrier to correctly and simply sychronize the
driver and test threads in
ClusterServiceIT#testClusterStateUpdateTasksAreExecutedInOrder.
2016-01-06 14:41:43 -05:00
Jason Tedor d1b4cf6778 Further simplify cluster state update task execution ordering test 2016-01-06 14:41:42 -05:00
Jason Tedor 18b42ce798 Simplify cluster state task execution ordering test 2016-01-06 14:41:42 -05:00
Jason Tedor 270b08b302 Add test that cluster state update tasks are executed in order
This commit adds a test that ensures that cluster state update tasks
are executed in order from the perspective of a single thread.
2016-01-06 14:41:38 -05:00
Jason Tedor ef16113697 Merge pull request #15735 from jasontedor/master-node-change-predicate
Refactor master node change predicate for reuse
2016-01-06 13:58:13 -05:00
Nik Everett add60a7560 [highlighting] Another test for boundary chars 2016-01-06 13:42:15 -05:00
Nicholas Knize 7df9ba6053 [TEST] Speed up GeoShapeQueryTests
This commit speeds up GeoShapeQueryTests by reducing the size of the random generated shapes and defaulting geo_shape indexes to use quadtree (more efficient for shapes) over geohash.
2016-01-06 12:41:04 -06:00
Martijn van Groningen 04b79c112f test: unmuted test
test failed, because now the percolator returns upto 10 matches whereas before this was unbounded. The test has been updated to take this in account by checking the total count instead of the number of matches
2016-01-06 19:10:55 +01:00
Jason Tedor 3b192cfc74 Merge pull request #15791 from jasontedor/relocating-shard-failure
Only fail the relocation target when a replication request on it fails

Closes #15790
2016-01-06 12:56:49 -05:00
Jason Tedor bb4d857e44 Redundant assertion in TransportReplicationActionTests#runReplicateTest 2016-01-06 12:53:45 -05:00
Jason Tedor c291c17142 Cleanup TransportReplicationActionTests#runReplicateTest
This commit cleans up some of the assertions in
TransportReplicationActionTests#runReplicateTest:
 - use a Map to track actual vs. expected requests
 - assert that no request was sent to the local node
 - use RoutingTable#shardRoutingTable convenience method
 - explicitly use false in boolean conditions
 - clarify requests are expected on replica shards when assigned and
   execution on replicas is true
 - test ShardRouting equality when checking the failed shard request
2016-01-06 12:53:45 -05:00
Jason Tedor 6413adb5bc Assert that replication requests are sent to the correct shard copies
This commit adds tighter assertions in
TransportReplicationActionTests#runReplicateTest that replication
requests are sent to the correct shard copies.
2016-01-06 12:53:45 -05:00
Jason Tedor 75106daf9c Only fail the relocation target when a replication request on it fails
This commit addresses an issue when handling a failed replication
request against a relocating target shard. Namely, if a replication
request fails against the target of a relocation we currently fail both
the source and the target. This leads to an unnecessary
recovery. Instead, only the target of the relocation should be failed.
2016-01-06 12:53:41 -05:00
Nik Everett f5898fb07f [highlighting] Test for boundary chars 2016-01-06 12:32:09 -05:00
Martijn van Groningen 81cffd1be3 test: mute test 2016-01-06 18:30:04 +01:00
Martijn van Groningen 247ce06fc3 percolator: if size is 0 then use TotalHitCountCollector
Fixes PercolateIT#testPercolateSizingWithQueryAndFilter test
2016-01-06 18:00:00 +01:00
Nik Everett 664d67d41d Add test for alias filter leaking into highlighter 2016-01-06 11:32:03 -05:00
Jason Tedor cd56366378 Assert that we fail the correct shard when a replication request fails
This commit adds an assertion to
TransportReplicationActionTests#runReplicateTest that when a replication
request fails, we fail the correct shard.
2016-01-06 11:01:02 -05:00
Martijn van Groningen 2d6adf6428 Percolator refactoring:
* Added percolator field mapper that extracts the query terms and indexes these terms with the percolator query.
* At percolate time these extracted terms are used to query percolator queries that are like to be evaluated. This can significantly cut down the time it takes to percolate. Whereas before all percolator queries were evaluated if they matches with the document being percolated.
* Changes made to percolator queries are no longer immediately visible, a refresh needs to happen before the changes are visible.
* By default the percolate api only returns upto 10 matches instead of returning all matching percolator queries.
* Made percolate more modular, so that it is easier to add unit tests.
* Added unit tests for the percolator.

Closes #12664
Closes #13646
2016-01-06 16:08:10 +01:00
Nik Everett 646d24ebdc Add a test that the typename isn't highlighted 2016-01-06 09:20:15 -05:00
Yannick Welsch de6dfe15a7 Add PathHierarchy type back to path_hierarchy tokenizer for backward compatibility with 1.x
Closes #15785
2016-01-06 14:37:33 +01:00
Yannick Welsch a6ec1434d6 [TEST] Reduce log level in NodeVersionAllocationDeciderTests 2016-01-06 14:35:47 +01:00
Simon Willnauer 8a90c8085d Merge pull request #15788 from s1monw/dont_delete_tlog_file
Never delete translog-N.tlog file when creation fails
2016-01-06 14:31:22 +01:00
Simon Willnauer 5c833750d7 apply feedback from @bleskes 2016-01-06 14:19:58 +01:00
Simon Willnauer 12b93e72f0 Never delete translog-N.tlog file when creation fails
We today delete the translog-N.tlog file if any subsequent operation fails
but we might actually be in a good state if for instance the creation of the writer
failes after we sucessfully baked the new translog generation into the checkpoint. In this situation
we used to delete the translog-N.tlog file and failed on the next recovery of the translog with a
NoSuchFileException | FileNotFoundException just like in https://discuss.elastic.co/t/cannot-recover-index-because-of-missing-tanslog-files/38336

This commit changes the behavior and cleans up that limbo state on recovery if we already have a generation+1 file written but not baked into
the checkpoint we remove that file but only if the previous ckp file has already been renamed otherwise we know we can't be in this state.
2016-01-06 13:10:21 +01:00
Simon Willnauer 56329d0f53 Never call a listerner under lock in InternalEngine
We has a postIndex|DeleteUnderLock listener callback to load percolator
queries which is entirely private to the index shard in the meanwhile. Yet,
it still calls an external callback while holding an indexing lock which is scary
since we have no control over how long the operation could possibly take.

This commit decouples the percolator registry entirely from the ShardIndexingService
by pessimistically fetching percolator documents from the the engine using realtime get.
Even in situations where the same document is changed concurrently we will eventually end up
in the correct state without loosing an update. This also moves the index throtteling stats directly into
the engine to entirely remove the need for the dependency between InternalEngine and ShardIndexingService.
2016-01-06 11:38:34 +01:00
Yannick Welsch 55cc88e1ae Fix version-based allocation decider to prevent peer recovery from node with older version
Relocating a non-primary shard from one node to another is actually done by recovering from the active
primary shard in the cluster, and not the node that we are logically relocating from.

Closes #15775
2016-01-06 10:07:39 +01:00
Michael McCandless 3744fb9dc0 merge master 2016-01-06 04:03:42 -05:00
Michael McCandless 5f4afe842c remove nocommits; fix test case 2016-01-06 03:41:23 -05:00
Adrien Grand 7e3ccf2ee3 Merge pull request #15746 from jpountz/fix/missing_terms_agg
Make `missing` on terms aggs work with all execution modes.
2016-01-06 09:32:39 +01:00
Jason Tedor d032dabed5 Merge pull request #15777 from jasontedor/safer-cluster-state-task-notifications
Safe cluster state task notifications
2016-01-05 16:56:24 -05:00
Jason Tedor 05c46c9d35 Safe cluster state task notifications
This commit addresses an issue where a cluster state task listener
throwing an exception could prevent other listeners from being notified,
and could prevent the executor from receiving notifications that a new
cluster state was published. Additionally, this commit also addresses a
similar issue for executors handling cluster state publication
notifications.
2016-01-05 16:44:59 -05:00
Igor Motov 9ca4386fba Lower logging level for registering/unregistering tasks with the task manager
At the debug level the task manager is too talkative.
2016-01-05 15:15:02 -05:00
Igor Motov a89dba27c2 Task Management: Add framework for registering and communicating with tasks
Adds task manager class and enables all activities to register with the task manager. Currently, the immutable Transport*Activity class represents activity itself shared across all requests. This PR adds and an additional structure Task that keeps track of currently running requests and can be used to communicate with these requests using TransportTaskAction.

Related to #15117
2016-01-05 12:24:43 -05:00
Simon Willnauer dff30ece05 Double check if stream must be flush to allow tests to make better assumptions of what is visible and what isn't after tragic events 2016-01-05 17:58:20 +01:00
Michael McCandless 1d46a00d43 move async-ness upwards 2016-01-05 11:27:05 -05:00
Michael McCandless 8e7719d136 add logger.warn if thread pool size is clipped; fix test failure 2016-01-05 10:59:40 -05:00
Simon Willnauer ea6718d878 Add pending review from @bleskes on #15771 2016-01-05 16:34:26 +01:00
Michael McCandless 99d6ec53fa fold in feedback 2016-01-05 09:53:13 -05:00
Simon Willnauer 5272c98136 format finally block 2016-01-05 15:50:10 +01:00
Simon Willnauer 469a179ab1 remove stale comments 2016-01-05 15:48:35 +01:00
Simon Willnauer 2370ace922 assert that tragic even exception is never null 2016-01-05 15:45:41 +01:00
Simon Willnauer f69502dd04 Simplify TranslogWriter to always write to a stream
We used to write into an in-memory buffer and if necessary also allow reading
from the memory buffer if the some translog locations that are not flushed to
the channel need to be read. This commit hides all writing behind a buffered output
stream and if ncecessary flushes all buffered data to the channel for reading. This allows
for several simplifcations like reusing javas build in BufferedOutputStream and removes the
need for read write locks on the translog writer. All thread safety is now achived using
the synchronized primitive.
2016-01-05 15:22:56 +01:00
Simon Willnauer 107859f347 Merge pull request #15766 from s1monw/add_more_io_exceptions
Add serialization support for more important IOExceptions
2016-01-05 12:00:35 +01:00
Simon Willnauer 3a41dfe1dd Add serialization support for more important IOExceptions
Several IOExceptions are always wrapped in an NotSerializableWrapper which is
annoying to read. These exceptions are important to get right across the network
and we should support the important ones that indicate problems on the Filesystem.

This commit also adds general support for IOException to preserve the parent type
across the network if no specific type is serializable.
2016-01-05 10:30:26 +01:00
Adrien Grand 1fcf9958b2 Fix GeoPointFieldMapperTests expectations. 2016-01-05 10:26:39 +01:00
Simon Willnauer bf7e2c333a Merge pull request #14632 from s1monw/indexing_stats_javadocs_
Add javadocs to IndexingStats.Stats
2016-01-05 09:37:42 +01:00
Simon Willnauer d4de8dbcfe Merge pull request #15762 from s1monw/issues/15754
Close recovered translog readers if createWriter fails
2016-01-05 08:23:11 +01:00
Simon Willnauer f6020a6ff2 Close recovered translog readers if createWriter fails
If we fail to create a writer all recovered translog readers are not
closed today which causes all open files to leak.

Closes #15754
2016-01-04 23:40:20 +01:00
Jason Tedor 06851b7224 Merge pull request #15736 from jasontedor/shard-state-action-cluster-state-refactoring
Make cluster state external to o.e.c.a.s.ShardStateAction
2016-01-04 16:22:12 -05:00
Jason Tedor 8a793b6564 Rename Shard(.*)ClusterStateHandler to Shard$1ClusterStateTaskExecutor 2016-01-04 16:13:39 -05:00
Nicholas Knize ff1aed80cc Revert GeoPointField.stored default back to false
As a default in V2, the GeoPointField.stored option was set to true. Since this consumes disk space with no positive benefit the default stored option is being reverted back to false.
2016-01-04 11:11:02 -06:00
Jason Tedor af523c4236 Fix order of logging parameters in ShardStateAction 2016-01-04 11:29:00 -05:00
Jason Tedor 7591f2047a Modify nested classes in ShardStateAction to be static 2016-01-04 11:17:18 -05:00
Jason Tedor fdb0c909ec Inline Shard(.*)TransportHandler#handleShard\1OnMaster methods 2016-01-04 11:10:52 -05:00
Jason Tedor 754bd66b63 Always log shard ID in ShardStateAction
This commit restores logging the ShardRouting#shardId at the front of
the log messages in ShardStateAction. The reason for this is so that
shard-level log messages have the format "[component][node][shard]
message".
2016-01-04 11:10:41 -05:00
Jason Tedor 1a131cf917 Inline ShardStateAction#innerShardFailed method 2016-01-04 11:10:23 -05:00
Adrien Grand c934f859c7 Make `missing` on terms aggs work with all execution modes.
There are two bugs:
 - the 'global_ordinals_low_cardinality' mode requires a fielddata-based impl so
   that it can extract the segment to global ordinal mapping
 - the 'global_ordinals_hash' mode abusively casts to the values source to a
   fielddata-based impl while it is not needed

Closes #14882
2016-01-04 11:13:17 +01:00
Adrien Grand 1a47226d9a Merge pull request #15663 from jpountz/remove/mapping_backcompat
Remove mapping backward compatibilit with pre-2.0.
2016-01-04 10:05:39 +01:00
Jason Tedor a70f76f763 Make cluster state external to o.e.c.a.s.ShardStateAction
This commit modifies the handling of cluster states in
o.e.c.a.s.ShardStateAction so that all necessary state is obtained
externally to the ShardStateAction#shardFailed and
ShardStateAction#shardStarted methods. This refactoring permits the
removal of the ClusterService field from ShardStateAction.
2016-01-03 19:15:49 -05:00
Jason Tedor c47340f2f1 Refactor master node change predicate for reuse
This commit migrates a ClusterStateObserver.ChangePredicate for
detecting a master node change into a separate class for reuse
elsewhere.
2016-01-03 12:48:21 -05:00
Jason Tedor 6a12b5e59a Cleanup o/e/c/ClusterStateObserver.java
This commit applies a minor code cleanup to
o/e/c/ClusterStateObserver.java. In particular
 - employ the diamond operator instead of explicitly specifying a
   generic type parameter
 - use 'L' instead of 'l' for specifying a long literal
 - remove redundant static modifier on a nested interface
 - remove redundant public access modifiers on interface methods
 - reformat the declaration of the four-argument ChangePredicate#apply
 - simplify the bodies of ValidationPredicate#apply
2016-01-03 12:35:25 -05:00
Jason Tedor 110fe6d36a Remove unused constant from o.e.c.n.DiscoveryNode 2016-01-03 11:56:32 -05:00
Jason Tedor 265f8f8512 Address unchecked warnings in o/e/c/c/Cache.java 2016-01-02 18:38:40 -05:00
Jason Tedor 61b7014023 Null parameter checks in BulkProcessor.Builder#builder 2016-01-01 19:52:58 -05:00
Nicholas Knize dc07affff1 Reconcile GeoPoint toString and fromString methods
GeoPoint.toString prints as a json array of values, but resetFromString expects comma delimited. This commit reconciles the methods.
2015-12-31 10:56:04 -06:00
Nicholas Knize ef33a74286 Fix multi-field support for GeoPoint types
This commit fixes multiField support for GeoPointFieldMapper by passing an externalValueContext to the multiField parser. Unit testing is added for multi field coverage.
2015-12-31 09:34:56 -06:00
Adrien Grand cb08c52a2a Merge pull request #15715 from jpountz/fix/dyn_mapping_and_template_multi_field
Fix dynamic mapping corner case.
2015-12-31 09:44:39 +01:00
Dave 3f9c0fbb58 MapperService: check index.mapper.dynamic during index creation
The MapperService doesn't currently check the
index.mapper.dynamic setting during index creation,
so indices can be created with dynamic mappings even
if this setting is false. Add a check that throws an
exception in this case. Fixes #15381
2015-12-30 21:34:02 +01:00
Jim Ferenczi 4986817c6d Fix NPE 2015-12-30 19:55:47 +01:00
Jim Ferenczi cce600ae57 Update test which assumes that an unknown similarity type is accepted. 2015-12-30 19:12:03 +01:00
Jim Ferenczi a4df067524 Fixes test with wrong similarity type (bm25 => BM25) 2015-12-30 18:27:06 +01:00
Adrien Grand 1576965831 Fix dynamic mapping corner case.
Today we fail if the field exists in another type and multi fields are defined
in a template.
2015-12-30 18:12:17 +01:00
Jim Ferenczi 992ffac509 Merge pull request #15446 from jimferenczi/classic_similarity
Renames `default` similarity into `classic`
2015-12-30 08:42:20 -08:00
Adrien Grand 6d3c9b074c Remove support for the `multi_field` type.
It is officially unsupported since version 1.0.
2015-12-30 12:03:15 +01:00
Adrien Grand a7e1eeb13d Merge pull request #15684 from jpountz/deprecate/field_boost
Remove object notation for core types.
2015-12-30 09:58:01 +01:00
Jim Ferenczi d898c0a6b0 Removes not relevant part of the bw compat test. 2015-12-30 03:54:26 +01:00
Nicholas Knize d027ceb76d [TEST] Update GeoDistanceQuery to use Lucene's maxRadialDistance
Removing maxRadialDistance method from ES GeoUtils in favor of Lucene 5.4 GeoDistanceUtils.maxRadialDistanceMeters.
2015-12-29 09:47:59 -06:00
Robert Muir 25914ae879 Merge pull request #15688 from rmuir/thirdPartyAudit3
Improve thirdPartyAudit check, round 3
2015-12-29 09:24:51 -05:00
Simon Willnauer 39cec9f2ff [TEST] Improve test speed
DedicatedClusterSnapshotRestoreIT#testRestoreIndexWithMissingShards took ~1.5 min to finish
due to timeouts that are applied if not all shards are allocated. Now that the index that has
unallocated shareds is not refreshed the test is more reasonable and runs in 15 sec
2015-12-29 14:53:15 +01:00
Daniel Mitterdorfer c813d21ffb Replace * import with explicit imports 2015-12-29 12:08:30 +01:00
Daniel Mitterdorfer 46a4aa9704 Tighten assertions in BulkProcessorRetryIT
With this commit we check more precisely on the result of a bulk
request. It could either be ok, fail or be rejected due to resource
constraints. Previously, we have relied that by default we never
get rejected.

However, this is a valid condition even when retrying. With this
commit we check that we either retried often enough that we don't
get rejected *and* if we got rejected that we maxed out the number
of specified retries.
2015-12-29 11:43:41 +01:00
Simon Willnauer 60cbb2d7bc [TEST] Protect UpgradeIT from using too many replicas 2015-12-29 10:41:04 +01:00
Robert Muir 180ab2493e Improve thirdPartyAudit check, round 3 2015-12-28 22:38:55 -05:00
Yannick Welsch c6182cbd37 Fail replica shards before primary shards
As failing primaries also fail associated replicas, we must fail replicas first so that their nodes are properly added to ignore list

Closes #15686
2015-12-28 22:37:18 +01:00
Simon Willnauer 387bdbd322 Don't limit recoveries in RoutingTableTests 2015-12-28 21:36:34 +01:00
Adrien Grand bffbad3c19 Remove object notation for core types.
When specifying a string field, you can either do:

```
{
  "foo": "bar"
}
```

or

```
{
  "foo": {
    "value": "bar",
    "boost": 42
  }
}
```

The latter option is now removed.

Closes #15388
2015-12-28 19:49:18 +01:00
Yannick Welsch 7e134da7d3 Cache result of RoutingNodes.node(...) in ModelNode 2015-12-28 19:14:48 +01:00
Yannick Welsch eea791de15 Remove superfluous method numPrimaries 2015-12-28 19:14:48 +01:00
Yannick Welsch e855282258 Simplify numShards calculation 2015-12-28 19:14:48 +01:00
Yannick Welsch 1536d7fe37 For rebalancing an index, only consider nodes that currently have a shard of that index or where the index can be allocated
This allows to prune a large number of nodes in case of hot/warm setup
2015-12-28 19:14:48 +01:00
Yannick Welsch 207dfc457d Remove superfluous method 2015-12-28 19:14:48 +01:00
Yannick Welsch fc0a33be05 Eliminate adding/removing shard to simulate weight of added shard / removed shard
Removal of the pattern node.addShard() -> calculate weight -> node.removeShard() which is expensive as, beside map lookups, it invalidates caching of precomputed values in ModelNode and ModelIndex. Replaced by adding an additional parameter to the weight function which accounts for the added / removed shard.
2015-12-28 19:14:48 +01:00
Yannick Welsch 5bd31a6cca Fix typo in method name 2015-12-28 19:14:48 +01:00
Yannick Welsch 67905b384f Remove unused class 2015-12-28 19:14:48 +01:00
Yannick Welsch 1bb7ca8a85 Removed unused methods 2015-12-28 19:14:48 +01:00
Yannick Welsch 0fd19008b1 Precalculate avgShardsPerNode 2015-12-28 19:14:48 +01:00
Yannick Welsch 507bb11345 Split two-element array into proper variables 2015-12-28 19:14:48 +01:00
Adrien Grand c6cf84336f Merge pull request #15679 from jpountz/fix/text_parsing
Make text parsing less lenient.
2015-12-28 16:47:05 +01:00
Adrien Grand 5eb7555ffb Make text parsing less lenient.
It now requires that the parser is on a value.
2015-12-28 16:06:42 +01:00
Simon Willnauer ba755a554f Merge pull request #15372 from s1monw/trash_recovery_threads
Remove recovery threadpools and throttle outgoing recoveries on the master
2015-12-28 15:43:14 +01:00
Simon Willnauer dcb164db3b Merge pull request #15598 from lks21c/master
Put space into last comment line
2015-12-28 15:41:10 +01:00
Jason Tedor 84c4ab6c18 Correctly release threads from starting gate in o.e.c.ClusterServiceIT 2015-12-28 08:13:55 -05:00
Jason Tedor 35cc749c9a Correctly release threads from starting gate in o.e.c.c.CacheTests 2015-12-28 08:07:37 -05:00
Simon Willnauer e8daad66d1 apply more feedback from @dakrone 2015-12-28 12:52:38 +01:00
Simon Willnauer a20b07d581 Merge branch 'master' into trash_recovery_threads 2015-12-28 12:45:14 +01:00
Simon Willnauer 0a816cd343 Add tests for equals and hashCode and fix FiltersFunctionScoreQuery equals and hashCode impls
Relates to #15676
2015-12-28 12:34:34 +01:00
Adrien Grand 23f736a0c1 Remove ParseContext.ignoredValue.
This API is unused.
2015-12-24 15:28:09 +01:00
Adrien Grand af122f4151 Remove mapping backward compatibilit with pre-2.0.
This removes the backward compatibility layer with pre-2.0 indices, notably
the extraction of _id, _routing or _timestamp from the source document when a
path is defined.
2015-12-24 13:47:37 +01:00
Adrien Grand e2fbdcfb4f Merge pull request #15633 from jpountz/fix/dynamic_mappings
Improve cross-type dynamic mapping updates.
2015-12-24 10:24:08 +01:00
Adrien Grand 3015eb3088 Improve cross-type dynamic mapping updates.
Today when dynamically mapping a field that is already defined in another type,
we use the regular dynamic mapping logic and try to copy some settings to avoid
introducing conflicts. However this is quite fragile as we don't deal with every
existing setting. This proposes a different approach that will just reuse the
shared field type.

Close #15568
2015-12-24 10:22:35 +01:00
Adrien Grand 79b14215aa Merge pull request #15636 from jpountz/feature/bool_fields
Add sub-fields support to `bool` fields.
2015-12-24 10:21:49 +01:00
Robert Muir d144ba24a5 Merge pull request #15588 from rmuir/hdfs2-only
merge current hdfs improvements to master
2015-12-23 18:17:22 -05:00
Jason Tedor 6ab922c5a0 Reduce runtime of CidrsTests#testValidCombinations
This commit reduces the running time of CidrsTests#testValidCombinations
by hoisting some costly repeated operations outside of the inner test
loop.
2015-12-23 18:02:41 -05:00
Adrien Grand f40b72633b Add sub-fields support to `bool` fields.
`bool` is our only core mapper that does not support sub fields.

Close #6587
2015-12-23 17:45:46 +01:00
Adrien Grand d8d8666877 Remove `index_name` back compat.
Since 2.0 we enforce that fields have the same full and index names. So in 3.x
we can remove the ability to have different names on the same field.
2015-12-23 14:55:26 +01:00
Adrien Grand 07658f58a8 FunctionScoreQuery should implement two-phase iteration.
FunctionScoreQuery should do two things that it doesn't do today:
 - propagate the two-phase iterator from the wrapped scorer so that things are
   still executed efficiently eg. if a phrase or geo-distance query is wrapped
 - filter out docs that don't have a high enough score using two-phase
   iteration: this way the score is only checked when everything else matches

While doing these changes, I noticed that minScore was ignored when scores were
not needed and that explain did not take it into account, so I fixed these
issues as well.
2015-12-23 12:02:21 +01:00
Adrien Grand a2072fe927 Merge pull request #15539 from jpountz/fix/immutable_document_mapper
Make mapping updates more robust.
2015-12-23 09:55:42 +01:00
Adrien Grand f535c27024 Make mapping updates more robust.
This changes a couple of things:

Mappings are truly immutable. Before, each field mapper stored a
MappedFieldTypeReference that was shared across fields that have the same name
across types. This means that a mapping update could have the side-effect of
changing the field type in other types when updateAllTypes is true. This works
differently now: after a mapping update, a new copy of the mappings is created
in such a way that fields across different types have the same MappedFieldType.
See the new Mapper.updateFieldType API which replaces MappedFieldTypeReference.

DocumentMapper is now immutable and MapperService.merge has been refactored in
such a way that if an exception is thrown while eg. lookup structures are being
updated, then the whole mapping update will be aborted. As a consequence,
FieldTypeLookup's checkCompatibility has been folded into copyAndAddAll.

Synchronization was simplified: given that mappings are truly immutable, we
don't need the read/write lock so that no documents can be parsed while a
mapping update is being processed. Document parsing is not performed under a
lock anymore, and mapping merging uses a simple synchronized block.
2015-12-23 09:55:07 +01:00
Lee Hinman 862cdad8dc Merge remote-tracking branch 'dakrone/freebsd-test-fix' 2015-12-22 15:41:35 -07:00
Jason Tedor 4c9a9b4b2d Reduce number of threads in o.e.c.c.CacheTests
This commit reduces the randomized number of threads in the cache tests
to reduce the amount of time that the tests take to execute.
2015-12-22 16:55:55 -05:00
Lee Hinman 482843e27b Fix build to run correctly on FreeBSD
This adds the required changes/checks so that the build can run on
FreeBSD.

There are a few things that differ between FreeBSD and Linux:

- CPU probes return -1 for CPU usage
- `hot_threads` cannot be supported on FreeBSD

From OpenJDK's `os_bsd.cpp`:

```c++
bool os::is_thread_cpu_time_supported() {
  #ifdef __APPLE__
  return true;
  #else
  return false;
  #endif
}
```

So this API now returns (for each FreeBSD node):

```
curl -s localhost:9200/_nodes/hot_threads
::: {Devil Hunter Gabriel}{q8OJnKCcQS6EB9fygU4R4g}{127.0.0.1}{127.0.0.1:9300}
   hot_threads is not supported on FreeBSD
```

- multicast fails in native `join` method - known bug:
  https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=193246

Which causes:

```
1> Caused by: java.net.SocketException: Invalid argument
1>    at java.net.PlainDatagramSocketImpl.join(Native Method)
1>    at java.net.AbstractPlainDatagramSocketImpl.join(AbstractPlainDatagramSocketImpl.java:179)
1>    at java.net.MulticastSocket.joinGroup(MulticastSocket.java:323)
1>    at org.elasticsearch.plugin.discovery.multicast.MulticastChannel$Plain.buildMulticastSocket(MulticastChannel.java:309)
```

So these tests are skipped on FreeBSD.

Resolves #15562
2015-12-22 12:36:04 -07:00
Daniel Mitterdorfer f9a601c7da Make BulkProcessorRetryIT less sensitive
In this commit we increase the queue size of the bulk pool in
BulkProcessorRetryIT to make it less sensitive.

As this test case should stress the pool so bulk processor needs to
back off but not so much that the backoff policy will give up at
some point (which is a valid condition), we still keep it below the
default queue size of 50.
2015-12-22 17:37:30 +01:00
Adrien Grand 6f3fc5d75d Speed up CodecTests.
Some tests, but in particular CodecTests, are slow because they test all
versions that ever existed even though they should only test supported
versions.
2015-12-22 15:16:08 +01:00
Simon Willnauer f5e4cd4616 Remove recovery threadpools and throttle outgoing recoveries on the master
Today we throttle recoveries only for incoming recoveries. Nodes that have a lot
of primaries can get overloaded due to too many recoveries. To still keep that at bay
we limit the number of threads that are sending files to the target to overcome this problem.

The right solution here is to also throttle the outgoing recoveries that are today unbounded on
the master and don't start the recovery until we have enough resources on both source and target nodes.

The concurrency aspects of the recovery source also added a lot of complexity and additional threadpools
that are hard to configure. This commit removes the concurrent streamns notion completely and sends files
in the thread that drives the recovery simplifying the recovery code considerably.
Outgoing recoveries are not throttled on the master via a allocation decider.
2015-12-22 14:59:43 +01:00
Simon Willnauer d353dcb138 Merge pull request #15592 from s1monw/remove_deprecated_query_cache_settings
Remove deprecated query cache settings
2015-12-22 10:34:00 +01:00
kwangsik 70d38760e6 Put space into last comment line 2015-12-22 18:27:50 +09:00
Simon Willnauer 8135a4ac9f Remove `index.compound_on_flush` setting and default to `true`
We added this undocumented realtime setting as backup plan long ago
but to date we haven't had a situation where it was a problem. It's reducing
the number of filehandles in the NRT case dramatically and should always be enabled.
2015-12-22 09:56:26 +01:00
Simon Willnauer 3a5d3a3bb0 Remove deprecated query cache settings 2015-12-22 09:39:12 +01:00
Simon Willnauer 8c898048bc Merge pull request #15584 from s1monw/move_translog_syn_to_index_service
Move async translog sync logic into IndexService
2015-12-22 09:28:11 +01:00
Robert Muir 010d1a89c5 Merge branch 'master' into hdfs2-only 2015-12-22 00:40:54 -05:00
Jun Ohtani 51f6519e71 Analysis : Fix no response from Analyze API without specified index
fix test
2015-12-22 11:53:29 +09:00
Jun Ohtani 267cd65506 Analysis : Fix no response from Analyze API without specified index
Fix error handling in TransportSingleShardAction without shardIt

Closes #15148
2015-12-22 10:13:52 +09:00
Michael McCandless 7ef43a22ff limit the max size of bulk and index thread pools to bounded number of processors 2015-12-21 19:18:19 -05:00
Simon Willnauer 98f9749ee1 fix visibility 2015-12-21 22:12:32 +01:00
Simon Willnauer 87494dd7db fix spelling and remove wildcard imports 2015-12-21 22:06:51 +01:00
Simon Willnauer c3f901400c Move async translog sync logic into IndexService
Today the logic to async - commit the translog is in every translog instance
itself. While the setting is a per index setting we manageing it per shard. This
polluts the translog code and can more easily be managed in IndexService.
2015-12-21 21:36:02 +01:00
Robert Muir 0ed45c5bfb remove filesystem leniency 2015-12-21 14:16:53 -05:00
Simon Willnauer 445be98e4c Merge pull request #15574 from s1monw/drop_simple_translog
Drop support for simple translog and hard-wire buffer to 8kb
2015-12-21 17:28:55 +01:00
Adrien Grand a6bf860cb5 Merge pull request #15575 from jpountz/upgrade/5.5.0-snapshot-1721183
Upgrade to lucene-5.5.0-snapshot-1721183.
2015-12-21 17:17:42 +01:00
Adrien Grand cf52e96c42 Upgrade to lucene-5.5.0-snapshot-1721183.
Some files that implement or use the Scorer API had to be changed because of
https://issues.apache.org/jira/browse/LUCENE-6919.
2015-12-21 17:02:08 +01:00
Simon Willnauer fcfd98e9e8 Drop support for simple translog and hard-wire buffer to 8kb
Today we have two variants of translogs for indexing. We only recommend the buffered
one which also has a 20% advantage in indexing speed. This commit removes the option and defaults
to the buffered case. It also hard-wires the translog buffer to 8kb instead of 64kb. We used to
adjust that buffer based on if the shard is active or not, this code has also been removed and
instead we just keep an 8kb buffer arround.
2015-12-21 16:44:35 +01:00
Jim Ferenczi 81fd2169cf Renames "default" similarity into "classic".
Replaces deprecated DefaultSimilarity by ClassicSimilarity.
Fixes #15102
2015-12-21 16:22:53 +01:00
Jim Ferenczi 96e29be7c6 Merge pull request #15571 from jimferenczi/min_should_match
Min should match greater than the number of optional clauses should return no result
2015-12-21 16:06:51 +01:00
Jim Ferenczi 4d32cc0b9f Queries.calculateMinShouldMatch returns the number of "min should match" clauses that the user wanted
even if the number of optional clauses is smaller than the provided number.
In such case the query now returns no result.
Closes #15521
2015-12-21 16:06:06 +01:00
Simon Willnauer 0cdbcdab64 added 2.3.0-SNAPSHOT as a Version constant 2015-12-21 15:48:16 +01:00
Simon Willnauer 7fe2eddfec fix test 2015-12-21 15:26:56 +01:00
Simon Willnauer afc1cc19af Simplify translog-based flush settings
This commit removes `index.translog.flush_threshold_ops` and `index.translog.disable_flush`
in favor of `index.translog.flush_threshold_size`. The number of operations is meaningless by itself and
can easily be turned into a size value with knowledge of the data. Disabling the flush is only useful in
tests and we can set the size value to a really high value. If users really need to do this they can
also apply a very high value like `1PB`.
2015-12-21 15:15:00 +01:00
Simon Willnauer eb64a81d05 Remove `index.merge.scheduler.notify_on_failure` and default to `true`
This setting was undocumented and should not be set by any user. We should
fail the shard instead.

Closes  #15570
2015-12-21 14:39:58 +01:00
Adrien Grand ac393b7a31 Make mappings tests more realistic.
DocumentMapperParser has both parse and parseCompressed methods. Except that the
parse methods are ONLY used from the unit tests. This commit removes the parse
method and moves all tests to parseCompressed so that they test more
realistically how mappings are managed.

Then I renamed parseCompressed to parse given that this is the only alternative
anyway.
2015-12-21 10:44:00 +01:00
Jim Ferenczi 1ec44dcdda Merge pull request #15371 from jimferenczi/alias_routing
Resolves the conflict between alias routing and parent routing by applying the alias routing and ignoring the parent routing.
2015-12-21 09:58:45 +01:00
Jim Ferenczi 776e5d8096 Separates routing and parent in all documentrequest in order to be able to distinguish an explicit routing value from a parent routing.
Resolves conflicts between parent routing and alias routing with the following rule:
    * The parent routing is ignored if there is an alias routing that matches the request.
Closes #3068
2015-12-21 09:44:57 +01:00
Robert Muir 5d976f1431 add sniper for broken hadoop 2015-12-18 20:16:15 -05:00
Zachary Tong 5cfefe08b0 Fix collector's class name on response output 2015-12-18 16:47:06 -05:00
Ryan Ernst 690fb2cd3f Rename InternalFilters.Bucket to InternalFilters.InternalBucket to avoid name collision 2015-12-18 13:22:20 -08:00
Ryan Ernst beec7ca9db Merge branch 'master' into wildcard_imports 2015-12-18 13:18:08 -08:00
Ryan Ernst 4ea19995cf Remove wildcard imports 2015-12-18 12:43:47 -08:00
Zachary Tong 4700cb9409 [TEST] Check length of profiled results in tests 2015-12-18 15:19:04 -05:00
Zachary Tong 41139c6b15 Ensure profile results are immutable after serialization 2015-12-18 15:19:00 -05:00
Daniel Mitterdorfer 56e4752d28 Align handling of interrupts in BulkProcessor
With this commit we implement a cancellation policy in
BulkProcessor which is aligned for the sync and the async case
and also document it.

Closes #14833.
2015-12-18 19:34:23 +01:00
Robert Muir 7e53076112 Merge pull request #15545 from rmuir/rat
add gradle licenseHeaders to precommit
2015-12-18 13:27:37 -05:00
Robert Muir 266bece2fe add missing license headers 2015-12-18 13:19:09 -05:00
Robert Muir 6ba374d632 add missing license headers 2015-12-18 13:15:34 -05:00
Areek Zillur 9d9b557cea Don't allow nodes with missing custom data types to join cluster 2015-12-18 11:33:29 -05:00
Colin Goodheart-Smithe 15588a4991 Aggregations: Run pipeline aggregations for empty buckets added in the Range Aggregation
Closes #15471
2015-12-18 14:42:05 +00:00
Christoph Büscher 2f97ff0925 Merge pull request #15010 from cbuescher/refactor-shapebuilders
Make remaining ShapeBuilders implement Writeable
2015-12-18 11:11:34 +01:00
Simon Willnauer 55f77db45b Merge pull request #15278 from s1monw/settings_prototype
Add infrastructure to transactionally apply and reset dynamic settings
2015-12-18 10:57:29 +01:00
Simon Willnauer 186242145f [TEST] use new settings infra 2015-12-18 10:26:57 +01:00
Simon Willnauer 5b991b9d5e Check for tragic event on all kinds of exceptions not only ACE and IOException
It's important to close not matter what exception caused a tragic event. Today
we only check on IOException and AlreadyClosedExceptions. The test had a bug and
threw an IAE instead causing the translog not to be closed.
2015-12-18 10:16:11 +01:00