3138 Commits

Author SHA1 Message Date
Nik Everett
4772b1fe9d Remove tests for backwards compatibility we don't support any more 2016-01-08 17:25:26 -05:00
Nik Everett
d8526f4d00 Handle some deprecation warnings
Suppress lots of them with comments. Fix a few. Removes lots of pre-built
stuff used to support pre-2.0 indexes.
2016-01-08 17:17:33 -05:00
Nik Everett
6bb01984b6 Fix blended terms for non-strings
It had some funky errors, like lenient:true not working and queries with
two integer fields blowing up if there was no analyzer defined on the
query. This throws a bunch more tests at it and rejiggers how non-strings
are handled so they don't wander off into scary QueryBuilder-land unless
they have a nice strong analyzer to protect them.

Closes #15860
2016-01-08 16:53:40 -05:00
Michael McCandless
d3d41fc22b Merge pull request #15585 from mikemccand/max_indexing_thread_pool_size
Limit the max size of bulk and index thread pools to bounded number of processors
2016-01-08 14:36:21 -05:00
Adrien Grand
8568480a74 Fix initial sizing of BytesStreamOutput.
It currently tries to align to the page size (16KB) by default. However, this
might waste a significant memory (if many BytesStreamOutputs are allocated)
and is also useless given that BytesStreamOutput does not recycle (on the
contrary to ReleasableBytesStreamOutput). So the initial size has been changed
to 0.

Closes #15789
2016-01-08 20:08:31 +01:00
Jason Tedor
871d1b4885 Remove and forbid use of j.u.c.ThreadLocalRandom
This commit removes and now forbids all uses of
java.util.concurrent.ThreadLocalRandom across the codebase. The
underlying issue with ThreadLocalRandom is that it can not be
seeded. This means that if ThreadLocalRandom is used in production code,
then tests that cover any code path containing ThreadLocalRandom will be
prevented from being reproducible by use of ThreadLocalRandom. Instead,
using org.elasticsearch.common.random.Randomness#get will give
reproducible sources of random when running under tests and otherwise
still give an instance of ThreadLocalRandom when running as production
code.
2016-01-08 12:23:48 -05:00
Jason Tedor
21f5b0ff35 Remove dead o.e.c.m.UnboxedMathUtils
This commit removes the dead UnboxedMathUtils from the codebase.
2016-01-08 11:58:39 -05:00
javanna
3e08c24991 clarify comment in TemplateService, explain why we have this class 2016-01-08 14:31:57 +01:00
Simon Willnauer
71796e2319 [TEST] Close failable translog in a controlled way otherwise assertions are off in the test 2016-01-08 13:10:09 +01:00
Martijn van Groningen
b9dd99636d dynamic option is a string, not a boolean 2016-01-08 11:36:52 +01:00
javanna
8675784e3f adapt error message when node.ingest is set to false and we fail executing a pipeline 2016-01-08 10:39:39 +01:00
javanna
ae69d46f92 move processors that have no deps to core, also move to core rest spec and tests and set node.inget to true by default 2016-01-08 10:39:39 +01:00
Martijn van Groningen
e35e9bd736 Removed index template and instead create .ingest with settings and mapping when we find out that the index is missing during the put pipeline call. 2016-01-08 10:35:52 +01:00
Adrien Grand
581fd49dac Merge pull request #15836 from jpountz/fix/bitset_cache_duplicates
BitSetFilterCache duplicates its content.
2016-01-08 09:59:28 +01:00
Igor Motov
8fbb3686cd Improve stability of the testBatchingShardUpdateTask test
On slow machines when this test randomly picks a large number of shards it can occasionally take more than 32.5 seconds to snapshot all shards. That is causing the test to miss the second to last assert in awaitsBusy at 32.5 seconds and then timeout in BlockingClusterStateListener at 60 seconds. Due to the timeout, the pending task queue is cleaned before the last awaitsBusy assert at 65 seconds and as a result the last assert runs on a completely empty queue and fails with a very confusing assert error.

This commit makes the timeout in BlockingClusterStateListener to occur after the last assert in assertBusyPendingTasks and therefore allows assertBusyPendingTasks to perform the last assert before cleaning the pending tasks queue takes place.

 This commit also reduces the maximum number of shards used in the test to 10 in order to speed up this test.
2016-01-07 19:33:50 -05:00
Adrien Grand
3ef9ec25f8 BitSetFilterCache duplicates its content.
We have a bug that makes all per-index bitset caches store bitsets for all
indices. In the case that you have many indices, which is fairly common with
time-based data, this could translate to a lot of wasted memory.

Closes #15820
2016-01-07 18:50:14 +01:00
Britta Weber
f93b4cb215 sync translog to disk after recovery from primary
Otherwise if that node is shutdown and restarted it might will have lost all operations
that were in the translog.
2016-01-07 16:27:40 +01:00
javanna
79c1377661 [TEST] set node.ingest to true otherwise we may try to modify values for a threadpool that hasn't been started 2016-01-07 16:16:14 +01:00
javanna
03fe38681e renamed qa package o.e.plugin.ingest to o.e.ingest
This way InternalTemplateService constructor can be set back to package private visibility
2016-01-07 15:51:52 +01:00
javanna
694eeffe92 catch the right exception... 2016-01-07 15:36:34 +01:00
Martijn van Groningen
0d70d92aa0 Also ignore rejected execution exception when installing index template 2016-01-07 15:25:35 +01:00
Martijn van Groningen
c44b83dc29 Also ignore rejected execution exception when retying restart 2016-01-07 15:21:21 +01:00
Adrien Grand
8bd54dbf5a Merge pull request #15828 from jpountz/enhancement/stricter_metadata_parsing
Make MetaData parsing less lenient.
2016-01-07 15:20:51 +01:00
Adrien Grand
6ce7a972bc Make MetaData parsing less lenient.
Today this simply ignores everything that is not recognized.
2016-01-07 15:20:16 +01:00
javanna
1ea690e814 Merge branch 'feature/ingest' into enhancement/move_to_core 2016-01-07 15:13:31 +01:00
javanna
7d7e0db91d Merge branch 'master' into feature/ingest 2016-01-07 15:09:13 +01:00
javanna
234371811d awaited fix a couple of index template tests till we have removed the ingest index template 2016-01-07 15:06:05 +01:00
Martijn van Groningen
fa9aab91eb Ignore rejected execution exception 2016-01-07 14:52:15 +01:00
Martijn van Groningen
ba97b7f7bd Don't activate ingest on tribe nodes 2016-01-07 14:36:23 +01:00
Nik Everett
52f28888d5 Merge pull request #15813 from nik9000/xlint1
Remove Xlint:-override,-fallthrough,-static
2016-01-07 08:34:40 -05:00
javanna
eca1594969 start ingest thread pool only when node.ingest is set to true 2016-01-07 14:33:56 +01:00
Boaz Leskes
d5e6eb58a8 Log uncaught exceptions from scheduled once tasks
`ScheduledThreadPoolExecutor` allows you to schedule tasks to run once or periodically at the future. If such a task throws an exception, that exception is caught and reported in the future that `ScheduledThreadPoolExecutor#schedule` returns. However, we typically do not capture the future / do not test it for errors. This results in exception being swallowed and not reported. To mitigate this we now wrap any command in a LoggingRunnable  (already used for periodic tasks).  Also, RunnableCommand is changed not to swallow exception but percolate them further for reporting by the future.

Closes #15824
2016-01-07 14:04:35 +01:00
Martijn van Groningen
0b06648205 template installation is async, so use assertBusy 2016-01-07 14:00:50 +01:00
javanna
0bfe6de75c move all transport actions under same package org.elasticsearch.action.ingest 2016-01-07 13:52:42 +01:00
Martijn van Groningen
e6cc79a976 test: added ingest script context 2016-01-07 13:48:36 +01:00
javanna
95bc0ed7a2 move constants to IngestActionFilter 2016-01-07 13:40:28 +01:00
javanna
2803ae09dc addProcessor -> registerProcessor 2016-01-07 13:25:25 +01:00
javanna
18aabd67c8 adapt qa tests for when ingest.node is set to false
CRUD and simulate apis work now fine, every node has the pipelines in memory, but node.ingest disables ingestion, meaning that any index or bulk request with a pipeline id is going to fail
2016-01-07 13:21:06 +01:00
javanna
52c2a273f9 amended error message 2016-01-07 13:21:06 +01:00
Martijn van Groningen
7e56f65939 since ingest is part of core the ingest template always gets installed, so the assumptions this test had had to be updated 2016-01-07 12:58:13 +01:00
Martijn van Groningen
4dca3cb38f fix thread pools, use management the do the operation and response handeling 2016-01-07 12:14:13 +01:00
javanna
c185c1339a remove grok and join usage from IngestClientIT 2016-01-07 11:30:02 +01:00
javanna
e036b5896d move registerFilter up 2016-01-07 11:29:43 +01:00
Martijn van Groningen
9ec2e140b8 Merge branch 'master' into feature/ingest 2016-01-07 10:44:21 +01:00
Simon Willnauer
e7f9d685f1 [TEST] Test that translog can recover after random IOException
This commit adds a new test that can throw an IOException at any point in time
and ensures that all previously synced documents can be successfully recovered after hitting
an excepiton.

Relates to #15788
2016-01-07 10:17:31 +01:00
Adrien Grand
67d233cecd Remove warmers and the warmer API.
Warmers are now barely useful and will be removed in 3.0. Note that this only
removes the warmer API and query-based warmers. We still have warmers internally
for eg. global ordinals.

Close #15607
2016-01-07 09:57:07 +01:00
Martijn van Groningen
604d59a95e muted test 2016-01-07 09:54:59 +01:00
Nik Everett
20e7fa97db Remove Xlint:-override,-fallthrough,-static
Adds `@SuppressWarnings("fallthrough")` in two places where the fallthrough
is used to implement well known hashing algorithms.
2016-01-06 22:27:14 -05:00
Nik Everett
74c132afc6 Standardize some methods on varargs
Right now we define the same sort of methods as taking String arrays and
string varargs. We should standardize on one and varargs is easier to
call so lets use varargs!
2016-01-06 21:01:58 -05:00
Jason Tedor
a583edb2df Merge pull request #15801 from jasontedor/cyclic-barriers-for-boaz
Use CyclicBarriers for sychronizing driver and test threads
2016-01-06 20:09:28 -05:00