Commit Graph

345 Commits

Author SHA1 Message Date
Jason Tedor 15d3d74444 Merge branch 'master' into feature/seq_no
* master: (904 commits)
  Removes unused methods in the o/e/common/Strings class.
  Add note regarding thread stack size on Windows
  painless: restore accidentally removed test
  Documented fuzzy_transpositions in match query
  Add not-null precondition check in BulkRequest
  Build: Make run task you full zip distribution
  Build: More pom generation improvements
  Add test for wrong array index
  Take return type from "after" field.
  painless: build descriptor of array and field load/store in code; fix array index to adapt type not DEF
  Build: Add developer info to generated pom files
  painless: improve exception stacktraces
  painless: Rename the dynamic call site factory to DefBootstrap and make the inner class very short (PIC = Polymorphic Inline Cache)
  Remove dead code.
  Avoid race while retiring executors
  Allow only a single extension for a scripting engine
  Adding REST tests to ensure key_as_string behavior stays consistent
  [test] Set logging to 11 on reindex test
  [TEST] increase logger level until we know what is going on
  Don't allow `fuzziness` for `multi_match` types cross_fields, phrase and phrase_prefix
  ...
2016-05-14 20:23:59 -04:00
Nik Everett 0a300320cd [test] Set logging to 11 on reindex test
It has failures we can't explain and we need logs to be able to do
anything useful with the failures:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+g1gc/359/consoleFull
2016-05-13 11:48:01 -04:00
Tanguy Leroux 6d288dec11 Clean up tests in Reindex module 2016-05-11 09:54:52 +02:00
Tanguy Leroux 8c52e8814b Remove ReindexResponse in favor of BulkIndexByScrollResponse 2016-05-09 17:03:16 +02:00
Adrien Grand 7d8708716e QueryBuilder does not need generics. #18133
QueryBuilder has generics, but those are never used: all call sites use
`QueryBuilder<?>`. Only `AbstractQueryBuilder` needs generics so that the base
class can contain a default implementation for setters that returns `this`.
2016-05-06 08:38:20 +02:00
Nik Everett 230697c202 [reindex] Switch throttle to Float.POSITIVE_INFITINTY/"unlimited"
All other values are errors.

Add java test for throttling. We had a REST test but it only ran against
one node so it didn't catch serialization errors.

Add Simple round trip test for rethrottle request
2016-05-04 16:14:32 -04:00
Nik Everett cc1a55423c Reindex: properly mark things as child tasks
Do this by creating a Client subclass that automatically assigns the
parentTask to all requests that come through it. Code that doesn't want
to set the parentTask can call `unwrap` on the Client to get the inner
client instance that doesn't set the parentTask. Reindex uses this for
its ClearScrollRequest so that the request will run properly after the
reindex request has been canceled.
2016-04-22 14:00:11 -04:00
Nik Everett 51621f9d75 Remove ChildTaskRequest and always pass parentTaskId when building a task
Passing parentTaskId forces the caller to handle the parentTaskId.
2016-04-22 11:26:18 -04:00
Martijn van Groningen c5ad2e2865 Changed indexed scripts to be stored in the cluster state instead of the `.scripts` index.
Also added max script size soft limit for stored scripts.

Closes #16651
2016-04-22 13:42:55 +02:00
Nik Everett d3b1306069 Reindex: never report negative throttled_until
Just clamp the value at 0. It isn't useful to tell the user "this
thread should have woken 5ms ago".

Closes #17783
2016-04-15 16:53:23 -04:00
Martijn van Groningen 2928fd6ef3 Cleanup query builder for inner hits construction.
* Inner hits can now only be provided and prepared via setter in the nested, has_child and has_parent query.
* Also made `score_mode` a required constructor parameter.
* Moved has_child's min_child/max_children validation from doToQuery(...) to a setter.
2016-04-14 14:43:21 +02:00
Nik Everett cca3154c43 Rename isSourceEmpty to hasSource
And add a test case for {} to reindex.
2016-04-13 08:19:58 -04:00
Nik Everett c2e745bf3b reindex: Guard against user disabling fields 2016-04-13 08:19:58 -04:00
Nik Everett 0f9804b0e2 reindex: gracefully handle when _source is disabled
Closes #17666
2016-04-13 08:19:58 -04:00
Nik Everett 14d37baa4b [reindex] Don't get rejected
BulkByScrollTaskTest#testDelayAndRethrottle was getting rejected exceptions
every once in a while. This was reproducible ~20% of the time for me. I
added a CyclicBarrier to prevent the test from shutting down the thread pool
before the threads get finished.
2016-03-31 14:50:14 -04:00
Nik Everett 0c762fca35 Fix test mistake 2016-03-31 12:27:35 -04:00
Nik Everett 7f794e7b77 Test for invalid scroll_size 2016-03-31 12:21:32 -04:00
Nik Everett 78ab6c5b7f [reindex] Dynamic throttle!
This allows the user to update the reindex throttle on the fly, with changes
that speed up the throttling being applied immediately and changes that
slow down the throttling being applied during the next batch. This means
that if a user throttles reindex in such a way that it tries to sleep for
16 years and then realizes that they've done something wrong then they
can change the throttle and reindex will wake up again. We don't apply
slow downs immediately so we never get in danger of losing the scan context.

Also, if reindex is canceled while it is sleeping (how it honor throttling)
then it'll immediately wake up and cancel itself.
2016-03-30 16:40:42 -04:00
Adrien Grand 068c788ec8 Disable fielddata on text fields by defaults. #17386
`text` fields will have fielddata disabled by default. Fielddata can still be
enabled on an existing index by setting `fielddata=true` in the mappings.
2016-03-30 14:35:32 +02:00
Clinton Gormley 3087d2b882 Fixed bad YAML in reindex REST test: 50_routing.yaml 2016-03-29 15:03:09 +02:00
Clinton Gormley 52daed0732 Update-by-query rest tests: fixed bad yaml and deleted a client-dependent test 2016-03-29 14:58:29 +02:00
Clinton Gormley 5f24581de3 The reindex body is now required, which changes the exception thrown by the REST test 2016-03-29 14:09:59 +02:00
Clinton Gormley b87beeb05f Rename update-by-query REST tests to update_by_query 2016-03-29 13:13:49 +02:00
Clinton Gormley 97606850e8 Renamed update-by-query REST spec to update_by_query 2016-03-29 11:45:20 +02:00
Nik Everett 0e6141e675 Replace is_true: took with took >= 0
This prevents tests from failing on machines that can finish the request
less than half a millisecond.
2016-03-28 13:03:48 -04:00
Boaz Leskes 91021e3019 merge from master 2016-03-25 15:50:48 +01:00
Nik Everett 48aaebf23d [reindex] Wait for headers
The test was checking that we'd set the headers properly but in some cases
the request had yet to come in because it was running on another thread.
Now we wait for the headers to show up before failing the test.

Closes #17299
2016-03-24 09:55:49 -04:00
Nik Everett aaa4d57fff [reindex] Don't attempt to refresh on noop
If the user asks for a refresh but their reindex or update-by-query
operation touched no indexes we should just skip the resfresh call
entirely. Without this commit we refresh *all* indexes which is totally
wrong.

Closes #17296
2016-03-23 18:12:40 -04:00
Boaz Leskes 7c8cdf4a71 merged from master 2016-03-22 19:21:28 +01:00
Nik Everett da96b6e41d [reindex] Add thottling support
The throttle is applied when starting the next scroll request so that its
timeout can include the throttle time.
2016-03-22 12:34:14 -04:00
Boaz Leskes 858610d0d1 merge from master 2016-03-19 13:57:40 +01:00
Nik Everett 7197172047 [reindex] Properly register status
Without this commit fetching the status of a reindex from a node that isn't
coordinating the reindex will fail. This commit properly registers reindex's
status so this doesn't happen. To do so it moves all task status registration
into NetworkModule and creates a method to register other statuses which the
reindex plugin calls.
2016-03-16 07:40:49 -04:00
Nik Everett ebc12690bc [reindex] Move refresh tests to unit test
The refresh tests were failing rarely due to refreshes happening
automatically on indexes with -1 refresh intervals. This commit moves
the refresh test into a unit test where we can check if it was attempted
so we never get false failures from background refreshes.

It also stopped refresh from being run if the reindex request was canceled.
2016-03-10 17:48:22 -05:00
Nik Everett b2eec96045 [reindex] Make search failure cause rest failure
Indexing failures have caused the reindex http request to fail for a while
now. Both search and indexing failures cause it to abort. But search
failures didn't cause a non-200 response code from the http api. This
fixes that.

Also slips in a fix to some infrequently failing rest tests.

Closes #16037
2016-03-10 13:47:49 -05:00
Nik Everett b8d931d23c [reindex] Timeout if sub-requests timeout
Sadly, it isn't easy to simulate a timeout during an integration test, you
just have to cause one. Groovy's sleep should do the job.
2016-03-10 13:05:23 -05:00
Boaz Leskes 838c7ddd82 fix indexing compilation issue 2016-03-10 12:12:14 +01:00
Nik Everett 38241a5d8b [reindex] Implement CompositeIndicesRequest
Implements CompositeIndicesRequest on UpdateByQueryRequest and
ReindexRequest so that plugins can reason about the request. In both cases
this implementation is imperfect but useful because instead of listing
all requests that make up the request it instead attempts to make dummy
requests that represent the requests that it will later make.
2016-03-09 16:29:23 -05:00
Nik Everett 6d0efae713 Teach list tasks api to wait for tasks to finish
_wait_for_completion defaults to false. If set to true then the API will
wait for all the tasks that it finds to stop running before returning. You
can use the timeout parameter to prevent it from waiting forever. If you
don't set a timeout parameter it'll default to 30 seconds.

Also adds a log message to rest tests if any tasks overrun the test. This
is just a log (instead of failing the test) because lots of tasks are run
by the cluster on its own and they shouldn't cause the test to fail. Things
like fetching disk usage from the other nodes, for example.

Switches the request to getter/setter style methods as we're going that
way in the Elasticsearch code base. Reindex is all getter/setter style.

Closes #16906
2016-03-08 11:53:57 -05:00
Nik Everett 4d6cb34417 [reindex] Add ingest support 2016-03-04 10:05:13 -05:00
Adrien Grand 2b545df372 Fix modules/reindex to not use the string field anymore. 2016-03-03 11:11:00 +01:00
Nik Everett 18e5bb83c5 Disable problematic reindex test
This should get the builds back to normal while we wait on #16914 or
something like it to fix the test properly.
2016-03-02 13:02:29 -05:00
Nik Everett 942eb70956 Revert "Silence reindex's rest tests"
This reverts commit aa0ef84f5a.
2016-03-02 09:17:06 -05:00
Nik Everett aa0ef84f5a Silence reindex's rest tests
They are failing sporadically in CI.
2016-03-02 08:32:30 -05:00
Nik Everett aeed7ee218 Reindex: rename source to searchRequest
This makes the code easier to read for those familiar with the
Elasticsearch code base.
2016-02-29 14:57:16 -05:00
Nik Everett 18808b7576 Move reindex from a plugin to a module 2016-02-11 17:39:49 -05:00