Commit Graph

18664 Commits

Author SHA1 Message Date
Simon Willnauer 2c978941f5 fix test to wipe lenient index first 2016-01-13 10:03:20 +01:00
Simon Willnauer 050afe9171 add yet another test 2016-01-12 21:17:14 +01:00
Simon Willnauer 5204440471 add more tests and apply feedback from @mikemccand 2016-01-12 21:09:26 +01:00
Simon Willnauer 59211927b6 Move RefreshTask into IndexService and use since task per index
`refresh_interval` is a per index setting but we interpret and maintain it per shard. This
change moves the refresh task outside of IndexShard to the IndexService where it logically belongs
and reuses scheduling infrastructure used for translog fsync (async commit).

This change will use the same task for all shards of an index while previously we used on thread/task
per shard to refresh. This will also prevent too many concurrent refreshes if there are many indices and
shards allocated on a single node.
2016-01-12 17:46:40 +01:00
Michael McCandless b4a095d430 Merge pull request #14121 from mikemccand/fair_indexing_buffers
The indexing buffer on a node (default: 10% of the JVM heap) is now a "shared pool" across all shards on that node.  This way, shards doing intense indexing can use much more than other shards doing only light indexing, and only once the sum of all indexing buffers across all shards exceeds the node's indexing buffer will we ask shards to move recently indexed documents to segments on disk.
2016-01-12 04:59:10 -05:00
Jason Tedor d9fd6e2fe3 Merge pull request #15907 from jasontedor/load-average
Reintroduce five-minute and fifteen-minute load averages on Linux

Relates #12049, relates #14741
2016-01-11 23:43:40 -05:00
Jason Tedor 1de2081ed3 Reintroduce five-minute and fifteen-minute load averages on Linux
This commit reintroduces the five-minute and fifteen-minute load stats
on Linux, and changes the format of the load_average field back to an
array.
2016-01-11 23:42:47 -05:00
Michael McCandless 07e8370322 feedback 2016-01-11 17:16:28 -05:00
Nik Everett b29c416b7c Merge pull request #15848 from nik9000/xlint3
Remove -Xlint:-deprecation from all but core
2016-01-11 16:55:13 -05:00
Nik Everett dfe77879b9 Merge pull request #15850 from nik9000/more_deprecation
Handle some deprecation warnings
2016-01-11 14:23:48 -05:00
Clinton Gormley f782c8ac07 Merge pull request #15891 from sromocki/patch-1
Update crud.asciidoc, nouns mixed up
2016-01-11 20:01:16 +01:00
Nik Everett d8af49eb91 Revert "Fix blended terms for non-strings"
This reverts commit 6bb01984b6.

It caused test failures.
2016-01-11 10:05:48 -05:00
Alexander Reelsen 5bd4582824 Release script: Ensure push to the right region with latest rpm-s3 2016-01-11 14:44:28 +01:00
Nik Everett c2259a2edd Merge pull request #15869 from nik9000/cross_fields_numeric
Fix blended terms for non-strings
2016-01-11 08:19:14 -05:00
Martijn van Groningen ace1b33c3c test: simplify percolator recovery test by only testing what is needed
Running requests via the percolate or mpercolate api is irrelevant.
What is relevant is that when nodes come back that they report the expected number of matches.
2016-01-11 13:38:50 +01:00
Simon Willnauer 55b0eced04 Merge pull request #15882 from s1monw/issues/13278
Check lenient_expand_open after aliases have been resolved
2016-01-11 13:31:45 +01:00
Simon Willnauer d32d16f117 Check lenient_expand_open after aliases have been resolved
We fail today with ClusterBlockExceptions if an alias expands to a closed index
during search since we miss to check the index option down the road after we expanded
aliases.

Closes #13278
2016-01-11 13:31:14 +01:00
Michael McCandless 5e7144f5c5 IMC is now just another IndexingOperationListener 2016-01-11 06:07:19 -05:00
Michael McCandless f3de7783d2 merge master 2016-01-11 05:36:16 -05:00
Michael McCandless db832cce06 improve logging messages a bit 2016-01-11 04:52:26 -05:00
David Pilato 39bc001e33 Merge pull request #15879 from elastic/docs/hdfs
Change link for HDFS support to plugins docs
2016-01-11 10:20:20 +01:00
David Pilato bc181c0310 Fix s3 with hdfs 2016-01-11 09:56:17 +01:00
Clinton Gormley 1d337a1178 Merge pull request #15859 from paulmallet/patch-1
Update aliases.asciidoc
2016-01-11 09:51:15 +01:00
David Pilato c514b26c8b Change link for HDFS support to plugins docs
I came to this change when I read https://github.com/elastic/elasticsearch/pull/15591

HDFS plugin link has not been updated when we moved HDFS to elasticsearch repository (#15192).
2016-01-11 09:43:00 +01:00
Adrien Grand 77dbfbc988 Merge pull request #15864 from jpountz/fix/initial_sizing_of_BytesStreamOutput
Fix initial sizing of BytesStreamOutput.
2016-01-11 09:15:09 +01:00
Clinton Gormley 1cf48b08fb Merge pull request #15534 from jmluy/patch-2
Fix minor typos in inner hits document
2016-01-10 21:52:31 +01:00
Simon Willnauer 6fcb45de60 Merge pull request #15877 from s1monw/make_indexing_memory_controller_private
Make IndexingMemoryController private to IndicesService
2016-01-10 21:33:41 +01:00
Simon Willnauer 2f5e2c09c8 add @Override annotation 2016-01-10 21:09:26 +01:00
Simon Willnauer 91e8d156f5 Make IndexingMemoryController private to IndicesService
This commit detaches the IndexingMemoryController from guice and moves
it's creation and closing into IndicesService where it logically belongs.
2016-01-10 20:51:31 +01:00
Simon Willnauer 37f6b5ced7 Merge pull request #15875 from s1monw/cleanup_indexing_operation_listeners
Cleanup IndexingOperationListeners infrastructure
2016-01-10 20:22:42 +01:00
Simon Willnauer 80ef201514 Remove dead code and add missing @Override annotations 2016-01-10 20:04:25 +01:00
Jason Tedor 7873dddab6 Merge pull request #15874 from jasontedor/index-writer-is-locked-be-gone
Remove and forbid use of IndexWriter#isLocked

Closes #15846
2016-01-10 08:32:41 -05:00
Jason Tedor abaf816d00 Remove and forbid use of IndexWriter#isLocked
This commit removes and now forbids use of
org.apache.lucene.index.IndexWriter#isLocked as this method was
deprecated in LUCENE-6508. The deprecation is due to the fact that
checking if a lock is held before acquiring that lock is subject to a
time-of-check-to-time-of-use race condition. There were three uses of
IndexWriter#isLocked in the code base:
 - a logging statement in o.e.i.e.InternalEngine where we are already in
   an exceptional condition that the lock was held; in this case,
   logging whether or not the directory is locked is superfluous
 - in o.e.c.l.u.VersionsTests where we were verifying that a write lock
   is released upon closing an IndexWriter; in this case, the check is
   not needed as successfully closing an IndexWriter releases its
   write lock
 - in o.e.t.s.MockFSDirectoryService where we were verifying that a
   directory is not write-locked before (implicitly) trying to obtain
   such a write lock in org.apache.lucene.index.CheckIndex#<init> (this
   is the exact type of a situation that is subject to a race
   condition); in this case we can proceed by just (implicitly) trying
   to obtain the write lock and failing if we encounter a
   LockObtainFailedException
2016-01-10 08:28:44 -05:00
Clinton Gormley 3c58a93a6e Merge pull request #15856 from marcbachmann/patch-1
Document index deletion using comma separated indices
2016-01-10 10:48:32 +01:00
Clinton Gormley 9773cca58e Merge pull request #15870 from rjruizes/patch-1
fix nested multi-value query
2016-01-10 10:06:40 +01:00
Simon Willnauer 54d1e35d84 Cleanup IndexingOperationListeners infrastructure
This commit reduces the former ShardIndexinService to a simple stats/metrics
class, moves IndexingSlowLog to the IndexService level since it can be shared
across shards of an index and is now hidden behind IndexingOperationListener.

IndexingOperationListener is now a first class citizen in IndexShard and is passed
in from IndexService.
2016-01-09 21:50:17 +01:00
Jason Tedor 37880040a2 Remove unused imports from o/e/i/e/InternalEngine.java 2016-01-09 12:16:09 -05:00
Simon Willnauer 2dbad1d65a Merge pull request #15841 from simonw/patch-1
Fixed broken link on from-size documentation page
2016-01-08 23:51:18 +01:00
Nik Everett 4772b1fe9d Remove tests for backwards compatibility we don't support any more 2016-01-08 17:25:26 -05:00
Nik Everett d8526f4d00 Handle some deprecation warnings
Suppress lots of them with comments. Fix a few. Removes lots of pre-built
stuff used to support pre-2.0 indexes.
2016-01-08 17:17:33 -05:00
Nik Everett 6bb01984b6 Fix blended terms for non-strings
It had some funky errors, like lenient:true not working and queries with
two integer fields blowing up if there was no analyzer defined on the
query. This throws a bunch more tests at it and rejiggers how non-strings
are handled so they don't wander off into scary QueryBuilder-land unless
they have a nice strong analyzer to protect them.

Closes #15860
2016-01-08 16:53:40 -05:00
Michael McCandless d3d41fc22b Merge pull request #15585 from mikemccand/max_indexing_thread_pool_size
Limit the max size of bulk and index thread pools to bounded number of processors
2016-01-08 14:36:21 -05:00
Adrien Grand 8568480a74 Fix initial sizing of BytesStreamOutput.
It currently tries to align to the page size (16KB) by default. However, this
might waste a significant memory (if many BytesStreamOutputs are allocated)
and is also useless given that BytesStreamOutput does not recycle (on the
contrary to ReleasableBytesStreamOutput). So the initial size has been changed
to 0.

Closes #15789
2016-01-08 20:08:31 +01:00
Nik Everett 6250f4dbaa Remove deprecated azure settings 2016-01-08 13:13:14 -05:00
Jason Tedor 0b1a7fc2b8 Merge pull request #15862 from jasontedor/thread-local-random-be-gone
Remove and forbid use of j.u.c.ThreadLocalRandom

Closes #15294
2016-01-08 12:59:59 -05:00
Jason Tedor 871d1b4885 Remove and forbid use of j.u.c.ThreadLocalRandom
This commit removes and now forbids all uses of
java.util.concurrent.ThreadLocalRandom across the codebase. The
underlying issue with ThreadLocalRandom is that it can not be
seeded. This means that if ThreadLocalRandom is used in production code,
then tests that cover any code path containing ThreadLocalRandom will be
prevented from being reproducible by use of ThreadLocalRandom. Instead,
using org.elasticsearch.common.random.Randomness#get will give
reproducible sources of random when running under tests and otherwise
still give an instance of ThreadLocalRandom when running as production
code.
2016-01-08 12:23:48 -05:00
Jason Tedor 349352c85a Merge pull request #15861 from jasontedor/unboxed-math-utils-is-dead
Remove dead o.e.c.m.UnboxedMathUtils
2016-01-08 12:23:28 -05:00
Jason Tedor 21f5b0ff35 Remove dead o.e.c.m.UnboxedMathUtils
This commit removes the dead UnboxedMathUtils from the codebase.
2016-01-08 11:58:39 -05:00
Nik Everett 98fdb39d3d Remove deprecated settings 2016-01-08 11:17:56 -05:00
Simon Willnauer 71796e2319 [TEST] Close failable translog in a controlled way otherwise assertions are off in the test 2016-01-08 13:10:09 +01:00