Commit Graph

111 Commits

Author SHA1 Message Date
Jason Tedor d09d89f8c5 Remove only node preference
This commit removes the search preference _only_node as the same
functionality can be obtained by using the search preference
_only_nodes. This commit also adds a test that ensures that _only_nodes
will continue to support specifying node IDs.

Relates #18875
2016-06-17 15:27:46 -04:00
Jason Tedor 245def80f0 Add note that thread pool settings are node-level
This commit adds a note to the breaking changes docs that since commit
da74323141, thread pool settings are no
longer cluster-level settings and thus not dynamically updatable.
2016-06-17 15:19:52 -04:00
Jim Ferenczi fb2a48d0f0 Revert "Remove support for sorting terms aggregation by ascending count"
This is delayed after alpha4 since Kibana relies on it.
2016-06-17 17:14:01 +02:00
Jim Ferenczi 755721953b Remove support for sorting terms aggregation by ascending count
closes #17614
2016-06-17 15:06:49 +02:00
Jason Tedor e96722d91c Add search preference to prefer multiple nodes
The search preference _prefer_node allows specifying a single node to
prefer when routing a request. This functionality can be enhanced by
permitting multiple nodes to be preferred. This commit replaces the
search preference _prefer_node with the search preference _prefer_nodes
which supplants the former by specifying a single node and otherwise
adds functionality.

Relates #18872
2016-06-14 21:34:24 -04:00
Colin Goodheart-Smithe cfd3356ee3 Remove size 0 options in aggregations
This removes the ability to set `size: 0` in the `terms`, `significant_terms` and `geohash_grid` aggregations for the reasons described in https://github.com/elastic/elasticsearch/issues/18838

Closes #18838
2016-06-14 13:07:02 +01:00
Nik Everett 4b21157906 Remove setRefresh
It has been replaced with `setRefreshPolicy` which has support for
waiting until refresh with `setRefreshPolicy(WAIT_FOR)`.

Related to #1063
2016-06-08 13:50:59 -04:00
Lee Hinman c637fea84b Change the default of `include_global_state` from true to false for restores
This changes the default value to be false *only* for restore operations.

Resolves #18569
2016-06-08 10:48:36 -06:00
Lee Hinman 762bbdbd0c Revert "Change the default of `include_global_state` from true to false."
This reverts commit 052a62250c.
2016-06-07 15:07:37 -06:00
Lee Hinman 052a62250c Change the default of `include_global_state` from true to false.
Resolves #18569
2016-06-07 15:06:20 -06:00
Lee Hinman 32bd869b28 Merge remote-tracking branch 'dakrone/no-cluster-name-in-path' 2016-06-07 10:14:23 -06:00
Lee Hinman feb244c14a Remove cluster name from data path
Previously Elasticsearch used $DATA_DIR/$CLUSTER_NAME/nodes for the path
where data is stored, this commit changes that to be $DATA_DIR/nodes.

On startup, if the old folder structure is detected it will be used.
This behavior will be removed in Elasticsearch 6.0

Resolves #17810
2016-06-07 10:13:48 -06:00
Jason Tedor da74323141 Register thread pool settings
This commit refactors the handling of thread pool settings so that the
individual settings can be registered rather than registering the top
level group. With this refactoring, individual plugins must now register
their own settings for custom thread pools that they need, but a
dedicated API is provided for this in the thread pool module. This
commit also renames the prefix on the thread pool settings from
"threadpool" to "thread_pool". This enables a hard break on the settings
so that:
 - some of the settings can be given more sensible names (e.g., the max
   number of threads in a scaling thread pool is now named "max" instead
   of "size")
 - change the soft limit on the number of threads in the bulk and
   indexing thread pools to a hard limit
 - the settings names for custom plugins for thread pools can be
   prefixed (e.g., "xpack.watcher.thread_pool.size")
 - remove dynamic thread pool settings

Relates #18674
2016-06-06 22:09:12 -04:00
Nik Everett d8056c8213 Add support for waiting until a refresh occurs
This adds support for setting the refresh request parameter to
`wait_for` in the `index`, `delete`, `update`, and `bulk` APIs. When
`refresh=wait_for` is set those APIs will not return until their
results have been made visible to search by a refresh.

Also it adds a `forced_refresh` field to the response of `index`,
`delete`, `update`, and to each item in a bulk response. This will
be true for requests with `?refresh` or `?refresh=true` and will be
true for some requests (see below) with `refresh=wait_for` but ought
to otherwise always be false.

`refresh=wait_for` is implemented as a list of
`Tuple<Translog.Location, Consumer<Boolean>>`s in the new `RefreshListeners`
class that is managed by `IndexShard`. The dynamic, index scoped
`index.max_refresh_listeners` setting controls a maximum number of
listeners allowed in any shard. If more than that many listeners
accumulate in the engine then a refresh will be forced, the thread that
adds the listener will be blocked until the refresh completes, and then the
listener will be called with a `forcedRefresh` flag so it knows that it was
the "straw that broke the camel's back". These listeners are only used by
`refresh=wait_for` and that flag manifests itself as `forced_refresh` being
`true` in the response.

About half of this change comes from piping async-ness down to the appropriate
layer in a way that is compatible with the ongoing with with sequence ids.

Closes #1063

You can look up the winding story of all the commits here:
https://github.com/elastic/elasticsearch/pull/17986

Here are the commit messages in case they are intersting to you:
commit 59a753b89109828d2b8f0de05cb104fc663cf95e
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon Jun 6 10:18:23 2016 -0400

    Replace a method reference with implementing an interface

    Saves a single allocation and forces more commonality
    between the WriteResults.

commit 31f7861a85b457fb7378a6f27fa0a0c171538f68
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon Jun 6 10:07:55 2016 -0400

    Revert "Replace static method that takes consumer with delegate class that takes an interface"

    This reverts commit 777e23a6592c75db0081a53458cc760f4db69507.

commit 777e23a6592c75db0081a53458cc760f4db69507
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon Jun 6 09:29:35 2016 -0400

    Replace static method that takes consumer with delegate class that takes an interface

    Same number of allocations, much less code duplication.

commit 9b49a480ca9587a0a16ebe941662849f38289644
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon Jun 6 08:25:38 2016 -0400

    Patch from boaz

commit c2bc36524fda119fd0514415127e8901d94409c8
Author: Nik Everett <nik9000@gmail.com>
Date:   Thu Jun 2 14:46:27 2016 -0400

    Fix docs

    After updating to master we are actually testing them.

commit 03975ac056e44954eb0a371149d410dcf303e212
Author: Nik Everett <nik9000@gmail.com>
Date:   Thu Jun 2 14:20:11 2016 -0400

    Cleanup after merge from master

commit 9c9a1deb002c5bebb2a997c89fa12b3d7978e02e
Author: Nik Everett <nik9000@gmail.com>
Date:   Thu Jun 2 14:09:14 2016 -0400

    Breaking changes notes

commit 1c3e64ae06c07a85f7af80534fab88279adb30b4
Merge: 9e63ad6 f67e580
Author: Nik Everett <nik9000@gmail.com>
Date:   Thu Jun 2 14:00:05 2016 -0400

    Merge branch 'master' into block_until_refresh2

commit 9e63ad6de52d0b28f0b6d7203721baf1ebf6f56b
Author: Nik Everett <nik9000@gmail.com>
Date:   Thu Jun 2 13:21:27 2016 -0400

    Test for TransportWriteAction

commit 522ecb59d39b3c9e8df0d3b8df34b9e7aeaf0ce9
Author: Nik Everett <nik9000@gmail.com>
Date:   Thu Jun 2 10:30:18 2016 -0400

    Document deprecation

commit 0cd67b947f58867e704a1f0e66928a6fb5a11f11
Author: Nik Everett <nik9000@gmail.com>
Date:   Thu Jun 2 10:26:23 2016 -0400

    Deprecate setRefresh(boolean)

    Users should use `setRefresh(RefreshPolicy)` instead.

commit aeb1be3f2c501990b33fb1f8230d496035f498ef
Author: Nik Everett <nik9000@gmail.com>
Date:   Thu Jun 2 10:12:27 2016 -0400

    Remove checkstyle suppression

    It is fixed

commit 00d09a9caa638b6f90f4896b5502dd98d8fad56e
Author: Nik Everett <nik9000@gmail.com>
Date:   Thu Jun 2 10:08:28 2016 -0400

    Improve comment

commit 788164b898a6ee2878a273961230122b7386c3c9
Author: Nik Everett <nik9000@gmail.com>
Date:   Thu Jun 2 10:01:01 2016 -0400

    S/ReplicatedWriteResponse/WriteResponse/

    Now it lines up with WriteRequest.

commit b74cf3fe778352b140355afcaa08d3d4412d749d
Author: Nik Everett <nik9000@gmail.com>
Date:   Wed Jun 1 18:27:52 2016 -0400

    Preserve `?refresh` behavior

    `?refresh` means the same things as `?refresh=true`.

commit 30f972bdaeaaa0de6fe67746cdb8628aa86f5a8c
Author: Nik Everett <nik9000@gmail.com>
Date:   Wed Jun 1 17:39:05 2016 -0400

    Handle hanging documents

    If a document is added to the index during a refresh we weren't properly
    firing its refresh listener. This happened because the way we detect
    whether a refresh makes something visible or not is imperfect. It is
    ok because it always errs on the side of thinking that something isn't
    yet visible.

    So when a document arrives during a refresh the refresh listeners
    won't think it made it into a refresh when, often, it does. The way
    we work around this is by telling Elasticsearch that it ought to
    trigger a refresh if there are any pending refresh listeners even
    if there aren't pending documents to update. Lucene short circuits
    the refresh so it doesn't take that much effort, but the refresh
    listeners still get the signal that a refresh has come in and they
    still pick up the change and notify the listener.

    This means that the time that a listener can wait is actually slightly
    longer than the refresh interval.

commit d523b5702b60c7ba309fb0dcf3cd3a4798f11960
Author: Nik Everett <nik9000@gmail.com>
Date:   Wed Jun 1 14:34:01 2016 -0400

    Explain Integer.MAX_VALUE

commit 4ffb7c0e954343cc1c04b3d7be2ebad66d3a016b
Author: Nik Everett <nik9000@gmail.com>
Date:   Wed Jun 1 14:27:39 2016 -0400

    Fire all refresh listeners in a single thread

    Rather than queueing a runnable each.

commit 19606ec3bbe612095df45eba734c5b7eb2709c01
Author: Nik Everett <nik9000@gmail.com>
Date:   Wed Jun 1 14:09:52 2016 -0400

    Assert translog ordering

commit 6bb4e5c75e850f4a42518f06fbc955f7ec76d245
Author: Nik Everett <nik9000@gmail.com>
Date:   Wed Jun 1 13:17:44 2016 -0400

    Support null RefreshListeners in InternalEngine

    Just skip using it.

commit 74be1480d6e44af2b354ff9ea47c234d4870b6c2
Author: Nik Everett <nik9000@gmail.com>
Date:   Tue May 31 18:02:03 2016 -0400

    Move funny ShardInfo hack for bulk into bulk

    This should make it easier to understand because it is closer to where it
    matters....

commit 2b771f8dabd488e056cfdc9989608d18264ddfb0
Author: Nik Everett <nik9000@gmail.com>
Date:   Tue May 31 17:39:46 2016 -0400

    Pull listener out into an inner class with javadoc and stuff

commit 058481ad72019c0492b03a7a4ac32a48673697d3
Author: Nik Everett <nik9000@gmail.com>
Date:   Tue May 31 17:33:42 2016 -0400

    Fix javadoc links

commit d2123b1cabf29bce8ff561d4a4c1c1d5b42bccad
Author: Nik Everett <nik9000@gmail.com>
Date:   Tue May 31 17:28:09 2016 -0400

    Make more stuff final

commit 8453fc4f7850f6a02fb5971c17a942a3e3fd9f7b
Author: Nik Everett <nik9000@gmail.com>
Date:   Tue May 31 17:26:48 2016 -0400

    Javadoc

commit fb16d2fc7016c1e8e1621d481e8781c7ef43326c
Author: Nik Everett <nik9000@gmail.com>
Date:   Tue May 31 16:14:48 2016 -0400

    Rewrite refresh docs

commit 5797d1b1c4d233c0db918c0d08c21731ddccd05e
Author: Nik Everett <nik9000@gmail.com>
Date:   Tue May 31 15:02:34 2016 -0400

    Fix forced_refresh flag

    It wasn't being set.

commit 43ce50a1de250a9e073a2ca6cbf55c1b4c74b11b
Author: Nik Everett <nik9000@gmail.com>
Date:   Tue May 31 14:02:56 2016 -0400

    Delay translog sync and flush until after refresh

    The sync might have occurred for us during the refresh so we
    have less work to do. Maybe.

commit bb2739202e084703baf02cfa58f09517598cf14e
Author: Nik Everett <nik9000@gmail.com>
Date:   Tue May 31 13:08:08 2016 -0400

    Remove duplication in WritePrimaryResult and WriteReplicaResult

commit 2f579f89b4867a880396f2e7fcffc508449ff2de
Author: Nik Everett <nik9000@gmail.com>
Date:   Tue May 31 12:19:05 2016 -0400

    Clean up registration of RefreshListeners

commit 87ab6e60ca5ba945bf0fba84784b2bbe53506abf
Author: Nik Everett <nik9000@gmail.com>
Date:   Tue May 31 11:28:30 2016 -0400

    Shorten lock time in RefreshListeners

    Also use null to represent no listeners rather than an empty list.
    This saves allocating a new ArrayList every refresh cycle on every
    index.

commit 0d49d9c5720dadfb67da3fa760397bf6d874601c
Author: Nik Everett <nik9000@gmail.com>
Date:   Tue May 24 10:46:18 2016 -0400

    Flip relationship between RefreshListeners and Engine

    Now RefreshListeners comes to Engine from EngineConfig.

commit b2704b8a39382953f8f91a9743e894ee289f7514
Author: Nik Everett <nik9000@gmail.com>
Date:   Tue May 24 09:37:58 2016 -0400

    Remove unused imports

    Maybe I added them?

commit 04343a22647f19304d9dc716b3fac9b183227f63
Author: Nik Everett <nik9000@gmail.com>
Date:   Tue May 24 09:37:52 2016 -0400

    Javadoc

commit da1e765678890a02d61d8a29aa433274beb5e00c
Author: Nik Everett <nik9000@gmail.com>
Date:   Tue May 24 09:26:35 2016 -0400

    Reply with non-null

    Also move the fsync and flush to before the refresh listener stuff.

commit 5d8eecd0d904b497844b4c81c46477bd6178ed3a
Author: Nik Everett <nik9000@gmail.com>
Date:   Tue May 24 08:58:47 2016 -0400

    Remove funky synchronization in AsyncReplicaAction

commit 1ec71eea0f4e1228ae1497d982307be818ef4b65
Author: Nik Everett <nik9000@gmail.com>
Date:   Tue May 24 08:01:14 2016 -0400

    s/LinkedTransferQueue/ArrayList/

commit 7da36a4ceed2ccf7955138c3b005237fa41efcb4
Author: Nik Everett <nik9000@gmail.com>
Date:   Tue May 24 07:46:38 2016 -0400

    More cleanup for RefreshListeners

commit 957e9b77007c32ee75dde152c6622bab065d5993
Author: Nik Everett <nik9000@gmail.com>
Date:   Tue May 24 07:34:13 2016 -0400

    /Consumer<Runnable>/Executor/

commit 4d8bf5d4a70dcc56150c8d8d14165cd23d308b3c
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon May 23 22:20:42 2016 -0400

    explain

commit 15d948a348089bb2937eec5ac4e96f3ec67dbe32
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon May 23 22:17:59 2016 -0400

    Better....

commit dc28951d02973fc03b4d51913b5f96de14b75607
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon May 23 21:09:20 2016 -0400

    Javadocs and compromises

commit 8eebaa89c0a1ee74982fbe0d56d1485ca2ae09db
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon May 23 20:52:49 2016 -0400

    Take boaz's changes to their logic conclusion and unbreak important stuff like bulk

commit 7056b96ea412f275005b93e3570bcff895859ed5
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon May 23 15:49:32 2016 -0400

    Patch from boaz

commit 87be7eaed09a274cc6a99d1a3da81d2d7bf9dd64
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon May 23 15:49:13 2016 -0400

    Revert "Move async parts of replica operation outside of the lock"

    This reverts commit 13807ad10b6f5ecd39f98c9f20874f9f352c5bc2.

commit 13807ad10b6f5ecd39f98c9f20874f9f352c5bc2
Author: Nik Everett <nik9000@gmail.com>
Date:   Fri May 20 22:53:15 2016 -0400

    Move async parts of replica operation outside of the lock

commit b8cadcef565908b276484f7f5f988fd58b38d8b6
Author: Nik Everett <nik9000@gmail.com>
Date:   Fri May 20 16:17:20 2016 -0400

    Docs

commit 91149e0580233bf79c2273b419fe9374ca746648
Author: Nik Everett <nik9000@gmail.com>
Date:   Fri May 20 15:17:40 2016 -0400

    Finally!

commit 1ff50c2faf56665d221f00a18d9ac88745904bf5
Author: Nik Everett <nik9000@gmail.com>
Date:   Fri May 20 15:01:53 2016 -0400

    Remove Translog#lastWriteLocation

    I wasn't being careful enough with locks so it wasn't right anyway.
    Instead this builds a synthetic Tranlog.Location when you call
    getWriteLocation with much more relaxed equality guarantees. Rather
    than being equal to the last Translog.Location returned it is
    simply guaranteed to be greater than the last translog returned
    and less than the next.

commit 55596ea68b5484490c3637fbad0d95564236478b
Author: Nik Everett <nik9000@gmail.com>
Date:   Fri May 20 14:40:06 2016 -0400

    Remove listener from shardOperationOnPrimary

    Create instead asyncShardOperationOnPrimary which is called after
    all of the replica operations are started to handle any async
    operations.

commit 3322e26211bf681b37132274ee158ae330afc28b
Author: Nik Everett <nik9000@gmail.com>
Date:   Tue May 17 17:20:02 2016 -0400

    Increase default maximum number of listeners to 1000

commit 88171a8322a424e624d48960fb4c98dd43e4d671
Author: Nik Everett <nik9000@gmail.com>
Date:   Tue May 17 16:40:57 2016 -0400

    Rename test

commit 179c27c4f829f2c6ded65967652cf85adaf2ae52
Author: Nik Everett <nik9000@gmail.com>
Date:   Tue May 17 16:35:27 2016 -0400

    Move refresh listeners into their own class

    They still live at the IndexShard level but they live on their
    own in RefreshListeners which interacts with IndexShard using a
    couple of callbacks and a registration method. This lets us test
    the listeners without standing up an entire IndexShard. We still
    test the listeners against an InternalEngine, because the interplay
    between InternalEngine, Translog, and RefreshListeners is complex
    and important to get right.

commit d8926d5fc1d24b4da8ccff7e0f0907b98c583c41
Author: Nik Everett <nik9000@gmail.com>
Date:   Tue May 17 11:02:38 2016 -0400

    Move refresh listeners into IndexShard

commit df91cde398eb720143a85a8c6fa19bdc3a74e07d
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon May 16 16:01:03 2016 -0400

    unused import

commit 066da45b08148b266e4173166662fc1b3f66ed53
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon May 16 15:54:11 2016 -0400

    Remove RefreshListener interface

    Just pass a Translog.Location and a Consumer<Boolean> when registering.

commit b971d6d3301c7522b2e7eb90d5d8dd96a77fa625
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon May 16 14:41:06 2016 -0400

    Docs for setForcedRefresh

commit 6c43be821eaf61141d3ec520f988aad3a96a3941
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon May 16 14:34:39 2016 -0400

    Rename refresh setter and getter

commit e61b7391f91263a4c4d6107bfbc2a828bbcc805c
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon Apr 25 22:48:09 2016 -0400

    Trigger listeners even when there is no refresh

    Each refresh gives us an opportunity to pick up any listeners we may
    have left behind.

commit 0c9b0477085c021f503db775640d25668e02f635
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon Apr 25 20:30:06 2016 -0400

    REST

commit 8250343240de7e63118c663a230a7a314807a754
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon Apr 25 19:34:22 2016 -0400

    Switch to estimated count

    We don't need a linear time count of the number of listeners - a volatile
    variable is good enough to guess. It probably undercounts more than it
    overcounts but it isn't a huge problem.

commit bd531167fe54f1bde6f6d4ddb0a8de5a7bcc18a2
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon Apr 25 18:21:02 2016 -0400

    Don't try and set forced refresh on bulk items without a response

    NullPointerExceptions are bad. If the entire request fails then the user
    has worse problems then "did these force a refresh".

commit bcfded11515af5e0b3c3e36f3c2f73f5cd26512e
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon Apr 25 18:14:20 2016 -0400

    Replace LinkedList and synchronized with LinkedTransferQueue

commit 8a80cc70a76375a7593745884cb987535b37ca80
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon Apr 25 17:38:24 2016 -0400

    Support for update

commit 1f36966742f851b7328015151ef6fc8f95299af2
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon Apr 25 15:46:06 2016 -0400

    Cleanup translog tests

commit 8d121bf35eb265b8a0aee9710afeb1b054a113d4
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon Apr 25 15:40:53 2016 -0400

    Cleanup listener implementation

    Much more testing too!

commit 2058f4a808762c4588309f21b13b677245832f2c
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon Apr 25 11:45:55 2016 -0400

    Pass back information about whether we refreshed

commit e445cb0cb91ebdbcfdbf566696edb2bf1c84a882
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon Apr 25 11:03:31 2016 -0400

    Javadoc

commit 611cbeeaeb458f4b428bfc43a1ee6652adf4baff
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon Apr 25 11:01:40 2016 -0400

    Move ReplicationResponse

    now it is in the same package as its request

commit 9919758b644fd73895fb88cd6a4909a8387eb2e2
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon Apr 25 11:00:14 2016 -0400

    Oh boy that wasn't working

commit 247cb483c4459dea8e95e0e3bd2e4bf8d452c598
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon Apr 25 10:29:37 2016 -0400

    Basic block_until_refresh exposed to java client

    and basic "is it plugged in" style tests.

commit 46c855c9971cb2b748206d2afa6a2d88724be3ba
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon Apr 25 10:11:10 2016 -0400

    Move test to own class

commit a5ffd892d0a352ae7e9757f2640fc2a1fa656bf2
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon Apr 25 07:44:25 2016 -0400

    WIP

commit 213bebb6ece11b85d17e44af9a54fc2e5e332d39
Author: Nik Everett <nik9000@gmail.com>
Date:   Fri Apr 22 21:35:52 2016 -0400

    Add refresh listeners

commit a2bc7f30e6d4857a1224ef5a89909b36c8f33731
Author: Nik Everett <nik9000@gmail.com>
Date:   Fri Apr 22 21:11:55 2016 -0400

    Return last written location from refresh

commit 85033a87551da89f36a23d4dfd5016db218e08ee
Author: Nik Everett <nik9000@gmail.com>
Date:   Fri Apr 22 20:28:21 2016 -0400

    Never reply to replica actions while you have the operation lock

    This last thing was causing periodic test failures because we were
    replying while we had the operation lock. Now, we probably could get
    away with that in most cases but the tests don't like it and it isn't
    a good idea to do network io while you have a lock anyway. So this
    prevents it.

commit 1f25cf35e796835b3827b8a4110e09e5de61784c
Author: Nik Everett <nik9000@gmail.com>
Date:   Fri Apr 22 19:56:18 2016 -0400

    Cleanup

commit 52c5f7c3f04710901f503334239a611c0e21c85a
Author: Nik Everett <nik9000@gmail.com>
Date:   Fri Apr 22 19:33:00 2016 -0400

    Add a listener to shard operations

commit 5b142dc331214c8eef90587144f4b3f959f9eced
Author: Nik Everett <nik9000@gmail.com>
Date:   Fri Apr 22 18:03:52 2016 -0400

    Cleanup

commit 3d22b2d7ceb473db339259452a7c4f117ce86069
Author: Nik Everett <nik9000@gmail.com>
Date:   Fri Apr 22 17:59:55 2016 -0400

    Push the listener into shardOperationOnPrimary

commit 34b378943b8185451acf6350f661c0ad33b5836d
Author: Nik Everett <nik9000@gmail.com>
Date:   Fri Apr 22 17:48:47 2016 -0400

    Doc

commit b42b8da968d42cc7414020c7b199606a5dcce50a
Author: Nik Everett <nik9000@gmail.com>
Date:   Fri Apr 22 17:45:40 2016 -0400

    Don't finish early if the primary finishes early

    We use a "fake" pending shard that we resolve when the replicas have
    all started.

commit 0fc045b56e1e02a48c30383ac50a281d5af7e0b6
Author: Nik Everett <nik9000@gmail.com>
Date:   Fri Apr 22 17:30:06 2016 -0400

    Make performOnPrimary asyncS

    Instead of returning Tuple<Response, ReplicaRequest> it returns
    ReplicaRequest and takes a ActionListener<Response> as an argument.
    We call the listener immediately to preserve backwards compatibility
    for now.

commit 80119b9a26ede96a865af45904c3ac69d5b19b59
Author: Nik Everett <nik9000@gmail.com>
Date:   Fri Apr 22 16:51:53 2016 -0400

    Factor out common code in shardOperationOnPrimary

commit 0642083676702618f900fa842c08802a04c1a53e
Author: Nik Everett <nik9000@gmail.com>
Date:   Fri Apr 22 16:32:29 2016 -0400

    Factor out common code from shardOperationOnReplica

commit 8bdc415fedaaa9f2d0c555590a13ec4699a7c3f7
Author: Nik Everett <nik9000@gmail.com>
Date:   Fri Apr 22 16:23:28 2016 -0400

    Create ReplicatedMutationRequest

    Superclass for index, delete, and bulkShard requests.

commit 0f8fa846a2822c4293df32fed18c9b99660b39ff
Author: Nik Everett <nik9000@gmail.com>
Date:   Fri Apr 22 16:10:30 2016 -0400

    Create TransportReplicatedMutationAction

    It is the superclass of replication actions that mutate data: index, delete,
    and shardBulk. shardFlush and shardRefresh are replication actions but they
    do not extend TransportReplicatedMutationAction because they don't change
    the data, only shuffle it around.
2016-06-06 11:37:53 -04:00
Jason Tedor 8e2a7d0fe1 Rename boostrap.mlockall to bootstrap.memory_lock
The setting bootstrap.mlockall is useful on both POSIX-like systems
(POSIX mlockall) and Windows (Win32 VirtualLock). But mlockall is really
a POSIX only thing so the name should not be tied POSIX. This commit
renames the setting to "bootstrap.memory_lock".

Relates #18669
2016-06-01 16:25:51 -04:00
Martijn van Groningen 0e9f3addd2 Nested inner hits shouldn't use relative paths
Like on other places in the query dsl the full field name should be used.
Before this change this wasn't the case for nested inner hits when source filtering was used.
Highlighting has a workaround, which is now removed as the source of nested inner hits can only be refered by the full name.

Closes #16653
2016-05-27 13:41:45 +02:00
Jason Tedor d29844e597 Remove custom plugins path
This commit removes the ability to specify a custom plugins
path. Instead, the plugins path will always be a subdirectory called
"plugins" off of the home directory.
2016-05-26 10:16:25 -04:00
Nik Everett a93f578bf6 Move parsing of allocation commands into REST
Port them to the ObjectParser.

Don't let plugins register custom allocation commands
2016-05-24 11:59:05 -04:00
Nik Everett 72eb621bce Docs: Replace [source,json] with [source,js]
The syntax highlighter only supports [source,js].

Also adds a check to the rest test generator that runs during
the build that'll fail the build if it sees `[source,json]`.
2016-05-24 11:17:27 -04:00
Tanguy Leroux 1f011f9dea Remove Delete-By-Query plugin
closes #18469
2016-05-24 13:28:20 +02:00
Martijn van Groningen 27cc2fe4dc Moved the percolator from core to its own module
Significant changes:
* AbstractQueryTestCase has moved to the test framework module, in order for query builder tests in modules and plugins
* Added support to AbstractQueryTestCase to register plugins
* Lift the restriction that only one percolator could be added per index. This validation existed in MapperService, but because the percolator moved to a module it could no longer exist there. Instead of bringing it back it was removed. This validation existed since the percolator cache only supported one percolator query per document, since the percolator cache has been removed this restriction could removed as well.
* While moving percolator tests to the new module, also removed a couple of tests for the deprecated percolate and mpercolate api. These APIs are now sugar  APIs for bwc and rediect to the searvh and msearvh APIs. Some tests were still testing as if percolate and mpercolate API did the percolation, but this no longer the case and these tests could be removed.
2016-05-24 11:01:57 +02:00
Martijn van Groningen 80fee8666f percolator: Removed percolator cache
Before 5.0 for it was required that the percolator queries were cached in jvm heap as Lucene queries for two reasons:
1) Performance. The percolator evaluated all percolator queries all the time. There was no pre-selecting queries that are likely to match like we have today.
2) Updates made to percolator queries were visible in realtime, Today these changes are visible in near realtime. So updating no longer requires the percolator to have the queries in jvm heap.

So having the percolator queries in jvm heap via the percolator cache is now less attractive. Especially when there are many percolator queries then these queries can consume many GBs of jvm heap.
Removing the percolator cache does make the percolate query slower compared to how the execution time in 5.0.0-alpha1 and alpha2, but it is still faster compared to 2.x and before.
2016-05-20 14:52:16 +02:00
Christoph Büscher d3fe22c990 Improve adding clauses to `span_near` and `span_or` query
Currently the query builders expose the clauses of the span
query as a modifiable list. Instead we should make the that
getter return an unmodifiable list. Also renaming the method
used to add a clause from `clause(spanQuery)` to
`addClause(spanQuery)`.
2016-05-20 13:36:55 +02:00
Jason Tedor c257e2c51f Remove settings and system properties entanglement
Today when parsing settings during bootstrap, we add a system property
for every Elasticsearch setting. Additionally, settings can be set via
system properties. This commit simplifies this situation.
 - settings are no longer propogated to system properties
 - system properties can not be used to set settings
 - the "es." prefix on settings is no longer required (nor permitted)
 - test logging has a dedicated system property (tests.logger.level)

Relates #18198
2016-05-19 14:08:08 -04:00
Simon Willnauer d77c299cb9 Register `indices.query.bool.max_clause_count` setting (#18341)
* Register `indices.query.bool.max_clause_count` setting

This commit registers `indices.query.bool.max_clause_count` as a node
level setting and removes support for its synonym setting
`index.query.bool.max_clause_count`.

Closes #18336
2016-05-19 10:42:35 +02:00
Nik Everett fe4823eae0 Reindex should retry on search failures
This uses the same backoff policy we use for bulk and just retries until
the request isn't rejected.

Instead of `{"retries": 12}` in the response to count retries this now
looks like `{"retries": {"bulk": 12", "search": 1}`.

Closes #18059
2016-05-17 13:58:45 -04:00
Colin Goodheart-Smithe e2367b2f12 [DOCS] Updates the documentation for the recent changes in the profiler 2016-05-17 09:58:30 +01:00
Adrien Grand 864ed04059 Lessen leniency of the query dsl. #18276
This change does the following:
 - Queries that are currently unsupported such as prefix queries on numeric
   fields or term queries on geo fields now throw an error rather than returning
   a query that does not match anything.
 - Fuzzy queries on numeric, date and ip fields are now unsupported: they used
   to create range queries, we now expect users to use range queries directly.
   Fuzzy, regexp and prefix queries are now only supported on text/keyword
   fields (including `_all`).
 - The `_uid` and `_id` fields do not support prefix or range queries anymore as
   it would prevent us to store them more efficiently in the future, eg. by
   using a binary encoding.

Note that it is still possible to ignore these errors by using the `lenient`
option of the `match` or `query_string` queries.
2016-05-16 17:37:00 +02:00
Lee Hinman 864ba8dac1 Merge remote-tracking branch 'dakrone/there-can-be-only-one2' 2016-05-13 10:28:41 -06:00
Lee Hinman 9bcdafedda Allow only a single extension for a scripting engine
Previously multiple extensions could be provided, however, this can lead
to confusion with on-disk scripts (ie, "foo.js" and "foo.javascript")
having different content. Only a single extension is now supported.

The only language currently supporting multiple extensions was the
Javascript engine ("js" and "javascript"). It now only supports the
`.js` extension.

Relates to #10598
2016-05-13 09:54:31 -06:00
Lee Hinman d5b75491dc Merge remote-tracking branch 'dakrone/remove-script-sandbox' 2016-05-13 09:50:39 -06:00
Christoph Büscher a40c397c67 Don't allow `fuzziness` for `multi_match` types cross_fields, phrase and phrase_prefix
Currently `fuzziness` is not supported for the `cross_fields` type
of the `multi_match` query since it complicates the logic that
blends the term queries that cross_fields uses internally. At the
moment using this combination is silently ignored, which can lead to
confusions. Instead we should throw an exception in this case.
The same is true for phrase and phrase_prefix type.

Closes #7764
2016-05-13 17:32:14 +02:00
Lee Hinman efff3918d8 Remove support for mulitple languages per scripting engine 2016-05-13 09:24:31 -06:00
Lee Hinman a4060f7436 Remove vestiges of script engine sandboxing
This removes all the mentions of the sandbox from the script engine
services and permissions model. This means that the following settings
are no longer supported:

```yaml
script.inline: sandbox
script.stored: sandbox
```

Instead, only a `true` or `false` value can be specified.

Since this would otherwise break the default-allow parameter for
languages like expressions, painless, and mustache, all script engines
have been updated to have individual settings, for instance:

```yaml
script.engine.groovy.inline: true
```

Would enable all inline scripts for groovy. (they can still be
overridden on a per-operation basis).

Expressions, Painless, and Mustache all default to `true` for inline,
file, and stored scripts to preserve the old scripting behavior.

Resolves #17114
2016-05-13 09:24:31 -06:00
Adrien Grand 638da06c1d Add back support for `ip` range aggregations. #17859
This commit adds support for range aggregations on `ip` fields. However it will
only work on 5.x indices.

Closes #17700
2016-05-13 17:22:01 +02:00
Adrien Grand 866a5459f0 Make significant terms work on fields that are indexed with points. #18031
It will keep using the caching terms enum for keyword/text fields and falls back
to IndexSearcher.count for fields that do not use the inverted index for
searching (such as numbers and ip addresses). Note that this probably means that
significant terms aggregations on these fields will be less efficient than they
used to be. It should be ok under a sampler aggregation though.

This moves tests back to the state they were in before numbers started using
points, and also adds a new test that significant terms aggs fail if a field is
not indexed.

In the long term, we might want to follow the approach that Robert initially
proposed that consists in collecting all documents from the background filter in
order to compute frequencies using doc values. This would also mean that
significant terms aggregations do not require fields to be indexed anymore.
2016-05-11 16:52:58 +02:00
Jason Tedor 2bf585e642 Require /bin/bash in packaging
This commit adds a hard requirement to the RPM and Debian packages for
/bin/bash to be present, and adds a note regarding this to the migration
docs.

Relates #18259
2016-05-10 21:17:09 -04:00
Lee Hinman 1c54033e92 Merge branch 'pr/18068' 2016-05-10 08:27:43 -06:00
Alexander Kazakov 667a091205 Add note about cat field data API changes into migration doc 2016-05-10 16:41:21 +03:00
Jason Tedor 7d1fd17172 Remove plugin script parsing of system properties
The plugin script parses command-line options looking for Java system
properties and extracts these arguments to pass to the java command when
starting the JVM. Since elasticsearch-plugin allows arbitrary user
arguments to the JVM via ES_JAVA_OPTS, this parsing is unnecessary. This
commit removes this unnecessary 

Relates #18207
2016-05-09 13:06:18 -04:00
Clinton Gormley 3f594089c2 Renamed all AUTOSENSE snippets to CONSOLE (#18210) 2016-05-09 15:42:23 +02:00
Adrien Grand de8354dd7f Allow binary sort values. #17959
The `ip` field uses a binary representation internally. This breaks when
rendering sort values in search responses since elasticsearch tries to write a
binary byte[] as an utf8 json string. This commit extends the `DocValueFormat`
API in order to give fields a chance to choose how to render values.

Closes #6077
2016-05-06 09:27:02 +02:00
Adrien Grand 7d8708716e QueryBuilder does not need generics. #18133
QueryBuilder has generics, but those are never used: all call sites use
`QueryBuilder<?>`. Only `AbstractQueryBuilder` needs generics so that the base
class can contain a default implementation for setters that returns `this`.
2016-05-06 08:38:20 +02:00
Nik Everett 4b1c116461 Generate and run tests from the docs
Adds infrastructure so `gradle :docs:check` will extract tests from
snippets in the documentation and execute the tests. This is included
in `gradle check` so it should happen on CI and during a normal build.

By default each `// AUTOSENSE` snippet creates a unique REST test. These
tests are executed in a random order and the cluster is wiped between
each one. If multiple snippets chain together into a test you can annotate
all snippets after the first with `// TEST[continued]` to have the
generated tests for both snippets joined.

Snippets marked as `// TESTRESPONSE` are checked against the response
of the last action.

See docs/README.asciidoc for lots more.

Closes #12583. That issue is about catching bugs in the docs during build.
This catches *some* bugs in the docs during build which is a good start.
2016-05-05 13:58:03 -04:00
Isabel Drost-Fromm 6b9ac46402 Merge branch 'master' into enhancement/switch_geodistancesortbuilder_to_geovalidationmethod 2016-05-04 11:30:15 +02:00
Daniel Mitterdorfer 0a6f40c7f5 Enable HTTP compression by default with compression level 3
With this commit we compress HTTP responses provided the client
supports it (as indicated by the HTTP header 'Accept-Encoding').

We're also able to process compressed HTTP requests if needed.

The default compression level is lowered from 6 to 3 as benchmarks
have indicated that this reduces query latency with a negligible
increase in network traffic.

Closes #7309
2016-05-03 08:53:15 +02:00
Clinton Gormley 04bd55d61c Added perl migration script for indexed scripts to migration docs 2016-04-29 14:18:27 +02:00
Martijn van Groningen 6c3beaa2eb Drop top level inner hits in favour of inner hits defined in the query dsl.
Fix a limitation that prevent from hierarchical inner hits be defined in query dsl.

Removed the nested_path, parent_child_type and query options from inner hits dsl. These options are only set by ES
upon parsing the has_child, has_parent and nested queries are using their respective query builders.

These options are still used internally, when these options are set a new private copy is created based on the
provided InnerHitBuilder and configuring either nested_path or parent_child_type and the inner query of the query builder
being used.

Closes #11118
2016-04-29 11:17:24 +02:00
Isabel Drost-Fromm c1fa9cd18e Add note that coerce and ignore_malformed are deprecated for geo
distance sorting
2016-04-28 14:13:58 +02:00
Isabel Drost-Fromm a19c426e0f Deprecate coerce/ignore_malformed for GeoBoundingBoxQuery 2016-04-28 14:10:59 +02:00