honeymoose/OpenSearch
1
0
Fork 0
mirror of https://github.com/honeymoose/OpenSearch.git synced 2025-04-04 14:28:32 +00:00
Code Issues Packages Projects Releases Wiki Activity
ed1c5a0241
OpenSearch/docs/reference/docs.asciidoc

53 lines
1.0 KiB
Plaintext
Raw Normal View History

Migrated documentation into the main repo
2013-08-29 01:24:34 +02:00
[[docs]]
= Document APIs
[partintro]
--
Add a high level description of ES's data replication model (#22327) This commit adds a document describing our data replication model in high level terms. The goal is give people basic insight into how things work in order to better understand how read and writes interact, both during normal operations and under failures.
2017-01-10 13:17:48 +01:00
This section starts with a short introduction to Elasticsearch's <<docs-replication,data replication model>>, followed by a
detailed description of the following CRUD APIs:
[DOCS] Reorganised common API conventions
2013-10-13 16:46:56 +02:00
.Single document APIs
* <<docs-index_>>
* <<docs-get>>
* <<docs-delete>>
* <<docs-update>>
.Multi-document APIs
* <<docs-multi-get>>
* <<docs-bulk>>
Inclusion of link to Multi Delete (#22619) Based on the 2 categories it looks like the docs-delete-by-query was to be listed here under the Multi-document APIs section.
2017-01-16 06:58:18 -05:00
* <<docs-delete-by-query>>
Add support for waiting until a refresh occurs This adds support for setting the refresh request parameter to `wait_for` in the `index`, `delete`, `update`, and `bulk` APIs. When `refresh=wait_for` is set those APIs will not return until their results have been made visible to search by a refresh. Also it adds a `forced_refresh` field to the response of `index`, `delete`, `update`, and to each item in a bulk response. This will be true for requests with `?refresh` or `?refresh=true` and will be true for some requests (see below) with `refresh=wait_for` but ought to otherwise always be false. `refresh=wait_for` is implemented as a list of `Tuple<Translog.Location, Consumer<Boolean>>`s in the new `RefreshListeners` class that is managed by `IndexShard`. The dynamic, index scoped `index.max_refresh_listeners` setting controls a maximum number of listeners allowed in any shard. If more than that many listeners accumulate in the engine then a refresh will be forced, the thread that adds the listener will be blocked until the refresh completes, and then the listener will be called with a `forcedRefresh` flag so it knows that it was the "straw that broke the camel's back". These listeners are only used by `refresh=wait_for` and that flag manifests itself as `forced_refresh` being `true` in the response. About half of this change comes from piping async-ness down to the appropriate layer in a way that is compatible with the ongoing with with sequence ids. Closes #1063 You can look up the winding story of all the commits here: https://github.com/elastic/elasticsearch/pull/17986 Here are the commit messages in case they are intersting to you: commit 59a753b89109828d2b8f0de05cb104fc663cf95e Author: Nik Everett <nik9000@gmail.com> Date: Mon Jun 6 10:18:23 2016 -0400 Replace a method reference with implementing an interface Saves a single allocation and forces more commonality between the WriteResults. commit 31f7861a85b457fb7378a6f27fa0a0c171538f68 Author: Nik Everett <nik9000@gmail.com> Date: Mon Jun 6 10:07:55 2016 -0400 Revert "Replace static method that takes consumer with delegate class that takes an interface" This reverts commit 777e23a6592c75db0081a53458cc760f4db69507. commit 777e23a6592c75db0081a53458cc760f4db69507 Author: Nik Everett <nik9000@gmail.com> Date: Mon Jun 6 09:29:35 2016 -0400 Replace static method that takes consumer with delegate class that takes an interface Same number of allocations, much less code duplication. commit 9b49a480ca9587a0a16ebe941662849f38289644 Author: Nik Everett <nik9000@gmail.com> Date: Mon Jun 6 08:25:38 2016 -0400 Patch from boaz commit c2bc36524fda119fd0514415127e8901d94409c8 Author: Nik Everett <nik9000@gmail.com> Date: Thu Jun 2 14:46:27 2016 -0400 Fix docs After updating to master we are actually testing them. commit 03975ac056e44954eb0a371149d410dcf303e212 Author: Nik Everett <nik9000@gmail.com> Date: Thu Jun 2 14:20:11 2016 -0400 Cleanup after merge from master commit 9c9a1deb002c5bebb2a997c89fa12b3d7978e02e Author: Nik Everett <nik9000@gmail.com> Date: Thu Jun 2 14:09:14 2016 -0400 Breaking changes notes commit 1c3e64ae06c07a85f7af80534fab88279adb30b4 Merge: 9e63ad6 f67e580 Author: Nik Everett <nik9000@gmail.com> Date: Thu Jun 2 14:00:05 2016 -0400 Merge branch 'master' into block_until_refresh2 commit 9e63ad6de52d0b28f0b6d7203721baf1ebf6f56b Author: Nik Everett <nik9000@gmail.com> Date: Thu Jun 2 13:21:27 2016 -0400 Test for TransportWriteAction commit 522ecb59d39b3c9e8df0d3b8df34b9e7aeaf0ce9 Author: Nik Everett <nik9000@gmail.com> Date: Thu Jun 2 10:30:18 2016 -0400 Document deprecation commit 0cd67b947f58867e704a1f0e66928a6fb5a11f11 Author: Nik Everett <nik9000@gmail.com> Date: Thu Jun 2 10:26:23 2016 -0400 Deprecate setRefresh(boolean) Users should use `setRefresh(RefreshPolicy)` instead. commit aeb1be3f2c501990b33fb1f8230d496035f498ef Author: Nik Everett <nik9000@gmail.com> Date: Thu Jun 2 10:12:27 2016 -0400 Remove checkstyle suppression It is fixed commit 00d09a9caa638b6f90f4896b5502dd98d8fad56e Author: Nik Everett <nik9000@gmail.com> Date: Thu Jun 2 10:08:28 2016 -0400 Improve comment commit 788164b898a6ee2878a273961230122b7386c3c9 Author: Nik Everett <nik9000@gmail.com> Date: Thu Jun 2 10:01:01 2016 -0400 S/ReplicatedWriteResponse/WriteResponse/ Now it lines up with WriteRequest. commit b74cf3fe778352b140355afcaa08d3d4412d749d Author: Nik Everett <nik9000@gmail.com> Date: Wed Jun 1 18:27:52 2016 -0400 Preserve `?refresh` behavior `?refresh` means the same things as `?refresh=true`. commit 30f972bdaeaaa0de6fe67746cdb8628aa86f5a8c Author: Nik Everett <nik9000@gmail.com> Date: Wed Jun 1 17:39:05 2016 -0400 Handle hanging documents If a document is added to the index during a refresh we weren't properly firing its refresh listener. This happened because the way we detect whether a refresh makes something visible or not is imperfect. It is ok because it always errs on the side of thinking that something isn't yet visible. So when a document arrives during a refresh the refresh listeners won't think it made it into a refresh when, often, it does. The way we work around this is by telling Elasticsearch that it ought to trigger a refresh if there are any pending refresh listeners even if there aren't pending documents to update. Lucene short circuits the refresh so it doesn't take that much effort, but the refresh listeners still get the signal that a refresh has come in and they still pick up the change and notify the listener. This means that the time that a listener can wait is actually slightly longer than the refresh interval. commit d523b5702b60c7ba309fb0dcf3cd3a4798f11960 Author: Nik Everett <nik9000@gmail.com> Date: Wed Jun 1 14:34:01 2016 -0400 Explain Integer.MAX_VALUE commit 4ffb7c0e954343cc1c04b3d7be2ebad66d3a016b Author: Nik Everett <nik9000@gmail.com> Date: Wed Jun 1 14:27:39 2016 -0400 Fire all refresh listeners in a single thread Rather than queueing a runnable each. commit 19606ec3bbe612095df45eba734c5b7eb2709c01 Author: Nik Everett <nik9000@gmail.com> Date: Wed Jun 1 14:09:52 2016 -0400 Assert translog ordering commit 6bb4e5c75e850f4a42518f06fbc955f7ec76d245 Author: Nik Everett <nik9000@gmail.com> Date: Wed Jun 1 13:17:44 2016 -0400 Support null RefreshListeners in InternalEngine Just skip using it. commit 74be1480d6e44af2b354ff9ea47c234d4870b6c2 Author: Nik Everett <nik9000@gmail.com> Date: Tue May 31 18:02:03 2016 -0400 Move funny ShardInfo hack for bulk into bulk This should make it easier to understand because it is closer to where it matters.... commit 2b771f8dabd488e056cfdc9989608d18264ddfb0 Author: Nik Everett <nik9000@gmail.com> Date: Tue May 31 17:39:46 2016 -0400 Pull listener out into an inner class with javadoc and stuff commit 058481ad72019c0492b03a7a4ac32a48673697d3 Author: Nik Everett <nik9000@gmail.com> Date: Tue May 31 17:33:42 2016 -0400 Fix javadoc links commit d2123b1cabf29bce8ff561d4a4c1c1d5b42bccad Author: Nik Everett <nik9000@gmail.com> Date: Tue May 31 17:28:09 2016 -0400 Make more stuff final commit 8453fc4f7850f6a02fb5971c17a942a3e3fd9f7b Author: Nik Everett <nik9000@gmail.com> Date: Tue May 31 17:26:48 2016 -0400 Javadoc commit fb16d2fc7016c1e8e1621d481e8781c7ef43326c Author: Nik Everett <nik9000@gmail.com> Date: Tue May 31 16:14:48 2016 -0400 Rewrite refresh docs commit 5797d1b1c4d233c0db918c0d08c21731ddccd05e Author: Nik Everett <nik9000@gmail.com> Date: Tue May 31 15:02:34 2016 -0400 Fix forced_refresh flag It wasn't being set. commit 43ce50a1de250a9e073a2ca6cbf55c1b4c74b11b Author: Nik Everett <nik9000@gmail.com> Date: Tue May 31 14:02:56 2016 -0400 Delay translog sync and flush until after refresh The sync might have occurred for us during the refresh so we have less work to do. Maybe. commit bb2739202e084703baf02cfa58f09517598cf14e Author: Nik Everett <nik9000@gmail.com> Date: Tue May 31 13:08:08 2016 -0400 Remove duplication in WritePrimaryResult and WriteReplicaResult commit 2f579f89b4867a880396f2e7fcffc508449ff2de Author: Nik Everett <nik9000@gmail.com> Date: Tue May 31 12:19:05 2016 -0400 Clean up registration of RefreshListeners commit 87ab6e60ca5ba945bf0fba84784b2bbe53506abf Author: Nik Everett <nik9000@gmail.com> Date: Tue May 31 11:28:30 2016 -0400 Shorten lock time in RefreshListeners Also use null to represent no listeners rather than an empty list. This saves allocating a new ArrayList every refresh cycle on every index. commit 0d49d9c5720dadfb67da3fa760397bf6d874601c Author: Nik Everett <nik9000@gmail.com> Date: Tue May 24 10:46:18 2016 -0400 Flip relationship between RefreshListeners and Engine Now RefreshListeners comes to Engine from EngineConfig. commit b2704b8a39382953f8f91a9743e894ee289f7514 Author: Nik Everett <nik9000@gmail.com> Date: Tue May 24 09:37:58 2016 -0400 Remove unused imports Maybe I added them? commit 04343a22647f19304d9dc716b3fac9b183227f63 Author: Nik Everett <nik9000@gmail.com> Date: Tue May 24 09:37:52 2016 -0400 Javadoc commit da1e765678890a02d61d8a29aa433274beb5e00c Author: Nik Everett <nik9000@gmail.com> Date: Tue May 24 09:26:35 2016 -0400 Reply with non-null Also move the fsync and flush to before the refresh listener stuff. commit 5d8eecd0d904b497844b4c81c46477bd6178ed3a Author: Nik Everett <nik9000@gmail.com> Date: Tue May 24 08:58:47 2016 -0400 Remove funky synchronization in AsyncReplicaAction commit 1ec71eea0f4e1228ae1497d982307be818ef4b65 Author: Nik Everett <nik9000@gmail.com> Date: Tue May 24 08:01:14 2016 -0400 s/LinkedTransferQueue/ArrayList/ commit 7da36a4ceed2ccf7955138c3b005237fa41efcb4 Author: Nik Everett <nik9000@gmail.com> Date: Tue May 24 07:46:38 2016 -0400 More cleanup for RefreshListeners commit 957e9b77007c32ee75dde152c6622bab065d5993 Author: Nik Everett <nik9000@gmail.com> Date: Tue May 24 07:34:13 2016 -0400 /Consumer<Runnable>/Executor/ commit 4d8bf5d4a70dcc56150c8d8d14165cd23d308b3c Author: Nik Everett <nik9000@gmail.com> Date: Mon May 23 22:20:42 2016 -0400 explain commit 15d948a348089bb2937eec5ac4e96f3ec67dbe32 Author: Nik Everett <nik9000@gmail.com> Date: Mon May 23 22:17:59 2016 -0400 Better.... commit dc28951d02973fc03b4d51913b5f96de14b75607 Author: Nik Everett <nik9000@gmail.com> Date: Mon May 23 21:09:20 2016 -0400 Javadocs and compromises commit 8eebaa89c0a1ee74982fbe0d56d1485ca2ae09db Author: Nik Everett <nik9000@gmail.com> Date: Mon May 23 20:52:49 2016 -0400 Take boaz's changes to their logic conclusion and unbreak important stuff like bulk commit 7056b96ea412f275005b93e3570bcff895859ed5 Author: Nik Everett <nik9000@gmail.com> Date: Mon May 23 15:49:32 2016 -0400 Patch from boaz commit 87be7eaed09a274cc6a99d1a3da81d2d7bf9dd64 Author: Nik Everett <nik9000@gmail.com> Date: Mon May 23 15:49:13 2016 -0400 Revert "Move async parts of replica operation outside of the lock" This reverts commit 13807ad10b6f5ecd39f98c9f20874f9f352c5bc2. commit 13807ad10b6f5ecd39f98c9f20874f9f352c5bc2 Author: Nik Everett <nik9000@gmail.com> Date: Fri May 20 22:53:15 2016 -0400 Move async parts of replica operation outside of the lock commit b8cadcef565908b276484f7f5f988fd58b38d8b6 Author: Nik Everett <nik9000@gmail.com> Date: Fri May 20 16:17:20 2016 -0400 Docs commit 91149e0580233bf79c2273b419fe9374ca746648 Author: Nik Everett <nik9000@gmail.com> Date: Fri May 20 15:17:40 2016 -0400 Finally! commit 1ff50c2faf56665d221f00a18d9ac88745904bf5 Author: Nik Everett <nik9000@gmail.com> Date: Fri May 20 15:01:53 2016 -0400 Remove Translog#lastWriteLocation I wasn't being careful enough with locks so it wasn't right anyway. Instead this builds a synthetic Tranlog.Location when you call getWriteLocation with much more relaxed equality guarantees. Rather than being equal to the last Translog.Location returned it is simply guaranteed to be greater than the last translog returned and less than the next. commit 55596ea68b5484490c3637fbad0d95564236478b Author: Nik Everett <nik9000@gmail.com> Date: Fri May 20 14:40:06 2016 -0400 Remove listener from shardOperationOnPrimary Create instead asyncShardOperationOnPrimary which is called after all of the replica operations are started to handle any async operations. commit 3322e26211bf681b37132274ee158ae330afc28b Author: Nik Everett <nik9000@gmail.com> Date: Tue May 17 17:20:02 2016 -0400 Increase default maximum number of listeners to 1000 commit 88171a8322a424e624d48960fb4c98dd43e4d671 Author: Nik Everett <nik9000@gmail.com> Date: Tue May 17 16:40:57 2016 -0400 Rename test commit 179c27c4f829f2c6ded65967652cf85adaf2ae52 Author: Nik Everett <nik9000@gmail.com> Date: Tue May 17 16:35:27 2016 -0400 Move refresh listeners into their own class They still live at the IndexShard level but they live on their own in RefreshListeners which interacts with IndexShard using a couple of callbacks and a registration method. This lets us test the listeners without standing up an entire IndexShard. We still test the listeners against an InternalEngine, because the interplay between InternalEngine, Translog, and RefreshListeners is complex and important to get right. commit d8926d5fc1d24b4da8ccff7e0f0907b98c583c41 Author: Nik Everett <nik9000@gmail.com> Date: Tue May 17 11:02:38 2016 -0400 Move refresh listeners into IndexShard commit df91cde398eb720143a85a8c6fa19bdc3a74e07d Author: Nik Everett <nik9000@gmail.com> Date: Mon May 16 16:01:03 2016 -0400 unused import commit 066da45b08148b266e4173166662fc1b3f66ed53 Author: Nik Everett <nik9000@gmail.com> Date: Mon May 16 15:54:11 2016 -0400 Remove RefreshListener interface Just pass a Translog.Location and a Consumer<Boolean> when registering. commit b971d6d3301c7522b2e7eb90d5d8dd96a77fa625 Author: Nik Everett <nik9000@gmail.com> Date: Mon May 16 14:41:06 2016 -0400 Docs for setForcedRefresh commit 6c43be821eaf61141d3ec520f988aad3a96a3941 Author: Nik Everett <nik9000@gmail.com> Date: Mon May 16 14:34:39 2016 -0400 Rename refresh setter and getter commit e61b7391f91263a4c4d6107bfbc2a828bbcc805c Author: Nik Everett <nik9000@gmail.com> Date: Mon Apr 25 22:48:09 2016 -0400 Trigger listeners even when there is no refresh Each refresh gives us an opportunity to pick up any listeners we may have left behind. commit 0c9b0477085c021f503db775640d25668e02f635 Author: Nik Everett <nik9000@gmail.com> Date: Mon Apr 25 20:30:06 2016 -0400 REST commit 8250343240de7e63118c663a230a7a314807a754 Author: Nik Everett <nik9000@gmail.com> Date: Mon Apr 25 19:34:22 2016 -0400 Switch to estimated count We don't need a linear time count of the number of listeners - a volatile variable is good enough to guess. It probably undercounts more than it overcounts but it isn't a huge problem. commit bd531167fe54f1bde6f6d4ddb0a8de5a7bcc18a2 Author: Nik Everett <nik9000@gmail.com> Date: Mon Apr 25 18:21:02 2016 -0400 Don't try and set forced refresh on bulk items without a response NullPointerExceptions are bad. If the entire request fails then the user has worse problems then "did these force a refresh". commit bcfded11515af5e0b3c3e36f3c2f73f5cd26512e Author: Nik Everett <nik9000@gmail.com> Date: Mon Apr 25 18:14:20 2016 -0400 Replace LinkedList and synchronized with LinkedTransferQueue commit 8a80cc70a76375a7593745884cb987535b37ca80 Author: Nik Everett <nik9000@gmail.com> Date: Mon Apr 25 17:38:24 2016 -0400 Support for update commit 1f36966742f851b7328015151ef6fc8f95299af2 Author: Nik Everett <nik9000@gmail.com> Date: Mon Apr 25 15:46:06 2016 -0400 Cleanup translog tests commit 8d121bf35eb265b8a0aee9710afeb1b054a113d4 Author: Nik Everett <nik9000@gmail.com> Date: Mon Apr 25 15:40:53 2016 -0400 Cleanup listener implementation Much more testing too! commit 2058f4a808762c4588309f21b13b677245832f2c Author: Nik Everett <nik9000@gmail.com> Date: Mon Apr 25 11:45:55 2016 -0400 Pass back information about whether we refreshed commit e445cb0cb91ebdbcfdbf566696edb2bf1c84a882 Author: Nik Everett <nik9000@gmail.com> Date: Mon Apr 25 11:03:31 2016 -0400 Javadoc commit 611cbeeaeb458f4b428bfc43a1ee6652adf4baff Author: Nik Everett <nik9000@gmail.com> Date: Mon Apr 25 11:01:40 2016 -0400 Move ReplicationResponse now it is in the same package as its request commit 9919758b644fd73895fb88cd6a4909a8387eb2e2 Author: Nik Everett <nik9000@gmail.com> Date: Mon Apr 25 11:00:14 2016 -0400 Oh boy that wasn't working commit 247cb483c4459dea8e95e0e3bd2e4bf8d452c598 Author: Nik Everett <nik9000@gmail.com> Date: Mon Apr 25 10:29:37 2016 -0400 Basic block_until_refresh exposed to java client and basic "is it plugged in" style tests. commit 46c855c9971cb2b748206d2afa6a2d88724be3ba Author: Nik Everett <nik9000@gmail.com> Date: Mon Apr 25 10:11:10 2016 -0400 Move test to own class commit a5ffd892d0a352ae7e9757f2640fc2a1fa656bf2 Author: Nik Everett <nik9000@gmail.com> Date: Mon Apr 25 07:44:25 2016 -0400 WIP commit 213bebb6ece11b85d17e44af9a54fc2e5e332d39 Author: Nik Everett <nik9000@gmail.com> Date: Fri Apr 22 21:35:52 2016 -0400 Add refresh listeners commit a2bc7f30e6d4857a1224ef5a89909b36c8f33731 Author: Nik Everett <nik9000@gmail.com> Date: Fri Apr 22 21:11:55 2016 -0400 Return last written location from refresh commit 85033a87551da89f36a23d4dfd5016db218e08ee Author: Nik Everett <nik9000@gmail.com> Date: Fri Apr 22 20:28:21 2016 -0400 Never reply to replica actions while you have the operation lock This last thing was causing periodic test failures because we were replying while we had the operation lock. Now, we probably could get away with that in most cases but the tests don't like it and it isn't a good idea to do network io while you have a lock anyway. So this prevents it. commit 1f25cf35e796835b3827b8a4110e09e5de61784c Author: Nik Everett <nik9000@gmail.com> Date: Fri Apr 22 19:56:18 2016 -0400 Cleanup commit 52c5f7c3f04710901f503334239a611c0e21c85a Author: Nik Everett <nik9000@gmail.com> Date: Fri Apr 22 19:33:00 2016 -0400 Add a listener to shard operations commit 5b142dc331214c8eef90587144f4b3f959f9eced Author: Nik Everett <nik9000@gmail.com> Date: Fri Apr 22 18:03:52 2016 -0400 Cleanup commit 3d22b2d7ceb473db339259452a7c4f117ce86069 Author: Nik Everett <nik9000@gmail.com> Date: Fri Apr 22 17:59:55 2016 -0400 Push the listener into shardOperationOnPrimary commit 34b378943b8185451acf6350f661c0ad33b5836d Author: Nik Everett <nik9000@gmail.com> Date: Fri Apr 22 17:48:47 2016 -0400 Doc commit b42b8da968d42cc7414020c7b199606a5dcce50a Author: Nik Everett <nik9000@gmail.com> Date: Fri Apr 22 17:45:40 2016 -0400 Don't finish early if the primary finishes early We use a "fake" pending shard that we resolve when the replicas have all started. commit 0fc045b56e1e02a48c30383ac50a281d5af7e0b6 Author: Nik Everett <nik9000@gmail.com> Date: Fri Apr 22 17:30:06 2016 -0400 Make performOnPrimary asyncS Instead of returning Tuple<Response, ReplicaRequest> it returns ReplicaRequest and takes a ActionListener<Response> as an argument. We call the listener immediately to preserve backwards compatibility for now. commit 80119b9a26ede96a865af45904c3ac69d5b19b59 Author: Nik Everett <nik9000@gmail.com> Date: Fri Apr 22 16:51:53 2016 -0400 Factor out common code in shardOperationOnPrimary commit 0642083676702618f900fa842c08802a04c1a53e Author: Nik Everett <nik9000@gmail.com> Date: Fri Apr 22 16:32:29 2016 -0400 Factor out common code from shardOperationOnReplica commit 8bdc415fedaaa9f2d0c555590a13ec4699a7c3f7 Author: Nik Everett <nik9000@gmail.com> Date: Fri Apr 22 16:23:28 2016 -0400 Create ReplicatedMutationRequest Superclass for index, delete, and bulkShard requests. commit 0f8fa846a2822c4293df32fed18c9b99660b39ff Author: Nik Everett <nik9000@gmail.com> Date: Fri Apr 22 16:10:30 2016 -0400 Create TransportReplicatedMutationAction It is the superclass of replication actions that mutate data: index, delete, and shardBulk. shardFlush and shardRefresh are replication actions but they do not extend TransportReplicatedMutationAction because they don't change the data, only shuffle it around.
2016-06-06 10:45:44 -04:00
* <<docs-update-by-query>>
* <<docs-reindex>>
[DOCS] Reorganised common API conventions
2013-10-13 16:46:56 +02:00
NOTE: All CRUD APIs are single-index APIs. The `index` parameter accepts a single
index name, or an `alias` which points to a single index.
Migrated documentation into the main repo
2013-08-29 01:24:34 +02:00
--
Add a high level description of ES's data replication model (#22327) This commit adds a document describing our data replication model in high level terms. The goal is give people basic insight into how things work in order to better understand how read and writes interact, both during normal operations and under failures.
2017-01-10 13:17:48 +01:00
include::docs/data-replication.asciidoc[]
Migrated documentation into the main repo
2013-08-29 01:24:34 +02:00
include::docs/index_.asciidoc[]
include::docs/get.asciidoc[]
include::docs/delete.asciidoc[]
Port Delete By Query to Reindex infrastructure closes #16883
2016-05-13 15:27:02 +02:00
include::docs/delete-by-query.asciidoc[]
Migrated documentation into the main repo
2013-08-29 01:24:34 +02:00
include::docs/update.asciidoc[]
Docs: Display reindex/update by query API and fix build doc issue The documentation existed, but was not linked anywhere. Also fixed the docs to make sure they build with this enabled.
2016-03-18 10:44:16 +01:00
include::docs/update-by-query.asciidoc[]
Migrated documentation into the main repo
2013-08-29 01:24:34 +02:00
include::docs/multi-get.asciidoc[]
include::docs/bulk.asciidoc[]
Docs: Display reindex/update by query API and fix build doc issue The documentation existed, but was not linked anywhere. Also fixed the docs to make sure they build with this enabled.
2016-03-18 10:44:16 +01:00
include::docs/reindex.asciidoc[]
[DOCS] Moved termvector and mtermvectors from search to docs
2014-01-22 14:10:07 +01:00
include::docs/termvectors.asciidoc[]
Migrated documentation into the main repo
2013-08-29 01:24:34 +02:00
[DOCS] Moved termvector and mtermvectors from search to docs
2014-01-22 14:10:07 +01:00
include::docs/multi-termvectors.asciidoc[]
Add support for waiting until a refresh occurs This adds support for setting the refresh request parameter to `wait_for` in the `index`, `delete`, `update`, and `bulk` APIs. When `refresh=wait_for` is set those APIs will not return until their results have been made visible to search by a refresh. Also it adds a `forced_refresh` field to the response of `index`, `delete`, `update`, and to each item in a bulk response. This will be true for requests with `?refresh` or `?refresh=true` and will be true for some requests (see below) with `refresh=wait_for` but ought to otherwise always be false. `refresh=wait_for` is implemented as a list of `Tuple<Translog.Location, Consumer<Boolean>>`s in the new `RefreshListeners` class that is managed by `IndexShard`. The dynamic, index scoped `index.max_refresh_listeners` setting controls a maximum number of listeners allowed in any shard. If more than that many listeners accumulate in the engine then a refresh will be forced, the thread that adds the listener will be blocked until the refresh completes, and then the listener will be called with a `forcedRefresh` flag so it knows that it was the "straw that broke the camel's back". These listeners are only used by `refresh=wait_for` and that flag manifests itself as `forced_refresh` being `true` in the response. About half of this change comes from piping async-ness down to the appropriate layer in a way that is compatible with the ongoing with with sequence ids. Closes #1063 You can look up the winding story of all the commits here: https://github.com/elastic/elasticsearch/pull/17986 Here are the commit messages in case they are intersting to you: commit 59a753b89109828d2b8f0de05cb104fc663cf95e Author: Nik Everett <nik9000@gmail.com> Date: Mon Jun 6 10:18:23 2016 -0400 Replace a method reference with implementing an interface Saves a single allocation and forces more commonality between the WriteResults. commit 31f7861a85b457fb7378a6f27fa0a0c171538f68 Author: Nik Everett <nik9000@gmail.com> Date: Mon Jun 6 10:07:55 2016 -0400 Revert "Replace static method that takes consumer with delegate class that takes an interface" This reverts commit 777e23a6592c75db0081a53458cc760f4db69507. commit 777e23a6592c75db0081a53458cc760f4db69507 Author: Nik Everett <nik9000@gmail.com> Date: Mon Jun 6 09:29:35 2016 -0400 Replace static method that takes consumer with delegate class that takes an interface Same number of allocations, much less code duplication. commit 9b49a480ca9587a0a16ebe941662849f38289644 Author: Nik Everett <nik9000@gmail.com> Date: Mon Jun 6 08:25:38 2016 -0400 Patch from boaz commit c2bc36524fda119fd0514415127e8901d94409c8 Author: Nik Everett <nik9000@gmail.com> Date: Thu Jun 2 14:46:27 2016 -0400 Fix docs After updating to master we are actually testing them. commit 03975ac056e44954eb0a371149d410dcf303e212 Author: Nik Everett <nik9000@gmail.com> Date: Thu Jun 2 14:20:11 2016 -0400 Cleanup after merge from master commit 9c9a1deb002c5bebb2a997c89fa12b3d7978e02e Author: Nik Everett <nik9000@gmail.com> Date: Thu Jun 2 14:09:14 2016 -0400 Breaking changes notes commit 1c3e64ae06c07a85f7af80534fab88279adb30b4 Merge: 9e63ad6 f67e580 Author: Nik Everett <nik9000@gmail.com> Date: Thu Jun 2 14:00:05 2016 -0400 Merge branch 'master' into block_until_refresh2 commit 9e63ad6de52d0b28f0b6d7203721baf1ebf6f56b Author: Nik Everett <nik9000@gmail.com> Date: Thu Jun 2 13:21:27 2016 -0400 Test for TransportWriteAction commit 522ecb59d39b3c9e8df0d3b8df34b9e7aeaf0ce9 Author: Nik Everett <nik9000@gmail.com> Date: Thu Jun 2 10:30:18 2016 -0400 Document deprecation commit 0cd67b947f58867e704a1f0e66928a6fb5a11f11 Author: Nik Everett <nik9000@gmail.com> Date: Thu Jun 2 10:26:23 2016 -0400 Deprecate setRefresh(boolean) Users should use `setRefresh(RefreshPolicy)` instead. commit aeb1be3f2c501990b33fb1f8230d496035f498ef Author: Nik Everett <nik9000@gmail.com> Date: Thu Jun 2 10:12:27 2016 -0400 Remove checkstyle suppression It is fixed commit 00d09a9caa638b6f90f4896b5502dd98d8fad56e Author: Nik Everett <nik9000@gmail.com> Date: Thu Jun 2 10:08:28 2016 -0400 Improve comment commit 788164b898a6ee2878a273961230122b7386c3c9 Author: Nik Everett <nik9000@gmail.com> Date: Thu Jun 2 10:01:01 2016 -0400 S/ReplicatedWriteResponse/WriteResponse/ Now it lines up with WriteRequest. commit b74cf3fe778352b140355afcaa08d3d4412d749d Author: Nik Everett <nik9000@gmail.com> Date: Wed Jun 1 18:27:52 2016 -0400 Preserve `?refresh` behavior `?refresh` means the same things as `?refresh=true`. commit 30f972bdaeaaa0de6fe67746cdb8628aa86f5a8c Author: Nik Everett <nik9000@gmail.com> Date: Wed Jun 1 17:39:05 2016 -0400 Handle hanging documents If a document is added to the index during a refresh we weren't properly firing its refresh listener. This happened because the way we detect whether a refresh makes something visible or not is imperfect. It is ok because it always errs on the side of thinking that something isn't yet visible. So when a document arrives during a refresh the refresh listeners won't think it made it into a refresh when, often, it does. The way we work around this is by telling Elasticsearch that it ought to trigger a refresh if there are any pending refresh listeners even if there aren't pending documents to update. Lucene short circuits the refresh so it doesn't take that much effort, but the refresh listeners still get the signal that a refresh has come in and they still pick up the change and notify the listener. This means that the time that a listener can wait is actually slightly longer than the refresh interval. commit d523b5702b60c7ba309fb0dcf3cd3a4798f11960 Author: Nik Everett <nik9000@gmail.com> Date: Wed Jun 1 14:34:01 2016 -0400 Explain Integer.MAX_VALUE commit 4ffb7c0e954343cc1c04b3d7be2ebad66d3a016b Author: Nik Everett <nik9000@gmail.com> Date: Wed Jun 1 14:27:39 2016 -0400 Fire all refresh listeners in a single thread Rather than queueing a runnable each. commit 19606ec3bbe612095df45eba734c5b7eb2709c01 Author: Nik Everett <nik9000@gmail.com> Date: Wed Jun 1 14:09:52 2016 -0400 Assert translog ordering commit 6bb4e5c75e850f4a42518f06fbc955f7ec76d245 Author: Nik Everett <nik9000@gmail.com> Date: Wed Jun 1 13:17:44 2016 -0400 Support null RefreshListeners in InternalEngine Just skip using it. commit 74be1480d6e44af2b354ff9ea47c234d4870b6c2 Author: Nik Everett <nik9000@gmail.com> Date: Tue May 31 18:02:03 2016 -0400 Move funny ShardInfo hack for bulk into bulk This should make it easier to understand because it is closer to where it matters.... commit 2b771f8dabd488e056cfdc9989608d18264ddfb0 Author: Nik Everett <nik9000@gmail.com> Date: Tue May 31 17:39:46 2016 -0400 Pull listener out into an inner class with javadoc and stuff commit 058481ad72019c0492b03a7a4ac32a48673697d3 Author: Nik Everett <nik9000@gmail.com> Date: Tue May 31 17:33:42 2016 -0400 Fix javadoc links commit d2123b1cabf29bce8ff561d4a4c1c1d5b42bccad Author: Nik Everett <nik9000@gmail.com> Date: Tue May 31 17:28:09 2016 -0400 Make more stuff final commit 8453fc4f7850f6a02fb5971c17a942a3e3fd9f7b Author: Nik Everett <nik9000@gmail.com> Date: Tue May 31 17:26:48 2016 -0400 Javadoc commit fb16d2fc7016c1e8e1621d481e8781c7ef43326c Author: Nik Everett <nik9000@gmail.com> Date: Tue May 31 16:14:48 2016 -0400 Rewrite refresh docs commit 5797d1b1c4d233c0db918c0d08c21731ddccd05e Author: Nik Everett <nik9000@gmail.com> Date: Tue May 31 15:02:34 2016 -0400 Fix forced_refresh flag It wasn't being set. commit 43ce50a1de250a9e073a2ca6cbf55c1b4c74b11b Author: Nik Everett <nik9000@gmail.com> Date: Tue May 31 14:02:56 2016 -0400 Delay translog sync and flush until after refresh The sync might have occurred for us during the refresh so we have less work to do. Maybe. commit bb2739202e084703baf02cfa58f09517598cf14e Author: Nik Everett <nik9000@gmail.com> Date: Tue May 31 13:08:08 2016 -0400 Remove duplication in WritePrimaryResult and WriteReplicaResult commit 2f579f89b4867a880396f2e7fcffc508449ff2de Author: Nik Everett <nik9000@gmail.com> Date: Tue May 31 12:19:05 2016 -0400 Clean up registration of RefreshListeners commit 87ab6e60ca5ba945bf0fba84784b2bbe53506abf Author: Nik Everett <nik9000@gmail.com> Date: Tue May 31 11:28:30 2016 -0400 Shorten lock time in RefreshListeners Also use null to represent no listeners rather than an empty list. This saves allocating a new ArrayList every refresh cycle on every index. commit 0d49d9c5720dadfb67da3fa760397bf6d874601c Author: Nik Everett <nik9000@gmail.com> Date: Tue May 24 10:46:18 2016 -0400 Flip relationship between RefreshListeners and Engine Now RefreshListeners comes to Engine from EngineConfig. commit b2704b8a39382953f8f91a9743e894ee289f7514 Author: Nik Everett <nik9000@gmail.com> Date: Tue May 24 09:37:58 2016 -0400 Remove unused imports Maybe I added them? commit 04343a22647f19304d9dc716b3fac9b183227f63 Author: Nik Everett <nik9000@gmail.com> Date: Tue May 24 09:37:52 2016 -0400 Javadoc commit da1e765678890a02d61d8a29aa433274beb5e00c Author: Nik Everett <nik9000@gmail.com> Date: Tue May 24 09:26:35 2016 -0400 Reply with non-null Also move the fsync and flush to before the refresh listener stuff. commit 5d8eecd0d904b497844b4c81c46477bd6178ed3a Author: Nik Everett <nik9000@gmail.com> Date: Tue May 24 08:58:47 2016 -0400 Remove funky synchronization in AsyncReplicaAction commit 1ec71eea0f4e1228ae1497d982307be818ef4b65 Author: Nik Everett <nik9000@gmail.com> Date: Tue May 24 08:01:14 2016 -0400 s/LinkedTransferQueue/ArrayList/ commit 7da36a4ceed2ccf7955138c3b005237fa41efcb4 Author: Nik Everett <nik9000@gmail.com> Date: Tue May 24 07:46:38 2016 -0400 More cleanup for RefreshListeners commit 957e9b77007c32ee75dde152c6622bab065d5993 Author: Nik Everett <nik9000@gmail.com> Date: Tue May 24 07:34:13 2016 -0400 /Consumer<Runnable>/Executor/ commit 4d8bf5d4a70dcc56150c8d8d14165cd23d308b3c Author: Nik Everett <nik9000@gmail.com> Date: Mon May 23 22:20:42 2016 -0400 explain commit 15d948a348089bb2937eec5ac4e96f3ec67dbe32 Author: Nik Everett <nik9000@gmail.com> Date: Mon May 23 22:17:59 2016 -0400 Better.... commit dc28951d02973fc03b4d51913b5f96de14b75607 Author: Nik Everett <nik9000@gmail.com> Date: Mon May 23 21:09:20 2016 -0400 Javadocs and compromises commit 8eebaa89c0a1ee74982fbe0d56d1485ca2ae09db Author: Nik Everett <nik9000@gmail.com> Date: Mon May 23 20:52:49 2016 -0400 Take boaz's changes to their logic conclusion and unbreak important stuff like bulk commit 7056b96ea412f275005b93e3570bcff895859ed5 Author: Nik Everett <nik9000@gmail.com> Date: Mon May 23 15:49:32 2016 -0400 Patch from boaz commit 87be7eaed09a274cc6a99d1a3da81d2d7bf9dd64 Author: Nik Everett <nik9000@gmail.com> Date: Mon May 23 15:49:13 2016 -0400 Revert "Move async parts of replica operation outside of the lock" This reverts commit 13807ad10b6f5ecd39f98c9f20874f9f352c5bc2. commit 13807ad10b6f5ecd39f98c9f20874f9f352c5bc2 Author: Nik Everett <nik9000@gmail.com> Date: Fri May 20 22:53:15 2016 -0400 Move async parts of replica operation outside of the lock commit b8cadcef565908b276484f7f5f988fd58b38d8b6 Author: Nik Everett <nik9000@gmail.com> Date: Fri May 20 16:17:20 2016 -0400 Docs commit 91149e0580233bf79c2273b419fe9374ca746648 Author: Nik Everett <nik9000@gmail.com> Date: Fri May 20 15:17:40 2016 -0400 Finally! commit 1ff50c2faf56665d221f00a18d9ac88745904bf5 Author: Nik Everett <nik9000@gmail.com> Date: Fri May 20 15:01:53 2016 -0400 Remove Translog#lastWriteLocation I wasn't being careful enough with locks so it wasn't right anyway. Instead this builds a synthetic Tranlog.Location when you call getWriteLocation with much more relaxed equality guarantees. Rather than being equal to the last Translog.Location returned it is simply guaranteed to be greater than the last translog returned and less than the next. commit 55596ea68b5484490c3637fbad0d95564236478b Author: Nik Everett <nik9000@gmail.com> Date: Fri May 20 14:40:06 2016 -0400 Remove listener from shardOperationOnPrimary Create instead asyncShardOperationOnPrimary which is called after all of the replica operations are started to handle any async operations. commit 3322e26211bf681b37132274ee158ae330afc28b Author: Nik Everett <nik9000@gmail.com> Date: Tue May 17 17:20:02 2016 -0400 Increase default maximum number of listeners to 1000 commit 88171a8322a424e624d48960fb4c98dd43e4d671 Author: Nik Everett <nik9000@gmail.com> Date: Tue May 17 16:40:57 2016 -0400 Rename test commit 179c27c4f829f2c6ded65967652cf85adaf2ae52 Author: Nik Everett <nik9000@gmail.com> Date: Tue May 17 16:35:27 2016 -0400 Move refresh listeners into their own class They still live at the IndexShard level but they live on their own in RefreshListeners which interacts with IndexShard using a couple of callbacks and a registration method. This lets us test the listeners without standing up an entire IndexShard. We still test the listeners against an InternalEngine, because the interplay between InternalEngine, Translog, and RefreshListeners is complex and important to get right. commit d8926d5fc1d24b4da8ccff7e0f0907b98c583c41 Author: Nik Everett <nik9000@gmail.com> Date: Tue May 17 11:02:38 2016 -0400 Move refresh listeners into IndexShard commit df91cde398eb720143a85a8c6fa19bdc3a74e07d Author: Nik Everett <nik9000@gmail.com> Date: Mon May 16 16:01:03 2016 -0400 unused import commit 066da45b08148b266e4173166662fc1b3f66ed53 Author: Nik Everett <nik9000@gmail.com> Date: Mon May 16 15:54:11 2016 -0400 Remove RefreshListener interface Just pass a Translog.Location and a Consumer<Boolean> when registering. commit b971d6d3301c7522b2e7eb90d5d8dd96a77fa625 Author: Nik Everett <nik9000@gmail.com> Date: Mon May 16 14:41:06 2016 -0400 Docs for setForcedRefresh commit 6c43be821eaf61141d3ec520f988aad3a96a3941 Author: Nik Everett <nik9000@gmail.com> Date: Mon May 16 14:34:39 2016 -0400 Rename refresh setter and getter commit e61b7391f91263a4c4d6107bfbc2a828bbcc805c Author: Nik Everett <nik9000@gmail.com> Date: Mon Apr 25 22:48:09 2016 -0400 Trigger listeners even when there is no refresh Each refresh gives us an opportunity to pick up any listeners we may have left behind. commit 0c9b0477085c021f503db775640d25668e02f635 Author: Nik Everett <nik9000@gmail.com> Date: Mon Apr 25 20:30:06 2016 -0400 REST commit 8250343240de7e63118c663a230a7a314807a754 Author: Nik Everett <nik9000@gmail.com> Date: Mon Apr 25 19:34:22 2016 -0400 Switch to estimated count We don't need a linear time count of the number of listeners - a volatile variable is good enough to guess. It probably undercounts more than it overcounts but it isn't a huge problem. commit bd531167fe54f1bde6f6d4ddb0a8de5a7bcc18a2 Author: Nik Everett <nik9000@gmail.com> Date: Mon Apr 25 18:21:02 2016 -0400 Don't try and set forced refresh on bulk items without a response NullPointerExceptions are bad. If the entire request fails then the user has worse problems then "did these force a refresh". commit bcfded11515af5e0b3c3e36f3c2f73f5cd26512e Author: Nik Everett <nik9000@gmail.com> Date: Mon Apr 25 18:14:20 2016 -0400 Replace LinkedList and synchronized with LinkedTransferQueue commit 8a80cc70a76375a7593745884cb987535b37ca80 Author: Nik Everett <nik9000@gmail.com> Date: Mon Apr 25 17:38:24 2016 -0400 Support for update commit 1f36966742f851b7328015151ef6fc8f95299af2 Author: Nik Everett <nik9000@gmail.com> Date: Mon Apr 25 15:46:06 2016 -0400 Cleanup translog tests commit 8d121bf35eb265b8a0aee9710afeb1b054a113d4 Author: Nik Everett <nik9000@gmail.com> Date: Mon Apr 25 15:40:53 2016 -0400 Cleanup listener implementation Much more testing too! commit 2058f4a808762c4588309f21b13b677245832f2c Author: Nik Everett <nik9000@gmail.com> Date: Mon Apr 25 11:45:55 2016 -0400 Pass back information about whether we refreshed commit e445cb0cb91ebdbcfdbf566696edb2bf1c84a882 Author: Nik Everett <nik9000@gmail.com> Date: Mon Apr 25 11:03:31 2016 -0400 Javadoc commit 611cbeeaeb458f4b428bfc43a1ee6652adf4baff Author: Nik Everett <nik9000@gmail.com> Date: Mon Apr 25 11:01:40 2016 -0400 Move ReplicationResponse now it is in the same package as its request commit 9919758b644fd73895fb88cd6a4909a8387eb2e2 Author: Nik Everett <nik9000@gmail.com> Date: Mon Apr 25 11:00:14 2016 -0400 Oh boy that wasn't working commit 247cb483c4459dea8e95e0e3bd2e4bf8d452c598 Author: Nik Everett <nik9000@gmail.com> Date: Mon Apr 25 10:29:37 2016 -0400 Basic block_until_refresh exposed to java client and basic "is it plugged in" style tests. commit 46c855c9971cb2b748206d2afa6a2d88724be3ba Author: Nik Everett <nik9000@gmail.com> Date: Mon Apr 25 10:11:10 2016 -0400 Move test to own class commit a5ffd892d0a352ae7e9757f2640fc2a1fa656bf2 Author: Nik Everett <nik9000@gmail.com> Date: Mon Apr 25 07:44:25 2016 -0400 WIP commit 213bebb6ece11b85d17e44af9a54fc2e5e332d39 Author: Nik Everett <nik9000@gmail.com> Date: Fri Apr 22 21:35:52 2016 -0400 Add refresh listeners commit a2bc7f30e6d4857a1224ef5a89909b36c8f33731 Author: Nik Everett <nik9000@gmail.com> Date: Fri Apr 22 21:11:55 2016 -0400 Return last written location from refresh commit 85033a87551da89f36a23d4dfd5016db218e08ee Author: Nik Everett <nik9000@gmail.com> Date: Fri Apr 22 20:28:21 2016 -0400 Never reply to replica actions while you have the operation lock This last thing was causing periodic test failures because we were replying while we had the operation lock. Now, we probably could get away with that in most cases but the tests don't like it and it isn't a good idea to do network io while you have a lock anyway. So this prevents it. commit 1f25cf35e796835b3827b8a4110e09e5de61784c Author: Nik Everett <nik9000@gmail.com> Date: Fri Apr 22 19:56:18 2016 -0400 Cleanup commit 52c5f7c3f04710901f503334239a611c0e21c85a Author: Nik Everett <nik9000@gmail.com> Date: Fri Apr 22 19:33:00 2016 -0400 Add a listener to shard operations commit 5b142dc331214c8eef90587144f4b3f959f9eced Author: Nik Everett <nik9000@gmail.com> Date: Fri Apr 22 18:03:52 2016 -0400 Cleanup commit 3d22b2d7ceb473db339259452a7c4f117ce86069 Author: Nik Everett <nik9000@gmail.com> Date: Fri Apr 22 17:59:55 2016 -0400 Push the listener into shardOperationOnPrimary commit 34b378943b8185451acf6350f661c0ad33b5836d Author: Nik Everett <nik9000@gmail.com> Date: Fri Apr 22 17:48:47 2016 -0400 Doc commit b42b8da968d42cc7414020c7b199606a5dcce50a Author: Nik Everett <nik9000@gmail.com> Date: Fri Apr 22 17:45:40 2016 -0400 Don't finish early if the primary finishes early We use a "fake" pending shard that we resolve when the replicas have all started. commit 0fc045b56e1e02a48c30383ac50a281d5af7e0b6 Author: Nik Everett <nik9000@gmail.com> Date: Fri Apr 22 17:30:06 2016 -0400 Make performOnPrimary asyncS Instead of returning Tuple<Response, ReplicaRequest> it returns ReplicaRequest and takes a ActionListener<Response> as an argument. We call the listener immediately to preserve backwards compatibility for now. commit 80119b9a26ede96a865af45904c3ac69d5b19b59 Author: Nik Everett <nik9000@gmail.com> Date: Fri Apr 22 16:51:53 2016 -0400 Factor out common code in shardOperationOnPrimary commit 0642083676702618f900fa842c08802a04c1a53e Author: Nik Everett <nik9000@gmail.com> Date: Fri Apr 22 16:32:29 2016 -0400 Factor out common code from shardOperationOnReplica commit 8bdc415fedaaa9f2d0c555590a13ec4699a7c3f7 Author: Nik Everett <nik9000@gmail.com> Date: Fri Apr 22 16:23:28 2016 -0400 Create ReplicatedMutationRequest Superclass for index, delete, and bulkShard requests. commit 0f8fa846a2822c4293df32fed18c9b99660b39ff Author: Nik Everett <nik9000@gmail.com> Date: Fri Apr 22 16:10:30 2016 -0400 Create TransportReplicatedMutationAction It is the superclass of replication actions that mutate data: index, delete, and shardBulk. shardFlush and shardRefresh are replication actions but they do not extend TransportReplicatedMutationAction because they don't change the data, only shuffle it around.
2016-06-06 10:45:44 -04:00
include::docs/refresh.asciidoc[]
Reference in New Issue Copy Permalink