kimchy
|
0de24218ff
|
improve local gateway, store started shards and metadata in separated files, metadata stored on master nodes, and started shards on data nodes.
|
2010-09-01 21:32:07 +03:00 |
|
kimchy
|
7709cd1bc2
|
improve GC logging
|
2010-09-01 15:02:56 +03:00 |
|
kimchy
|
7d6aa6c027
|
local gateway: don't store the local state when not a data/master node
|
2010-09-01 00:07:10 +03:00 |
|
kimchy
|
ba5cd19989
|
gateway test to make sure it recovers from the latest version
|
2010-08-31 21:13:42 +03:00 |
|
kimchy
|
a3efa21d71
|
Mapper: Failure to handle null valued objects ("obj1" : null), closes #341.
|
2010-08-31 13:33:21 +03:00 |
|
kimchy
|
f3292f6778
|
upgrade netty version to 3.2.2
|
2010-08-30 23:57:37 +03:00 |
|
kimchy
|
289cfdf25a
|
improve replica allocation, fix test to wait for recovery on node2
|
2010-08-30 23:37:09 +03:00 |
|
kimchy
|
9503fca2ae
|
fix logging
|
2010-08-30 22:53:47 +03:00 |
|
kimchy
|
61764c5b69
|
add another test for local gateway, fix race when loading the current state of a node
|
2010-08-30 19:29:00 +03:00 |
|
kimchy
|
908fba44e7
|
fsync translog when closing and not deleting, allocate failed shard to another started shard
|
2010-08-30 16:55:57 +03:00 |
|
kimchy
|
d9979f8dfe
|
more work on local gateway, start integration test it
|
2010-08-30 01:25:45 +03:00 |
|
kimchy
|
8ed54c24bd
|
change nodeFile to nodeLocation
|
2010-08-29 23:25:23 +03:00 |
|
kimchy
|
5b5d42445d
|
move the shared fs gateway base class to fs package
|
2010-08-29 23:22:55 +03:00 |
|
kimchy
|
269616f35e
|
more work on getting local gateway to work, apply transaction log operations when primary "recovering"
|
2010-08-29 16:56:04 +03:00 |
|
kimchy
|
4f4471483d
|
initial work on local gateway
|
2010-08-29 01:24:23 +03:00 |
|
kimchy
|
163b7be639
|
refactor gateway service to allow for more custom gateway implemenations
|
2010-08-28 00:34:15 +03:00 |
|
kimchy
|
a10ab7fe08
|
better exception thrown when failing to handle responses
|
2010-08-27 14:14:28 +03:00 |
|
kimchy
|
e85c47db74
|
improve stack trace when failing to send a transaction request
|
2010-08-27 13:52:58 +03:00 |
|
kimchy
|
e5b041c8ef
|
simplify gateway snapshot operation
|
2010-08-27 00:41:55 +03:00 |
|
kimchy
|
19ddee5ec9
|
add a test for time interval
|
2010-08-26 18:12:51 +03:00 |
|
kimchy
|
a49af6072b
|
add objects option to range filter builder
|
2010-08-26 16:39:39 +03:00 |
|
kimchy
|
d56b4d266e
|
move query hint only to broadcast requests that use it
|
2010-08-26 14:31:54 +03:00 |
|
kimchy
|
dfa24f6d03
|
improve caching of stored metadata fetched from nodes during allocation in order to reuse existing shards
|
2010-08-25 23:12:07 +03:00 |
|
kimchy
|
2910b6ab7f
|
add callback for shards_started and shards_failed to better handle caching on node allocations level
|
2010-08-25 22:20:27 +03:00 |
|
kimchy
|
3ebd03f69a
|
doc
|
2010-08-25 22:03:11 +03:00 |
|
kimchy
|
9d615a4f0b
|
Query DSL: Improve `or` and `and` filters to accept just array of filters, closes #336.
|
2010-08-25 18:06:28 +03:00 |
|
kimchy
|
c7a36ed431
|
add an exception when on filtesr are provided to "and" and "not" filters
|
2010-08-25 17:56:50 +03:00 |
|
kimchy
|
fcdc5f872a
|
improve (perf) getting last committed version of the index to create a new translog with
|
2010-08-25 15:12:26 +03:00 |
|
kimchy
|
9d08e56d8a
|
cache nodes store metadata for on a shard basis when iteratring over the unassigned nodes
|
2010-08-25 15:03:25 +03:00 |
|
kimchy
|
f36d80f66d
|
don't go and find local storage for a shard on nodes if its not going to be allocated
|
2010-08-25 15:01:09 +03:00 |
|
kimchy
|
4bfd0a8c26
|
cache commit points during smart allocation based on local storage when recovering from gateway
|
2010-08-25 01:12:16 +03:00 |
|
kimchy
|
244cd42298
|
allow to use just hosts when using unicast discover, will use the transport level ports
|
2010-08-25 00:44:04 +03:00 |
|
kimchy
|
fc812a306b
|
change logging
|
2010-08-24 23:33:03 +03:00 |
|
kimchy
|
ba5814a72f
|
change log level
|
2010-08-24 23:25:20 +03:00 |
|
kimchy
|
31aed01220
|
increase timeout to 5 seconds
|
2010-08-24 23:18:58 +03:00 |
|
kimchy
|
fdc82dd5a3
|
ignore parsing of wrong file format in the gateway (log warn), use index vesrion as the translog id
|
2010-08-24 23:12:18 +03:00 |
|
kimchy
|
d0f5bc3403
|
add russian letter tokenizer
|
2010-08-24 23:11:22 +03:00 |
|
kimchy
|
49b4659322
|
Indexing Buffer Size: Refine default setting to 10% (from 40%) and add more settings, closes #334.
|
2010-08-24 18:12:45 +03:00 |
|
kimchy
|
7ae8d4c669
|
thread pool type setting is wrong, fix blocking max setting to have a minimum of 10, use cached TP where needed
|
2010-08-24 15:28:54 +03:00 |
|
kimchy
|
b8ab50828c
|
only list and delete files we created
|
2010-08-24 14:48:49 +03:00 |
|
kimchy
|
df23d8f7ec
|
don't snapshot a translog that has no content
|
2010-08-24 03:58:21 +03:00 |
|
kimchy
|
6c29142b91
|
throw recovery failure if failing to parse all commit points
|
2010-08-24 01:28:39 +03:00 |
|
kimchy
|
b52d854711
|
smarter handling of throttling when preferring allocating to an existing work location
|
2010-08-23 21:39:54 +03:00 |
|
kimchy
|
78b6879ecd
|
move rebalance only when active logic into node allocation, add canRebalance hook point
|
2010-08-23 18:28:35 +03:00 |
|
kimchy
|
149962691e
|
move rebalance only when active logic into node allocation, add canRebalance hook point
|
2010-08-23 18:27:01 +03:00 |
|
kimchy
|
4300a6ca18
|
move rebalance only when active logic into node allocation, add canRebalance hook point
|
2010-08-23 18:26:17 +03:00 |
|
kimchy
|
0e47898902
|
add /_shutdown to the REST endpoints
|
2010-08-23 18:01:47 +03:00 |
|
kimchy
|
5900e01a0d
|
list remote nodest storage information with a timeout
|
2010-08-23 18:00:37 +03:00 |
|
kimchy
|
1461da5b49
|
move prefetch allocation to be a NodeAllocation
|
2010-08-23 17:22:44 +03:00 |
|
kimchy
|
edbd586ee5
|
add load test for recovery from gateway
|
2010-08-23 16:33:33 +03:00 |
|
kimchy
|
939d583550
|
initial chunk of work on refactoring the gateway, syntatic gateway files names, commit points that include translog information
|
2010-08-23 15:42:03 +03:00 |
|
kimchy
|
b49d331c86
|
initial chunk of work on refactoring the gateway, syntatic gateway files names, commit points that include translog information
|
2010-08-23 15:28:12 +03:00 |
|
kimchy
|
62cc4d554a
|
better logging message for reused vs. recovered from gateway
|
2010-08-23 15:25:06 +03:00 |
|
kimchy
|
5bd37f6f47
|
initial chunk of work on refactoring the gateway, syntatic gateway files names, commit points that include translog information
|
2010-08-23 14:48:49 +03:00 |
|
kimchy
|
1517fa3d28
|
initial chunk of work on refactoring the gateway, syntatic gateway files names, commit points that include translog information
|
2010-08-23 01:45:03 +03:00 |
|
kimchy
|
36ad3b246a
|
addlow to add node allocation ot the module
|
2010-08-22 12:35:59 +03:00 |
|
kimchy
|
664f28492d
|
improvements to the module system, allow for a module to spawn other modules, and pre process modules before injector creation
|
2010-08-22 10:42:25 +03:00 |
|
kimchy
|
aa28b93610
|
refactor how throttling is done, instead of doing it after a shard is allocated to a node, and then wait till its allowed to recover, do it on the allocation level, and don't allocate a shard to a node that has N number of recoveries going on it
|
2010-08-22 02:47:34 +03:00 |
|
kimchy
|
7592862646
|
add a callback for allocation as well
|
2010-08-22 01:00:37 +03:00 |
|
kimchy
|
a3852766ff
|
only check for prefered allocation on data nodes
|
2010-08-22 00:32:43 +03:00 |
|
kimchy
|
3f701365b0
|
move the allocate replica after primary active logic to a node allocation
|
2010-08-22 00:14:56 +03:00 |
|
kimchy
|
5ded04c5cb
|
refactor shard allocation to be more pluggable for node allocation "deciders"
|
2010-08-21 21:59:31 +03:00 |
|
kimchy
|
2865ceef85
|
more on sorted nodes when allocating shards above average number of shards per node
|
2010-08-21 19:55:38 +03:00 |
|
kimchy
|
76e1a6b1bf
|
check also for negative values
|
2010-08-21 19:51:14 +03:00 |
|
kimchy
|
5d2c005622
|
pass to canAllocate (though nothing is done there now) the actual routing nodes
|
2010-08-21 17:52:24 +03:00 |
|
kimchy
|
1ddd969fa7
|
pass to canAllocate (though nothing is done there now) the actual routing nodes
|
2010-08-21 17:52:02 +03:00 |
|
kimchy
|
11dbbebb7a
|
renamve var to better reflect its state (started, not active)
|
2010-08-21 17:50:08 +03:00 |
|
kimchy
|
3117341f44
|
Shards Allocation: Only rebalance a shard if all its instances are already active, closes #331.
|
2010-08-21 17:49:19 +03:00 |
|
kimchy
|
38e6649a7e
|
Cluster Health API: Add `initializing_shards` and `unassigned_shards` to the response, closes #330.
|
2010-08-21 16:03:29 +03:00 |
|
kimchy
|
113ea1bb1b
|
don't throttle gateway recovery based on peer recoveries (gateway recoveries are important to do as fast as possible). still throttle peer recoveries based on both.
|
2010-08-21 15:37:12 +03:00 |
|
kimchy
|
e12bdd9faf
|
rename RETRY stage in recovery to THROTTLE
|
2010-08-21 15:13:34 +03:00 |
|
kimchy
|
0c72bb2125
|
better writing of local md5 caches
|
2010-08-21 02:16:36 +03:00 |
|
kimchy
|
1ef1e9b25b
|
don't spawn on received cluster state
|
2010-08-20 19:36:39 +03:00 |
|
kimchy
|
800e450e3a
|
handle local md5 cases, don't include them in the list of files the directory has
|
2010-08-20 19:30:26 +03:00 |
|
kimchy
|
9acae60dbf
|
also have a minimum value for concurrent recoveries, at 3
|
2010-08-20 17:28:31 +03:00 |
|
kimchy
|
f62f7b8ffe
|
check the progress in case the index size has not been set yet...
|
2010-08-20 17:17:32 +03:00 |
|
kimchy
|
9f0d5f8ef1
|
better default concurrent recoveries, based on number of processors, tap it at 10
|
2010-08-20 17:06:56 +03:00 |
|
kimchy
|
8c25be6dee
|
better exception message with primary not started exception on index/delete/delete_by_query apis
|
2010-08-20 16:36:25 +03:00 |
|
kimchy
|
07285d81d1
|
add stream done when there is no file (should not happen)
|
2010-08-20 15:59:46 +03:00 |
|
kimchy
|
b609162be3
|
don't ignore recovery on throttling unless the shard is closed
|
2010-08-20 15:58:24 +03:00 |
|
kimchy
|
4e74001bde
|
move marking recovering before creating a status
|
2010-08-20 15:24:49 +03:00 |
|
kimchy
|
4c00e89f5e
|
move memory monitor benchamrk to single indexing stress test
|
2010-08-20 15:05:21 +03:00 |
|
kimchy
|
ef64f5d970
|
move memory monitor benchamrk to single indexing stress test
|
2010-08-20 13:41:33 +03:00 |
|
kimchy
|
c0552bdc70
|
Can't set "no stopwords" on analyzer, closes #329.
|
2010-08-20 03:22:43 +03:00 |
|
kimchy
|
6f1a9fbfb9
|
create md5 files on the local index storage as well, so they won't have to be recomputed when performing initial recovery
|
2010-08-20 02:28:14 +03:00 |
|
kimchy
|
acae0650c8
|
bail out of recovery when close and still waiting on throttling
|
2010-08-20 01:24:02 +03:00 |
|
kimchy
|
a05c367c5e
|
don't snapshot gateway when cleaning shard
|
2010-08-20 00:59:37 +03:00 |
|
kimchy
|
4a2e076c6a
|
do a reuse work prefetching not on the cluster state thread, so it won't block while fetching md5s from other nodes
|
2010-08-20 00:45:17 +03:00 |
|
kimchy
|
a5c5f65da5
|
fix recovery status when all index got reused
|
2010-08-19 22:46:58 +03:00 |
|
kimchy
|
7437acfcea
|
add progress to indices status API
|
2010-08-19 22:29:31 +03:00 |
|
kimchy
|
8d0e5b239c
|
recator source rest building into common code
|
2010-08-19 20:24:12 +03:00 |
|
kimchy
|
7b1093fb16
|
anoter compressed string test
|
2010-08-19 18:45:15 +03:00 |
|
kimchy
|
19abe7a2a5
|
when using compressed source, extract the compressed source directly into the response without any buffering
|
2010-08-19 18:29:36 +03:00 |
|
kimchy
|
46ccee8f89
|
when deleting an index, remove any blocks associated with it
|
2010-08-19 17:21:14 +03:00 |
|
kimchy
|
c40935ae14
|
store the mapping definition in compressed form internally to save memory
|
2010-08-19 17:06:36 +03:00 |
|
kimchy
|
a2d10d490d
|
allocate list when deserializing, NPE galore...
|
2010-08-19 16:40:51 +03:00 |
|
kimchy
|
04ae1581a0
|
fix NPE introduced on optimization of index status on docs status creation
|
2010-08-19 15:37:06 +03:00 |
|
kimchy
|
70c344b079
|
add compressed string
|
2010-08-19 15:33:57 +03:00 |
|
kimchy
|
1a18387fab
|
add `standard_html_strip` analyzer that combines the standard analyzer with html_strip char filter.
|
2010-08-19 15:10:31 +03:00 |
|
kimchy
|
22ea5e6608
|
fix primary election on replica relocation
|
2010-08-19 14:59:50 +03:00 |
|
kimchy
|
9c0bbe9bb5
|
log failure to notify master on shard started/failed
|
2010-08-19 13:49:48 +03:00 |
|
kimchy
|
fd569a9523
|
renme tests to reflect replica and not backup
|
2010-08-19 13:11:53 +03:00 |
|
kimchy
|
27e92cd505
|
fix a case where node might not exists on relocating, don't fail the replication action (index / delete / ...) because of it
|
2010-08-19 13:08:20 +03:00 |
|
kimchy
|
51656552a5
|
better mehtod name
|
2010-08-19 11:20:06 +03:00 |
|
kimchy
|
965d7303cf
|
index status action - some docs and extract common docs status
|
2010-08-19 09:34:02 +03:00 |
|
kimchy
|
a3107bc5b1
|
add snapshot status (either current one or the latest one executed) to indices status api
|
2010-08-18 17:11:29 +03:00 |
|
kimchy
|
91aada2706
|
boolean values cannot be searched with expected queries, closes #327.
|
2010-08-18 15:11:23 +03:00 |
|
kimchy
|
de71a1ce99
|
filtered more_like_this doesn't parse correctly, closes #326.
|
2010-08-18 14:36:34 +03:00 |
|
kimchy
|
2259ef671b
|
Gateway: Failure to read full translog from the gateway, closes #328.
|
2010-08-18 14:29:46 +03:00 |
|
kimchy
|
3f9034b41c
|
add gateway recovery status to the indices status API exposing both on going and summary when recovering from a gateway
|
2010-08-17 22:34:56 +03:00 |
|
kimchy
|
311520d146
|
add peer recovery status to the indices status API exposing both on going and summary when recovering from a peer shard
|
2010-08-17 21:33:11 +03:00 |
|
kimchy
|
5fb80c391b
|
add peer recovery status to the indices status API exposing both on going and summary when recovering from a peer shard
|
2010-08-17 21:23:05 +03:00 |
|
kimchy
|
96fc16dddb
|
add ability to iterate over assigned shards, and add an extension point to control shard routing iteration in the broadcast based action support
|
2010-08-17 14:04:12 +03:00 |
|
kimchy
|
7833cb1c76
|
Document IDs need to be URL decoded when indexed, closes #324.
|
2010-08-17 10:47:02 +03:00 |
|
kimchy
|
c989d3a928
|
fix limit on UTF writing (the new mechanism) to be higher than short
|
2010-08-17 10:21:26 +03:00 |
|
kimchy
|
e1f3fee4e4
|
internally store on going state of snapshot to gateway
|
2010-08-17 10:03:19 +03:00 |
|
kimchy
|
29e981d28d
|
internally store on going state of recovery from gateway
|
2010-08-17 08:17:29 +03:00 |
|
kimchy
|
92fd9af2b9
|
more info stored on shard recovery process, start time and recovery time
|
2010-08-16 18:22:19 +03:00 |
|
kimchy
|
92aa91d7b2
|
update the current recvoerd files size in peer recovery
|
2010-08-16 09:11:51 +03:00 |
|
kimchy
|
a012e0f1bd
|
allow to configure translog batch size (internal) and compress translog operations when doing peer recvoery
|
2010-08-16 09:03:48 +03:00 |
|
kimchy
|
1bdce4c7ef
|
capture current state of peer level recovery
|
2010-08-16 09:00:34 +03:00 |
|
kimchy
|
57ee1bdc55
|
improve utf based stream input / output handling
|
2010-08-15 21:19:36 +03:00 |
|
kimchy
|
14237317fc
|
support compression of the source field
|
2010-08-15 17:57:14 +03:00 |
|
kimchy
|
ee33ee457a
|
allow to explicitly disable compression in peer recovery (defaults to true)
|
2010-08-15 15:13:45 +03:00 |
|
kimchy
|
f6c58ff35d
|
change default calibrate_size_by_deletes for merge policy to true from false (this is what upcoming lucene version will use, and makes much more sense)
|
2010-08-15 09:01:09 +03:00 |
|
kimchy
|
6d509a5e4e
|
explicitly clean stream handles
|
2010-08-15 03:08:01 +03:00 |
|
kimchy
|
c18904eb96
|
Recovery: Using transport compression explicitly when recovering an index from a peer shard, closes #322.
|
2010-08-15 03:01:35 +03:00 |
|
kimchy
|
1ee2f80e68
|
Transport: add global compression support compressing all internal transport communication (using lzf), closes #321.
|
2010-08-15 02:57:22 +03:00 |
|
kimchy
|
c10544479f
|
refactor cached streams into a separate class
|
2010-08-14 13:47:34 +03:00 |
|
kimchy
|
01b3d42cee
|
separet recovery tests to one with small buffer size and on with bigger one
|
2010-08-14 13:44:42 +03:00 |
|
kimchy
|
e1226626bc
|
improve thread cached bytes allocation
|
2010-08-14 02:26:12 +03:00 |
|
kimchy
|
7c5f0fe405
|
minor transport refactoring, add options to send response, though no options to set yet
|
2010-08-14 02:25:58 +03:00 |
|
kimchy
|
b3afca2589
|
remove compressor package, move lzf to a different package
|
2010-08-14 02:25:30 +03:00 |
|
kimchy
|
0adb2edba2
|
clean code
|
2010-08-14 01:23:04 +03:00 |
|
kimchy
|
75ee01a6b4
|
clean builder code when dealing with maps
|
2010-08-14 00:37:33 +03:00 |
|
kimchy
|
4a9c7d672e
|
clean code
|
2010-08-13 23:50:16 +03:00 |
|
kimchy
|
d12c757824
|
Gateway: Chunk based storage broken, fails to recover from gateway, closes #318.
|
2010-08-13 14:03:52 +03:00 |
|
kimchy
|
12ef12f7aa
|
better handling of closing a shard / index while retrying recovery
|
2010-08-12 21:44:27 +03:00 |
|
kimchy
|
2bd9a63467
|
Query DSL: custom score script, allow to use `_score` as well as `score` as the underlying query score, closes #316.
|
2010-08-12 18:41:29 +03:00 |
|
kimchy
|
98bc8285ea
|
Analysis: Add `char_filter` on top of `tokenizer`, `filter`, and `analyzer`. Add an `html_strip` char filter, closes #315.
|
2010-08-12 18:16:30 +03:00 |
|
kimchy
|
e29925684a
|
Term Facets: Add `reverse_cout` and `reverse_term` to `order` options, closes #314.
|
2010-08-12 15:26:26 +03:00 |
|
kimchy
|
b1d1f1ff94
|
improve peer recovery shard state handling. when throttling a recovery, don't restore the non recovering state and then move to recovery again with each retry, keep the recovering state while retrying
|
2010-08-12 12:31:27 +03:00 |
|
kimchy
|
c0a7dc327c
|
range facet: from is used in to_str, should be to
|
2010-08-11 23:40:42 +03:00 |
|
kimchy
|
a6d84f5942
|
no need to spawn a thread when response to a ping response
|
2010-08-11 13:20:13 +03:00 |
|
kimchy
|
fbea92e688
|
REST API hang on a bogus call, closes #297.
|
2010-08-11 13:18:33 +03:00 |
|
kimchy
|
ee26d55296
|
Remove memory monitor and move translog operations threshold to shard level setting, closes #312.
|
2010-08-11 12:54:00 +03:00 |
|
kimchy
|
df4ece8cef
|
remove no longer required commented code ;)
|
2010-08-11 11:43:52 +03:00 |
|
kimchy
|
9f8e5cb2c0
|
add a simple integration level test for scroll
|
2010-08-10 20:48:05 +03:00 |
|
kimchy
|
7977edd5db
|
a filter based on the type of the document is built in several places in the code, which is wasteful. Instead, create a filter based on the type on the document mapper level, and reuse it where applicable.
|
2010-08-10 18:34:49 +03:00 |
|
kimchy
|
5f986ef422
|
build a match all filter internal implementation and use it where appropiate (instead of using query filter wrapper around a match all query)
|
2010-08-10 18:18:51 +03:00 |