Commit Graph

855 Commits

Author SHA1 Message Date
kimchy 8d669ff54e move sorting to be done using the new fielddata cache, and not using lucene built in FieldCache 2010-07-08 14:42:35 +03:00
kimchy 97b5b41522 enable eager cache eviction based on readers, requires a nasty hack to Lucene, basically overriding two classes to hook into when a reader actually gets closed (and only enabled when elasticsearch is before lucene in the classpath, which we take for in the startup scripts) 2010-07-07 17:15:26 +03:00
kimchy a36b06c682 ignore failed snapshot on a non started index 2010-07-07 16:59:33 +03:00
kimchy a8c317e355 fix terms facets test to take into account distributed terms request 2010-07-07 16:33:39 +03:00
kimchy 5ff46911b6 ignore recovery when no node to recover from (it has disconnected, for example) 2010-07-07 16:30:28 +03:00
kimchy 3fb68c52da ignore snapshot on a shard thas has just been created 2010-07-07 16:30:17 +03:00
kimchy 11a8d637c0 ignore snapshot on a shard thas has just been created 2010-07-07 16:26:19 +03:00
kimchy e40aa540fe ignore recovery when no node to recover from (it has disconnected, for example) 2010-07-07 16:25:55 +03:00
kimchy 9578ad3ef1 Terms Facets: Allow to specify a set of terms to exclude in the request, closes #246. 2010-07-07 14:41:34 +03:00
kimchy 2b5458daf6 implement a simple file system based translog, not the defeault yet, should provide better memory management 2010-07-07 09:30:07 +03:00
kimchy b2bdb149f9 log shard close failure when closing an index 2010-07-06 22:50:40 +03:00
kimchy 6aa9be238c use the atomic integer counter to represent the size 2010-07-06 18:45:19 +03:00
kimchy 267016758d improve handling of memory caching with file system, only force compound file when really needed (when an extension that exists within the compound file is part of the memory cached extensions) 2010-07-06 18:21:36 +03:00
kimchy ac01bb86e8 clean lock only before opening an index writer 2010-07-06 18:13:05 +03:00
kimchy 2908418d37 checn the default byte buffer cache to 200mb, now that its global per node 2010-07-06 17:58:30 +03:00
kimchy 070cb5b295 allocate replica shards (backups) only after the primaries have been allocated, this results in better theoretical shard allocation, and the ability to reuse local storage index files (comared against the backup) 2010-07-06 17:58:05 +03:00
kimchy 1d39bb4d51 cache md5 computation for unallocated shard and reused it 2010-07-06 09:00:24 +03:00
kimchy d4f86899e3 more work on reuse work dir, improved transaction log to move to stream based 2010-07-06 00:12:40 +03:00
kimchy b078c9206a more work on reuse work dir, added md5 logic to identify what need to be recovered or not, note, requires reindexing 2010-07-05 14:13:55 +03:00
kimchy 79b69764e0 doc 2010-07-01 15:03:11 +03:00
kimchy e8b261c165 better shutdown handling 2010-07-01 10:28:57 +03:00
kimchy ad5525f575 fix unwrapping of exceptions 2010-07-01 10:27:53 +03:00
kimchy f6c9c49645 better logging when failing to read full content of a transport message 2010-07-01 03:00:57 +03:00
kimchy c75a56ca17 better shutdown handling 2010-07-01 02:41:00 +03:00
kimchy c2d02e4e3a wrong serialization of shardId 2010-07-01 02:18:35 +03:00
kimchy d848ab3df3 rename executors to esexecutors 2010-07-01 00:06:34 +03:00
kimchy b0d736b489 make the shard routing an actual component 2010-06-30 23:39:27 +03:00
kimchy 811559f393 add another test if the pool size is greater than the core pool size before a worker can exit 2010-06-30 23:08:05 +03:00
kimchy 6709518c4e add the ability to list shard index blobs without actually having one created 2010-06-30 22:54:59 +03:00
kimchy 66096e97f2 Remove the terms API (replaced by facets in search), closes #242. 2010-06-30 19:49:41 +03:00
kimchy df7e3f55a4 improve cluster blocks data structures for simpler usage 2010-06-30 16:06:31 +03:00
kimchy 4049da97fb time value to support parsing days as well 2010-06-30 13:00:25 +03:00
kimchy 088e0b5a64 Automatic management of indexing buffer size, closes #241. 2010-06-30 01:46:57 +03:00
kimchy fdb2eff998 Automatic management of indexing buffer size, closes #241. 2010-06-29 22:54:54 +03:00
kimchy 03ac38fdba better benchmark on memory monitor 2010-06-29 22:12:43 +03:00
kimchy 3f6cd46736 allow to configure tcp network modules (transport, memcached and http) to use blocking io (just enabling it to do some performance tests) 2010-06-29 21:07:38 +03:00
kimchy 01ff81fa89 fix log message 2010-06-29 19:49:28 +03:00
kimchy bc803b4fcf close the thread pool on test 2010-06-29 18:35:03 +03:00
kimchy 703657699a refactor new thread pools in, remove dynamic ones 2010-06-29 18:20:31 +03:00
kimchy de4b21e986 backups don't become primaries on node failures, primaries are first allocated and then start initializing from gateway 2010-06-29 18:20:13 +03:00
kimchy 514df4ee3f refactor the shard routing abstraction, no need for it 2010-06-29 18:02:10 +03:00
kimchy fe5e5a073b work on scaling and blocking thread pool 2010-06-29 18:01:54 +03:00
kimchy 9051ae3a65 work on scaling and blocking thread pool 2010-06-29 17:50:05 +03:00
kimchy 297001de53 update benchmark code 2010-06-29 08:59:28 +03:00
kimchy aa5791a14f move block metadata checks to the transport level 2010-06-29 08:51:36 +03:00
kimchy a82bf74c1c expose blocks in the cluster state rest api 2010-06-29 01:00:07 +03:00
kimchy 5f538b1ba3 Block operation performed on cluster until it recovered from the gateway, closes #239. 2010-06-28 22:53:13 +03:00
kimchy b970f0dcc2 fix toString of bytes value 2010-06-28 17:53:07 +03:00
kimchy 63523f4525 Query DSL: field query does not take into account `allow_leading_wildcards`, closes #236. 2010-06-28 12:33:36 +03:00
kimchy cb9be9973b add the ability to define meta _attributes for mapping, basically a place to store additional unstructured data on the mapping 2010-06-28 09:10:08 +03:00
kimchy 3770924300 allow to control the buffer size of the gateway blob container 2010-06-27 23:26:32 +03:00
kimchy a7c13826da more work on reusing work directory, clean unallocated shards when they are not needed 2010-06-27 18:19:12 +03:00
kimchy b43f0f5965 add streamable to stores files metadata 2010-06-27 14:18:14 +03:00
kimchy 4ae45946a2 nicer logging 2010-06-27 13:50:27 +03:00
kimchy fc7392a98b fix logging 2010-06-27 13:07:21 +03:00
kimchy 90c995dda5 add todo 2010-06-27 12:28:30 +03:00
kimchy 1e7481d5ab fix logging 2010-06-27 12:15:58 +03:00
kimchy c56609a0ad fix default merge size to not truncate on long 2010-06-27 12:14:38 +03:00
kimchy bbb9c3883f fix jdoc 2010-06-27 12:05:30 +03:00
kimchy f1f695b4d9 fix logging 2010-06-27 09:39:07 +03:00
kimchy b9590b5ed6 fix logging 2010-06-27 09:37:44 +03:00
kimchy edd8786a81 close open index input on failure 2010-06-27 02:13:59 +03:00
kimchy 33d357dbb4 Memory Store / FS Memory: Create a node level memory store cache and allocator, closes #235. 2010-06-26 22:34:30 +03:00
kimchy 00d2abef40 refactor facets builder API to make it more manageable 2010-06-26 01:36:42 +03:00
kimchy 5df2257319 add test for facets with search size 0 2010-06-25 22:46:07 +03:00
kimchy 83010f7ee1 more apis on the index level store to help clean unused work shards when needed 2010-06-25 03:39:42 +03:00
kimchy 811856e1f8 better method name 2010-06-25 01:41:06 +03:00
kimchy 98e93322ae use java user.dir as the default "path.home" location 2010-06-24 22:11:13 +03:00
kimchy 4709f00790 add to index store the ability to get the backing store free / total space 2010-06-24 18:05:21 +03:00
kimchy edf0075025 more work on reusing work node level data, now when recovering from peers 2010-06-24 15:34:02 +03:00
kimchy 57169d4233 initial work on reusing work node level data when recovering from gateway 2010-06-24 00:32:05 +03:00
kimchy 0d7cebbaf0 upgrade to async http client 1.0.0 2010-06-23 21:51:29 +03:00
kimchy 296b6b2f57 use custom similarity in search (if there is one) 2010-06-23 14:30:14 +03:00
kimchy a7fdd36f6e clean files that are in the store but not in the gateway 2010-06-23 12:21:04 +03:00
kimchy d5f35f3cee upgrade to netty 3.2.1.Final 2010-06-23 10:52:16 +03:00
kimchy 2121aaad3b part of the index store enhacement 2010-06-23 10:49:06 +03:00
kimchy c4aff14e15 remove field options from field cache and freqs, simpler and they are not used in any case 2010-06-23 09:48:46 +03:00
kimchy cccce2b114 fix work with cluster setting 2010-06-23 08:00:29 +03:00
kimchy 3b4584040b add index level store constuct (on top of the shard level store) 2010-06-23 07:47:18 +03:00
kimchy 8e62bb6934 Cluster State API: Allow to filter the state on nodes, routing_table, metadata, and indices, closes #234. 2010-06-22 16:19:10 +03:00
kimchy dcb0d18519 javadoc 2010-06-22 16:09:49 +03:00
kimchy 622d4041b8 handle cases where the node does not exists in the disco nodes 2010-06-22 10:59:24 +03:00
kimchy 7c931f34fa ignore state exceptions when trying to flush 2010-06-22 10:26:44 +03:00
kimchy 0c6b82aeca fix javadoc 2010-06-22 09:26:37 +03:00
kimchy 3d84af2a40 refactor hdfs gateway to use the new common blobstore 2010-06-22 09:10:00 +03:00
kimchy 21627bca10 return after notifiying listener on failure 2010-06-21 23:29:13 +03:00
kimchy 893de86e9e fix missing break statemetns (via Lukas) 2010-06-21 22:45:02 +03:00
kimchy 7ed7c6db4e Gateway: Internal refactoring, requires manual upgrade when using fs gateway, closes #232. 2010-06-21 22:40:12 +03:00
kimchy 733abdcac3 more unboxed math 2010-06-20 14:42:11 +03:00
kimchy 701920c6c1 remove unused API in field mapper (that also cause compilation to break on eclipse) 2010-06-20 12:12:01 +03:00
kimchy e846ed6465 Lucene: Upgrade to latest 3.0.2 version, closes #231. 2010-06-20 01:04:39 +03:00
kimchy 384f8a4f42 Return the maxScore per search and score for each search hit, closes #205. 2010-06-20 00:23:27 +03:00
kimchy 0f2147aeec add some prepare APIs to the groovy APIs 2010-06-19 21:09:03 +03:00
kimchy 891c69f8de refactor builder requets into common base class 2010-06-18 14:26:33 +03:00
kimchy adef9d69ff Gateway: Allow to configure a `recovery_after_time` and `recover_after_nodes`, closes #223. 2010-06-18 05:32:51 +03:00
kimchy 22926f0026 Analysis: When specifying empty array for stopwords, use an empty list for stopwords, closes #230. 2010-06-18 00:34:18 +03:00
kimchy 8e4c139c8b Analysis: fields that are `not_analyzed` should automatically default to keyword analyzer, closes #229. 2010-06-17 23:23:53 +03:00
kimchy 60bd8f34d2 add debug logging of the ping responses received 2010-06-17 22:37:47 +03:00
kimchy f64b96e3c8 add an exception if there is no master node set 2010-06-17 22:19:43 +03:00
kimchy 7b145fd4e1 Expose the mappings of a given index or index/type, closes #222. Added the ability to filter returned data from the cluster state API, and use it to expose REST GET endpoint for "/_mapping", "/{index}/_mapping", "/{index}/{type}/_mapping" ({index} can also be "_all"). 2010-06-17 17:56:07 +03:00
kimchy 3bc8c307f1 simplify netty transport to use single channel 2010-06-17 12:50:01 +03:00
kimchy 597f020156 fork node connected and disconnected on a different thread pool 2010-06-17 12:47:42 +03:00
kimchy 662ded3801 connect to a node when it joins the cluster on the disco level, so if it fails, it will be propagated back and the node will not be added to the cluster state 2010-06-17 12:46:53 +03:00
kimchy 86134d32a8 nicer logging 2010-06-17 12:46:05 +03:00
kimchy d1c7bde07e make the rest cluster state return mappings as object keyed type 2010-06-17 02:45:17 +03:00
kimchy 38005826e7 detect write errors early and notify transport handler 2010-06-17 02:17:54 +03:00
kimchy 8af925a0b1 add internal flags to help simulate forcefull disconnections 2010-06-17 01:05:23 +03:00
kimchy a60e542718 cancel the timeout if we fail to send the request 2010-06-16 23:07:41 +03:00
kimchy b0c8255cd9 clean the cluster state when starting (handles restarts better) 2010-06-16 21:59:26 +03:00
kimchy a59679b0c6 better failure messages 2010-06-16 21:47:58 +03:00
kimchy 4d26acd6da add aliases to cluster state rest response 2010-06-16 06:08:36 +03:00
kimchy a547c1ccc0 cluster state to return native json (if possible) as part of the mappings 2010-06-16 05:48:30 +03:00
kimchy 626989c55e don't return the source when asking for scripted fields 2010-06-16 04:48:06 +03:00
kimchy 2d6f61b3c1 Facets: Script Histogram facet, closes #228. 2010-06-16 04:22:00 +03:00
kimchy bb24b56c66 Facets: Script statistical facets, closes #227. 2010-06-16 02:18:24 +03:00
kimchy c2786038e2 Script Filter: Support providing a custom script as a filter, closes #226. 2010-06-16 01:32:04 +03:00
kimchy be3b779caa Tcp Transport: Reduce `transport.tcp.connection_per_node` from 5 to 1, closes #225. 2010-06-15 23:36:31 +03:00
kimchy 8e589b5a04 nicer code 2010-06-15 23:14:04 +03:00
kimchy eaf124123d reduce long name 2010-06-15 23:09:09 +03:00
kimchy 65039ff21d Zen Discovery: When a master node is forcefully killed, other nodes might not monitor the other elected master, closes #224 2010-06-15 23:08:52 +03:00
kimchy eb68891ae5 move size value/unit to bytes size value/unit, and create a ten based size value/unit 2010-06-15 18:26:44 +03:00
kimchy a18e58b394 move size value/unit to bytes size value/unit, and create a ten based size value/unit 2010-06-15 18:16:10 +03:00
kimchy f75abbaa00 big refactoring thanks to proper jarjar built from source with asm 3.3 allowing to jarjar guice and others, includes package relocations 2010-06-15 17:59:04 +03:00
kimchy d5bd77cc38 big refactoring thanks to proper jarjar built from source with asm 3.3 allowing to jarjar guice and others, includes package relocations 2010-06-15 17:28:05 +03:00
kimchy 636bfe8466 big refactoring thanks to proper jarjar built from source with asm 3.3 allowing to jarjar guice and others, includes package relocations 2010-06-15 16:51:38 +03:00
kimchy 48979ab6c8 improve gateway recovery when using delay index creation, close the loophole when the master was shut down before the delay expired and now other node becoming master will do the recovery 2010-06-15 10:27:26 +03:00
kimchy 98df1b3433 Gateway: Allow to configure a delay till index creation from gateway will occur, closes #223. 2010-06-15 01:25:55 +03:00
kimchy e0d20af743 ignore when flush can't be done 2010-06-14 13:25:18 +03:00
kimchy 0a1bc874c3 Search: Allow to specify script fields to be loaded, closes #221. 2010-06-14 12:50:33 +03:00
kimchy c087bbe804 use List for fieldNames to reduce extra array construction 2010-06-14 04:36:03 +03:00
kimchy 953779ccea allow to pass parameters to custom score script 2010-06-14 04:12:57 +03:00
kimchy 1b32c1ccf4 add cleanable thread local to single values 2010-06-14 03:42:43 +03:00
kimchy 751c56f31e Query DSL: custom score (scripted) query, closes #220. 2010-06-14 03:15:23 +03:00
kimchy 2d0785078c move to field.data package 2010-06-13 18:05:59 +03:00
kimchy a9fc276a3e refactor the custom boost factor query into a more general function boost query 2010-06-13 16:51:19 +03:00
kimchy ec481159d6 pass score docs in order in custom boost factor query 2010-06-13 12:41:16 +03:00
kimchy 2daa54a2bb don't clean thread locals explicitly, let GC clean them 2010-06-12 12:45:38 +03:00
kimchy d515fe8644 add internal support for Rest/HTTP OPTIONS and HEAD 2010-06-11 00:13:08 +03:00
kimchy 5205a183e8 use deletion aware constant score query 2010-06-10 15:04:36 +03:00
kimchy 668e84a99e add specific doc object, DocFieldData and specific types extension to allow in the future for simpler doc level operations 2010-06-09 23:35:31 +03:00
kimchy 497b4a4f10 filtres should implement equals and hashcode 2010-06-09 21:23:19 +03:00
kimchy 64479a11c3 Search Facets: Histogram facets to allow to specify different key and value fields 2010-06-09 00:50:07 +03:00
kimchy 5ca050ffcb Search Facets: Facets to allow to define filters on them, closes #217. 2010-06-08 22:30:21 +03:00
kimchy 6257b78243 Query DSL: Add `and`, `or`, and `not` filters, closes #216 2010-06-08 18:04:34 +03:00
kimchy 47b3a81bec Search Facets: Histogram Facets, closes #215 2010-06-08 15:45:58 +03:00
kimchy d36cea07f3 refactor some facet support 2010-06-08 11:46:30 +03:00
kimchy 874993557c add sum of squares, variance, and std deviation for statistical facet 2010-06-08 10:39:45 +03:00
kimchy 26ed029d8a Query DSL: Support fuzzy query, closes #213. 2010-06-07 18:13:57 +03:00
kimchy c772e7c337 better comment 2010-06-06 14:18:58 +03:00
kimchy f0c56ce18e nicer code, internalize Facets and create a clean external interface 2010-06-06 03:14:34 +03:00
kimchy 81deb833de numeric statistical facets support 2010-06-05 23:58:05 +03:00
kimchy 8aeb589a42 numeric statistical facets support 2010-06-05 23:44:34 +03:00
kimchy 57a21bedd2 update doc 2010-06-05 21:59:42 +03:00
kimchy 88ba980f79 refactoring of facets to be more open for different facet types 2010-06-05 21:49:10 +03:00
kimchy d9d9304f47 add TODO 2010-06-05 15:59:06 +03:00
kimchy 83ad15b9f2 improve cleaning algo, force clean only ones that have not been cleaned, and do a force clean only when we start reaching the full clean 2010-06-05 14:35:12 +03:00
kimchy 5cdba0383b Flush API: Allow to provide `full` parameter for a complete clean, closes #210. 2010-06-05 06:10:02 +03:00
kimchy ef1866aed4 add flush logging 2010-06-05 03:10:55 +03:00
kimchy bcbc0dd741 Search: Search requests hangs when no indices exists, closes #209. 2010-06-05 01:36:24 +03:00
kimchy d0eb836c4a Plugins: Allow to easily plug a custom DSL query/filter parsers, closes #208. 2010-06-05 01:07:12 +03:00
kimchy 9f8644472e more tuning of field data capabilities, make term facet collector handle any type (though it really make sense mainly for string types) 2010-06-04 14:03:43 +03:00
kimchy 0901322337 rename field facet to term facet 2010-06-04 07:03:30 +03:00
kimchy 6e15e3dd24 add helper methods to field data 2010-06-04 06:15:35 +03:00
kimchy 13d7f10b7b add helper methods to field data 2010-06-04 04:42:44 +03:00
kimchy 66c9f2f834 Search Facets: Field Facets, closes #207 2010-06-04 02:31:49 +03:00
kimchy dbb7df742e protected if the cached stream got evicted on us 2010-06-03 20:48:57 +03:00
kimchy c68787aa4c most master node outside of the nodes section, so all elements in nodes will have the same strucutre 2010-06-03 14:30:54 +03:00
kimchy d9ec629473 clean log 2010-06-03 11:42:20 +03:00
kimchy b2f90a2133 REST API does not expose node-master status, closes #203. 2010-06-02 17:04:43 +03:00
kimchy 6c8f49c37d REST API does not expose node-master status, closes #203. 2010-06-02 16:59:55 +03:00
kimchy c2d583064e FS Gateway: Allow to configure is native file copying will be used, closes #202. 2010-06-02 12:06:41 +03:00
kimchy 5ef421e779 Nodes Info API: Failed to generate REST response when node attributes are set, closes #201. 2010-06-02 01:10:43 +03:00
kimchy a7ad295f63 Zen Discovery: ungraceful shutdown of the master and start of replacement node might cause the cluster not to elect a new master, closes #200. 2010-06-01 22:29:38 +03:00
kimchy ec662cc019 always start the unicast ping discovery, so unicast discovery will work even when using multicast 2010-05-31 22:24:51 +03:00
kimchy 4b85407f5e work on a nicer java api 2010-05-29 22:23:39 +03:00
kimchy e59b413980 work on a nicer java api 2010-05-29 05:50:17 +03:00
kimchy 3406e77cec Streamline single value with boost queries: Allow for both value and query name to specify the query value, closes #192. 2010-05-28 12:23:51 +03:00
kimchy 493464ba02 improve logging 2010-05-28 05:26:51 +03:00
kimchy a7cce9cd38 Replication Actions: Allow to control replication type - `async` or `sync`, closes #196. 2010-05-28 03:56:04 +03:00
kimchy 84a5c1eac8 Replication Actions: Allow to control replication type - `async` or `sync`, closes #196. 2010-05-28 03:47:35 +03:00
kimchy 477024e3be cached termsSet used in dfs phase 2010-05-27 20:31:48 +03:00
kimchy 7e5bfb5d3b share the open channels handler 2010-05-27 18:25:59 +03:00
kimchy ded270fa03 add search id when failing to execute 2010-05-27 14:29:04 +03:00
kimchy 71ece4e31c rever back all interfaces binding in multicast 2010-05-27 13:12:33 +03:00
kimchy d7984e9962 rever back all interfaces binding in multicast 2010-05-27 12:43:05 +03:00
kimchy d14a8afaf0 add another trace logging 2010-05-27 12:12:18 +03:00
kimchy 101652d720 add another trace logging 2010-05-27 12:10:55 +03:00
kimchy 74fa3ed32f Zen Discovery: Improve Multicast Binding and Sending, closes #195. 2010-05-27 02:11:06 +03:00
kimchy 9433f4d651 Network Settings: Allow to use '_' as well as '#' for special host names, closes #194. 2010-05-26 16:33:35 +03:00
kimchy 40d4a350fd nicer code 2010-05-26 13:01:16 +03:00
kimchy 26364afd7e Failed to acquire lock, closes #193. 2010-05-26 12:29:13 +03:00
kimchy 463910da45 handle null doc id set 2010-05-26 11:20:15 +03:00
kimchy 9a49629d17 throw an exception when an action is not found 2010-05-26 09:16:10 +03:00
kimchy 32e4c405de add async http client 2010-05-24 23:27:07 +03:00
kimchy b7d11f1303 Gateway Storage: Improve it to support non breaking changes in the future, closes #191. 2010-05-24 18:27:48 +03:00
kimchy 116cfce6f2 FS Gateway: Change indices location (Requires manual upgrade), closes #188. 2010-05-23 08:37:41 +03:00
kimchy 5c6864e8b3 inital work on hadoop plugin - gateway hdfs support 2010-05-23 08:17:52 +03:00
kimchy 596f9db8d8 fix to build the facets map on getFacets 2010-05-22 21:12:07 +03:00
kimchy a372643ca9 remove using Lucene StringHelper in the code 2010-05-22 16:52:22 +03:00
kimchy 4c51ddd2ad update comment for next lucene version 2010-05-22 13:22:27 +03:00