Commit Graph

3294 Commits

Author SHA1 Message Date
Boaz Leskes 33e49a55f7 Increased timeout in ClusterServiceTests.testTimeoutUpdateTask to 100ms
The previous 2ms timeout was too small and caused a racing condition in timeout handling. This will be fixed but requires more work.
2013-12-16 10:02:24 +01:00
Simon Willnauer a668cd22d8 wait until cluster-state is ready after startup 2013-12-15 22:26:11 +01:00
Simon Willnauer d9e988aea9 Fix QuorumLocalGatewayTests#testChangeInitialShardsRecovery to start only one shard to not fulfill the quorum 2013-12-15 20:06:22 +01:00
Igor Motov d8ba92cfa8 Resolve potential deadlock state during EsThreadPoolExecutor shutdown
Fixes #4334

The deadlock occurs between monitor object of EsThreadPoolExecutor and mainLock of ThreadPoolExecutor. The shutdown method of EsThreadPoolExecutor obtains the lock on monitor first and waits for mainLock of ThreadPoolExecutor in ThreadPoolExecutor#shutdown for part of the processing, while EsThreadPoolExecutor#terminated is executed under mainLock and tries to obtain monitor to notify listeners.
2013-12-14 21:45:03 -05:00
Shay Banon 80ab75e54e upgrade to latest jsr166 2013-12-14 01:27:18 +01:00
Luca Cavanna 9b121baafc More common AcknowledgedRestResponseActionListener
Introduced use of AcknowledgedRestResponseActionListener in put/delete index template
2013-12-13 19:39:46 +01:00
Simon Willnauer d684a2b8da Don't try to run check-index in MockDirectoryWrapper
if the MockDirWrapper checks the index on close it closes all
closeables that is has to crash the index if something is not flushed yet.
For us this is a problem since the input is still used. We need to fix this in
lucene first.
2013-12-13 17:44:10 +01:00
Adrien Grand 19c8d18b14 Relax a bit the accuracy tests on SloppyMaths.
Tests now ensure that the computed distance is correct within 1‰ instead of 1‱.
2013-12-13 15:42:39 +01:00
Adrien Grand 36bd9cc432 Aggregations: Ordinals-based string bucketing support.
When the ValuesSource has ordinals, terms ordinals are used as a cache key to
bucket ordinals. This can make terms aggregations on String terms significantly
faster.

Close #4350
2013-12-13 15:34:02 +01:00
Martijn van Groningen cc61348013 Fixed test bug 2013-12-13 14:54:21 +01:00
Martijn van Groningen 10e2528cce Added the `force_source` option to highlighting that enforces to use of the _source even if there are stored fields.
The percolator uses this option to deal with the fact that the MemoryIndex doesn't support stored fields,
this is possible b/c the _source of the document being percolated is always present.

Closes #4348
2013-12-13 13:39:53 +01:00
Simon Willnauer 59e4e58683 Add trace logging to AllocationDeciders if decision is NO() 2013-12-13 12:08:26 +01:00
Sebastian Geidies 6ed126deaf Skipping execution of remaining Deciders if one of them returns a Decision.NO.
Reordering execution of the different AllocationDeciders, so the cheapest get executed first.
2013-12-13 11:52:48 +01:00
Boaz Leskes 2acccf257d Add `queries` to XContent output of PercolateStats
The queries stats tracked but not exposed to the Rest API

Closes #4440
2013-12-13 11:36:51 +01:00
Boaz Leskes b65d07b495 A failure during the response construction could cause a Node Operation never to return 2013-12-13 10:36:39 +01:00
Shay Banon 976bdd6a26 randomize transport tcp compression on nodes 2013-12-12 23:19:59 +01:00
Shay Banon 0469cca4be properly support global transport compression with new bytes transport request optimization 2013-12-12 22:29:54 +01:00
Shay Banon 0f4d81fcd4 refresh the mapping source directly into a compressed buffer
instead of building the mapping source, and then compressing it, we can generate it directly into a compressed buffer
2013-12-12 20:58:11 +01:00
Lee Hinman 77fcf71338 Add new `simple_query_string` query type
This adds support for Lucene's SimpleQueryParser by adding a new type
of query called the `simple_query_string`. The `simple_query_string`
query is designed to be able to parse human-entered queries without
throwing any exceptions.

Resolves #4159
2013-12-12 12:09:32 -07:00
Alexander Reelsen 348c3731c8 Fixing debian init script to work on ubuntu LTS
sysctl parameters must be separated like `-q -w` instead of `-qw`
2013-12-12 17:57:34 +01:00
Adrien Grand e2e19203d7 Remove IndexFieldData.getHighestNumberOfSeenUniqueValues().
Closes #4426
2013-12-12 15:44:05 +01:00
Shay Banon f0e1f9215b try and wait better for all mapping updates to be processes in test 2013-12-12 15:06:37 +01:00
Simon Willnauer 198d36dc15 Also wrap IndexInput if CFS is used 2013-12-12 12:14:47 +01:00
Simon Willnauer d95420cc67 Add more information to IndexInput failures related to abusing closed input 2013-12-12 10:44:01 +01:00
Adrien Grand 0554f8e875 Add missing mappers to MapperBuilders.
- completion
 - geo_point
 - geo_shape
 - parent
 - size
 - ttl

Close #4423
2013-12-12 09:44:55 +01:00
Martijn van Groningen b4d1d899d2 Replaced the synchronization method for enabling the realtime percolator from synchronized methods to use AtomicBoolean#compareAndSet instead.
Remove @TestLogging from percolator tests.
2013-12-11 22:54:35 +01:00
Luca Cavanna 2cbecd9cb3 Made sure that we never throw IndexMissingException in indices query and filter
It could happen although we internally use IgnoreIndices.MISSING, due to MetaData#concreteIndices contract, which throws IndexMissingException anyway if all requested indices are missing.
In case all the indices specified in the query/filter are missing, we just execute the no_match query/filter, no need to throw any error.

Closes #3428
2013-12-11 22:26:46 +01:00
Luca Cavanna 24434063b6 Introduced ok & acknowledged constant in AcknowledgedRestResponseActionListener 2013-12-11 21:28:54 +01:00
Luca Cavanna b1fa147968 More common AcknowledgedRestResponseActionListener
Introduced use of AcknowledgedRestResponseActionListener in put/delete alias & delete index
2013-12-11 21:28:15 +01:00
Luca Cavanna 037bf6a85f Unified default ack timeout to 30 seconds
Increased also default publish state timeout to 30 seconds (from 5 seconds) and introduced constant for it.
Introduced AcknowledgedRequest.DEFAULT_ACK_TIMEOUT constant.
Removed misleading default values coming from the REST layer.
Removed (in a bw compatible manner) the timeout support in put/delete index template as the timeout parameter was ignored.

Closes #4395
2013-12-11 20:54:27 +01:00
Martijn van Groningen f9ff733eb8 Renamed the methods in DocumentListenerType to actually reflect when these are invoked. 2013-12-11 17:51:09 +01:00
Martijn van Groningen 92c32dca9e Invoke the DocumentType listeners *before* the mappers are updated. This fixes an issue where if two or more concurrent percolate index requests are processed, the first request would the `.percolator` type mapping, but the real time percolator listener wouldn't be active, this would result in that the subsequent concurrent requests wouldn't be parsed and kept in memory and would never be included in any percolate api result. This issue any occurred when `.percolator` type is created on the fly.
Also made the call to PercolatorQueriesRegistry#enableRealTimePercolator and #disableRealTimePercolator synchronized, so that for the same shard the RealTimePercolatorOperationListener can't registered twice.
2013-12-11 17:41:10 +01:00
Shay Banon a3f1c428c2 bulk response has errors indication + status per item
closes #4002
2013-12-11 16:04:41 +01:00
Shay Banon 10cdb0ae22 Batch processing mapping updates can cause missed merged mappings when batching multiple types
when we bulk changes, we need to use the same index metadata builder across the tasks, otherwise we might remove mappings erroneously
 also, when we check if we can use a higher order mapping, we need to verify that its for the same mapping type
2013-12-11 14:34:59 +01:00
Martijn van Groningen a760f1f54a Make listeners a final field in ShardIndexingService, which fixed possible visibility issue. 2013-12-11 12:22:45 +01:00
Shay Banon aee388ec46 introduce an internal metadata uuid 2013-12-11 10:49:36 +01:00
Florian Schilling 937a4e9d9d SloppyMath
Added copy of SloppyMath.java from lucene 4.6+
and setup GeoDistance for new haversin method

closes #3862
2013-12-11 17:51:32 +09:00
Alexander Reelsen 81e13a870b Packaging: Ensure setting of sysctl vm.max_map_count
In order to be sure that memory mapped lucene directories are working
one can configure the kernel about how many memory mapped areas
a process may have. This setting ensure for the debian and redhat initscripts
as well as the systemd startup, that this setting is set high enough.

Closes #4397
2013-12-11 09:19:22 +01:00
Boaz Leskes 99b421925f Add wildcard support to field resolving in the Get Field Mapping API
Closes #4367
2013-12-10 23:46:37 +01:00
Shay Banon a9e259d438 Introduce BytesTransportRequest, allowing for downstream network optimization in buffers usage
When sending a request, mainly to multiple nodes, if we already have the "body" of the request in bytes, we can share it instead of copying it over to a new buffer. Also, it helps a lot when sending a relatively large body to multiple nodes, since it will use the same body buffer across all nodes
2013-12-10 22:49:31 +01:00
Alexander Reelsen c4f3da2b9d Preference only_node with unknown nodeId returns useful exception
When the search preference is set to only node, but this node is not a
data (or does not exist), we return a search exception, which indicates,
that this is actually a server problem.

However specifying a non-existing node id is a client problem
and should return a more useful error message than
{"error":"SearchPhaseExecutionException[Failed to execute phase [query_fetch], all shards failed]","status":503}
2013-12-10 20:15:14 +01:00
Martijn van Groningen 7d3b78c293 moved log statements 2013-12-10 19:54:27 +01:00
Richard Boulton 1037d071bf Fix bug in explain for function_score queries.
The explain output for function_score queries with score_mode=max or
score_mode=min was incorrect, returning instead the value of the last
function.  This change fixes this.
2013-12-10 17:59:00 +01:00
Simon Willnauer 6c189310b9 Remove 'term_index_interval' and 'term_index_divisor'
These settings are no longer relevant since they are codec /
postingsformat level settings since Lucene 4.0

Closes #3912
2013-12-10 16:54:08 +01:00
Martijn van Groningen 38cff53a0f Added more logging for percolator.
Lowered log level in percolator tests.
2013-12-10 15:45:04 +01:00
Adrien Grand 794a927c43 Use a dedicated cluster name for GeoDistanceSearchBenchmark (like we do for other benchmarks). 2013-12-10 14:42:14 +01:00
Luca Cavanna ac158f6a87 Made sure the new cluster state is not held by AckCountDownListener
(also made AckCountDownListener class static)
2013-12-10 11:15:46 +01:00
Simon Willnauer acf371ef57 Do not override thetas if sum is less or equals to `0`
Closes #4390
2013-12-10 10:41:50 +01:00
Simon Willnauer 2dfb1d98f4 move to ensureGreen in aggs tests to stabelize tests 2013-12-09 21:50:24 +01:00
Simon Willnauer 5771f9083c Added ElasticsearchIntegrationTest#ensureSearchable() to prepare indices to be searchable
Currently we sometimes see test failures that fail because not all replicase are
`searchable` which means they are not started yet or still recovering. Yet, the usual
situation is where two nodes have the same clusterstate but the one that acts as
the search target has not yet processed that clusterstate. The requester sees the
shard as started but it's not mark as such on the target node. For now the #ensureSearchable()
just delegates to #ensureYellow() to make sure the cluster is not red. In the future if we have
the possibilty to recover from situations like this in the search logic we can easily test
this by making the impl a no-op. Note: this problem only occurs if you have low number of docs
and the indexing is really quick such that first request are exectued but shards are not
fully `started`
2013-12-09 17:31:58 +01:00
Martijn van Groningen d3fde78394 Fix test failure. 2013-12-09 17:27:58 +01:00
Martijn van Groningen 1b481ef395 Support postings highlighter in percolate api.
The postings hl now uses a searcher that only encapsulate the view of segment the document being highlighted is in,
this should be better than using the top level engine searcher.

Closes #4385
2013-12-09 16:56:09 +01:00
Alexander Reelsen d2ee22f01c Allow to get a specific transport client inside of tests
Adding functionality to call cluster().transportClient() in tests in order
to get an arbitrary TransportClient object back, independently if the
transport client ratio in returning the normal clients is configured.

Also made sure, that if the normal client is already a transport client
(or a node client) we do not generate another one.
2013-12-09 14:39:22 +01:00
Simon Willnauer 7cd189b3d3 Trace log all shard failures if an exception is attached. 2013-12-09 11:55:13 +01:00
Simon Willnauer 44382ae629 remove ensureGreen for debugging 2013-12-09 11:04:33 +01:00
Simon Willnauer 7ff4681e5f Enable Trace Logging for org.elasticsearch.action.search.type on aggregations 2013-12-09 11:03:36 +01:00
Simon Willnauer 581eb618b3 Add Class and Package Level support for TestLogging 2013-12-09 11:02:13 +01:00
David Pilato 9d8361a542 Use random number of iteration for tests
Thanks @simonw for the review.
Related to #4361 and #4371.
2013-12-09 09:01:51 +01:00
David Pilato a2853f3855 Use random number of iteration for tests
Thanks @simonw for the review.
Related to #4361 and #4371.
2013-12-09 08:41:45 +01:00
Lee Hinman bc9698a347 Support 'yaml' as a format for the Analyze API
Fixes #4311
2013-12-08 15:08:00 -07:00
Simon Willnauer 2058a03547 Enable Trace Logging for SearchWhileRelocatingTests 2013-12-08 22:31:17 +01:00
Simon Willnauer 7e326bdb73 Beef up test to use more threads and iterations randomly 2013-12-08 20:56:24 +01:00
Simon Willnauer 191887dd08 Added refCount assertion due to LUCENE-5362 2013-12-08 20:55:57 +01:00
Simon Willnauer 11e2b1bef3 Move Thread.sleep test to awaitBusy 2013-12-08 20:55:08 +01:00
Simon Willnauer 1ef349db49 Add generic assertion to all aggs.bucket tests to check if all shards were successful 2013-12-08 19:34:17 +01:00
Shay Banon 36a76cc0ab add node and order to the source of update mapping 2013-12-08 03:32:10 +01:00
uboness 2e08977adc - fixing update mapping tests for index operations so the number of request will be based on the index thread pool size
- added update mapping tests for bulk operations
2013-12-07 22:57:18 +01:00
David Pilato 822acfa568 Query String caching could cause matched_filters not working
When searching with a query containing query_strings inside a bool query, the specified _name is randomly missing from the results due to caching.

Closes #4361.
Closes #4371.
2013-12-07 22:52:16 +01:00
Boaz Leskes b7d6cce4c9 Add an `usage` key to the CPU section of OsStats.toXContent.
This is just the sum of existing `sys` and `user`

Closes #4374
2013-12-07 21:26:22 +01:00
Shay Banon 104613c0b9 update the test to use the same API as 0.90
so backports will be simpler for now
2013-12-07 19:31:33 +01:00
Shay Banon 4a3978d1b7 Optimize dynamic mapping updates on master by processing latest one per index/node
Instead of processing all the bulk of update mappings we have per index/node, we can only update the last ordered one out of those (cause they are incremented on the node/index level). This will improve the processing time of an index that have large updates of mappings.
closes #4373
2013-12-07 19:20:16 +01:00
Simon Willnauer dd95895977 Apply slop correctly if phrase query is wrapped in a filtered query.
If a phrase query is wrapped in a filtered query due to type filtering
slop was not applied correctly. Also if the default field required a
type filter the filter was not applied.

Closes #4356
2013-12-06 23:02:09 +01:00
Adrien Grand 309ee7d581 Fix _all boosting.
_all boosting used to rely on the fact that the TokenStream doesn't eagerly
consume the input java.io.Reader. This fixes the issue by using binary search
in order to find the right boost given a token's start offset.

Close #4315
2013-12-05 18:40:25 +01:00
Andrew Raines 53be1fe9d0 Cat: Add descs for rest of actions. 2013-12-05 10:44:26 -06:00
Andrew Raines aec2b95401 Cat: Add nodes column descriptions. 2013-12-05 08:44:00 -06:00
Shay Banon 28eff2ba29 remove help command, list all cat commands in /_cat?h endpoint 2013-12-05 14:36:27 +01:00
Boaz Leskes 0e78404af3 Added a FsStats.total section that sums up all stats for the different path.dirs. Added `disk_io_size` and `disk_io_op` to the toXContent output, summing up `disk_read_size+disk_write_size` and `disk_writes+disk_reads` respectively.
Closes #4344 , Closes #4343
2013-12-05 12:18:22 +01:00
Martijn van Groningen 9f57dc1de3 test fix 2013-12-05 03:17:56 +01:00
Martijn van Groningen facd18086c test fix 2013-12-05 02:21:18 +01:00
Shay Banon e04474bcd8 Allow to disable sending a refresh-mapping to master node
When a node processed an index request, which caused it to update its own mapping, then it sends that mapping to the master. While the master process it, that node receives a state that includes an older version of the mapping. Now, there is a conflict, its not bad (i.e. the cluster state will eventually have the correct mapping), but we send for a refresh just in case form that node to the master.

With a system that has extreme cases of updates and frequent mapping changes, it might make sense to disable this feature. The indices.cluster.send_refresh_mapping setting can be introduced to support that (note, this setting need to be set on the data nodes)

Note, sending refresh mapping is more important when the reverse happens, and for some reason, the mapping in the master is ahead, or in conflict, with the actual parsing of it in the actual node the index exists on. In this case, the refresh mapping will result in warning being logged on the master node.

closes #4342
2013-12-05 01:18:39 +01:00
Shay Banon df4ffbe723 better error handling when processing refresh/update mapping
don't fail the whole bulk of updates, just the specific ones, and warn log it
2013-12-05 00:58:35 +01:00
Martijn van Groningen dd86db3347 Fixed incorrect results with `has_child` query with score mode if the parent type has nested object types. The inner objects (Separate Lucene docs) are also emitted as hits, which incorrectly decreased the count down short circuit mechanism in the `has_child` query.
Closes #4341
2013-12-05 00:57:10 +01:00
Simon Willnauer 2b42a0f94a Override DefaultExceptionHandler to filter out certain exceptions
We have the situation that some tests fail since they don't handle
EsRejectedExecutionException which gets thrown when a node shuts
down. That is ok to ignore this exception and not fail.
We also suffer from OOMs that can't create native threads but don't
get threaddumps for those failures. This patch prints the thread
stacks once we catch a OOM which can' create native threads.
2013-12-04 14:18:52 +00:00
Shay Banon 74bfa27e7e Fail geohash_cell filter if geohash prefix is not enabled
closes #4335
2013-12-04 13:16:55 +01:00
Martijn van Groningen eacbd8f88d Fixed parsing bug in percolator, where everything after the `doc` object was skipped.
Closes #4325
2013-12-04 01:11:24 +01:00
Adrien Grand 346d8efeb5 Fix implementation of currentValueHash in FieldDataSource.Bytes.SortedAndUnique.
Close #4330
2013-12-04 00:42:24 +01:00
Andrew Raines 9e4b5ea242 Introduce abstract superclass, group cat module bindings, and short-circuit helps. 2013-12-03 18:32:12 -05:00
Shay Banon adef349f74 MetaData to use new iterator in ImmutableOpenMap 2013-12-03 22:43:25 +01:00
Shay Banon fd014473d0 expose key and value iterators in ImmutableOpenMap 2013-12-03 22:29:54 +01:00
uboness 84d7929c3b - removed the getKeyAsString method from StringTerms (redundant method) 2013-12-03 18:18:23 +01:00
Martijn van Groningen 5fb15d3afa The registered queries were loaded after shard went to started phase, this resulted in the situation that the percolate api could be used while the registered queries still needed to be loaded from disk.
The newly added afterIndexShardPostRecovery method to InternalIndicesLifecycle, that the percolator now uses to trigger the loading of the registered queries will make sure that a shard doesn't go to started state before the queries have been loaded.
The percolate api (like other apis) will retry the execution on a different shard copy if a shard isn't in a started state preventing empty results if there registered queries to be loaded. Percolator tests fail sometimes for this reason.
2013-12-03 15:10:36 +01:00
Martijn van Groningen 9d1e3a7256 Added test logging 2013-12-03 11:27:27 +01:00
Nik Everett 8e34057bc0 Add support for combining fields to the FVH
The Fast Vector Highlighter can combine matches on multiple fields to
highlight a single field using `matched_fields`.  This is most
intuitive for multifields that analyze the same string in different
ways.  Example:
{
    "query": {
        "query_string": {
            "query": "content.plain:running scissors",
            "fields": ["content"]
        }
    },
    "highlight": {
        "order": "score",
        "fields": {
            "content": {
                "matched_fields": ["content", "content.plain"],
                "type" : "fvh"
            }
        }
    }
}

Closes #3750
2013-12-03 11:10:01 +01:00
Adrien Grand df08894b53 Fix broken test. 2013-12-03 10:57:11 +01:00
Alexander Reelsen 7c96627d76 Remove bash specific calls
Make sure that every call in a shell script is supported by /bin/sh and not a bashism, like $()

Closes #4314
2013-12-03 10:23:40 +01:00
Nik Everett 7690b40ec6 Allow string fields to store token counts
To use this one you send a string to a field of type 'token_count'.  This
makes the most sense with a multi-field.
2013-12-03 09:39:32 +01:00
Martijn van Groningen 43691bb735 Removed shortCircuit in MatchDocIdSet since it is no longer used. 2013-12-02 18:28:27 +01:00
Simon Willnauer 2c8ee3fbbe Moving to 1.0.0RC1 snap 2013-12-02 17:10:07 +01:00
Kevin Kluge 296cfbe390 release [1.0.0.Beta2] 2013-12-02 15:45:30 +00:00
Martijn van Groningen 5488bab003 Use EA#assertMatchCount() over just checking the match count. 2013-12-02 14:50:00 +01:00
Simon Willnauer e18bee50ef User minimum_master_nodes in tests that shutdown nodes some of those time out on Windows 2013-12-02 13:04:13 +01:00
Martijn van Groningen cf63ea212f Use EA#assertMatchCount() over just checking the match count. 2013-12-02 12:12:04 +01:00
uboness 2f835621ae added more wait for green status on aggs tests 2013-12-02 11:42:51 +01:00
Martijn van Groningen a1d58bf486 fix mapping error in p/c benchmarks 2013-12-02 11:31:31 +01:00
Martijn van Groningen 809c7d07f0 only wrap the parentFilter in ApplyAcceptedDocsFilter, not the other short circuit filters. 2013-12-02 11:05:55 +01:00
uboness cdc7dfbb2c Changed the "script_lang" parameter to "lang" in all value source based aggs - to be consistent with all other script based APIs. 2013-12-02 02:01:03 +01:00
Martijn van Groningen 079ac79617 Added random filtered query logic the to random parent query tests.
Fixed bug in ParentQuery.ChildScorer#advance(), the parent value was checked for the _uid field while the _parent should have been checked.
2013-12-02 00:18:59 +01:00
Martijn van Groningen 292e53fe77 The short circuit mechanism always needs to be wrapped with the live docs. In certain scenarios the live docs isn't passed down with acceptedDocs.
For example when a has_child is wrapped in a filtered query as query and the wrapped filter is cached.
The short circuit mechanism in that case counts down based on deleted docs, which then yields lower results than is expected.

Relates to #4306
2013-12-02 00:18:59 +01:00
Simon Willnauer 27f9f9bdb4 Log intermediate pending searchers as trace instead of debug 2013-12-01 19:44:56 +01:00
Simon Willnauer d2bd476e70 add more ensureGreen to aggs tests 2013-12-01 18:58:39 +01:00
Martijn van Groningen 04675f85cf Make the ChildrenQuery and ParentQuery not rely on the fact that score of 0 means that there isn't a match. 2013-12-01 17:22:58 +01:00
Martijn van Groningen 1285894f5f Also short circuit within segment when score mode is avg 2013-12-01 16:44:19 +01:00
Martijn van Groningen ac1e985670 Fixes related to accepted docs not taken into account:
* Removed the applyAcceptedDocs in ChildrenConstantScoreQuery, they need to be applied at all times. (because of short circuit mechanism)
* Moved ParentDocSet to FilteredDocIdSetIterator, because it fits better than MatchDocIdSet.
* Made similar changes to ParentConstantScoreQuery for consistency between the two queries. The bug accepted docs bug didn't occur in the ParentConstantScoreQuery.
* Updated random p/c tests to randomly update parent or child docs during the test run.

Closes #4306
2013-11-30 20:05:55 +01:00
Simon Willnauer 5d2c334bbd wait for green in tests for now 2013-11-30 19:41:49 +01:00
Alexander Reelsen 971b828e76 Cat API: Improve help and execute help before sending server requests
* Force the cat API classes to have simple help available by extending from a AbstractCatAction
* Use a set binder on guice creation to create the help on node start up
* Make sure that the help/field info is returned without querying any data
2013-11-30 16:22:30 +01:00
uboness 88d829ad78 added more assertions to aggs tests 2013-11-30 05:44:04 +01:00
uboness efea1298cd added assertions for shard failures in stats agg tests 2013-11-30 03:58:11 +01:00
Shay Banon ed01d53506 NullPointerException in FsTranslog when reverting transient log
This happens when reverting the trans transaction log on failure, and when that happens, actually we might have failed on the transient translog creation to being with....
fixes #4223
2013-11-30 00:30:32 +01:00
Luca Cavanna fe1d22af01 trimmed down SimpleQueryTests + more assertAcked 2013-11-29 21:15:28 +01:00
Luca Cavanna de206e72f1 Added simple test for minScore 2013-11-29 20:35:40 +01:00
Clinton Gormley bc393b6d79 Changed the minScore comparator from > to >=
Closes #4303
2013-11-29 20:29:20 +01:00
Adrien Grand 65541e6912 Fix computation of explanations for AllTermQuery.
The use of freq() instead of sloppyFreq() and the fact that `numMatches` was
not updated in `setFreqCurrentDoc` could lead to an inaccurate score in the
explanation.

Close #4298
2013-11-29 19:24:56 +01:00
Benjamin Devèze c20d4bb69e Add line feed for prettified main REST action 2013-11-29 16:42:50 +01:00
Luca Cavanna 8e54319a1d test classes cleanup (fixed warnings and typos) 2013-11-29 16:22:20 +01:00
Luca Cavanna b214943522 fixed typo in TestCluster 2013-11-29 14:55:24 +01:00
uboness 0d6a35b9a7 - Added support for term filtering based on include/exclude regex on the terms agg
- Added javadoc to the TermsBuilder

Closes #4267
2013-11-29 13:46:48 +01:00
Luca Cavanna 630641f292 Refactored put mapping api to make use of the new recently introduced generic ack mechanism
Note: we were previously waiting for ack only from all nodes that contain shards for the indices that the mapping updatewas applied to. This change introduces a wait for ack from all nodes, consitent with other api as the ack is meant more on the cluster state itself, which is held by all nodes and needs to be updated on all nodes anyway.

Closes #4228
2013-11-29 11:39:23 +01:00
Alexander Reelsen 0ec3eaf53d Completion postings format: Write hole character
Another issue found by randomized testing.
2013-11-29 10:40:33 +01:00
Andrew Raines 28ecafac11 Display placeholder for non-master-eligible nodes. 2013-11-29 02:35:45 -06:00
Andrew Raines 455c92d27b Check for load avg before attempting to display. 2013-11-29 02:32:54 -06:00
Andrew Raines 9aee9949e5 Convert _cat/count to TimestampedTable. 2013-11-29 02:19:56 -06:00
Andrew Raines d16aacb8d3 Supply field descriptions for TimestampedTable. 2013-11-29 02:19:15 -06:00
Andrew Raines 91999e14ce Add _cat/pending_tasks.
Closes #4251.
2013-11-29 01:09:06 -06:00
Andrew Raines 8f4c8e0d4a Should be limiting health request to concreteIndices.
Closes #4296.
2013-11-28 21:13:32 -06:00
imotov d2f91173b7 Increase priority of shard started task to URGENT 2013-11-28 19:20:18 -05:00
Martijn van Groningen 61aba89110 Added aggregation support to the percolate api.
Closes #4245
2013-11-28 17:26:07 +01:00
Martijn van Groningen 9fe2b8e074 Randomly use a cached filter.
Wrap the ChildrenQuery in a XFilteredQuery to also execute other code paths (scorer#advance)
2013-11-28 17:22:51 +01:00
Martijn van Groningen 2ca5fd64fe Fixed positive infinity bug that can occur in specific scenarios when score mode average is used.
Closes #4291
2013-11-28 17:21:29 +01:00
Simon Willnauer 06f520ebeb Use FilterDirectory from lucene instead of maintaining a copy
FilterDirectory has been ported to Lucene in LUCENE-5204 which makes
the class in elasticsearch obsolet. This commit removes the class and
moves the static utils that are not in lucene to `DirectoryUtils`
2013-11-28 14:37:48 +01:00
Alexander Reelsen 11de330246 Made hole character in XAnalyzingSuggester part of Postingsformat
* Hole charactor now can change with new releases
* Fixed bug where the SEP_LABEL constant was used instead of the sepLabel instance variable
* Replaced if- with switch-statement
2013-11-28 14:22:10 +01:00
Adrien Grand 1e85e4dd26 Add more logging to recent aggregations failures. 2013-11-28 13:22:39 +01:00
Adrien Grand 5968680b41 Fix test bug: we need at least one parent to have 1 child or more. 2013-11-28 09:40:52 +01:00
Simon Willnauer 71eb4532f8 Catch AlreadClosedException in RobinSearcher#release()
When we relocate a shard we might still have pending SearchContext
instances hanging around that will be used in "in-flight" searches
on the already relocated shard. This is a valid operation but if
we have already closed the underlying directory which happens during
cleanup concurrently the close call on the IndexReader can trigger
an AlreadyClosedException when the NRT reader tries to cleanup files
via the IndexWriter.

Closes #4273
2013-11-27 15:55:30 +01:00
Simon Willnauer ba3269dedc Fix assertions in testEquivalence and beef up the test 2013-11-27 15:21:09 +01:00
Adrien Grand 596d511466 Cleanup of IndexFieldDataService.getForField.
This method has 2 signatures and one of them is dangerous since it allows to
discard fielddata configuration of the field mapper. This commit changes the
percolator so that it uses fielddata configuration of the _id field mapper
instead of forcing the paged_bytes format.

Closes #4270
2013-11-27 14:56:09 +01:00
uboness fda6ca4869 Added support for shard_size in terms agg
Closes #4242
2013-11-27 14:16:58 +01:00
Simon Willnauer d18ce2733d Cleanup Rescorer Tests & add random indexing 2013-11-27 11:55:29 +01:00
Alexander Reelsen 4774439436 Fixed file-based template loading via config/templates
When parsing the json file, the first field is ignored as
parser.nextToken() seems to be called too often.

Closes #4235
2013-11-27 09:49:15 +01:00
Simon Willnauer e66c2793aa Add hole character to the reverved characters for completion suggester 2013-11-26 22:46:43 +01:00
Shay Banon f40959cc30 Cat API: Add h parameter to apis, allowing to return columns and descriptions for them
closes #4262
2013-11-26 22:14:18 +01:00
Alexander Reelsen 2a456e5716 Fixing CompletionFieldMapper.isReservedChar() to take all relevant chars in account 2013-11-26 20:42:37 +01:00
Martijn van Groningen 3c4fc119ab Only print the warning about '.' in type, when it is not a percolator type. 2013-11-26 19:01:43 +01:00
Martijn van Groningen dafb90e62a Include facets in the rest response. 2013-11-26 18:59:21 +01:00
Britta Weber 1510cd6923 Fix issue 3989
Multi term vector request never returned if shards for all requested documents
were non existent.

closes #3989
2013-11-26 17:04:18 +01:00
Britta Weber 3be5f3345e Make parsing of uri and body parameters consistent with single term vector api
uri parameters were not all parsed for the multi term vector request. This commit
makes sure that all parameters are parsed and used when creating the requests for the
multi term vector request.

In order to simplify both code and json request, the request structure now allows
two ways to use multi term vectors:

1. Give all parameters for each document requested in the docs array like this:

```
{
   "docs": [
      {
         "_index": "testidx",
         "_type": "test",
         "_id": "2",
         "terms": [
            "fox"
         ],
         "term_statistics": true
      },
      {
         "_index": "testidx",
         "_type": "test",
         "_id": "1",
         "terms": [
            "quick",
            "brown"
         ],
         "term_statistics": false
      }
   ]
}
```

2. Define a list of ids and give parameters in a separate parameters object like this:

```
{
   "ids": [
      "1",
      "2"
   ],
   "parameters": {
      "_index": "testidx",
      "_type": "test",
      "terms": [
         "brown"
      ]
   }
}
```

uri parameters are global parameters that are set for both cases. They are overwritten
by parameter definitions in the body.

Also, this commit adds the missing setParent(..) and setPreference(..) to TermVectorRequestBuilder.
2013-11-26 17:02:56 +01:00
Alexander Reelsen 6e1a04b370 Added unit test for current completion postingsformat 2013-11-26 12:52:07 +01:00
Alexander Reelsen 4e836a3459 Randomized CompletionSuggestSearchTests 2013-11-26 12:52:07 +01:00
Alexander Reelsen bf74f49fdd Updated Analyzing/Fuzzysuggester from lucene trunk
* Minor alignments (like setter to ctor)
* FuzzySuggester has a unicode aware flag, which is not exposed in the fuzzy completion request parameters
* Made XAnalyzingSuggester flags (PAYLOAD_SEP, END_BYTE, SEP_LABEL) to be written into the postings format, so we can retain backwards compatibility
* The above change also implies, that these flags can be set per instantiated XAnalyzingSuggester
* CompletionPostingsFormatTest now uses a randomProvider for writing data to check for bwc
2013-11-26 12:52:06 +01:00
Martijn van Groningen 9f5d01ca4c Cut DiscoveryNodes over to ImmutableOpenMap. 2013-11-26 11:57:15 +01:00
Simon Willnauer cf3ba7c51c Only use -Dtests.jvm.argline instead of numbered options 2013-11-26 11:11:19 +01:00
Simon Willnauer 9b839ca79f Add more reporting to asserting index searcher 2013-11-26 10:54:37 +01:00
Shay Banon 6f90a3e39a allow to parse directly the compressed mapping 2013-11-26 09:48:33 +01:00
Shay Banon 0610fc3ad2 use the proper abstraction of ImmutableOpenMap
we already use the open map / clone trick, might as well use the ImmutableOpenMap here
2013-11-26 02:48:31 +01:00
Martijn van Groningen a03556daa0 Added execution option to `range` filter, with the `index` and `fielddata` as values.
Deprecated `numeric_range` filter in favor for the `range` filter with `fielddata` as execution.

Closes #4034
2013-11-25 23:43:40 +01:00
Martijn van Groningen ac03fba9d3 Small cleanup 2013-11-25 23:18:36 +01:00
Martijn van Groningen 036febe110 Cut the following classes over to ImmutableOpenMap:
* MetaData
* IndexMetaData
* ClusterState
* IndexTemplateMetaData
2013-11-25 21:59:06 +01:00
Simon Willnauer 46ab6a1533 Include JVM Arg line in reproduce line 2013-11-25 14:35:34 +01:00
Shay Banon 32d073bbf8 Bulk should support shard timeout like the index api
closes #4220
2013-11-25 11:48:22 +01:00
Shay Banon 2ba7c1d4a1 acquireSearcher should fail with state failure when not set/closed
also fixes #4232
2013-11-25 11:24:55 +01:00
Shay Banon 9648a53fd5 acquireSearcher should fail with state failure when not set/closed
also fixes #4232
2013-11-25 11:24:19 +01:00
Martijn van Groningen efba3a50dd iterator doesn't implement #remove() 2013-11-25 10:32:35 +01:00
Igor Motov e3d4d73242 Local primaries should be preferred to relocating primaries
To reproduce the bug use -Dtests.seed=5AB62524C9AB0489
Fixes #4237
2013-11-24 21:20:01 -05:00
Simon Willnauer 8e17d636ef Upgrade to Lucene 4.6
This commit upgrades to Lucene 4.6 and contains the following improvements:

 * Remove XIndexWriter in favor of the fixed IndexWriter
 * Removes patched XLuceneConstantScoreQuery
 * Now uses Lucene passage formatters contributed from Elasticsearch in PostingsHighlighter
 * Upgrades to Lucene46 Codec from Lucene45 Codec
 * Fixes problem in CommonTermsQueryParser where close was never called.

Closes #4241
2013-11-24 21:08:38 +01:00
uboness 28adbd475d added a waitForGreen condition for the unmapped tests on missing agg 2013-11-24 20:26:12 +01:00
Martijn van Groningen 8fda8aaabf force default codec
reduce p/c searches and num child docs being generated
2013-11-24 15:35:11 +01:00
uboness c7f6c5266d initial commit of the aggregations module
Closes #3300
2013-11-24 03:13:08 -08:00
Simon Willnauer 34d567a133 Don't use mock wrapper on relocate stress test for now since it could be the reason for intermediate/occational failures 2013-11-23 09:22:16 +01:00
Alexander Reelsen fdc4f72e8a Added mlockall setting to process info output 2013-11-23 07:40:59 +01:00
Martijn van Groningen 688a6bd29b Added random test for ParentQuery and ChildrenQuery. 2013-11-22 18:38:46 +01:00
Martijn van Groningen c46fe4a0e8 Introduced EA#assertMatchCount() and make test used that. 2013-11-22 13:47:02 +01:00
Boaz Leskes 5e58c1b9e1 Added a bulk indexing while initializing test.
Relates to #4214
2013-11-22 13:23:42 +01:00
Igor Motov 4ffd8a663c Fix possible NPE in snapshot service if a shard doesn't have primary 2013-11-21 20:08:57 -05:00
Igor Motov 1d0bae0f50 Prevent modification or deletion of repositories while snapshots are running 2013-11-21 20:08:57 -05:00
Igor Motov 2f66bf0720 Fix typo in MockRamDirectoryService name 2013-11-21 20:08:57 -05:00
Luca Cavanna 22852d8040 AckTests: more assertAcked and added common method to retrieve local cluster state 2013-11-21 22:59:48 +01:00
Martijn van Groningen efbe1e84b9 Added random test for ParentConstantScoreQuery 2013-11-21 19:19:04 +01:00
Martijn van Groningen c5aee7200f removed the static field in ChildrenConstantScoreQueryTests 2013-11-21 18:29:41 +01:00
Martijn van Groningen 3f02392376 unset static field after test suite has been completed. 2013-11-21 16:01:03 +01:00
Martijn van Groningen 9e25c6a4db Added random test for ChildrenConstantScoreQuery. 2013-11-21 14:24:53 +01:00
Simon Willnauer 8d1e526616 By default don't fail on close sicne pending open files are common in ES
We have pending open files on a regular basis since we search while
relocating etc. and keep search contexts around that are cleaned up
later. We should rather let the close call pass on even if files are
open and only force failures on teardown in ElasticsearchIntegrationTest
2013-11-21 11:51:54 +01:00
Shay Banon f465a6b589 field level field data cache should take into account a null listener
also, cleanup the onRemoval code
2013-11-20 09:31:19 -08:00
Martijn van Groningen 1608ccc373 Use #assertAcked() for verification. 2013-11-20 12:56:08 +01:00
Alexander Reelsen 8257370415 Refactoring IndicesAnalysisService
Using enums where possible in order to clean up the code in IndicesAnalysisService

Also introduced a simpler generic caching mechanism, and tests.
2013-11-20 12:11:07 +01:00
Luca Cavanna adb83182a5 Added readTimeout, writeTimeout, readAcknowledged and writeAcknowledged methods variation without version as parameter, to be used by api that always read and write those parameters. Helps avoiding to call the variation that accepts a Version with actual version null. 2013-11-20 10:40:25 +01:00
Luca Cavanna ec3d858cc2 Added base ClusterStateUpdateRequest class that executes on multiple indices 2013-11-20 10:40:25 +01:00
Luca Cavanna 39c606c59a Refactored open/close index api to make use of the new recently introduced generic ack mechanism
Closes #4169
2013-11-20 10:40:25 +01:00
Luca Cavanna 8377f05408 Removed redundant masterNodeTimeout, already present in the ClusterStateUpdateRequest base class 2013-11-20 10:15:43 +01:00
David Pilato 260531bbb2 NPE in PluginsService when starting elasticsearch with a wrong user
When starting elasticsearch with a wrong linux user, it could generate a `NullPointerException` when `PluginsService` tries to list available plugins in `./plugins` dir.

To reproduce:

* create a plugins directory with `rwx` rights for root user only
* launch elasticsearch from another account (elasticsearch for example)

Related discussion: https://groups.google.com/forum/#!topic/elasticsearch/_WRW4Qfpo7M

Closes #4186.
Closes #4187.
2013-11-20 09:58:20 +01:00
Shay Banon bc54201cfc Running the node stats api while a shard is moving onto the node logs an exception
fixes #4203
2013-11-19 11:54:32 -08:00
Matt Weber a841a422f6 Add a field data based TermsFilter
Add FieldDataTermsFilter that compares terms out of
the fielddata cache. When filtering on a large
set of terms this filter can be considerably faster
than using a standard lucene terms filter.

Add the "fielddata" execution mode to the
terms filter parser to enable the use of
the new FieldDataTermsFilter.

Add supporting tests and documentation.

Closes #4209
2013-11-19 19:18:16 +01:00
Alexander Reelsen dab841d4ec Revert Ignore slow log configuration on shard creation
This reverts commit 10810f00d4 as this introduces an exception to the rule when parsing settings.
2013-11-19 16:11:59 +01:00
Alexander Reelsen 10810f00d4 Ignore slow log configuration on shard creation
In case of a misconfigured slow search/index configuration (unparseable
TimeValue) an exception is thrown.

This is not a problem when creating a shard of an index, as an exception
is returned and all is good. However, this is a huge problem, when
starting up a node, as the shard creation is repeated endlessly.

This patch changes the behaviour to go on as usual and just disable the
slowlog, as an improper configuration of logging should not affect the
allocation behaviour.

Closes #2730
2013-11-19 15:35:50 +01:00
Alexander Reelsen 88b854e2ea Making SearchRequest in PutWarmerRequest mandatory & validated
The search request inside of a put warmer request was nullable, but actually we have to have that request in the transport action.
Validation and appropriate test added.

Closes #4196
2013-11-19 15:02:08 +01:00
Alexander Reelsen 513611bd48 DateFieldMapper.merge() can change date format and include_in_all
In order to be able to add an additional date to a date field mapper,
the merge operation has to support this.

Closes #3727
2013-11-19 14:33:41 +01:00
Martijn van Groningen 244e78fbfb Fixed small test bug. 2013-11-19 11:26:45 +01:00
Lee Hinman c96cb99cef Move ?pretty=false handling to the builder instead of mutating the map 2013-11-18 16:03:36 -07:00
Matt Weber 9c15b86b89 Update FilterBuilder and QueryBuilder interfaces
Make the FilterBuilder interface consistent with the QueryBuilder
interface and replace usage of QueryBuilderException with
ElasticSearchIllegalArgumentException.
2013-11-18 20:22:54 +01:00
Benjamin Devèze 00be285c26 Handle pretty=false and missing line feed for pretty=true which is the default for main REST action 2013-11-18 07:39:15 -07:00
Martijn van Groningen 59f1ce5ada Added test for PR #4157 (deep pagination in a small index) 2013-11-18 12:58:55 +01:00
Mark Elliot 5e164a85ed Size docIdsToLoad appropriately in SearchService
Closes #4156
2013-11-18 12:40:13 +01:00
Alexander Reelsen 5e835f0071 Added test for Update API via native scripts
Motivation was to have a test ready as well as something other people could have a look at.
2013-11-18 11:42:29 +01:00
Luca Cavanna 38d895bf1a Added catch EsRejectedExecutionException when trying to reschedule river start 2013-11-18 10:40:09 +01:00
Simon Willnauer 5b1820b723 Don't take static memory of test base classes into account when checking for static leftovers. 2013-11-18 10:26:08 +01:00
Matt Weber 55300f3a3f Update BloomFilter
Allow the user to configure the number of hash functions as well as add
support for serializing/deserializing the bloom filter from a stream.
Add a hashCode to the bloom filter.
2013-11-18 09:46:04 +01:00
Igor Motov 2c1714d4b2 Re-enable snapshot/restore tests that were failing on windows 2013-11-15 13:05:46 -05:00
Britta Weber faf2380605 Allow native scripts to set the value of a script field to primitive arrays
Script fields could not be set to int[] and float[] by native
scripts because StreamInput and StreamOutput could not handle
them.

closes #4175
2013-11-15 18:58:12 +01:00
Andrew Raines c65dcd6dc8 Add load average to _cat/nodes. 2013-11-14 20:13:57 -05:00
Andrew Raines 7a348cd2d8 Add _cat/help (halp!) 2013-11-14 20:13:55 -05:00
Andrew Raines 72d052f753 Add full_id param to support displaying entire node ID. 2013-11-14 20:10:41 -05:00
Andrew Raines 2b8bf07bd3 Drop the heap and RAM used from _cat/nodes and add the RAM percentage.
In this view you never care about the actual heap used bytes; you only
want to know that your max is set to what you meant and what
percentage you're currently using.

Closes #4151.
2013-11-14 20:10:40 -05:00
Andrew Raines 5c085c1204 Fix misspellings. 2013-11-14 20:10:36 -05:00
Igor Motov c63701c5c7 Improve snapshot/restore test speed 2013-11-14 19:05:13 -05:00
Luca Cavanna b2ad34cf99 Refactored indices aliases api to make use of the new recently introduced generic ack mechanism
Closes #4114
2013-11-14 22:59:17 +01:00
Shay Banon 3b30192c8f fix failed compilation under jdk 6 2013-11-14 22:48:11 +01:00
Igor Motov 499cff3b1e Fix possible NPE in snapshot/restore during node shutdown 2013-11-14 13:22:55 -05:00
Luca Cavanna 6945f294e7 Added support for _name parameter in indices filter
Closes #4166
2013-11-14 18:43:08 +01:00
Luca Cavanna 4670a2e514 Trimmed down MatchedQueriesTests 2013-11-14 18:37:55 +01:00
Luca Cavanna 0aaa39d00a Minor improvements to indices filter and query & updated docs
Slightly simplified indices filter and query parsers code
Trimmed down tests where possible
2013-11-14 17:25:34 +01:00
Olivier Favre fa80ca97b2 Indices query/filter skip parsing altogether for irrelevant indices when possible
Closes #2416
2013-11-14 17:24:49 +01:00
Alexander Reelsen c8020760d7 multi_match query supports fields as single string and array
The multi_match query accepted only an array in the fields parameter. This patch allows to use a single string as well.

Also added tests for parsing in both cases.

Closes #4164
2013-11-14 15:54:03 +01:00
Shay Banon 826b8bd742 move bloom filter class to a more common package 2013-11-14 15:37:01 +01:00
Shay Banon f8afa4d67b remove deprecated method usage with jackson 2013-11-14 13:15:26 +01:00
Igor Motov 3c0cc22d36 Fix possible race condition during snapshot deletion 2013-11-13 23:21:17 -05:00
Igor Motov 0a7c6c9288 Fix file handle leak in URLBlobContainer 2013-11-13 23:21:16 -05:00
Boaz Leskes c324251fc2 Make SimpleNodeSampler populate the list of connected nodes using the information returned from the cluster
This is to allow people to introspect things like data settings and attributes. Also makes it consistent with the sniff sampler.

Closes #4162
2013-11-13 17:26:24 +01:00
Simon Willnauer ba61bbb31b Move [0.90.7] to a release 2013-11-13 15:53:57 +01:00
Martijn van Groningen 8e0291823a Fixed bug in bool filter where it doesn't emit docs as hits while they are hits.
There is an optimization that executes bit based (slow) filters in the end. Matched docs could be unset if they didn't match with any of these filters. The bug was that also iterator based (fast) filters should be checked.
This change checks all should filters in the end part (if must or must_not clauses exists), so it can now correctly unset matched docs. The current bool filters requires that at least one should clause must match for docs to be match regardless of any other clauses.

Closes #4130
2013-11-13 11:54:48 +01:00
Igor Motov c3e53f3889 Make mock repository blocking tests more deterministic and reproducible 2013-11-12 13:34:14 -05:00
Simon Willnauer 904c0abb3e Add random filter that compares XBooleanFilter with BooleanQuery while respective the min_should_match = 1 that the filter applies 2013-11-12 18:30:46 +01:00
Simon Willnauer 1e06c76467 Release semaphore if client call throws and exception
Closes #4153
2013-11-12 15:05:42 +01:00
Simon Willnauer 16ee742682 Cleanup test framework in order to release it as a jar file
This commit adds javadocs and removed unused methods from central
classes like ElasticsearchIntegrationTest. It also changes visibility
of many methods and classes that are only needed inside the test infrastructure.
2013-11-12 11:54:55 +01:00
Antoine Rey 2afdb4c8e7 [Javadoc] Fix copy/paste comment on YAML type 2013-11-12 10:19:41 +01:00
Igor Motov f9b43a9efa Failure in onComplete method of AbstractFsBlobContainer#readBlob should trigger onFailure 2013-11-11 20:57:50 -05:00
Igor Motov ab52ab48dd Randomize the number of snapshots in deleteSnapshotTest 2013-11-11 14:42:43 -05:00
Igor Motov e32f16b789 Fix file handle leak in readBlob method of AbstractFsBlobContainer 2013-11-11 14:41:53 -05:00
Igor Motov 2b57770543 Disable snapshot/restore tests that fail on windows with AwaitsFix while working on the fix 2013-11-11 10:05:36 -05:00
Martijn van Groningen 1895ba8271 Fixed NPE if matched parent docs is higher than short_circuit_cutoff.
Closes #4135
2013-11-11 14:40:48 +01:00
Martijn van Groningen 9fdcb0ad05 Upgraded to hppc version 0.5.3 2013-11-11 14:27:52 +01:00
Simon Willnauer a305fb08d6 Wipe snapshot repos after and before tests
On windows tests sometimes fail since files can not be deleted due
to existing repos still holding on to the files. The test
framework is picky about that since it could be a bug and fails
the test if a temp file can not be deleted.
2013-11-11 09:57:45 +01:00
Boaz Leskes 7d9154b8b5 IndexShardGatewayService should not call post_recovery if shard is in STARTED state.
At the end recovery, the IndexShardGatewayService will double check the gateway has moved the shard to POST_RECOVERY and if not, do it it self.
The shard state could have already move to started, causing the post_recovery call to throw an exception and the entire shard recovery to fail.

This can happened if, after the gateway moved the shard to POST_RECOVERY:
1) master sent a new cluster state indicating shard is initializing
2) IndicesClusterStateService#applyInitializingShard will send a shard started event
3) Master will mark shard as started and this will be processed quickly and move the shard to STARTED.

Closes #4147
2013-11-11 08:54:24 +01:00
Igor Motov 510397aecd Initial implementation of Snapshot/Restore API
Closes #3826
2013-11-10 18:26:56 -05:00
Boaz Leskes 81928bd323 fixed a broken trace logging line. 2013-11-10 22:59:19 +01:00
Boaz Leskes 2c5d483be1 Added heap_max(_in_bytes) and heap_used_percent to JVM node stats.
Closes #4145
2013-11-10 21:11:07 +01:00
Luca Cavanna b7cc378aeb Schedule retry if the river type is available but the _meta document isn't
With #3782 we changed the execution order of dynamic mapping updates and index operations. We now first send the mapping update to the master node, and then we index the document. This makes sense but caused issues with rivers as they are started due to the cluster changed event that is triggered on the master node right after the mapping update has been applied, but in order for the river to be started its _meta document needs to be available, which is not the case anymore as the index operation most likely hasn't happened yet. As a result in most of the cases rivers don't get started.
What we want to do is retry a few times if the _meta document wasn't found, so that the river gets started anyway.

Closes #4089, #3840
2013-11-10 21:02:06 +01:00
Igor Motov d390f5250b Add river creation test 2013-11-10 21:02:06 +01:00
Luca Cavanna 60117d3acf Added docs for River start and close methods
Closes #3787
2013-11-10 21:01:01 +01:00
Simon Willnauer 77239a76f8 Lazily fill CharTermAttribute if needed in CompletionTokenStream
This adds a delegate to CharTermAttributeImpl to be compatible
with the Percolator that needs a CharTermAttribute. Yet compared
to CharTermAttributImpl we only fill the BytesRef with UTF-8 since
we already have it and only if we need to convert to UTF-16 we do it.

Closes #4028
2013-11-10 20:43:00 +01:00
Andrew Ochsner 914fd29e70 Change default store impl to mmapfs on 64bit Linux.
Fixes #4134
2013-11-09 11:37:17 -07:00
Shay Banon ffe5e1861f close transport client to wait for ongoing samples
the transport client should not be executing at the same time as sampling happens, so connection open/close logic will be properly maintained
2013-11-09 14:54:50 +01:00
Simon Willnauer fb7a234040 s/AbstractIntegrationTest/ElasticsearchIntegrationTest 2013-11-08 23:56:44 +01:00
Clinton Gormley 7189310764 In ctor of GeoPointFieldMapper, geohash_prefix now implicitly enables geohash option
Also improved docs for geopoint type and geohash_cell filte

Closes #3951
2013-11-08 13:52:17 +01:00
Luca Cavanna 4aa59aff00 Added support for external query in postings highlighter
It is now possible to highlight an external query using the postings highlighter, relates to #3630

Closes #4121
2013-11-08 11:31:42 +01:00
Shay Banon 6f286c3382 Unable to create a nested filtered alias on a dataless master
fixes #4112
2013-11-08 11:22:43 +01:00
Shay Banon 7fc4947271 _default_ mapping not applied when using separate master/data nodes
fixes #4124
2013-11-08 10:55:05 +01:00
Simon Willnauer 76622f1a9b Reduce number of docs in test for #4093 - 30 docs are enough to trigger the bug 2013-11-08 09:59:30 +01:00
Luca Cavanna cdbd7918ee Used the actual index_name when making highlighting using the postings highlighter
Previously the field name specified in the search request was used, which isn't correct in case a custom index_name has been used for a field or the "path":"just_name" has been used in the mapping.

 Closes #4116
2013-11-07 17:19:52 +01:00
Olivier Favre 566ca24247 Fix missing affectation in SimpleChildQuerySearchTests.testParentChildQueriesCanHandleNoRelevantTypesInIndex() 2013-11-07 15:58:41 +01:00
Luca Cavanna 2071fb26c2 Trimmed down HighlighterSearchTests
Used common assertHighlight and concise mapping whenever possible
More indexRandom too
2013-11-07 15:44:33 +01:00
Luca Cavanna 06938e6fb0 Fixed fetch subphase to not recreate the HitContext for each hit, so that the object cache stays the same
Fixed also bug in the fast vector highlighter which was raised by enabling the object cache, due to null FieldQuery (NPE) in case the objects are taken from the cache

Added tests to check if there are issues when highlighting multiple fields at the same time

Closes #4106
2013-11-07 15:44:33 +01:00
Simon Willnauer 91e31e530f Move stresstests out of o.e.test 2013-11-07 14:44:05 +01:00
Simon Willnauer 03c326dafa Move o.e.junit to o.e.test.junit 2013-11-07 14:41:17 +01:00
Simon Willnauer e8f4445cc3 Move MockDirectories into o.e.test.store 2013-11-07 14:39:33 +01:00
Simon Willnauer 8ecdb65d94 Move IndexFieldDataServiceTests into corresponding package 2013-11-07 14:37:01 +01:00
Simon Willnauer a9239f8147 Guarantee sorted order for [Double|Long|Bytes]Values
Values returned by [Double|Long|Bytes]Values are sorted today which
is guaranteed by the underlying Lucene index. Several implementations can
make use of this property but the interfaces don't guarantee this behavior.
This commit adds the guarantees and makes use of them in several places.

Note: This change might require sorting for 3rd party implemenations of these
interaces.
2013-11-07 10:29:23 +01:00
Simon Willnauer f5f6259d03 Reduce number of documents in RobinEngineIntegrationTest to prevent memory peaks triggering OOM 2013-11-07 09:44:53 +01:00
Shay Banon 021aa09614 External method to set rootTypeParsers in DocumentMapperParser incorrect
fixes #4113
2013-11-07 01:06:57 +01:00
Simon Willnauer f6c2ee0ab2 Improve stability of RobinEngineIntegrationTest by reducing direct memory usage of the test env 2013-11-06 22:09:00 +01:00
Shay Banon c95c7096e5 Index Stats: Add support for segments stats
closes #4101
2013-11-06 20:44:01 +01:00
Simon Willnauer e1b6988886 move to [1.0.0.Beta2] SNAP 2013-11-06 16:19:50 +01:00
Simon Willnauer 77bc5d5ecf release [1.0.0.Beta1] 2013-11-06 15:32:43 +01:00
Simon Willnauer bb777a2dfe Apply fix for LUCENE-5330 pruning the IndexWriter queue to get rid of pending event
Closes #4093
2013-11-06 14:00:28 +01:00
Simon Willnauer d4873bd6a5 Use same caching strategy for STANDARD and DEFAULT analyzer 2013-11-06 10:58:40 +01:00
Luca Cavanna 47aa01f5d1 Improved highlight *manyDocs tests (fvh and postings) to have random refreshes, flushes and optimizes using indexRandom 2013-11-06 10:30:32 +01:00
Luca Cavanna a3e355d40e Fixed doc_id used in combination with context.searcher(), needs to be topLevelId rather than just docId
Improved test to catch this problem calling refresh more frequently and having the word to highlight in different positions in the text

Closes #4103
2013-11-06 01:20:27 +01:00
Shay Banon ebc8975efd NPE when closing XContentBuilder and using 'pretty' query parameter
fixes #4100
2013-11-06 01:10:58 +01:00
Igor Motov 03179497a3 Add /_cat endpoint 2013-11-05 15:40:48 -05:00
Simon Willnauer 9654631186 Change 'standart' analyzer to use emtpy stopword list by default.
The 'default' / 'standard' analyzer can be a trappy default sicne it filters
english stopwords by default. Yet a default should not be dedicated to a certain language
since elasticsearch is used in many different scenarios where a standard analysis chain
with specialization to english full-text might be rather counter productive.

This commit changes the 'standard' analyzer to use an empty stopword list for indices
that are created from 1.0.0.Beta1 version onwards but will maintain backwards compatibiliy
for older indices.

Closes #3775
2013-11-05 21:07:21 +01:00
Shay Banon 1586339ee0 better support for single value to settings builder where the type gets lost
relates to #4097
2013-11-05 21:04:27 +01:00
Shay Banon 7c32269f4f Dist. Percolation: Use .percolator instead of _percolator for type name
Use .percolator as the internal (hidden) type name for percolators within the index. Seems nicer name to represent "hidden" types within an index.
closes #4090
2013-11-05 20:02:59 +01:00
Lee Hinman af183a1619 Fix `indices.recovery.concurrent_small_file_streams` not being dynamically settable
Fixes #4094
2013-11-05 10:03:21 -07:00
Boaz Leskes 0ef2493b2c Throw an exception if a type's mapping root node is not equal to the type in question.
Also, fix all the problems it brought up in tests.
Removed OverrideTypeMappingTests as it is no longer relevant.
Better naming for the default percolator mapping and change it's content use _default_ as root node.

Closes #4038
2013-11-05 11:54:25 +01:00
Simon Willnauer 552852f299 Remove unreleased versions from master & fix 0.90.6 Lucene Version 2013-11-05 10:34:44 +01:00
Igor Motov 0f785d5779 Optimize concrete index resolution in TransportMoreLikeThisAction 2013-11-04 22:05:37 -05:00
Leonardo Menezes 5e4928ca92 Resolve aliases on redirects inside TransportMoreLikeThisAction
fixes #4045
2013-11-04 22:05:37 -05:00
Shay Banon 5f170cb4fd Remove Index Reader warmer introduced in 0.90.6
It create the following challenges:
- it automatically load all the norms for all fields. This should be an opt in feature similar to the new loading feature in field data. Will open a separate issue for it.
- It automatically loads all doc values for all fields (if they have it), overriding effectively the loading option of field data when its backed by doc values.

closes #4078
2013-11-04 18:32:50 +01:00
Vojtech Hyza 47969efae9 Keep word filter through an error if `keep_word_path` was specified.
Closes #4073
2013-11-04 16:49:07 +01:00
Simon Willnauer 0be5eca5d2 move to [0.90.7] SNAP 2013-11-04 14:55:55 +01:00
Luca Cavanna 5474cffe8f Added support for highlighting multi term queries using the postings highlighter
Closes #4042
2013-11-04 12:55:22 +01:00
Igor Motov 123bc98d81 Fix possible NPE in ClusterState.toString()
ClusterState.toString() can throw NPE if at least on index template exists in the cluster state.
2013-11-04 06:36:45 -05:00
Igor Motov c724f0de5d Initial implementation of ResourceWatcherService
Closes #4062
2013-11-03 21:55:54 -05:00
Shay Banon 1b799e9a3e use immutable open map to list the files in store 2013-11-03 21:31:42 +01:00
Boaz Leskes c1e853c51d More detailed logging for FullRollingRestartTests.testFullRollingRestart
set indices.cluster  & cluster.service to TRACE
2013-11-03 16:50:22 +01:00
Shay Banon ed10758a6b add keys/values to ImmutableOpenMap 2013-11-03 14:34:29 +01:00
Shay Banon 671d2dd650 verify operations on store dir are executed when its open
call ensureOpen and properly set the open flag
also, better handling of failures and error message during listAll in local recovery
2013-11-03 14:23:58 +01:00
Shay Banon 8f88d0aa4a URI routing parameter support with multi search API
Support URI level routing parameter for the multi search API, that will act as the default routing (unless explicitly set).
closes #4058
2013-11-03 02:30:19 +01:00
Shay Banon a8535e247e URI routing parameter does not work with Bulk API
closes #4053
2013-11-03 02:16:09 +01:00
Simon Willnauer b0a8abeb88 Add missing .get() call to actually execute the update setting in RecoveryWhileUnderLoadTests 2013-11-02 09:16:25 +01:00
Shay Banon 6be85cece4 cleanup DiscoveryNodes#Builder construction
have a better method name, and only allow to create a copy of the builder based on existing nodes when constructing the builder
2013-11-02 03:38:22 +01:00
Shay Banon e551ec282f cleanup ClusterState#Builder construction
have a better method name, and only allow to create a copy of the builder based on existing cluster state when constructing the builder
2013-11-02 02:52:01 +01:00
Shay Banon ef56c68f67 have IndexMetaData Builder pattern similar to other state builders 2013-11-02 02:13:25 +01:00
Shay Banon fef647cb92 cleanup RoutingTable#Builder construction
have a better method name, and only allow to create a copy of the builder based on existing routing table when constructing the builder
2013-11-02 01:27:51 +01:00
Shay Banon d111e169a4 cleanup MetaData#Builder construction
have a better method name, and only allow to create a copy of the builder based on existing meta data when constructing the builder
2013-11-02 01:04:33 +01:00
Shay Banon acc5f584d4 fix test to not rely on execution of processed / ack order
also, make sure the order is consistent (as much as possible) when calling ack to processed
2013-11-02 00:04:12 +01:00
Andrew Raines 7d2c4afa98 _cat/nodes improvements.
* Handle info/stats null possibilities if cluster is in duress.

* Shorten node id.
2013-11-01 17:33:54 -05:00
Andrew Raines f9a1726300 Add _cat/health.
% curl localhost:9200/_cat/health\?v=1\&ts=0
cluster  status nodeTotal nodeData shards pri relo init unassign
kluster green          1        1     20  20    0    0        0

% curl localhost:9200/_cat/health\?ts=0
kluster green 1 1 20 20 0 0 0

% curl localhost:9200/_cat/health\?v=1
epoch      time     cluster  status nodeTotal nodeData shards pri relo init unassign
1383341092 16:24:52 kluster green          1        1     20  20    0    0        0

% curl localhost:9200/_cat/health
1383341119 16:25:19 kluster green 1 1 20 20 0 0 0

Closes #4050.
2013-11-01 16:37:23 -05:00
Andrew Raines 48f4ba06c0 Add TimestampedTable subclass. 2013-11-01 16:37:22 -05:00
Andrew Raines 720edafa0d Disable RestTable headers by default. 2013-11-01 16:37:22 -05:00
Simon Willnauer caee7c8831 Stableize recoverWhileRelocating by expanding replicas at the end 2013-11-01 22:16:24 +01:00
Shay Banon f720531328 protect ImmutableOpenMap builder from using the built map after build has been called 2013-11-01 20:53:24 +01:00
Shay Banon 633e781204 remove unneeded check... 2013-11-01 19:44:25 +01:00
Luca Cavanna afa0ab6226 Fixed ack behaviour when no ack is expected from any node or timeout is set to 0
We now return acknowledged true when no wait is needed (mustAck always returns false). We do wait for the master node to complete its actions though. Previously it would try to timeout and hang due to a CountDown#fastForward call when the internal counter is set to 0

We now return acknowledged false without starting the timeout thread when the timeout is set 0, as starting the wait and immediately stopping the thread seems pointless.

Added coverage for ack in ClusterServiceTests
2013-11-01 18:24:45 +01:00
Shay Banon c63741db04 Introduce a new ImmutableOpenMap
this map is a "true" immutable map, encapsulating an open impl, and has a builder that allows it to be built easily.
the builder has the optimization of using clone if its being built based on an existing immutable map.
2013-11-01 18:20:25 +01:00
Luca Cavanna a021c1d4f2 Improved acknowledgement test for cluster reroute with dry_run flag
Usually acknowledged true means that all nodes have digested the change and are in sync. When no changes are made, there is no need to push a new cluster state, no need to wait for ack either, but can't guarantee that all nodes are in sync.

When using cluster reroute with dryRun flag no changes are made, this test was based on the wrong assumption that acknowledged meant all nodes were in sync, which is not the case here.

Changed the test to only read the cluster state from the master, to check that nothing changed there after processing the cluster state update.
2013-11-01 15:45:31 +01:00
Simon Willnauer c78c5469e4 Randomize exceptions when file is deleted while still open.
This somehow emulates a behavior on windows but we should test the
other code paths as well.
2013-11-01 11:47:22 +01:00
Martijn van Groningen d660b18298 Fixed bug (NPE) in mpercolate api, defaults from percolating existing documents where also applied for the regular percolate api, these defaults were none existing (null). 2013-11-01 11:17:16 +01:00
Martijn van Groningen 65fc90728e The mpercolate api rest endpoint should also support http get. 2013-11-01 11:09:17 +01:00
Luca Cavanna 29363d8139 Removed needless concreteIndices call, already called in doExecute and set to the request object 2013-10-31 20:58:52 +01:00
Luca Cavanna 3ee9f27fbe Fixed delete mapping to return acknowledged false when ack times out 2013-10-31 16:28:44 +01:00
Martijn van Groningen 81089fb228 Fixed concurrency issue in simple id cache. The lget() of a map can only be used if the map isn't shared. 2013-10-31 16:20:29 +01:00
Shay Banon c68016bb83 better timeout handling waiting for primary to be active for indexing
take into account the correct delta timeout when scheduling it, since now we can retry again after removing a listener
2013-10-31 15:54:37 +01:00
Simon Willnauer fcfc41209b use Math.max rather than an if statement 2013-10-31 14:36:24 +01:00
Simon Willnauer 59588913b3 Evilize RecoveryWhileUnderLoadTests a bit more 2013-10-31 14:25:28 +01:00
Shay Banon 5f2b7dc266 improve moving from post_recovery to started
we need to move to started from post recovery on cluster level changes, we need to make sure we handle a global state change of relocating, which can happen (and not pass through started)
2013-10-31 14:23:58 +01:00
Shay Banon 5cd56e5cea Improve new field mapping introduction performance
Improve the introduction of new fields into the concrete parsed mappings by not relying on immutable maps and copying over entries, but instead using open maps (which will also use less memory), and using clone to perform the copy on write logic
2013-10-31 11:07:52 +01:00
Simon Willnauer f01d3a18ad Simplify FieldData API
Removing #getValue and #hasValue to have a simple and consistent API
for multiple values.
2013-10-31 10:10:35 +01:00
Alexander Reelsen dfcb3ca2d4 RegexpQueryBuilder now implements MultiTermQueryBuilder
This allows the RegexpQueryBuilder to be used in span queries

Added tests for all span multi term queries.
Also updated the documentation and removed mentioning of numeric range
queries for span queries (they have to be terms).

Closes #3392
2013-10-31 09:12:57 +01:00
Igor Motov d43f3d83d8 The +index pattern without a wildcard in the index list is handled inconsistently
Fixes #3979
2013-10-30 16:36:55 -04:00
Igor Motov 3b4b05e2c9 Fix bug in TransportShardReplicationOperationAction retry mechanism
This issue was causing some index requests against shards in POST_RECOVERY state to hang.
2013-10-30 16:36:55 -04:00
Andrew Raines 5d46e69154 Add ES version to _cat/nodes. 2013-10-30 15:35:41 -05:00
Andrew Raines 256605f065 Add heap used percentage to _cat/nodes.
Closes #4020.
2013-10-30 13:50:29 -05:00
Andrew Raines b2e8ec2924 Add node name & ip to _cat/recovery.
Closes #4016.
2013-10-30 12:38:31 -05:00
Boaz Leskes 8819f91d47 Add a GetFieldMapping API
This new API allows to get the mapping for a specific set of fields rather than get the whole index mapping and traverse it.
The fields to be retrieved can be specified by their full path, index name and field name and will be resolved in this order.
In case multiple field match, the first one will be returned.

Since we are now generating the output (rather then fall back to the stored mapping), you can specify `include_defaults`=true on the request to have default values returned.

Closes #3941
2013-10-30 16:16:36 +01:00
Andrew Raines f16eb7a243 Add _cat/allocation/{nodes}.
Visualizes how shards distribute across the cluster.

% curl localhost:9200/_cat/allocation
shards diskUsed diskAvail diskRatio ip        node
     2   85.6gb   379.4gb     22.6% 127.0.0.1 MacTaggert, Moira
     2   85.6gb   379.4gb     22.6% 127.0.0.1 Jarvis, Edwin

% curl localhost:9200/_cat/allocation/Jar\*
shards diskUsed diskAvail diskRatio ip        node
     2   85.6gb   379.4gb     22.6% 127.0.0.1 Jarvis, Edwin
2013-10-30 10:09:45 -05:00
Shay Banon 8aa2ee7bac indices cluster state service should execute as one of the first listeners
its preferable to execute the indices cluster state service as quickly as possible, as one of the first listeners, so it will apply the cluster state to the local state
potentially, it should even execute before it state is "visible" (through the state call), but that's another change...
2013-10-30 15:53:31 +01:00
Shay Banon 8a62619fb9 throw index already exists failure when it exists as alias on creation
make sure to throw the already exists exception, so when indexing into an alias, and it has not propagated yet through the cluster state, it will end up being ignored if it already exists
2013-10-30 13:47:31 +01:00
Boaz Leskes ed39e79d8f minor improvement to FullRollingRestartTests failure output 2013-10-30 12:08:54 +01:00
Simon Willnauer 454dc53483 Add back hashed BytesValues optimization that got lost in a previous commit.
Some FieldData consumers require hash values per byte. We provide an optimization
that allows to cache the hashes internally if the consumer knows that they are needed
this optimization got lost in a previous commit. This commit adds them back and folds
the dedicated method into AtomicFieldData#getBytesValues(true|false)
2013-10-30 11:48:00 +01:00
Boaz Leskes 542f021d6f Also resend shard started message if shard state is in POST_RECOVERY and master thinks it's initializing.
Closes #4009
2013-10-30 11:42:57 +01:00
Shay Banon 8f21d2831a explicitly enable http when running plugin tests
add an option to force a setting when building a node, overriding if its explicitly set as system property
2013-10-30 02:09:43 +01:00
Luca Cavanna 9cb54c0f2e Added getters for fs.path, fs.mount and fs.dev in node stats api
Closes #4004
2013-10-29 14:49:07 +01:00
Alexander Reelsen b6d2be7a51 Improving test stability for CompletionTests
Randomized testing caught an issue showing that the test should
flush after indexing a document, as always the same document is
indexed.
2013-10-29 13:11:34 +01:00
Luca Cavanna 9e6fab3a6d Added support for acknowledgements to update cluster settings api
As a side note, the internal reroute call is now part of the ack mechanism. That means that if the response contains acknowledged flag, the internal reroute that was eventually issued was acknowledged too. Also, even if the request is not acknowledged, the reroute is issued before returning, which means that there is no need to manually call reroute afterwards to make sure the new settings are immediately applied.

Closes #3995
2013-10-29 11:45:59 +01:00
Luca Cavanna d10bd2495d Added support for acknowledgements to cluster reroute api
Closes #3985
2013-10-29 11:45:59 +01:00
Luca Cavanna f23a1e23b1 Added support for node acknowledgements in delete mapping api
Closes #3984
2013-10-29 11:45:59 +01:00
Luca Cavanna 5f1ebf20f7 Added support for acknowledgement in update index settings api
Added support for serialization based on version to AcknowledgedResponse. Useful in api that don't support yet the acknowledged flag in the response.
Moved also ack warmer tests to more specific AckTests class

Close #3983
2013-10-29 11:45:59 +01:00
Luca Cavanna f425d4af2f Added ack call when the cluster state has not changed
The missing call could cause a cluster state update to hang in case there is no change no apply
2013-10-29 11:45:58 +01:00
Luca Cavanna 5639002a8e Fixed test cluster scope to be able to run it multiple times if needed 2013-10-29 11:10:31 +01:00
Igor Motov 01d51ef434 Fix String.format to use Locale.ROOT in RestRecoveryAction 2013-10-28 23:37:11 -04:00
David Pilato 5d90abf701 mget API should support global routing parameter
mget API support `_routing` field but not `routing` parameter.

Reproduction here:

```sh
curl -XDELETE "http://localhost:9200/test/"; echo
curl -XPUT "http://localhost:9200/test/" -d'{
   "settings": {
      "number_of_replicas": 0,
      "number_of_shards": 5
   }
}'; echo

curl -XPUT 'http://localhost:9200/test/order/1-1?routing=key1' -d '{
   "productName":"doc 1"
}'; echo
curl -XPUT 'http://localhost:9200/test/order/1-2?routing=key1' -d '{
   "productName":"doc 2"
}'; echo
curl -XPUT 'http://localhost:9200/test/order/1-3?routing=key1&refresh=true' -d '{
   "productName":"doc 3"
}'; echo

curl -XPOST 'http://localhost:9200/test/order/_mget?pretty' -d '{
    "docs" : [
        {
            "_index" : "test",
            "_type" : "order",
            "_id" : "1-1",
            "_routing" : "key1"
        },
        {
            "_index" : "test",
            "_type" : "order",
            "_id" : "1-2",
            "_routing" : "key1"
        },
        {
            "_index" : "test",
            "_type" : "order",
            "_id" : "1-3",
            "_routing" : "key1"
        }
    ]
}'; echo

curl -XPOST 'http://localhost:9200/test/order/_mget?pretty&routing=key1' -d '{
	"ids": [
		"1-1",
		"1-2",
		"1-3"
	]
}'; echo
```

Closes #3996.
2013-10-28 21:05:55 +01:00
Lee Hinman e3db12bf50 Add the 'recovery' _cat API endpoint
This addes the _cat/recovery/{index} API endpoint, which displays
information about the status of recovering shards. An example of the
output:

index shard node                   target    recovered     %
test2 0     Fwo7c_6MSdWM0uM1Ho4t-g 147304414  19236101 13.1%
test  0     Fwo7c_6MSdWM0uM1Ho4t-g 145891423 119640535 82.0%

Fixes #3969
2013-10-28 13:43:02 -06:00
Alexander Reelsen b1b52b641d Making unit test future proof by using old version, in case versions change 2013-10-28 18:46:46 +01:00
Alexander Reelsen ec0880df45 Add version to prebuilt analyzers
This patch takes the version of the created index into account when a
prebuilt analyzer is created.
So, if an index was created with 0.90.4, then the prebuilt analyzers
will be the same than on the 0.90.4 release.

One reason for this feature is the possibility to change pre built
analyzers like the standard one.

The patch tries to reuse analyzers as mutch as possible. So even if
version X.Y.Z and X.Y.A use the same lucene analyzers, the same instance
is reused in order to prevent overcreation of lucene analyzer instances.

Closes #3790
2013-10-28 18:11:35 +01:00
Britta Weber c9dab6991e rename and document "index.mapping.date.parse_upper_inclusive" setting for date fields
The setting causes the upper bound for a range query/filter to be rounded up,
therefore the name `round_ceil` seems to make more sense.

Also this commit removes the redundant fourth parameter to DateMathParser.parse(..)
which was never used.
was:    parse(String text, long now, boolean roundUp, boolean upperInclusive)
is now: parse(String text, long now, boolean roundCeil)

closes #3914
2013-10-28 15:48:31 +01:00
Adrien Grand 2e8bbe9e30 Make ContextIndexSearcher delegate more method calls to Engine.searcher().
When running tests, Engine.searcher() is going to be an AssertingIndexSearcher
so we definitely don't want to discard it. This commit fixes it as well as the
bugs it found.

Closes #3987
2013-10-28 13:12:09 +01:00
Simon Willnauer ce891e93b6 Assert on failures in QueryRescorerTests 2013-10-28 11:09:48 +01:00
Boaz Leskes 6b52391dbd Added a ClusterHealthResponse.toString implementation for better error reporting in FullRollingRestartTests timeout failures. 2013-10-28 09:54:54 +01:00
Boaz Leskes 68de46ff05 Use XContentParser.isBooleanValue to detect all boolean values for the `_source` parameter for both _mget and _search
Closes #3981
2013-10-26 22:19:05 +02:00
Boaz Leskes 0f8149dca4 Added Booleans.isBoolean to allow detecting whether a char sequence contains a boolean value. Added XContentParser.isBooleanValue to allow testing if current value is of boolean nature.
Boolean values:
boolean type values
numeric (0 == false)
one of the following strings: "true","false","on","off","yes","no","0","1"
2013-10-26 22:15:35 +02:00
Simon Willnauer 2255ecc980 Don't use TransportClient on FullClusterRestartTests
There seems to be an issue with this test since it shuts down random
nodes and TransportClients seem to be confused due to that. For
now we disable them to figure out if this is the cause of the sporadic
timeouts.
2013-10-25 22:04:44 +02:00
Adrien Grand b0b3748cae Re-apply pull-request #3956 now that the CLA has been signed. 2013-10-25 19:05:20 +02:00
Alexander Reelsen f9154de04f CompletionFieldMapper: Return error if weight is no integer
In order to make sure that people do not get confused, if they
index a float as weight, it makes more sense to reject it instead of
silently parsing it to an integer and using it.

The CompletionFieldMapper now checks for the type of the number which
is being read and throws and exception if the number is something else
than int or long.

Closes #3977
2013-10-25 16:38:50 +02:00
Simon Willnauer e79f714b4b Unify REST responses for Acknowledged[Request|Response]
All Rest handlers that use AcknowledgedResponse look identical. Since
we gonna have more of them in the future we can abstract the logic away.
2013-10-25 14:31:28 +02:00
Helton Alponti 0e9c049da1 Fixed NPE in multi_match query when using lenient and field weight
Closes #3797
2013-10-25 00:49:17 +02:00
Luca Cavanna 48ac9747a8 Added third highlighter type based on lucene postings highlighter
Requires field index_options set to "offsets" in order to store positions and offsets in the postings list.
Considerably faster than the plain highlighter since it doesn't require to reanalyze the text to be highlighted: the larger the documents the better the performance gain should be.
Requires less disk space than term_vectors, needed for the fast_vector_highlighter.
Breaks the text into sentences and highlights them. Uses a BreakIterator to find sentences in the text. Plays really well with natural text, not quite the same if the text contains html markup for instance.
Treats the document as the whole corpus, and scores individual sentences as if they were documents in this corpus, using the BM25 algorithm.

Uses forked version of lucene postings highlighter to support:
- per value discrete highlighting for fields that have multiple values, needed when number_of_fragments=0 since we want to return a snippet per value
- manually passing in query terms to avoid calling extract terms multiple times, since we use a different highlighter instance per doc/field, but the query is always the same

The lucene postings highlighter api is  quite different compared to the existing highlighters api, the main difference being that it allows to highlight multiple fields in multiple docs with a single call, ensuring sequential IO.
The way it is introduced in elasticsearch in this first round is a compromise trying not to change the current highlight api, which works per document, per field. The main disadvantage is that we lose the sequential IO, but we can always refactor the highlight api to work with multiple documents.

Supports pre_tag, post_tag, number_of_fragments (0 highlights the whole field), require_field_match, no_match_size, order by score and html encoding.

Closes #3704
2013-10-24 23:38:00 +02:00
Simon Willnauer 08b4ca66c3 Use at least one query to prevent division by zero in PercolatorFacetsTests 2013-10-24 21:45:12 +02:00
Britta Weber ebd328e340 remove trove import from mvel script
all subclasses of trove maps have been removed in 088e05b368
2013-10-24 19:44:45 +02:00
Simon Willnauer 3a34aa735e Upgrade to Lucene 4.5.1 2013-10-24 18:37:44 +02:00
Luca Cavanna 8e6c4ce8e8 Minor changes to no_match_size highlight parameter and highlight tests (#1171)
- Randomly store the field to highlight
- Updated test to use common assertions
- Restored previously commented out testCommonTermsQuery
2013-10-24 14:38:32 +02:00
Nik Everett 14a709f563 Highlighting can return excerpt with no highlights
You can configure the highlighting api to return an excerpt of a field
even if there wasn't a match on the field.

The FVH makes excerpts from the beginning of the string to the first
boundary character after the requested length or the boundary_max_scan,
whichever comes first.  The Plain highlighter makes excerpts from the
beginning of the string to the end of the last token before the requested
length.

Closes #1171
2013-10-24 14:38:32 +02:00
Simon Willnauer fb9cd5a562 Use abstract classes as super typs for Acknowledge operations
Currently we have a marker interface for Acknowledged[Request|Response],
this makes not much sense since we duplicate the code in each subclass
or class that implements the interface. We can simply use abstract
classes and have it implemented only once.
2013-10-24 14:01:43 +02:00
Simon Willnauer 7867de4f5b Refactor FieldData iterations
This commit primarily folds [Double|Bytes|Long|GeoPoint]Values.Iter
into [Double|Bytes|Long|GeoPoint]Values. Iterations now don't require
a auxillary class (Iter) but instead driven by native for loops. All
[Double|Bytes|Long|GeoPoint]Values are stateful and provide `setDocId`
and `nextValue` methods to iterate over all values in a document.
This has several advantage:

 * The amout of specialized classes is reduced
 * Iteration is clearly stateful ie. Iters can't be confused to be local.
 * All iterations are size bounded which prevents runtime checks and
   allows JIT optimizations / loop un-rolling and most iterations are
   branch free.
 * Due to the bounded iteration the need for a `hasNext` method call
   is removed.
 * Value iterations feels more native.

This commit also adds consistent documentation and unifies the calcualtion
if SortMode is involved.

This commit also changes the runtime behavior of BytesValues#getValue() such that it
will never return `null` anymore. If a document has no value in a field
this method still returns a `BytesRef` with a `length` of 0. To identify
documents with no values #hasValue() or #setDocument(int) should be used.
The latter should be preferred if the value will be consumed in the case
the document has a value.
2013-10-24 10:33:56 +02:00
Adrien Grand 7bd1a55f6e Revert "Fix for has_child can cause an infinite loop (100% CPU) when used in bool query."
Temporarily revert the commit while waiting for the CLA to be signed.
2013-10-24 09:43:58 +02:00
Josh Canfield adadc72da3 Fix for has_child can cause an infinite loop (100% CPU) when used in bool query.
Closes #3955
2013-10-24 09:24:14 +02:00
Shay Banon 35b573ff24 Transport: Have a separate channel for recovery
Have a separate channel for recovery, so it won't overflow the "low" channel which is also used for bulk indexing.

Also, rename the channel names to be more descriptive. Change low to bulk (for bulk based operations, currently just bulk indexing), med to reg (for "regular" operations), and high to state (for state based communication). The new channel for recovery will be named recovery, and the ping channel will remain the same.

closes #3954
2013-10-23 15:55:27 -07:00
Adrien Grand d18192b39f Add doc values to TermsFacetSearchBenchmark. 2013-10-23 10:58:35 +02:00
Shay Banon a3122a88e4 Java API: Setting track scores does not affect scan search type
When setting track scores, the scan search type will return the scores for each document. The Java API builder does not properly set this value (it only sets it if a sort in in place, which is not relevant for scan search type).

closes #3949
2013-10-22 18:04:22 -07:00
Shay Banon 5bc3825c70 externalize writing "raw" fields to helper method 2013-10-21 12:19:14 -07:00
Martijn van Groningen 8d49aa398f Added facet support to the percolate api.
Closes #3851
2013-10-21 19:13:28 +07:00
Luca Cavanna b7d8c275eb Fix small typo in terms lookup tests mapping (count api tests) 2013-10-18 17:55:21 +02:00
Matt Weber e6fc416adc Fix small typo in terms lookup tests mapping. 2013-10-18 17:55:21 +02:00
Simon Willnauer f749db26e8 Allow awareness attributes to be reset via the API
Currently we don't allow resetting the awareness
attribute via the API since it requires at least one
non-empty string to update the setting. This commit
allows resetting this using an empty string.

Closes #3931
2013-10-18 16:25:00 +02:00
Blake Smith 03a89297ba Fix teh typos in javadocs 2013-10-18 12:19:37 +02:00
Shay Banon c9b0e1de6c Settings queue_size on index/bulk TP can cause rejection failures when executed over network
The #3526 fix was not complete, it handled cases of on node execution, but didn't properly handle cases where it was executed over the network, and forcing the execution of the replica operation when done over the wire.

This relates to #3854

closes #3929
2013-10-17 17:06:44 +03:00
Boaz Leskes 2593b6e644 Terms facet will now throw a parsing exception if no field, fields or a script is supplied.
Previously you'd get an NPE. Also added extra exception when called with unknown parameters.
2013-10-17 09:26:43 +02:00
Martijn van Groningen cc9ab111a0 Prohibit indexing a document with parent for a type that doesn't have a `_parent` field configured and prohibit adding a _parent field to an existing mapping.
Closes #3848 #3849
2013-10-15 18:29:44 +02:00
Simon Willnauer 89de3ab627 Added simple count down class that allows to be fast forwarded
Closes #3910
2013-10-15 17:47:51 +02:00
Luca Cavanna fcf13e0fa7 Delete warmer api to support acknowledgements
Added support for acknowledgements in delete warmer api using the generic mechanism introduced in#3786

Closes #3833
2013-10-15 17:47:50 +02:00
Luca Cavanna 31142ae471 Put warmer api to support acknowledgements
Added support for acknowledgements in put warmer api using the generic mechanism introduced in #3786

Closes #3831
2013-10-15 17:47:50 +02:00
Luca Cavanna 55f1eab09a Added generic cluster state update ack mechanism
Added new AckedClusterStateUpdateTask interface that can be used to submit cluster state update tasks and allows actions to be notified back when a set of (configurable) nodes have acknowledged the cluster state update. Supports a configurable timeout, so that we wait for acknowledgement for a limited amount of time (will be provided in the request as it curently happens, default 10s).

Internally, a low level AckListener is created (InternalClusterService) and passed to the publish method, so that it can be notified whenever each node responds to the publish request. Once all the expected nodes have responded or the timeoeout has expired, the AckListener notifies the action which will return adding the proper acknowledged flag to the response.

Ideally, this new mechanism will gradually replace the existing ones based on custom endpoints and notifications (per api).

Closes #3786
2013-10-15 17:47:50 +02:00
Alexander Reelsen 4d19239ec4 Add support for Lucene SuggestStopFilter
The suggest stop filter is an improved version of the stop filter, which
takes stopwords only into account if the last char of a query is a
whitespace. This allows you to keep stopwords, but to allow suggesting for
"a".

Example: Index document content "a word". You are now able to suggest for
"a" and get back results in the completion suggester, if the suggest stop
filter is used on the query side, but will not get back any results for
"a " as this is identified as a stopword.

The implementation allows to set the `remove_trailing` parameter for a
custom stop filter and thus use the suggest stop filter instead of the
standard stop filter.
2013-10-15 16:12:02 +02:00