Commit Graph

6940 Commits

Author SHA1 Message Date
Simon Willnauer 8e8526c746 Mark translog as upgraded in the engine even if a legacy generation exists
Today we mark a translog as upgraded by adding a marker to the engine commit.
Yet, this commit was only added if there was no translog present before ie. only
if we have a fresh engine which is missing the entire point. Yet, this commit
adds a backwards index tests that ensures we can open old indices more than once
ie. mark the index as upgraded.

Closes #11858
2015-06-25 14:23:50 +02:00
Britta Weber 74aed85c36 highlighter: Save unwrap BytesRefHash.MaxBytesLengthExceededException
In Lucene 5x the exception thrown when highlighter encounters a huge term
is a BytesRefHash.MaxBytesLengthExceededException but in Lucene 4x it is
wrapped in a RuntimeException. Therefore, it seems saver to unwrap this.
2015-06-25 12:32:59 +02:00
Isabel Drost-Fromm 4f7ed2132e Remove duplicate operator enums
As we now have an enum Operator that comes with many useful helper methods switching to use
that instead of the enums defined separately. Also switches to using the new enum's helper
methods where applicable removing duplicate parsing logic.

This breaks backwards compatibility. Documenting the break in
migrate_query_refactoring.asciidoc

Relates to #10217
2015-06-25 10:47:39 +02:00
Adrien Grand 7cf3096c3c Merge pull request #11790 from jbarthelmes/master
Add a null-check for XContentBuilder#field for BigDecimals
2015-06-25 09:45:28 +02:00
Ryan Ernst e8e74ae569 Mappings: Remove close() from Mapper
There was only a single actual "use" of close, for a threadlocal
in VersionFieldMapper. However, that threadlocal is completely
unnecessary, so this change removes the threadlocal and
close() altogether.
2015-06-24 14:31:07 -07:00
Ryan Ernst 26ff6e452e Fix _ttl default field type to include index/search analyzer (numeric 64bit) 2015-06-24 09:08:50 -07:00
Robert Muir b00be9fe4a Merge pull request #11850 from rmuir/enable_j9
allow IBM J9 2.8+ in version check
2015-06-24 11:36:15 -04:00
Ryan Ernst da07c3acbf Switch to guava version of Function, since the jdk version is only
available in java8
2015-06-24 08:15:48 -07:00
Ryan Ernst 791d111cc0 Merge branch 'master' into pr/8871 2015-06-24 08:05:17 -07:00
Simon Willnauer dc67bd0021 Render strucutred exception in multi search
MultiMatch still only returns the exception message but should return the
actual exception and render it in a structured fashion
2015-06-24 16:21:41 +02:00
Robert Muir 077b9e0e58 allow IBM J9 2.8+ in version check 2015-06-24 09:56:18 -04:00
Tanguy Leroux 0ce18954a0 Merge pull request #11836 from wolfgangkarall/patch-1
Fix endless looping if starting fails
2015-06-24 15:42:46 +02:00
Simon Willnauer fcdcce3bba Consolidate shard level abstractions
This commit consolidates several abstractions on the shard level in
ordinary classes not managed by the shard level guice injector.

Several classes have been collapsed into IndexShard and IndexShardGatewayService
was cleaned up to be more lightweight and self-contained. It has also been moved into
the index.shard package and it's operation is renamed from recovery from "gateway" to recovery
from "store" or "shard_store".

Closes #11847
2015-06-24 15:18:04 +02:00
Simon Willnauer c57951780e Simplify ShardRouting abstraction
This commit folds ShardRouting, ImmutableShardRouting and MutableShardRouting
into ShardRouting. All mutators are package private anyway today so it's just
unnecessary abstraction.
ShardRoutings are now frozen once they are added to the IndexRoutingTable
to prevent modifications outside of the allocation code.
2015-06-24 14:35:57 +02:00
Adrien Grand aaa9ee35ea Add tests to XContentParser.list(). 2015-06-24 12:03:15 +02:00
Jörg Prante 411a719a86 add list parse methods to XContentParser 2015-06-24 11:41:22 +02:00
Christoph Büscher a2122fdc2b Merge branch 'master' into feature/query-refactoring 2015-06-24 11:29:59 +02:00
Adrien Grand 04968d72a1 Remove XContentParser.map[Ordered]AndClose().
It is a bit weird to have an API to read a map then close the parser.
2015-06-24 11:10:30 +02:00
Adrien Grand 121a699261 Merge pull request #11816 from jpountz/enhancement/return_metadata
Always return metadata in get/search APIs.
2015-06-24 10:32:53 +02:00
Adrien Grand e4d475c700 Always return metadata in get/search APIs.
This commit makes the get and search APIs always return `_parent`, `_routing`,
`_timestamp` and `_ttl` in addition to `_id` and `_type`. This way, consumers
always have all required information in order to reindex a document.
2015-06-24 10:22:14 +02:00
Isabel Drost-Fromm b166259ede Fix SimpleQueryStringBuilder wildcard handling
Forgot to commit the very last change yesterday which led to analyzeWildcard to remain at the default value always.

Relates to #11274
2015-06-24 09:39:19 +02:00
Adrien Grand 29ec4e015d Merge pull request #11843 from dpursehouse/immutable-settings-update
Replace references to ImmutableSettings with Settings
2015-06-24 08:33:20 +02:00
Adrien Grand ea175492b0 Merge pull request #11833 from jpountz/fix/smaller_filter_cache
Give the filter cache a smaller maximum number of cached filters.
2015-06-24 08:23:26 +02:00
David Pursehouse b49e66c3a1 Replace references to ImmutableSettings with Settings
ImmutableSettings was merged into Settings in commit 4873070.

Change-Id: I06bd0150381d131593920c2328c46beacf49661f
2015-06-24 14:54:53 +09:00
Igor Motov f379a13fb4 Extract all shard-level snapshot operation into dedicated SnapshotShardsService
Currently the SnapshotsService is concerned with both maintaining the global snapshot lifecycle on the master node as well as responsible for keeping track of individual shards on the data nodes. This refactoring separates two areas of concerns by moving all shard-level operations into a separate SnapshotShardsService.

Closes #11756
2015-06-23 18:01:32 -04:00
Alex Ksikes 3284ec9b55 doEquals for CommonsTerms and SimpleString queries 2015-06-23 15:25:24 -05:00
Alex Ksikes 29d16cd1e8 Refactors CommonTermsQuery
Refactors CommonTermsQuery analogous to TermQueryBuilder. Still left to do are
the tests to compare between builder and actual Lucene query.

Relates to #10217

This PR is against the query-refactoring branch.
2015-06-23 15:15:56 -05:00
Isabel Drost-Fromm 7afa37c62e Merge pull request #11274 from MaineC/feature/simple-query-string-refactoring
Refactors SimpleQueryStringBuilder/Parser
2015-06-23 22:01:33 +02:00
Ryan Ernst 573c85251e Added better error message when field types are not the same 2015-06-23 12:34:49 -07:00
Isabel Drost-Fromm e170c8e498 Refactors SimpleQueryStringBuilder/Parser
This commit makes SimpleQueryStringBuilder streamable, add hashCode and equals. Adds a dedicated builder/parser unit test, fixes formatting, adds JavaDoc where needed, adjust the handling of default values according to https://github.com/elastic/dev/blob/master/design/queries/general-guidelines.md

Switched to using toLanguageTag/forLanguageTag when parsing Locales. Using LocaleUtils from either Elasticsearch or Apache commons resulted in Locales not passing the roundtrip test. For more info see https://issues.apache.org/jira/browse/LUCENE-4021

Relates to #10217
2015-06-23 21:23:14 +02:00
Wolfgang Karall dc6f3e3eac Fix endless looping if starting fails
log_end_msg does not break the loop or exit the script results in endlessly printing 'failed' in red to the screen in case of a failed start.
2015-06-23 20:52:27 +02:00
Ryan Ernst 542c25e78d Remove MapperAnalyzer 2015-06-23 09:53:38 -07:00
Ryan Ernst 33339ab288 Addressed PR comments 2015-06-23 09:00:14 -07:00
Adrien Grand f29bf63fa8 Give the filter cache a smaller maximum number of cached filters.
Currently the filter cache is configured to have a maximum size in bytes of 10%
of the JVM memory, and a maximum number of cached filters (across all segments
of all shard on the same node) of 100000. I would like to change the latter to
a more reasonable value of 1000.

Given that we track the most 256 most recently used filters per index and only
cache those that have been seen 5 times or more, a single index cannot have more
than 50 hot filters, so a maximum number of cached filters of 1000 per node
should be more than necessary.
2015-06-23 17:44:07 +02:00
Shay Banon 435ce7f251 Remove scheduled routing
Today, we have scheduled reroute that kicks every 10 seconds and checks if a
reroute is needed. We use it when adding nodes, since we don't reroute right
away once its added, and give it a time window to add additional nodes.

We do have recover after nodes setting and such in order to wait for enough
nodes to be added, and also, it really depends at what part of the 10s window
you end up, sometimes, it might not be effective at all. In general, its historic
from the times before we had recover after nodes and such.

This change removes the 10s scheduling, simplifies RoutingService, and adds
explicit reroute when a node is added to the system. It also adds unit tests
to RoutingService.

closes #11776
2015-06-23 17:21:09 +02:00
Simon Willnauer 49bbc42ac8 Encapsualte common code in methods
This commit groups duplicated code in methods to make the actual decision
easier to read. There is no change in functionality in this change.
2015-06-23 15:48:52 +02:00
Christoph Büscher b2606d98df Minor fix: renaming read/writeNamedWritableList() to read/writeNamedWriteabelList() for consistency
The method for reading and writing single instances is called `read/writeNamedWriteable()`,
so the list version should be spelled the same.
2015-06-23 15:25:16 +02:00
Martijn van Groningen fe330b868a percolator: Fail nicely if `nested` query with `inner_hits` is used in a percolator query.
Closes #11672
2015-06-23 15:03:31 +02:00
Colin Goodheart-Smithe f26311e88b Aggregations: Rename `series_arithmetic` agg to `bucket_script` 2015-06-23 14:00:17 +01:00
Simon Willnauer 79f3e78ce2 Log warn message if leftover shard is detected 2015-06-23 14:36:57 +02:00
javanna 7653376ece Query refactoring: simplify equals implementation and dedup code
Closes #11818
2015-06-23 13:59:23 +02:00
Christoph Büscher 0da9f2038f Merge pull request #11823 from cbuescher/feature/query-refactoring-not
Query refactoring: NotQueryBuilder and Parser
2015-06-23 13:48:04 +02:00
Christoph Büscher 682b499f46 Query refactoring: NotQueryBuilder and Parser
Moving the query building functionality from the parser to the builders
new toQuery() method analogous to other recent query refactorings.

Relates to #10217
Closes #11823
2015-06-23 13:46:37 +02:00
Tanguy Leroux 95caa73518 [Packaging] Fix missing dependencies for RPM/DEB packages
Since elasticsearch doesn't shade artifacts anymore (see #11522), the dependencies list for RPM/DEB must be updated. Now we package all maven libs by default except the generated -shaded/-tests/-test-cours JARs and slf4j-api (marked as optionnal).
2015-06-23 13:16:16 +02:00
Shay Banon 15d799361b Set randomized node/index settings in the right place
Don't set node settings in the index template, and try and set less index settings in the node settings
closes #11767
2015-06-23 13:11:17 +02:00
Clinton Gormley 4c48f51500 Deleted slf4j.* from licenses/
Although it is in the target/lib/ directory, slf4j is not included in the
ZIP/tar files
2015-06-23 12:50:31 +02:00
Christoph Büscher fdc4d0121e Merge pull request #11817 from cbuescher/feature/query-refactoring-or
Query refactoring: OrQueryBuilder and Parser
2015-06-23 11:54:24 +02:00
Christoph Büscher f2573da77b Added null checks when adding inner queries 2015-06-23 11:45:00 +02:00
Adrien Grand 18ec76aae8 Better test reuse of existing mappings for dynamic mappings. 2015-06-23 11:44:07 +02:00
Simon Willnauer e97d3d8a49 Fix typo 2015-06-23 11:00:39 +02:00
javanna 99147228d7 Merge branch 'master' into feature/query-refactoring
Conflicts:
	core/src/main/java/org/elasticsearch/index/query/GeoShapeQueryBuilder.java
	core/src/main/java/org/elasticsearch/index/query/TermsQueryBuilder.java
2015-06-23 10:16:21 +02:00
javanna b78211ddaa Java api: make BoostableQueryBuilder public again
FunctionScoreQueryBuilder is outside of the index.query package, missed that in my previous commit.
2015-06-23 10:01:39 +02:00
javanna 99dee81d76 Java api: make BoostableQueryBuilder package private 2015-06-23 09:32:24 +02:00
javanna 90a7b4868e Java api: add missing boost support to TermsQueryBuilder
Relates to #11744
Closes #11810
2015-06-23 09:26:10 +02:00
javanna e7702468a2 Java api: add missing boost support to GeoShapeQueryBuilder
Relates to #11744
Relates to #11810
2015-06-23 09:25:53 +02:00
Ryan Ernst 00bcce79f7 Simplify root mapper builder creation 2015-06-22 14:30:40 -07:00
Ryan Ernst adcc0683b8 Mappings: Restrict fields with the same name in different types to have the same core settings
We currently are very lax about allowing data types to conflict for the
same field name, across document types. This change makes the underlying
map in MapperService a 1-1 map of field name to field type, and throws
exception when new types are not compatible.

To still allow changing a type, with parameters that are allowed to be
changed, but for a field that exists in multiple types, a new parameter
to index creation and put mapping API is added: update_all_types.
This defaults to false, and the exception messages suggest using
this parameter when trying to modify a setting that is allowed to be
modified but is being limited by this restriction.

There are also a couple changes which try to base fields from new types
for dynamic mappings, and root mappers, on existing settings. For
dynamic mappings this is important if the dynamic defaults have been
changed. For root mappings, this is mostly just for backcompat when
pre 2.0 root mappers could have their field type changed.

fixes #8871
2015-06-22 14:00:10 -07:00
Christoph Büscher 6f8c0ce966 Query refactoring: OrQueryBuilder and Parser
Moving the query building functionality from the parser to the builders
new toQuery() method analogous to other recent query refactorings.

Relates to #10217
2015-06-22 22:03:57 +02:00
Boaz Leskes 1df2d3015e Add OS name to _nodes and _cluster/nodes
we currently don't expose this.

This adds the following to the OS section of `_nodes`:

```
"os": {
     "name": "Mac OS X",
     ...
 }
 ```

 and the following to the OS section of `_cluster/stats`:

 ```
      "os": {
         ...
          "names": [
             {
                "name": "Mac OS X",
                "count": 1
             }
          ],
          ...
       },
 ```

Closes #11807
2015-06-22 20:36:29 +02:00
Christoph Büscher 33b3323a63 Merge pull request #11717 from cbuescher/feature/query-refactoring-fieldmaskingspan
Query refactoring: FieldMaskingSpanQueryBuilder and Parser
2015-06-22 20:07:31 +02:00
Ryan Ernst 12e7cbe92b Mappings: Lockdown _timestamp
This is a follow up to #8143 and #6730 for _timestamp. It removes
support for `path`, as well as any field type settings, and
enables docvalues for _timestamp, for 2.0.  Users who need to
adjust these settings can use a date field.
2015-06-22 10:21:03 -07:00
Christoph Büscher b6cdc46a61 Query refactoring: QueryFilterBuilder and Parser
Moving the query building functionality from the parser to the builders
new toQuery() method analogous to other recent query refactorings. In
this case this also includes FQueryFilterParser, since both queries are
closely related.

Relates to #10217
Closes #11729
2015-06-22 18:17:01 +02:00
Zachary Tong 5d94febcb1 Aggregations: Moving average forecasts should not include current datapoint.
- Fixes tests, and removes a few special snowflake, fragile tests.
- Removes concrete implementation of predict() and moves it into
  each model so that the logic is clearer.  Because there is some
  shared checks/assertions, those remain in predict() and the main
  prediction happens in doPredict()
2015-06-22 11:13:35 -04:00
Christoph Büscher 107635b194 Query refactoring: FieldMaskingSpanQueryBuilder and Parser
Moving the query building functionality from the parser to the builders
new toQuery() method analogous to other recent query refactorings.

Relates to #10217
Closes #11717
2015-06-22 17:03:32 +02:00
Alexander Reelsen eb23530e06 Internal: Readd accidental removed functionality in InternalClusterService
The commit about adding cluster health response features also removed
accidentally some functionality, that resulted in wrong instanceof checks
in InternalClusterService and thus in test failures because the cluster
state task that was added via an anonymous was missing the cast.

This commit readds the abstract class with slight renaming.

Commit id was: 88f8d58c8b
2015-06-22 16:34:02 +02:00
Simon Willnauer 1f3670733a Load percolator queries before shard is marked POST_RECOVERY
If we mark the shard as being in POST_RECOVERY before the percolator
is fully set up we might expose it to the user as fully searchable before
all queries are loaded. This can lead to wrong results especially in tests
when a shard is concurrently marked as STARTED.

This commit also removes unneded abstractions on IndexShard where readoperations
should be allowed when the purose is a write.
2015-06-22 16:23:27 +02:00
Shay Banon 55c55677a3 Remove reroute with no reassign
Its not used in our codebase anymore, so no need for it
closes #11804
2015-06-22 15:53:56 +02:00
javanna fabe91633f Query refactoring: make QueryBuilder an interface again and rename current abstract class to AbstractQueryBuilder
It is handy to have a base interface, not just an abstract class, for all of our query builders. This gives us  more flexibility especialy with complex class hierarchies. For instance SpanTermQueryBuilder extends BaseTermQueryBuilder, but also needs to be marked as a SpanQueryBuilder. The latter is a marker interface that should extend QueryBuilder which is not possible unless QueryBuilder actually is an interface.

Also remove queryId method as it created confusion, getName is good enough for the purpose, and override the return type of toQuery method for SpanQueryBuilders to SpanQuery.

Closes #11796
2015-06-22 15:11:04 +02:00
Alexander Reelsen 88f8d58c8b Cluster Health: Add max wait time for pending task and active shard percentage
In order to get a quick overview using by simply checking the cluster state
and its corresponding cat API, the following two attributes have been added
to the cluster health response:

* task max waiting time, the time value of the first task of the
  queue and how long it has been waiting
* active shards percent: The percentage of the number of shards that are in
  initializing state

This makes the cluster health API handy to check, when a fully restarted
cluster is back up and running.

Closes #10805
2015-06-22 15:04:47 +02:00
Artur Cygan b063427899 Fix indentation in deb init script 2015-06-22 14:10:19 +02:00
Colin Goodheart-Smithe 772d0cc6e7 Build: Make rest-spec-api a project so eclipse build works
The change makes rest-spec-api a project in the same way as we build dev-tools. it packages the tests and api in a bundle using the maven-remote-resources-plugin and uses the same plugin in the plugins and core pom to unpack the rest-api-spec into the target directory and references the rest tests there in the test resources.

The main stimulus for this change is that for those using Eclipse the current build does not work. After running `mvn eclipse:eclipse` the Eclipse IDE errors because the rest-api-spec is outside of the project scope, meaning that every time the command is run (required whenever any dependencies change), the class path of all the projects has to be manually fixed.
2015-06-22 11:41:44 +01:00
Simon Willnauer ec6946e979 [TEST] Enable testcase since #11482 is merged 2015-06-22 12:19:03 +02:00
Alexander Reelsen 38ddc8159c Dates: Allow for negative unix timestamps
This fixes an issue to allow for negative unix timestamps.
An own printer for epochs instead of just having a parser has been added.
Added docs that only 10/13 length unix timestamps are supported
Added docs in upgrade documentation

Fixes #11478
2015-06-22 11:56:31 +02:00
Simon Willnauer 895a7dd03c [TEST] assume assertions are enabled 2015-06-22 11:25:02 +02:00
Christoph Büscher 1a34d91641 Merge branch 'master' into feature/query-refactoring 2015-06-22 11:04:31 +02:00
Simon Willnauer af4b7ee000 Merge pull request #11795 from s1monw/abstract_runnable_ping
Use abstract runnable in scheduled ping
2015-06-22 10:27:19 +02:00
Simon Willnauer c7285d9d19 [TEST] Remove sleeps and latch timeouts from ClusterServiceTests
Tests relying on sleeps and latch timeouts are prone to weird timing issues
and hard to read / understand error messages. This commit moves towards a more
deterministic error model and replaces empty fails with real exceptions.
2015-06-22 09:51:51 +02:00
Simon Willnauer a45c05d907 Use abstract runnable in scheduled ping 2015-06-22 09:40:06 +02:00
Johannes Barthelmes 38ceeb5c30 Fix #11699
Add a null-check for XContentBuilder#field for BigDecimals
2015-06-21 21:08:07 +02:00
Simon Willnauer 2e762f078d fix typo 2015-06-21 13:50:04 +02:00
Ryan Ernst 4c869cdf61 quite noisy test failure, related to #11769 2015-06-19 22:30:55 -07:00
Igor Motov 576b825d1c Improve logging of repository verification exceptions.
Some repository verification exceptions are currently only returned to the users but not logged on the nodes where the exceptions occurred, which makes troubleshooting difficult.

Closes #11760
2015-06-19 22:09:03 -04:00
Ryan Ernst aa4369f55b fix parent field mapper to copy customFieldDataSettings on merge 2015-06-19 16:48:34 -07:00
Ryan Ernst 0a5e763697 Removed hardcoded seed from test 2015-06-19 16:08:51 -07:00
Ryan Ernst de2dea04fb Rename MappedFieldType.validateCompatible to checkCompatibility 2015-06-19 15:59:30 -07:00
Ryan Ernst fa8a300f71 Merge pull request #11783 from rjernst/refactor/field-type-merge
Move merge simulation of fieldtype settings to fieldtype method
2015-06-19 15:45:01 -07:00
Ryan Ernst a71b128738 Address PR comments and fix customFieldDataSettings to still be copied
on merge
2015-06-19 15:41:36 -07:00
Ryan Ernst 434b1c94b3 Mappings: Move merge simulation of fieldtype settings to fieldtype method.
For #8871, we need to be able to check field types are compatible,
without comparing FieldMappers.  This change moves the simulation
checks (which generate merge conflicts) for any properties of
MappedFieldTypes into a new method, validateCompatible.

This also simplifies the merge code which merges settings
between the old and new fieldtypes. Previously, each subclass
of FieldMapper would have to set its own fieldtype settings.
However, now that we have .clone(), which perfectly copies
all properties (with subclasses accounted for), we can now
do a simple clone when merging.

Finally, this fixes a subtle bug in merging, in which if
merging has conflicts, and we were not simulating, we would
still update the field type, even though it was not compatible!

NOTE: there is one test failure I am trying to track down with
timestamp merging. Otherwise, all tests pass.
2015-06-19 13:42:16 -07:00
Areek Zillur a8c2886b3f Internal: Mark store as corrupted instead of deleting state file on engine failure
Currently, we delete the shard _state file on engine failure.
This behaviour does not persist the engine failure reason for later inspection.

This commit marks the shard store as corrupted instead of deleting
the _state file to ensure the store index can not be opened after and
the engine failure is persisted.
2015-06-19 14:46:26 -04:00
Alex Ksikes e26b1763e0 CommonTermsQuery fix for ignored coordination factor
CommonTermsQueryParser does not check for disable_coords, only for
disable_coord. Yet the builder only outputs disable_coords, leading to
disabling the coordination factor to be ignored in the Java API.

Closes #11730
Closes #11780
2015-06-19 10:15:37 -05:00
Ryan Ernst 04c3095f5e Merge pull request #11770 from rjernst/fix/null_value_merge
Hide more fieldType access and cleanup null_value merging
2015-06-19 08:15:33 -07:00
Zachary Tong ae742c4a03 Aggregations: moving_avg model parser should accept any numeric, not just doubles
Also changes the models to throw ParseExceptions instead of SearchParseExceptions, so that
the validation can be unit-tested.

Fixes #11487
2015-06-19 10:36:15 -04:00
javanna 4db8c018f5 [TEST] null out NamedWriteableRegistry in BaseQueryTestCase 2015-06-19 13:41:25 +02:00
Iarwa1n ef8ee12259 Fix exception message in RestRequest
Closes #11305
2015-06-19 11:06:23 +01:00
javanna 7162c45ca7 Merge branch 'master' into feature/query-refactoring 2015-06-19 10:23:48 +02:00
javanna 88583a67de [TEST] remove @Repeat usages, manually run multiple iterations locally instead 2015-06-19 10:22:32 +02:00
Ryan Ernst caa40df328 Mappings: Hide more fieldType access and cleanup null_value merging
There were some missed uses of AbstractFieldMapper.fieldType in #11764.
This change also moves null_value merging into AbstractFieldMapper.
2015-06-18 15:58:30 -07:00
Ryan Ernst bfc4b8dd10 Merge pull request #11764 from rjernst/fix/field-type-ref
Replace fieldType access in mappers with getter
2015-06-18 13:01:07 -07:00
Adrien Grand 1bfa722d43 Make sure messages are fully read even in case of EOS markers.
When using compression over the network, you might sometimes see warnings that
the stream was not fully read. This is because DeflaterOutputStream adds an
end-of-stream marker. When deserializing, we need to poll for one byte using
InputStream.read() to make sure to decode this EOS marker.

For the record, it does not strike all the time today because we perform
buffering when decompressing to avoid performing too many JNI calls, but it
is easy to make this warning happen all the time by decreasing the size of
the buffer we use.

Close #11748
2015-06-18 21:55:53 +02:00
Ryan Ernst f09e0d7fb4 Mappings: Replace fieldType access in mappers with getter
A small refactoring to allow changing how the field type is stored later.
2015-06-18 11:47:38 -07:00
Shay Banon 341e980c52 [TEST] Use the correct renamed setting
and make the default value setting private
2015-06-18 20:40:07 +02:00
Simon Willnauer 1b2a3d0af6 Add @Repeat to forbidden APIs
@Repeat should not be committed just like @Seed.
Use -Pdev to run annotated methods.
2015-06-18 20:34:02 +02:00
Shay Banon 6d2ca116b9 [TEST] assertBusy on hasUnassigned
on fast machines, node leave might not move shards to unassigned right away, wait for it
2015-06-18 20:30:25 +02:00
Shay Banon 256f3d21ce Reset registeredNextDelaySetting on reroute
Need to reset the registered setting in order to make sure the nex round will capture the right delay interval

also randomize setting and name the setting properly

closes #11759
2015-06-18 20:20:49 +02:00
Jack Conradson 917aeb7278 Scripting: Allow executable expression scripts for aggregations
Added several classes to support expressions being used for numerical
calculations in aggregations.  Expressions will still not compile
when used with mapping and update script contexts.

Closes #11596
Closes #11689
2015-06-18 10:57:38 -07:00
Simon Willnauer 87946ba3b2 Allow security rule for advanced SSL configutation
if plugins need to install SSL factories etc. we have to allow
to `setFactory` in the security policy.
2015-06-18 16:50:35 +02:00
Christoph Büscher cfae809e6d Tests: Using UTC time zone in RangeQueryBuilderTest to make forbidden apis checker happy 2015-06-18 16:29:30 +02:00
Christoph Büscher cccc71f1e4 Merge branch 'master' into feature/query-refactoring
Conflicts:
	core/src/main/java/org/elasticsearch/index/query/SimpleQueryStringBuilder.java
2015-06-18 16:19:27 +02:00
Shay Banon 792a545633 Optional Delayed Allocation on Node leave
Allow to set delayed allocation timeout on unassigned shards when a node leaves the cluster. This allows to wait for the node to come back for a specific period in order to try and assign the shards back to it to reduce shards movements and unnecessary relocations.

The setting is an index level setting under `index.unassigned.node_left.delayed_timeout` and defaults to 0 (== no delayed allocation). We might want to change the default, but lets do it in a different change to come up with the best value for it. The setting can be updated dynamically.

When shards are delayed, a log message with "info" level will notify how many shards are being delayed.

An implementation note, we really only need to care about delaying allocation on unassigned replica shards. If the primary shard is unassigned, anyhow we are going to wait for a copy of it, so really the only case to delay allocation is for replicas.

close #11712
2015-06-18 16:06:02 +02:00
Christoph Büscher 3f4b493114 Query Refactoring: DisMaxQueryBuilder and Parser
Moving the query building functionality from the parser to the builders
new toQuery() method analogous to other recent query refactorings.

Relates to #10217
Closes #11703
2015-06-18 15:45:07 +02:00
Simon Willnauer 30cc984565 [TEST] fix TermsAggregationSearchBenchmark 2015-06-18 15:06:49 +02:00
Simon Willnauer 5808102d19 Upgrade AWS dependency to 1.10.0
This also apgrades JodaTime to 2.8.0.
2015-06-18 14:04:38 +02:00
Simon Willnauer 85dccdb8ab Use UTC as timeszone for index creation date 2015-06-18 11:15:40 +02:00
Simon Willnauer 2a63249441 Add DateTime ctors without timezone to forbidden APIs
Using DateTime with default timezone is asking for trouble and should
be added to forbidden APIs
2015-06-18 10:43:45 +02:00
Isabel Drost-Fromm 2fa56899d6 Merge pull request #11696 from MaineC/feature/simple-query-string-addboost
Add support for query boost to SimpleQueryStringBuilder.

Relates to #10217
2015-06-18 10:10:57 +02:00
Shay Banon cc97a4159e remove 1.7 version check 2015-06-18 09:34:39 +02:00
David Pilato 6a466b9a00 Replace Iterators#emptyIterator by JDK one
`com.google.common.collect.Iterators#emptyIterator()` is marked as deprecated and will be removed in May 2016. We should use JDK7 `Collections#emptyIterator()`
2015-06-18 08:47:31 +02:00
Igor Motov b2000a48a4 Snapshot/Restore: Add snapshot name validation logic to all snapshot operation
Make sure snapshot name validation occurs earlier in all snapshot operations.
2015-06-17 19:13:35 -04:00
Igor Motov 36da42c93b Show human readable Elasticsearch version that created index and date when index was created
By setting human parameter to true, it's now possible to see human readable versions of Elasticsearch that created and updated the index as well as the date when the index was created.

Closes #11484
2015-06-17 18:52:01 -04:00
Alex Ksikes 3f6dae1a73 More Like This: renamed `ignore_like` to `unlike`
This changes the parameter name `ignore_like` to the more user friendly name
`unlike`. This later feature generates a query from the terms in `A` but not
from the terms in `B`. This translates to a result set which is like `A` but
unlike `B`. We could have further negatively boosted any documents that have
some `B`, but these documents already do not receive any contribution from
having `B`, and would therefore negatively compete with documents having `A`.

Closes #11117
2015-06-17 17:18:50 -05:00
Simon Willnauer 90f9b5f60d Allow to opt-out of loading packaged REST tests
this is really just a workaround for plugins to run their own
REST tests instead of the core ones. It opts out of the rest test
loading from the core jar file and tries to load from the classpath instead.
Eventually we need to fix this infrastrucutre to move away from parameterized
tests such that subclasses can override behavior.

Closes #11721
2015-06-17 20:37:59 +02:00
Clinton Gormley 05d512f417 Packaging: Add LICENSE, NOTICE, and sha1 files and tests for all core dependencies
Added a licenses/ directory to core which contains a sha1 file for each JAR
dependency, and one or more LICENSE files and one NOTICE file for each
project.

Also adds dev-tools/src/main/resources/license-check/check_license_and_sha.pl
which checks that the licenses/ dir is up to date during a mvn verify,
and which can be used to update the sha1 files when upgrading dependencies.

Closes #2794
Closes #10684
Closes #11705
2015-06-17 18:06:00 +02:00
Michael McCandless 6fd893e10d Balance new shard allocations more evenly across multiple path.data
This change adds a simplistic heuristic to try to balance new shard
allocations across multiple data paths on one node so that e.g. if
there are two path.data and both have roughly the same free space, if
10 shards are suddenly allocated, we will put 5 on one path and 5 on
the other (vs 10 on a single path today).

Closes #11185

Closes #11122
2015-06-17 10:28:38 -04:00
Simon Willnauer 0434ecfb03 Merge pull request #11464 from nirmalc/nodes-preference
Search `preference` based on node specification
2015-06-17 12:33:51 +02:00
Christoph Büscher 9171ff0959 Merge branch 'master' into feature/query-refactoring 2015-06-17 11:54:00 +02:00
Isabel Drost-Fromm 5e9106ae5e Fixing review comments. 2015-06-17 11:18:32 +02:00
Isabel Drost-Fromm 907cdc61f7 Add boost to SimpleQueryStringBuilder.
As per discussion in #11274 this adds support for query boosting
to the SimpleQueryStringQuery.
2015-06-17 10:49:52 +02:00
Adrien Grand 17fac6dad5 Merge pull request #11568 from jpountz/remove/rivers
Rivers removal.
2015-06-17 08:20:48 +02:00
Adrien Grand 6e15e08f5e Merge pull request #11693 from jpountz/fix/simplify_timestamp_dv
Simplify doc values handling for _timestamp.
2015-06-17 08:19:33 +02:00
Nirmal Chidambaram 72a9d34eb8 5925 - Allow node specification in preference
-Allow node selector api's with new preference
ONLY_NODES ( selector apis like https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster.html)

-Update documentation
2015-06-16 11:49:12 -05:00
Adrien Grand 891689ed1e Merge pull request #11697 from jpountz/fix/synchronous_fielddata_removal
Reclaim memory from fielddata synchronously when an explicit clear is performed.
2015-06-16 17:36:29 +02:00
Shay Banon 4e0d65453f Add Unassigned meta data
Unassigned meta includes additional information as to why a shard is unassigned, this is especially handy when a shard moves to unassigned due to node leaving or shard failure.

The additional data is provided as part of the cluster state, and as part of `_cat/shards` API.

The additional meta includes the timestamp that the shard has moved to unassigned, allowing us in the future to build functionality such as delay allocation due to node leaving until a copy of the shard is found.
closes #11653
2015-06-16 17:01:48 +02:00
Christoph Büscher 33668a8df0 Merge pull request #11621 from cbuescher/feature/query-refactoring-boostingquery
Query refactoring: BoostingQueryBuilder and Parser
2015-06-16 16:51:03 +02:00
Tanguy Leroux 3b4ca8e399 Create PID_DIR in init.d script
Since the /var/run/elasticsearch directory is cleaned when the operating system starts, the init.d script must ensure that the PID_DIR is correctly created.

Closes #11594
2015-06-16 16:16:29 +02:00
Christoph Büscher 5a16ddc55f Query refactoring: AndQueryBuilder and Parser
Moving the query building functionality from the parser to the builders
new toQuery() method analogous to other recent query refactorings.

Relates to #10217
Closes #11628
2015-06-16 14:17:23 +02:00
Christoph Büscher 620211800e Query refactoring: BoostingQueryBuilder and Parser
As part of the refactoring of queries this PR splits the parsers parse() method
into a parsing and a query building part, adding serialization, hashCode() and
equals() to the builder.

Relates to #10217
Closes #11621
2015-06-16 13:57:06 +02:00
Adrien Grand 28e88e5bc0 Reclaim memory from fielddata synchronously when an explicit clear is performed.
The cache we are using for fielddata reclaims memory lazily/asynchronously. While this
helps with throughput this is an issue when a clear operation is issued manually since
memory is not reclaimed immediately. Since our clear methods already perform in linear
time anyway, this commit changes the fielddata cache to reclaim memory synchronously
when a clear command is issued manually. However, it remains lazy in case cache entries
are invalidated because segments or readers are closed, which is important since such
events happen all the time.

Close #11695
2015-06-16 12:08:39 +02:00
javanna 0a526be344 Query DSL: fix support for _name in some queries
Some of our Java api builders had wrong logic when it comes to serializing the query in json format, resulting in missing fields like _name. Also, regexp parser was ignoring the _name field.

Closes #11694
2015-06-16 12:04:51 +02:00
Christoph Büscher d25acdfad2 Merge pull request #11629 from cbuescher/feature/query-refactoring-constantscore
Query refactoring: ConstantScoreQueryBuilder and Parser
2015-06-16 11:52:17 +02:00
javanna 5d3bedfe96 Merge branch 'master' into feature/query-refactoring
Conflicts:
	core/src/main/java/org/elasticsearch/index/query/ExistsQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/ScriptQueryBuilder.java
2015-06-16 11:12:00 +02:00
Adrien Grand 2bb923a06d Simplify doc values handling for _timestamp.
`_timestamp` uses NumericDocValues instead of SortedNumericDocValues like other
numeric fields since it is guaranteed to be single-valued. However, we don't
need a different fielddata impl for it since DocValues.getSortedNumeric already
falls back to NUMERIC doc values if SORTED_NUMERIC are not available.
2015-06-16 10:52:06 +02:00
Adrien Grand cdd13253de Tests: Add test case from #11692 2015-06-16 10:34:51 +02:00
David Pilato 49cd9f3be4 Merge pull request #11676 from dadoonet/maven/clean-pom
[maven] clean pom.xml

In Maven parent project, in dependency management, we should only declare which versions of 3rd party jars we want to use but not force any scope.
It makes then more obvious in modules what is exactly the scope of any dependency.

For example, one could imagine importing `jimfs` as a `compile` dependency in another module/plugin with:

```xml
<dependency>
   <groupId>com.google.jimfs</groupId>
   <artifactId>jimfs</artifactId>
</dependency>
```

But it won't work as expected as the default maven `scope` should be `compile` but here it's `test` as defined in the parent project.

So, if you want to use this lib for tests, you should simply define:

```xml
<dependency>
   <groupId>com.google.jimfs</groupId>
   <artifactId>jimfs</artifactId>
   <scope>test</scope>
</dependency>
```

We also remove `maven-s3-wagon` from gce plugin as it's not used.
2015-06-16 09:12:03 +02:00
Ryan Ernst 98a5dbb10a Merge pull request #11686 from rjernst/remove/smart-object-wrapper
Remove SmartNameObjectMapper
2015-06-16 00:00:03 -07:00
Ryan Ernst a23c5b8bbc Make conditional clearer 2015-06-15 23:59:09 -07:00
Ryan Ernst 6e75f9fc93 fix field type tests 2015-06-15 17:03:32 -07:00
Ryan Ernst 7ab0009e41 Merge pull request #11644 from rjernst/refactor/field-type-equality
Add equals/hashcode to fieldtypes
2015-06-15 15:42:53 -07:00
Ryan Ernst 5601cea083 Mappings: Remove SmartNameObjectMapper
This was previously a container for an ObjectMapper, along with the
DocumentMapper that ObjectMapper came from. However, there was
only one use of needing the associated DocumentMapper, and that
wasn't actually used.
2015-06-15 15:40:01 -07:00
Boaz Leskes 2fb8df0985 Test: NetworkPartitionTests should make sure MockTransportService is used. 2015-06-15 23:40:52 +02:00
Ryan Ernst a08f51e13b Address PR comments 2015-06-15 10:55:37 -07:00
Christoph Büscher 1c0b7b3b06 Query refactoring: ConstantScoreQueryBuilder and Parser
Moving the query building functionality from the parser to the builders
new toQuery() method analogous to other recent query refactorings.

Relates to #10217
Closes #11629
2015-06-15 18:41:37 +02:00
David Pilato bd5c7d0ea2 [maven] clean pom.xml
In Maven parent project, in dependency management, we should only declare which versions of 3rd party jars we want to use but not force any scope.
It makes then more obvious in modules what is exactly the scope of any dependency.

For example, one could imagine importing `jimfs` as a `compile` dependency in another module/plugin with:

```xml
<dependency>
   <groupId>com.google.jimfs</groupId>
   <artifactId>jimfs</artifactId>
</dependency>
```

But it won't work as expected as the default maven `scope` should be `compile` but here it's `test` as defined in the parent project.

So, if you want to use this lib for tests, you should simply define:

```xml
<dependency>
   <groupId>com.google.jimfs</groupId>
   <artifactId>jimfs</artifactId>
   <scope>test</scope>
</dependency>
```

We also remove `maven-s3-wagon` from gce plugin as it's not used.
2015-06-15 17:08:15 +02:00
Adrien Grand e817393c8d Merge pull request #11669 from jpountz/remove/esoteric_fielddata_formats
Remove non-default fielddata formats.
2015-06-15 16:19:48 +02:00
Britta Weber 997550f5a2 Merge pull request #11595 from brwe/netowrk-partition-test
[TEST] remove node from nodes list if disruption is removed from node
2015-06-15 16:09:01 +02:00
Adrien Grand 14c9c239bc Remove non-default fielddata formats.
Now that doc values are the default for fielddata, specialized in-memory
formats are becoming an esoteric option. This commit removes such formats:
 - `fst` on string fields,
 - `compressed` on geo points.

I also removed documentation and tests that the fielddata cache is shared if
you change the format, since this is only true for in-memory fielddata formats
(given that for doc values, the caching is done directly in Lucene).
2015-06-15 14:05:23 +02:00
Christoph Büscher 11492b8962 Tests: fix RangeQueryBuilderTest, only use timestamp when date mapper is available
Also renaming field names to 'mapped_<fieldtype>' to avoid misunderstandings in error
messages.
2015-06-15 12:24:10 +02:00
Adrien Grand 36cab80cc8 Remove SimpleLuceneTests.testAddDocAfterPrepareCommit.
It is unnecessary to test features we are not using.
2015-06-15 11:26:28 +02:00
Adrien Grand ff279275b3 Upgrade to Lucene 5.2.1. 2015-06-15 10:51:05 +02:00
Simon Willnauer 9210aeacea Restore interrupt status when InterruptedException is caught 2015-06-15 09:40:28 +02:00
Shay Banon 87f0d09ace Add version 1.7.0
it was added in 1.x, but not in master
2015-06-14 23:43:45 +02:00
Simon Willnauer 02e540384d [TEST] Use NoMergePolicy in EngineTest#testVerboseSegments
These tests rely on that there are no merges but since MP is now
random this can break the test.
2015-06-14 12:20:28 +02:00
Simon Willnauer 0ac09fde95 [TEST] Use NoMergePolicy in EngineTest#testSegments
These tests rely on that there are no merges but since MP is now
random this can break the test.
2015-06-13 21:36:13 +02:00
Simon Willnauer 58ccb39dee Use CopyOnWrite list since list is concurrently modified as well as iterated 2015-06-13 21:17:57 +02:00
Simon Willnauer 09a30e7eee Remove deprecated script APIs
The script APIs have been deprecated long ago we can now remove them.
This commit still keeps the parsing code since it might be used in a
query that is still stuck in transaction log. This issue should be discussed
elsewhere.

Closes #11619
2015-06-13 10:11:43 +02:00
Ryan Ernst 0ef5a27556 Mappings: Add equals/hashcode to fieldtypes
In order to restrict a single set of field type settings for a given
field name across an index, we need the ability to compare field types.
This change adds equals and hashcode, as well as tests for every field
type.
2015-06-12 16:32:15 -07:00
Igor Motov e34cb18f8a Core: reduce the size of the XContent parsing exception
The content that caused the exception can be potentially very big and in most cases it's not very useful for debugging.
2015-06-12 19:06:56 -04:00
Shay Banon 3a97f322f7 Simplify ShardRouting and centralize move to unassigned
Make sure there is a single place where shard routing move to unassigned, so we can add additional metadata when it does, also, simplify shard routing implementations a bit
closes #11634
2015-06-12 22:51:21 +02:00
Shaunak Kashyap 2498cf9fab Revert "Removing top-level filter parameter from search API." 2015-06-12 11:09:58 -07:00
Shaunak Kashyap ffc4255041 Merge pull request #11600 from ycombinator/gh-8862
Removing top-level filter parameter from search API.
2015-06-12 10:07:40 -07:00
Christoph Büscher 5cdb1e6537 Merge branch 'master' into feature/query-refactoring 2015-06-12 15:57:03 +02:00
Michael McCandless f152614f92 use System.nanoTime for ThreadPool's estimated time, since it's less likely to go backwards 2015-06-12 09:16:23 -04:00
Simon Willnauer 5f40f97740 Fold ShardGetService creation away from Guice into IndexShard
it's always acccessed via IndexShard and has crazy circular dependencies or
rather had. It just makes IndexShard ctor bigger for no reason.
2015-06-12 13:45:10 +02:00
Simon Willnauer fab27f1bb0 Create ShardSuggestService/Metrics manually outside of guice
This ShardSuggestService is a simple metrics counter and doesn't need
to be injected. It just makes IndexShard ctor bigger for no reason.
2015-06-12 13:36:27 +02:00
Christoph Büscher 9fd61f786e Tests: Fix RangeQueryBuilderTest, adding separate test for unmapped field & timestamp 2015-06-12 12:21:44 +02:00
Colin Goodheart-Smithe a216062d88 Aggregations: allow users to perform simple arithmetic operations on histogram aggregations
Closes #11029
2015-06-12 09:25:52 +01:00
Boaz Leskes df8a3006fc Merge pull request #11615 from bleskes/async_fetch_non_existent_nodes
Internal: AsyncShardFetch can hang if there are new nodes in cluster state
2015-06-12 09:57:25 +02:00
Robert Muir b2ced13f3d Tests: fix test bugs so tests pass on IBM J9 (at least once) 2015-06-12 01:11:32 -04:00
Christoph Büscher b8dd7e688e Tests: Fix ExistsQueryBuilderTest after changes to builder merged in from master 2015-06-12 00:11:56 +02:00
Boaz Leskes 532fa0663b Internal: AsyncShardFetch can hang if there are new nodes in cluster state
The AsyncShardFetch retrieves shard information from the different nodes in order to detirment the best location for unassigned shards. The class uses TransportNodesListGatewayStartedShards and TransportNodesListShardStoreMetaData in order to fetch this information. These actions, inherit from TransportNodesAction and are activated using a list of node ids. Those node ids are extracted from the cluster state that is used to assign shards.

If we perform a reroute and adding new news in the same cluster state update task, it is possible that the AsyncShardFetch administration is based on
a different cluster state then the one used by TransportNodesAction to resolve nodes. This can cause a problem since TransportNodesAction filters away unkown nodes, causing the administration in AsyncShardFetch to get confused.

This commit fixes this allowing to override node resolving in TransportNodesAction and uses the exact node ids transfered by AsyncShardFetch
2015-06-11 23:16:09 +02:00
Christoph Büscher a125a31dfb Tests: Mark ExitsQueryTests as @AwaitsFix. 2015-06-11 22:53:44 +02:00
Igor Motov 93beea1f67 Snapshot/Restore: Move in-progress snapshot and restore information from custom metadata to custom cluster state part
Information about in-progress snapshot and restore processes is not really metadata and should be represented as a part of the cluster state similar to discovery nodes, routing table, and cluster blocks. Since in-progress snapshot and restore information is no longer part of metadata, this refactoring also enables us to handle cluster blocks in more consistent manner and allow creation of snapshots of a read-only cluster.

Closes #8102
2015-06-11 15:21:18 -04:00
Simon Willnauer 440580dd55 Cleanup MergeScheduler infrastrucutre
This commit cleans up all the MergeScheduler infrastructure
and simplifies / removes all unneeded abstractions. The MergeScheduler
itself is now private to the Engine and all abstractions like Providers
that had support for multiple merge schedulers etc. are removed.

Closes #11602
2015-06-11 20:49:42 +02:00
Lee Hinman 483a15a12b Use task's class name if not a TimedPrioritizeRunnable
This is helpful to track down the origin of pending_tasks that aren't
expected. In tests we catch this with an assert, but in production
asserts may not be enabled so we should at least add the class name.
2015-06-11 12:39:24 -06:00
Christoph Büscher 82cb0e66b4 Reverting previous commit 2015-06-11 19:32:21 +02:00
Shaunak Kashyap acfb3196f4 Removing top-level filter parameter from search API 2015-06-11 09:40:31 -07:00
Christoph Büscher 78eaffb646 Fixing edge case in RangeQueryBuilder when using time zone
When specifying a time zone, RangeQueryBuilder currently throws an
exeption when the field does not use DateFieldMapper, but if there
are no mappers at all for that field it doesn't complain. To be
consistent we also throw expection now.
2015-06-11 18:09:04 +02:00
Christoph Büscher c928852d4a Query Refactoring: ExistsQueryBuilder and Parser
Refactors ExistsQueryBuilder and Parser, splitting the parse() method into a parsing
and a query building part. Also moving newFilter() utility method from parser to query builder.

Changes in the BaseQueryTestCase include introduction of randomized version to test disabled
FieldNamesFieldMappers and also getting rid of the need for createEmptyBuilder() method by
using registered prototype constants.

Relates to #10217
Closes #11427
2015-06-11 14:40:21 +02:00
srogljan 40f85ccefb UNICODE_CHARACTER_CLASS fix 2015-06-11 13:18:18 +02:00
Simon Willnauer b0fe0096c4 [TEST] Prevent merges from being kicked off after two docs 2015-06-11 12:49:56 +02:00
Christoph Büscher 458d62e4a0 Refactors BoolQueryBuilder and Parser. Splits the parse(QueryParseContext ctx) method into a parsing
and a query building part, adding NamedWriteable implementation for serialization and hashCode(),
equals() for testing.

This change also adds tests using fixed set of leaf queries (Terms, Ids, MatchAll) as nested Queries in test query setup.
Also QueryParseContext is adapted to return QueryBuilder instances for inner filter parses instead of
previously returning Query instances, keeping old methods in place but deprecating them.

Relates to #10217
Closes #11427
2015-06-11 12:42:30 +02:00
Simon Willnauer f77804dad3 Bake in TieredMergePolicy
Today we provide the ability to plug in MergePolicy and
we provide the once lucene ships with. We do not recommend to change
the default and even only a small number of expert users would ever touch
this. This commit removes the ancient log byte size and log doc count
merge policy providers, simplifies the MergePolicy wiring and makes the
tiered MP the one and only default. All notions of a merge policy has been
removed from the docs and should be deprecated in the previous version.

Closes #11588
2015-06-11 11:58:30 +02:00
javanna 22c5205e67 Merge branch 'master' into feature/query-refactoring 2015-06-11 09:59:54 +02:00
Britta Weber e7a99501b0 [TEST] remove node from nodes list if disruption is removed from node
If we don't remove the node from the nodes list then later clearDisruption might fail
in case we shut down the node before.
2015-06-11 09:08:10 +02:00
Ryan Ernst d372bf7d7a Merge pull request #11586 from rjernst/fix/field-names-null
Mappings: Shortcut exists and missing queries when no types/docs exist
2015-06-10 14:58:57 -07:00
Simon Willnauer 657d6dd9cf Remove MergeScheduler pluggability
Nobody should really plug in a different merge scheduler for elasticsearch.
This is too expert and might cause catastrophic failures.
2015-06-10 20:28:30 +02:00
Boaz Leskes 650950c7f4 Merge pull request #11583 from bleskes/mapper_exception_unwrap
Recovery: Fix MapperException detection during translog ops replay
2015-06-10 19:06:53 +02:00
Ryan Ernst de4295cd7f Mappings: Shortcut exists and missing queries when no types/docs exist
There used to be a null check for _field_names mapper not existing. This
was recently removed. However, there is a corner case when the mapper
may be missing: when no types or docs exist at all in the index.

This change adds back a null check and just returns no docs.
2015-06-10 09:51:43 -07:00
Igor Motov 831cfa52d5 Snapshot/Restore: change metadata file format 2015-06-10 11:13:32 -04:00
Boaz Leskes e9d275d3a9 Recovery: fix MapperException detection during translog ops replay
The current ExceptionsHelper.unwrapCause(exception) requires the incoming exception to support ElasticsearchWrapperException , which TranslogRecoveryPerformer.BatchOperationException doesn't implement. I opted for a more generic solution
2015-06-10 17:12:14 +02:00
Boaz Leskes 5951f2580d Merge pull request #11576 from bleskes/wait_on_mapping_remove
Test: rename waitForConcreteMappingsOnAll & waitForMappingOnMaster to assertConcreteMappingsOnAll & assertMappingOnMaster
2015-06-10 14:37:59 +02:00
Adrien Grand fbe617c37b Internal: Make CompressedXContent.equals fast again.
We had to make CompressedXContent.equals decompress data to fix some
correctness issues which had the downside of making equals() slow. Now we store
a crc32 alongside compressed data which should help avoid decompress data in
most cases.

Close #11247
2015-06-10 13:37:08 +02:00
Boaz Leskes 356f0ffde2 Test: rename waitForConcreteMappingsOnAll & waitForMappingOnMaster to assertConcreteMappingsOnAll & assertMappingOnMaster
Now that mapping updates are sync and done before indexing we don't really need the waiting component. Also, removed many places were they were used as safe guard against delayed mapping updates, which are now not needed.
2015-06-10 13:27:57 +02:00
javanna ff9041dc48 Transport: allow to de-serialize arbitrary objects given their name
As part of the query refactoring, we want to be able to serialize queries by having them extend Writeable, rather than serializing their json. When reading them though, we need to be able to identify which query we have to create, based on its name.

For this purpose we introduce a new abstraction called NamedWriteable, which is supported by StreamOutput and StreamInput through writeNamedWriteable and readNamedWriteable methods. A new NamedWriteableRegistry is introduced also where named writeable prototypes need to be registered so that we are able to retrieve the proper instance of query given its name and then de-serialize it calling readFrom against it.

Closes #11553
2015-06-10 12:46:31 +02:00
Adrien Grand ac7ce2b899 Rivers removal.
While we had initially planned to keep rivers around in 2.0 to ease migration,
keeping support for rivers is challenging as it conflicts with other important
changes that we want to bring to 2.0 like synchronous dynamic mappings updates.
Nothing impossible to fix, but it would increase the complexity of how we
deal with dynamic mappings updates and manage rivers, while handling dynamic
mappings updates correctly is important for resiliency and rivers are on the go.
So removing rivers in 2.0 may well be a better trade-off.
2015-06-10 09:22:09 +02:00
javanna 5f66f68135 Merge branch 'master' into feature/query-refactoring
Conflicts:
	core/src/main/java/org/elasticsearch/index/query/RangeQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/SpanTermQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/TermQueryParser.java
2015-06-10 09:09:43 +02:00
Ryan Ernst 6c73647325 Mappings: Remove leftover sugar methods from FieldMapper
These methods are now all in MappedFieldType. This removes the remaining
callers of the methods on FieldMapper, and cuts down the FieldMapper
API to no longer include them.
2015-06-09 14:02:06 -07:00
Simon Willnauer 63ed54017e [TEST] Remove deprecated API 2015-06-09 20:43:01 +02:00
Ryan Ernst 9d57818cee fix field names to be disabled again pre 1.3 2015-06-09 11:42:06 -07:00
Simon Willnauer b0466bbacd [TEST] Add simple test for selecting fiels to be included in mlt query 2015-06-09 20:16:47 +02:00
Ryan Ernst 1fdae75025 Mappings: Make index level mapping apis use MappedFieldType
The MapperService is the "index wide view" of mappings. Methods on it
are used at query time to lookup how to query a field. This
change reduces the exposed api so that any information returned
is limited to that api exposed by MappedFieldType. In the future,
MappedFieldType will be guaranteed to be the same across all
document types for a given field.

Note CompletionFieldType needed some more settings moved to it. Other
than that, this change is almost purely cosmetic.
2015-06-09 10:04:28 -07:00
Ryan Ernst bf805168e9 Merge branch 'master' into refactor/null-value 2015-06-09 09:46:19 -07:00
Ryan Ernst 6719b1b79e add javadocs to null value functions 2015-06-09 09:46:05 -07:00
Alexander Reelsen a54d4e4aa8 Versioning: Adding 1.6.1 development version & 1.6.0 bwc index 2015-06-09 16:30:02 +02:00
Christoph Büscher 71f84b3a0e Fix using java 8 method in LimitQueryBuilder 2015-06-09 15:57:33 +02:00
Christoph Büscher aadfd1794c Query refactoring: refactored LimitQueryBuilder and Parser and added test
Split the parse method into a parsing and a query building part, adding serialization
and hashCode(), equals() for better testing. Add basic unit test for Builder and Parser.

Closes #11551
2015-06-09 14:48:13 +02:00
Britta Weber 2edff1bc4a [TEST] stop disruption before we check for deleted shards
The shard can potentially not be deleted if the obsever that checks for the shard
STARTED is not registered because the registering is delayed by the disruption.
If the sum of delays is more than 10s then the wait for shard deletion will time out.
2015-06-09 14:28:46 +02:00
javanna 42acee3587 Merge branch 'master' into feature/query-refactoring
Conflicts:
	core/src/main/java/org/elasticsearch/index/query/IdsQueryBuilder.java
2015-06-09 10:48:32 +02:00
Alexander Reelsen 3bda78e43b ResourceWatcher: Rename settings to prevent watcher clash
The ResourceWatcher used settings prefixed `watcher.`, which
potentially could clash with the watcher plugin.

In order to prevent confusion, the settings have been renamed to
`resource.reload` prefixes.

This also uses the deprecation logging infrastructure introduced
in #11033 to log deprecated settings and their alternative at
startup.

Closes #11175
2015-06-09 10:02:49 +02:00
Alexander Reelsen 57a94a151d IdsQueryBuilder: Allow to add a list in addition to array
In case a developer gets a list of ids from another data source,
it does not make a lot of sense, to convert it to an array first,
and then internally in IdsQueryBuilder elasticsearch creates a
list out of this.

Closes #5089
2015-06-09 09:31:19 +02:00
Michael McCandless 18609fff0d add node setting to send SegmentInfos debug output to System.out 2015-06-08 18:40:58 -04:00
Ryan Ernst ba15a2f6cb Mapping: Move null value handling into MappedFieldType
In order for exists queries to use the null value for
a field, null value needs to be part of the field type (should
differ between document types). This change moves null value
into the field type, as well as simplifies the null value
methods available to remove supportsNullValue().
2015-06-08 15:08:24 -07:00
Boaz Leskes 16d9480d78 Merge pull request #11536 from bleskes/recovery_translog_op_count_on_mapping_retry
Fix recovered translog ops stat counting when retrying a batch
2015-06-08 18:17:50 +02:00
Christoph Büscher b096984f04 Adapting DummyQueryParser to interfaces on featuer branch 2015-06-08 17:04:07 +02:00
Nirmal Chidambaram 931b9f9c74 Filtered out non data-nodes in relevant cat api
Closes #9214
Closes #9287
2015-06-08 16:05:42 +02:00
javanna 8cdd042cec Merge branch 'master' into feature/query-refactoring 2015-06-08 15:51:22 +02:00
javanna 1b496d09c3 [TEST] moved custom query parser tests to proper location 2015-06-08 15:50:43 +02:00
Boaz Leskes 10adb71445 Recovery: fix recovered translog ops stat counting when retrying a batch
#11363 introduced a retry logic for the case where we have to wait on a mapping update during the translog replay phase of recovery. The retry throws or recovery stats off as it may count ops twice.
2015-06-08 15:32:06 +02:00
javanna 7f673fbdfd Merge branch 'master' into feature/query-refactoring
Conflicts:
	core/src/test/java/org/elasticsearch/index/query/SimpleIndexQueryParserTests.java
	core/src/test/java/org/elasticsearch/index/query/guice/MyJsonQueryParser.java
	core/src/test/java/org/elasticsearch/index/query/plugin/PluginJsonQueryParser.java
2015-06-08 15:14:13 +02:00
javanna 2ef0fcfd6a Plugins: one single (global) way to register custom query parsers
There are different ways to register custom query parsers through plugins, a couple of them work per index via index settings, which is probably even too flexible. There also three different ways to add a global custom query parser through either IndicesQueriesModule or IndicesQueriesRegistry. This commit consolidates the registration of custom query parsers via IndicesQueriesModule#addQuery(Class<? extends QueryParser>). The complexity of supporting parsers per index is not needed hence it got removed. Also the other ways of registering global custom parsers are dropped in favour of the one mentioned above.

Closes #11481
2015-06-08 12:19:53 +02:00
Colin Goodheart-Smithe f336cea35e Scripting: Execute Scripting Engine before searching for inner templates in template query
The search template and template query did not run the template through the script engine before searching for an inner template. This meant that parsing for the inner template failed because the template was not always valid JSON (if it contained mustache code) when it was parsed to find the inner template. This has been fixed and Tests added to check for the failing behaviour.

Tests are from https://github.com/elastic/elasticsearch/pull/8393
2015-06-08 10:44:58 +01:00
Christoph Büscher b282ac6192 Moving parser tests and base classes to correct location 2015-06-08 09:00:49 +02:00
jaymode 78630e03a2 make prompt placeholders consistent with existing placeholders
In #10918, we introduced the prompt placeholders. These were had a different format
than our existing placeholders. This changes the prompt placeholders to follow the
format of the existing placeholders.

Relates to #11455
2015-06-06 10:41:07 -04:00
Christoph Büscher f1426abe3a Merge branch 'master' into feature/query-refactoring 2015-06-06 14:38:04 +02:00
Simon Willnauer 4c981ff4bf [BUILD] Don't shade core artifacts
This commit adds an additioal jar that is shaded and keeps all the
artifacts that are used by default on the server-side unshaded. Users
that need a shaded jar can now use the `shaded` classifyer to pull
the shaded minimized jar in instead. Including the shaded jar in a
downstream project looks like this:

```XML
<dependency>
  <groupId>org.elasticsearch</groupId>
  <artifactId>elasticsearch</artifactId>
  <classifier>shaded</classifier>
</dependency>
```
2015-06-05 21:52:09 +02:00
Boaz Leskes 6aa27a16c6 GatewayAllocator: reset rerouting flag after error
After asynchronously fetching shard information the gateway allocator issues a reroute via  a cluster state update task. #11421 introduced an optimization trying to avoid submitting unneeded reroutes when results for many shards come in together. This is done by having a rerouting flag, indicating a pending reroute is coming and thus any new incoming shard info doesn't need to issue a reroute. This flag wasn't reset upon an error in the reroute update task. Most notably - if a master node had to step during to a min_master_node violation, it could reject an ongoing reroute. Lacking to reset the flag causing it to skip any future reroute, when the node became master again.

Closes #11519
2015-06-05 21:21:09 +02:00
Igor Motov 1d02212b1c Snapshot/Restore: blob store shouldn't try deleting the write.lock file at the end of the restore process
Since we are creating write.lock earlier now, blob store shouldn't attempt deleting this file during clean up at the end of the restore process. The file is locked and the blog store doesn't succeed, but it generates a lot of useless warnings "failed to delete file [write.lock] during snapshot cleanup".

Closes #11517
2015-06-05 08:54:21 -10:00
Christoph Büscher b1a566fdf2 moving files not existing on master to core after introduction of multi module 2015-06-05 16:26:23 +02:00
Christoph Büscher 0f856cc7f9 Merge branch 'master' into feature/query-refactoring 2015-06-05 16:21:21 +02:00
Simon Willnauer 29d06605c0 add core module 2015-06-05 13:12:05 +02:00
Simon Willnauer 15a6244834 create core module 2015-06-05 13:12:03 +02:00