Commit Graph

2867 Commits

Author SHA1 Message Date
Britta Weber e0aa481bf5 Merge pull request #15213 from brwe/copy-to-in-multi-fields-exception
throw exception if a copy_to is within a multi field

Copy to within multi field is ignored from 2.0 on, see #10802.
Instead of just ignoring it, we should throw an exception if this
is found in the mapping when a mapping is added. For already
existing indices we should at least log a warning.
We remove the copy_to in any case.

related to #14946
2015-12-08 14:41:07 +01:00
Simon Willnauer 44df6814d4 fix imports 2015-12-08 14:39:16 +01:00
Simon Willnauer 2e27ee393f add rest API to reset settings 2015-12-08 14:39:16 +01:00
Simon Willnauer fbbb04b87e Add infrastructure to transactionally apply and reset dynamic settings
This commit adds the infrastructure to make settings that are updateable
resetable and changes the application of updates to be transactional. This means
setting updates are either applied or not. If the application failes all values are rejected.

This initial commit converts all dynamic cluster settings to make use of the new infrastructure.
All cluster level dynamic settings are not resettable to their defaults or to the node level settings.
The infrastructure also allows to list default values and descriptions which is not fully implemented yet.

Values can be reset using a list of key or simple regular expressions. This has only been implemented on the java
layer yet. For instance to reset all recovery settings to their defaults a user can just specify `indices.recovery.*`.

This commit also adds strict settings validation, if a setting is unknown or if a setting can not be applied the entire
settings update request will fail.
2015-12-08 14:39:15 +01:00
Simon Willnauer 7d6663e6e5 use AtomicBoolean instead of a simple boolean 2015-12-08 14:32:48 +01:00
Robert Muir a6ba351fb2 punch thru symlinks when loading plugins/modules
this ensures the codebase URL matches the permission grant (see matching toRealPath in Security.java)
in the case of symlinks or other shenanigans.

this is best effort, if we really want to support symlinks in any way, we need
e.g. qa or vagrant tests that configure a bunch of symlinks for things and ensure that in jenkins.
this should be easier to do with gradle, as we can just create a symlink'd home if we want
2015-12-08 08:04:11 -05:00
Simon Willnauer 449f9e7258 don't allow writing single bytes on recovery outputstream 2015-12-08 12:41:19 +01:00
Simon Willnauer ceb2797857 Handle cancel exceptions on recovery target if the cancel comes from the source
Today we only handle correctly if the `ExecutionCancelledException` comes from the
local execution. Yet, this can also come from remove and should be handled identically.
2015-12-08 12:01:22 +01:00
Simon Willnauer 8b8de5cb4b Don't send chunks after a previous send call failed. 2015-12-08 11:54:37 +01:00
Simon Willnauer 37b60bd76b more accurate chunk size 2015-12-08 11:20:55 +01:00
Simon Willnauer 414c04eb66 Restore chunksize of 512kb on recovery and remove configurability
This commit restores the chunk size of 512kb lost in a previous but unreleased
refactoring. At the same time it removes the configurability of:
 * `indices.recovery.file_chunk_size` - now fixed to 512kb
 * `indices.recovery.translog_ops` - removed without replacement
 * `indices.recovery.translog_size` - now fixed to 512kb
 * `indices.recovery.compress` - file chunks are not compressed due to lucene's compression but translog operations are.

The compress option is gone entirely and compression is used where it makes sense. On sending files of the index
we don't compress as we rely on the lucene compression for stored fields etc.

Relates to #15161
2015-12-08 11:20:55 +01:00
Boaz Leskes 0809e4a65f typo fix 2015-12-08 09:36:10 +01:00
Boaz Leskes 82b502c21f Make IndexShard operation be more explicit about whether they are expected to run on a primary or replica
This commit cherry picks some infrastructure changes from the `feature/seq_no` branch to make merging from master easier.

More explicitly, IndexShard current have  prepareIndex and prepareDelete methods that are called both on the primary as the replica, giving it a different origin parameter. Instead, this commits creates two explicit prepare*OnPrimary and prepare*OnReplica methods. This has the extra added value of not expecting the caller to use an Engine enum.

Also, the commit adds some code reuse between TransportIndexAction and TransportDeleteAction and their TransportShardBulkAction counter parts.

Closes #15282
2015-12-08 09:08:08 +01:00
Adrien Grand 47bcb33006 Merge pull request #15289 from jpountz/fix/mapper_tokenized
Mapper parsers should not check for a `tokenized` property.
2015-12-08 09:02:38 +01:00
Ryan Ernst 25d60e152f Tribe: Fix tribe node to load config file for internal client nodes
The tribe node creates one local client node for each cluster it
connects to. Refactorings in #13383 broke this so that each local client
node now tries to load the full elasticsearch.yml that the real tribe
node uses.

This change fixes the problem by adding a TribeClientNode which is a
subclass of Node. The Environment the node uses is now passed in (in
place of Settings), and the TribeClientNode simply does not use
InternalSettingsPreparer.prepareEnvironment.

The tests around tribe nodes are not great. The existing tests pass, but
I also manually tested by creating 2 local clusters, and configuring and
starting a tribe node. With this I was able to see in the logs the tribe
node connecting to each cluster.

closes #13383
2015-12-07 20:18:06 -08:00
Jason Tedor 618c2f7af5 Add missing license header to o.e.m.Probes 2015-12-07 22:07:20 -05:00
Simon Willnauer 4e80a5e099 Merge pull request #15234 from s1monw/remove_ancient_settings
Remove ancient deprecated and alternative recovery settings
2015-12-07 21:15:49 +01:00
Adrien Grand c6a73dfcd5 Mapper parsers should not check for a `tokenized` property.
I don't recall of this property of any of our field mappers and it's not in our
docs so I suspect it's very old. The removal of this property will not fail
version upgrades since none of the field mappers use it in toXContent.
2015-12-07 19:15:15 +01:00
Adrien Grand 3a58af04c0 Merge pull request #15245 from jpountz/fix/check_type_name
Make MappedFieldType.checkTypeName part of MappedFieldType.checkCompatibility.
2015-12-07 18:03:07 +01:00
Christoph Büscher c57672c9b3 Adding checks and tests for exceptions on unknown fieldnames 2015-12-07 16:52:53 +01:00
Christoph Büscher 6a7eedd8ae Switching HighlighterBuilder to use ParseFields 2015-12-07 14:56:05 +01:00
Jason Tedor 914486795f Addtional simplifications in IndexingMemoryController
This commit removes some unneeded null checks from
IndexingMemoryController that were left over from the work in #15251,
and simplifies the try-catch block in
IndexingMemoryController#updateShardBuffers.
2015-12-07 08:42:11 -05:00
Jim Ferenczi da380af10b Merge pull request #15216 from jimferenczi/copy_to_dynamic_object
Fix copy_to when the target is a dynamic object field.
2015-12-07 14:16:18 +01:00
Jim Ferenczi 8558a40894 Fix copy_to when the target is a dynamic object field.
Fixes #11237
2015-12-07 14:14:00 +01:00
Christoph Büscher aa69c4a20b Add fromXContent method to HighlightBuilder
For the search refactoring the HighlightBuilder needs a way to
create new instances by parsing xContent. For bwc this PR start
by moving over and slightly modifying the parsing from
HighlighterParseElement and keeps parsing for top level highlighter
and field options separate. Also adding tests for roundtrip
of random builder (rendering it to xContent and parsing it and
making sure the original builder properties are preserved)
2015-12-07 11:24:27 +01:00
Ryan Ernst 5402277462 Fix plugin service check for missing descriptor to allow ioexception 2015-12-06 23:09:00 -08:00
Robert Muir 1329ef487a Merge pull request #15262 from rmuir/filter_classes_in_scripts
Filter classes loaded by scripts
2015-12-06 10:29:56 -05:00
Michael McCandless fe5570dba8 use dummy mac address if anything goes wrong asking for the real one 2015-12-06 06:10:43 -05:00
Robert Muir 2169a123a5 Filter classes loaded by scripts
Since 2.2 we run all scripts with minimal privileges, similar to applets in your browser.
The problem is, they have unrestricted access to other things they can muck with (ES, JDK, whatever).
So they can still easily do tons of bad things

This PR restricts what classes scripts can load via the classloader mechanism, to make life more difficult.
The "standard" list was populated from the old list used for the groovy sandbox: though
a few more were needed for tests to pass (java.lang.String, java.util.Iterator, nothing scary there).

Additionally, each scripting engine typically needs permissions to some runtime stuff.
That is the downside of this "good old classloader" approach, but I like the transparency and simplicity,
and I don't want to waste my time with any feature provided by the engine itself for this, I don't trust them.

This is not perfect and the engines are not perfect but you gotta start somewhere. For expert users that
need to tweak the permissions, we already support that via the standard java security configuration files, the
specification is simple, supports wildcards, etc (though we do not use them ourselves).
2015-12-05 21:46:52 -05:00
Jason Tedor 74a81b4a26 Remove logging statement when no shards are active 2015-12-05 11:30:17 -05:00
Jason Tedor fe0b9be472 Simplify shard inactive logging
This commit simplifies shard inactive debug logging to only log when the
physical shard is marked as inactive. This eliminates duplicate logging
that existed in IndexShard#checkIdle and
IndexingMemoryController#checkIdle, and eliminates excessive logging
that was occurring when the shard was already inactive as a result of
the work in #15252.
2015-12-05 11:20:23 -05:00
Jason Tedor 73a0cc6488 Clarify variable name in IndexingMemoryController#availableShards 2015-12-05 09:23:46 -05:00
Ryan Ernst 7ac094fdc5 Remove unused dependency on rest spec in core integ tests 2015-12-04 19:48:40 -08:00
Ryan Ernst 62a0e0bc2b Fix plugin test to account for possibly extra dir by mock fs 2015-12-04 17:55:31 -08:00
Robert Muir 506fa617da Merge pull request #15253 from rmuir/getClassLoader
ban RuntimePermission("getClassLoader")
2015-12-04 18:34:08 -05:00
Ryan Ernst 50716c3941 Plugins: Add nicer error message when an existing plugin's descriptor is missing
Currently, when a user tries to install an old plugin (pre 2.x) on a 2.x
node, the error message is cryptic (just printing the file path that was
missing, when looking for the descriptor). This improves the message to
be more explicit that the descriptor is missing, and suggests the
problem might be the plugin was built before 2.0.

closes #15197
2015-12-04 14:11:37 -08:00
Jason Tedor bbef8acd3c Cleanup ClusterServiceIT#testClusterStateBatchedUpdates
This commit addresses some issues that arose during the review of #14899
but were lost during squash while integrating into master.
 - the number of test threads is dropped to at most eight
 - a local variable is renamed for clarity
 - task priorities are randomized
2015-12-04 16:36:29 -05:00
Jason Tedor e7952e2023 Fix test bug in ClusterServiceIT#testClusterStateBatchedUpdates
This commit fixes a test bug in
ClusterService#testClusterStateBatchedUpdates. In particular, in the
case that an executor did not receive a task assignment from the random
assignments, it would not have an entry in the map of executors to
counts of assigned tasks. The fix is to just check if each executor has
an entry in the counts map.
2015-12-04 16:25:36 -05:00
Jason Tedor b1a67b1c69 Simplify IndexingMemoryController#checkIdle
This commit further simplifies IndexingMemoryController#checkIdle after
the changes in #15251.
2015-12-04 16:10:43 -05:00
Robert Muir 46377778a9 Merge branch 'master' into getClassLoader 2015-12-04 15:58:36 -05:00
Robert Muir b0c64910b0 ban RuntimePermission("getClassLoader")
this gives more isolation between modules and plugins.
2015-12-04 15:58:02 -05:00
Jason Tedor 5a391f116b Merge pull request #15251 from jasontedor/stateless-indexing-memory-controller
IndexingMemoryController should not track shard index states
2015-12-04 15:18:40 -05:00
Jason Tedor 56328160c9 Absorb core ShardsIndicesStatusChecker logic into body of run 2015-12-04 15:16:22 -05:00
Jason Tedor 5341404f01 IndexingMemoryController should not track shard index states
This commit modifies IndexingMemoryController to be stateless. Rather
than statefully tracking the indexing status of shards,
IndexingMemoryController can grab all available shards, check their idle
state, and then resize the buffers based on the number of and which
shards are not idle.

The driver for this change is a performance regression that can arise in
some scenarios after #13918. One scenario under which this performance
regression can arise is if an index is deleted and then created
again. Because IndexingMemoryController was previously statefully
tracking the state of shards via a map of ShardIds, the new shards with
the same ShardIds as previously existing shards would not be detected
and therefore their version maps would never be resized from the
defaults. This led to an explosion in the number of merges causing a
degradation in performance.

Closes #15225
2015-12-04 15:01:47 -05:00
Ryan Ernst 01d48e2062 Merge branch 'master' into jigsaw 2015-12-04 11:29:49 -08:00
Lee Hinman 46ca2d679d Merge remote-tracking branch 'dakrone/remove-get-forcemerge' 2015-12-04 10:21:57 -07:00
Adrien Grand be033f1362 Make MappedFieldType.checkTypeName part of MappedFieldType.checkCompatibility. 2015-12-04 17:15:49 +01:00
Yannick Welsch 8dcc692373 Ignore RejectedExecutionException in NodesFaultDetection
Closes #15212
2015-12-04 16:35:20 +01:00
Adrien Grand e5ed5b908f Check mapping compatibility up-front.
Today we only check mapping compatibility when adding mappers to the
lookup structure. However, at this stage, the mapping has already been merged
partially, so we can leave mappings in a bad state. This commit removes the
compatibility check from Mapper.merge entirely and performs it _before_ we
call Mapper.merge.

One minor regression is that the exception messages don't group together errors
that come from MappedFieldType.checkCompatibility and Mapper.merge. Since we
run the former before the latter, Mapper.merge won't even have a chance to let
the user know about conflicts if conflicts were discovered by
MappedFieldType.checkCompatibility.

Close #15049
2015-12-04 14:02:04 +01:00
Adrien Grand 3f18487afa Merge pull request #15177 from jpountz/remove/MergeMappingException
Remove MergeMappingException.
2015-12-04 13:55:54 +01:00
Christoph Büscher 5d0689581b Geo: Remove `translated` state from LineStringBuilder
The `translated` flag makes LineStringBuilder stateful and gets set
to true under certain conditions when building a Shape or Geometry
from the ShapeBuilder. This makes building operations not be idempotent,
so calling build() more than once on a LineStringBuilder might change the
builder itself. This PR fixes this by replacing the instance variable by
a local `translated` flag that is only updated internally during the
building process and created again on any subsequent calls to build()
or buildGeometry().
2015-12-04 12:57:52 +01:00
Adrien Grand 3f86adddbf Remove MergeMappingException.
Failures to merge a mapping can either come as a MergeMappingException if they
come from Mapper.merge or as an IllegalArgumentException if they come from
FieldTypeLookup.checkCompatibility. I think we should settle on one: this pull
request replaces all usage of MergeMappingException with
IllegalArgumentException.
2015-12-04 12:56:26 +01:00
javanna fae494aa84 Java api: allow to specify ttl as a time value (either as a string or using TimeValue)
The ttl could be specified as a time value only via the REST layer. That is now possible via java api too, either as a string or as a proper TimeValue. The internal format in IndexRequest becomes now TimeValue, which will then still converted to a long before storing the document.

Closes #15047
2015-12-04 11:51:22 +01:00
Yannick Welsch fef043a5bf Persist currently started allocation IDs to index metadata
Closes #14964
2015-12-04 11:31:48 +01:00
Yannick Welsch 342665300b Add capabilities for serializing map-based cluster state diffs
- Supports ImmutableOpenIntMap besides java.util.Map and ImmutableOpenMap
- Map keys can be any value (not only String)
- Map values do not have to implement Diffable interface. In that case custom value serializer needs to be provided.
2015-12-04 11:24:22 +01:00
Simon Willnauer 536e7e1268 Remove ancient deprecated and alternative recovery settings
Several settings have been deprecated or are replaced with new settings after refactorings
in version 1.x. This commit removes the support for these settings.

The settings are:

 * `index.shard.recovery.translog_size`
 * `index.shard.recovery.translog_ops`
 * `index.shard.recovery.file_chunk_size`
 * `index.shard.recovery.concurrent_streams`
 * `index.shard.recovery.concurrent_small_file_streams`
 * `indices.recovery.max_size_per_sec`
2015-12-04 09:48:00 +01:00
Robert Muir 688935df1c fix broken plugin warning 2015-12-04 02:38:56 -05:00
Robert Muir c78c8bae08 improve logger output on startup 2015-12-04 02:23:34 -05:00
Robert Muir 699b140d69 update lists, rename stuff to be clearer, remove public method that is unused 2015-12-04 01:43:07 -05:00
Robert Muir 7160c5ec15 list modules separately in pluginservice 2015-12-04 01:13:17 -05:00
Robert Muir 763747df71 don't let pluginmanager install modules 2015-12-03 23:36:02 -05:00
Robert Muir 208b46d94b add leniency for tests 2015-12-03 23:25:57 -05:00
Ryan Ernst f2cebd9b3b Merge branch 'master' into jigsaw 2015-12-03 20:02:44 -08:00
Robert Muir 794264023a add module loading logic (untested) 2015-12-03 23:01:03 -05:00
Robert Muir da8fe687fc add modules/ infra (but no loading via pluginservice yet) 2015-12-03 22:34:24 -05:00
Ryan Ernst b50d94efab Merge pull request #15228 from rjernst/cli_error
Cli tools: Use toString instead of getMessage for exceptions
2015-12-03 19:01:33 -08:00
Ryan Ernst f78c732ca9 Cli tools: Use toString instead of getMessage for exceptions
When not in debug mode, we currently only print the message of an
exception. However, this is not usually useful without knowing what the
exception type was. This change makes cli tools use toString() on the
exception so we get the type + message.
2015-12-03 17:27:45 -08:00
Jason Tedor 6613b3a0d6 Merge pull request #15221 from jasontedor/avoid-trace-logging-allocations-in-transport-broadcast-by-node-action
Avoid trace logging allocations in TransportBroadcastByNodeAction
2015-12-03 17:17:01 -05:00
Lee Hinman f709b7283f Remove `GET` option for /_forcemerge
POST should be used to indicate this is not just a retrieval operation.

Resolves #15165
2015-12-03 13:56:15 -07:00
Jason Tedor de0e1f5e2f Avoid trace logging allocations in TransportBroadcastByNodeAction
This commit wraps the trace logging statements in
TransportBroadcastByNodeAction in trace enabled checks to avoid
unnecessarily allocating objects.

The most egregious offenders were the two trace logging statements in
BroadcastByNodeTransportRequestHandler#onShardOperation. Aside from the
usual object allocations that occur when invoking ESLogger#trace (the
allocated object array for the varargs Object... parameter), these two
logging statements were invoking ShardRouting#shortSummary generating a
bunch of char arrays and Strings (from the StringBuilder, and so a bunch
of array copies as well). In a scenario where there are a lot of shards
and this method is being invoked frequently (e.g., constantly hitting
the _stats endpoint), these two unprotected trace logging statements
were generating a lot of unnecessary allocations.
2015-12-03 14:38:16 -05:00
Jason Tedor b58d82f66c Remove obsolete flag in ShardStateAction$ShardRoutingEntry 2015-12-03 14:08:35 -05:00
Jason Tedor 928d53a884 Apply shard starts in a single batch 2015-12-03 14:04:10 -05:00
Jason Tedor 99eac0e7d7 Use general cluster state batching mechanism for shard started
This commit modifies the handling of shard started cluster state updates
to use the general cluster state batching mechanism. An advantage of
this approach is we now get correct per-listener notification on
failures.
2015-12-03 14:04:10 -05:00
Jason Tedor 182c22f23f Merge pull request #15016 from jasontedor/shard-failure-batch
Use general cluster state batching mechanism for shard failures
2015-12-03 14:02:48 -05:00
Jason Tedor b70d97f36b Remove unnecessary method in AllocationService 2015-12-03 13:42:05 -05:00
Jason Tedor 73a2d3085a Rename variable for clarity in ShardFailedClusterStateHandler#execute 2015-12-03 13:41:34 -05:00
Adrien Grand d2e07f21e0 Merge pull request #15217 from jpountz/fix/simplify_MetaDataMappingService
Simplify MetaDataMappingService.
2015-12-03 16:26:08 +01:00
Adrien Grand 3ff91baedf Simplify MetaDataMappingService.
Now that we create all types on indices that receive a mapping update, creation
of temporary indices could get simpler.
2015-12-03 16:06:51 +01:00
Jason Tedor d1fd1fb908 Remove unnecessary early-out in MetaDataMappingService#executeRefresh
This commit removes a simple early-out check in
MetaDataMappingService#executeRefresh. The early-out is unnecessary
because the cluster state task execution framework will not invoke
ClusterStateTaskExecutor#execute if the list of tasks is empty.
2015-12-03 09:05:19 -05:00
Jason Tedor 310e98f51d Update stale Javadoc in MetaDataMappingService#executeRefresh
This commit updates a stale Javadoc on
MetaDataMappingService#executeRefresh. Previously this method handled
refresh and update tasks. Update tasks have been removed and the method
was renamed, but the Javadoc was not updated to reflect this.
2015-12-03 09:05:19 -05:00
Martijn van Groningen d0fbd9d1c4 index template: Disallow index template pattern to be the same as an alias name
This can cause index creation to fail down the line.
2015-12-03 13:35:16 +01:00
Boaz Leskes c5301e853a reduced the number of shards in RecoveryWhileUnderLoadIT.testRecoveryWhileRelocating 2015-12-03 10:00:48 +01:00
Boaz Leskes 9930e6883d Move IndicesService.canDeleteShardContent to use IndexSettings
Just a minor cleanup/simplification

Closes #15059
Closes #15150
2015-12-03 09:55:07 +01:00
Adrien Grand 6ff82260a7 Merge pull request #15029 from jpountz/remove/onIndexService
Remove Plugin.onIndexService.
2015-12-03 09:46:30 +01:00
Adrien Grand 612ef4488e Add missing license header. 2015-12-03 09:41:06 +01:00
Adrien Grand 79ff2ddd20 Merge pull request #15202 from jpountz/fix/clear_empty_query_cache
Fix NPE when a segment with an empty cache gets closed.
2015-12-03 09:38:08 +01:00
Jason Tedor ab45d69b16 Remove unused import in o.e.c.l.Releasables 2015-12-02 16:29:04 -05:00
Adrien Grand 7d25623b1f Fix NPE when a segment with an empty cache gets closed.
This is due to the fact that the query cache will still call the
onDocIdSetEviction callback in this case but with a number of entries equal to
zero.

Close #15043
2015-12-02 21:59:20 +01:00
Areek Zillur 3d8c32008e [TEST] geo completions work with geo point fields 2015-12-02 13:01:33 -05:00
Jason Tedor 199a05311e Remove unchecked warnings rendered unnecessary 2015-12-02 10:55:25 -05:00
Jason Tedor 05430a788a Remove and forbid use of the type-unsafe empty Collections fields
This commit removes and now forbids all uses of the type-unsafe empty
Collections fields Collections#EMPTY_LIST, Collections#EMPTY_MAP, and
Collections#EMPTY_SET. The type-safe methods Collections#emptyList,
Collections#emptyMap, and Collections#emptySet should be used instead.
2015-12-02 10:41:59 -05:00
Adrien Grand 79188ed38d Treat mappings at an index-level feature.
Today we try to have type-level granularity when dealing with mappings. This
does not play well with the cross-type validations that we are adding. For
instance we prevent the `_parent` field to point to an existing type. This
validation would be skipped today in the case of dedicated master nodes, since
those master nodes would only create the type that is being updated when
updating a mapping.
2015-12-02 16:32:12 +01:00
Robert Muir b7f497627f Merge pull request #15172 from rmuir/nuke_unused_deps
remove unused core dependencies
2015-12-02 07:20:11 -05:00
Martijn van Groningen ec74586680 metadata: If an alias and index share the same name then a descriptive error should be thrown
In 1.x it is possible via index templates to create an index with an alias with the same name as the index. The index name must match the index template and have an alias with the same name as the index being created.
2015-12-02 12:23:51 +01:00
Ryan Ernst d68c6673a2 Build: Cleanup precommit task gradle code
This change attempts to simplify the gradle tasks for precommit. One
major part of that is using a "less groovy style", as well as being more
consistent about how tasks are created and where they are configured. It
also allows the things creating the tasks to set up inter task
dependencies, instead of assuming them (ie decoupling from tasks
eleswhere in the build).
2015-12-01 22:36:54 -08:00
Robert Muir ef7258413f improve exception and remove complex test, such indexes will not be loaded into a closed state.
we will not load them at all.
2015-12-01 23:03:54 -05:00
Robert Muir 6e5c301739 remove unused dependencies 2015-12-01 22:22:11 -05:00
Jason Tedor e52faa81cc Fix typo in field name in MetaDataMappingService 2015-12-01 13:16:28 -05:00
Adrien Grand 304695e7ee Don't treat _default_ as a regular type.
This adds safety that you can't index into the `_default_` type (it was possible
before), and can't add default mappers to the field type lookups (was not
happening in tests but I think this is still a good check).

Also MapperService.types() now excludes `_default` so that eg. the `ids` query
does not try to search on this type anymore.
2015-12-01 18:03:11 +01:00
Jason Tedor 413688b0ca Apply shard failures in a single batch 2015-12-01 11:19:47 -05:00
Adrien Grand 77095a3e97 Merge pull request #15144 from jpountz/fix/ignored_merge_failure
Mappings: Don't ignore merge failures.
2015-12-01 16:37:32 +01:00
Jason Tedor d7f4dd0767 Use general cluster state batching mechanism for shard failures
This commit modifies the handling of shard failure cluster state updates
to use the general cluster state batching mechanism. An advantage of
this approach is we now get correct per-listener notification on
failures.
2015-12-01 10:19:23 -05:00
Christoph Büscher 6f2c36dcb7 Merge pull request #15113 from cbuescher/highligh-builder-refactoring
Make HighlighterBuilder implement Writable, equals and hashCode
2015-12-01 15:28:33 +01:00
Christoph Büscher 027a9b1844 Refactor HighlighterBuilder
This change pulls out the common fields that HighlighterBuilder shares with
its nested Field class into a new abstract CommonHighlighterOptions superclass
which also gets equals() and hashCode() method and methods to serialize the
common fields to a StreamOutput and read them from a stream.

Relates to #15044
2015-12-01 15:17:09 +01:00
javanna c67a332486 Query DSL: Enforce distance is greater than 0 in geo distance query
Validation is not done as part of the distance setter method and tested in GeoDistanceQueryBuilderTests. Fixed GeoDistanceTests to adapt to the new validation.

Closes #15135
2015-12-01 14:07:32 +01:00
Jason Tedor 13dbed9c92 Add each mapping at most once on batch mapping updates
When creating an index on master for the purpose of updating mappings, a
mapping being updated could needlessly be merged multiple times. This
commit ensures that each mapping is merged at most once while preparing
to update mappings.
2015-12-01 07:52:32 -05:00
Jason Tedor 09006ace11 Add the default mapping at most once on batch mapping updates
When creating an index on master for the purpose of updating mappings,
the default mapping could needlessly be added multiple times. This
commit ensures that the default mapping is added at most once while
preparing to update mappings.
2015-12-01 07:52:32 -05:00
Jason Tedor eea72a6d86 Preserve existing mappings on batch mapping updates
This commit addresses an issues introduced in #14899 to apply mapping
updates in batches. The issue is that an existing mapping for a type
could be lost if that type came in a batch that already contained a
mapping update for another type on the same index. The underlying issue
was that the existing mapping would not be merged in because the merging
logic was only tripped once per index, rather than for all types seeing
updates for each index. Resolving this issue is simply a matter of
ensuring that all existing types seeing updates are merged in.

Closes #15129
2015-12-01 07:52:32 -05:00
David Pilato fb79d064d7 Merge remote-tracking branch 'origin/master' 2015-12-01 12:46:09 +01:00
Adrien Grand cebd7bdd7f Mappings: Don't ignore merge failures. 2015-12-01 12:22:07 +01:00
javanna d17d629422 [TEST] stabilize SimpleSearchIT.testQueryNumericFieldWithRegex
Provide mappings explicitly instead of relying on dynamic mapping, also add missing ensureGreen.

Closes #15105
2015-12-01 12:13:06 +01:00
David Pilato 854099f1d5 Reject refresh usage in bulk items when using and fix NPE when no source
The REST bulk API rejects use of `refresh` at the item level. But the Java API lets the user setting it.

We need to have the same behavior and don't let think the user he can define `refresh` per bulk item.

Note that the user can still define `refresh` on the bulk itself.

Also a user can create with Java API an IndexRequest without any source which is causing a NPE when evaluating the bulk item size.

Closes #7361.
Closes #15120.
2015-12-01 12:00:30 +01:00
Yannick Welsch 059a675aa5 [TEST] mark test as awaitsfix: RareClusterStateIT.testDeleteCreateInOneBulk() 2015-12-01 11:41:51 +01:00
Britta Weber e2f87b0c52 multi field names may not contain dots
related to #14957
2015-12-01 11:27:26 +01:00
Boaz Leskes e256c6fdd1 AwaitsFix testDynamicUpdates
pending on https://github.com/elastic/elasticsearch/issues/15129
2015-12-01 09:57:22 +01:00
Robert Muir 44f21b24d7 Fix unit tests to bind to port 0.
I will followup with ITs and other modules. By fixing this, these tests become more reliable (will never sporatically
fail due to other stuff on your machine: ports are assigned by the OS), and it allows us to move forward with
gradle parallel builds, in my tests this is a nice speedup, but we can't do it until tests are cleaned up
2015-11-30 17:22:58 -05:00
Jason Tedor fba74c9615 Add docs for cluster state update task batching 2015-11-30 08:59:24 -05:00
Jason Tedor ffb3e0a845 ClusterStateTaskListener#onNoLongerMaster now throws NotMasterException 2015-11-30 08:59:24 -05:00
Jason Tedor 72e18ec681 Add builder to create cluster state executor results 2015-11-30 08:59:24 -05:00
Jason Tedor 9aa9447016 Simplify loop in InternalClusterService#runTasksForExecutor 2015-11-30 08:59:23 -05:00
Jason Tedor 7caee2fa4d Explicitly correspond cluster state tasks and execution results 2015-11-30 08:59:23 -05:00
Jason Tedor c3f97e7642 Simplify InternalClusterService#submitStateUpdateTask with lambdas 2015-11-30 08:59:23 -05:00
Jason Tedor 1fb6a1f669 Simplify grouping of cluster state update tasks 2015-11-30 08:59:23 -05:00
Jason Tedor eaba3d5cb3 Add test for cluster state batch updates 2015-11-30 08:59:23 -05:00
Boaz Leskes f0f89e708d Split cluster state update tasks into roles
This commit splits cluster state update tasks into roles. Those roles
are:
 - task info
 - task configuration
 - task executor
 - task listener

All tasks that have the same executor will be executed in batches. This
removes the need for local batching as was previously in
MetaDataMappingService.

Additionally, this commit reintroduces batching on mapping update calls.

Relates #13627
2015-11-30 08:59:23 -05:00
Christoph Büscher a0fe93fa67 Tests: Correction in AbstractShapeBuilderTestCase
Removed check that two shapes that are different according
to equals() have different hashCode since that is not required
by the contract of hashCode.
2015-11-30 11:09:41 +01:00
Jim Ferenczi e182072b6f Merge pull request #15017 from jimferenczi/fields_option
Refuse to load fields from _source when using the `fields` option and support wildcards.
2015-11-30 11:01:21 +01:00
Jim Ferenczi 731833cfc6 Fixes #14489
Do not to load fields from _source when using the `fields` option.
  Non stored (non existing) fields are ignored by the fields visitor when using the `fields` option.

Fixes #10783
  Support * wildcard to retrieve stored fields when using the `fields` option.
  Supported pattern styles are "xxx*", "*xxx", "*xxx*" and "xxx*yyy".
2015-11-30 11:00:32 +01:00
Robert Muir b7523bb974 Add workaround for JDK-6427854
See e.g. http://build-us-00.elastic.co/job/es_feature_ingest/2831/consoleFull

The bug can still happen, so we should let netty do this workaround
2015-11-29 16:12:52 -05:00
Michael McCandless 94954ece3a improve javadocs 2015-11-29 06:13:47 -05:00
Michael McCandless 8933947b95 only pull Fields once from the reader 2015-11-29 06:00:11 -05:00
Robert Muir 29853744a3 Merge pull request #15055 from rmuir/fun_with_windows
set ActiveProcessLimit=1 on windows
2015-11-27 16:57:34 -05:00
Christoph Büscher b375084305 Tests: Correction in AbstractQueryTestCase
Removed check that two query builder that are different according
to equals() have different hashCode since that is not required
by the contract of hashCode.
2015-11-27 17:36:57 +01:00
Daniel Mitterdorfer 2f87e0a4d9 Declare NetworkService non-final to support mocking in tests 2015-11-27 15:51:19 +01:00
Robert Muir ee827198ff set ActiveProcessLimit=1 on windows
This blocks process creation, similar to the approach for BSD.
2015-11-27 01:50:09 -05:00
Robert Muir aa56bfa562 Merge pull request #15039 from rmuir/setrlimit
set RLIMIT_NPROC = 0 on bsd/os X systems.
2015-11-26 21:18:57 -05:00
Jason Tedor 5c5e02e0bf Remove outdated comment and cleanup RestVerifyRepositoryAction 2015-11-26 19:11:11 -05:00
Simon Willnauer e363e0c2aa Merge pull request #15012 from s1monw/issues/14866
Prevent writing to closed channel if translog is already closed
2015-11-26 14:47:38 +01:00
Simon Willnauer 59be738e40 Merge pull request #15035 from s1monw/simplify_monitoring_service
Simplify MonitorService construction and detach from guice
2015-11-26 14:05:32 +01:00
Simon Willnauer 9f6598b18d Fix compile errors 2015-11-26 13:41:00 +01:00
Britta Weber 16b84b3999 run bwc test also as integ test and share methods
We had no itegration test before with long terms and several shards only
a bwc test.

related to #14948
2015-11-26 10:57:33 +01:00
Simon Willnauer 3dfa146632 Move ensureOpen calls under lock
We used to check on several places if we are still open but non of these
places did the check under the lock which leaves a small window where we
potentially get closed but still access an already closed channel or another
IO resource.
2015-11-26 10:14:45 +01:00
Simon Willnauer dfdc738a9f Simplify MonitorService construction and detach from guice
This is a pretty trivial change that moves most of the monitor service related
object creation from guice into the monitor service. This is a babystep towards removing
guice on the node level as well. Instead of opening huge PRs I try to do this in baby-steps
that are easier to digest.
2015-11-26 09:50:41 +01:00
Robert Muir 2341330aba set RLIMIT_NPROC = 0 on bsd/os X systems.
This BSD-specific limit prevents child process creation.
2015-11-26 02:58:53 -05:00
Christoph Büscher 609d9db470 Geo: Fix toString() in GeoDistanceRangeQuery and GeoPolygonQuery
Minor typo in the two queries toString() method. They were former
filters the toString() seems to have been forgotten while renaming.
2015-11-25 21:52:20 +01:00
Adrien Grand f8a414c681 Remove Plugin.onIndexService. 2015-11-25 19:34:33 +01:00
Simon Willnauer 9c1f930b8c Prevent writing to closed channel if translog is already closed
We handle AlreadyClosedExceptions gracefully wherever IndexShard / Engine
is used. In some cases, instead of throwing the appropriate exception we
bubble up ChannelClosedException instead which causes shard failures etc.
Today, it seems like that this can only happen if the engine is closed without
acquireing the lock which means that the shard has failed already so the impact is really
just a confusing log message. Yet, this change enforces throwing the right exception
if the translog is already closed.

Closes #14866
2015-11-25 13:54:13 +01:00
Adrien Grand e2447c7ce0 Merge pull request #14981 from jpountz/fix/assert_version_sanity
Check that the declared versions match the backward indices.
2015-11-25 10:49:30 +01:00
Isabel Drost-Fromm 473b19400f Merge pull request #14827 from MaineC/bug-fix/10021-error-listeners
Adds exception objects to log messages.
2015-11-25 10:42:38 +01:00
Christoph Büscher e507ef807a Merge pull request #14933 from cbuescher/shapes-refactor-point
Make PointBuilder, CircleBuilder & EnvelopeBuilder implement Writable
2015-11-25 09:57:20 +01:00
Christoph Büscher 1f94448e92 Making EnvelopeBuilder writable and adding equals/hashCode 2015-11-25 08:43:25 +01:00
Adrien Grand 5fe3ce89e0 Check that the declared versions match the backward indices. 2015-11-24 20:40:02 +01:00
Clinton Gormley c512cc5248 Added bwc indices for 2.0.1 and 2.1.0 and versions 2.0.2-SNAPSHOT and 2.1.1-SNAPSHOT 2015-11-24 19:32:18 +01:00
Yannick Welsch ff9c33b6c8 Use ObjectParser to parse AllocationID 2015-11-24 12:13:54 +01:00
Britta Weber 556d80501c aggs: fix significant terms reduce for long terms
Significant terms were not reduced correctly if they were long terms.
Also, clean up the bwc test a little. Upgrades are not needed.

related to #13522
2015-11-24 11:04:48 +01:00
Adrien Grand 5f33fbdb75 Register field mappers at the node level.
This moves the registration of field mappers from the index level to the node
level and also ensures that mappers coming from plugins are treated no
differently from core mappers.
2015-11-24 08:59:37 +01:00
Jason Tedor b31f2850c0 Do not be lenient when parsing CIDRs
This commit fixes some leniency in the parsing of CIDRs. The leniency
that existed before includes not validating the octet values nor
validating the network mask; in some cases issues with these values were
silently ignored. Parsing is now done according to the guidelines in RFC
4632, page 6.

Closes #14862
2015-11-23 20:46:23 -05:00
Michael McCandless e13b0d4bde upgrade lucene to 5.4.0-snapshot-1715952 2015-11-23 17:13:49 -05:00
Jason Tedor dcfdc0dbf3 Remove leftover TODOs in o.e.c.r.RoutingTable 2015-11-23 12:57:06 -05:00
Yannick Welsch 3dcd35d1c8 Fix issue where shard state metadata is written while containing no allocation id
Such a write can happen when upgrading shard state metadata using the MultiDataPathUpgrader

Relates to #14831
2015-11-23 18:14:02 +01:00
Daniel Mitterdorfer b804a0c4a6 Improve stability of UpdateIT
With this commit, we reduce the amount of work
that UpdateIT does and add progress logging.

Closes #14877
2015-11-23 17:54:57 +01:00
Yannick Welsch fccad13029 Persist allocation ID with shard state metadata on nodes
Closes #14831
2015-11-23 17:35:33 +01:00
Isabel Drost-Fromm 37d7629146 Merge pull request #14249 from MaineC/enhancement/8964
Switch query parsers to use ParseField
2015-11-23 14:16:40 +01:00
Boaz Leskes c1264b926f Improve trace logging in TransportReplicationAction and error reporting at RecoveryWhileUnderLoadIT
Things that helped me traced down an issue.

Closes #14931
2015-11-23 13:52:32 +01:00
Boaz Leskes 4a6f3c7840 Make sure the remaining delay of unassigned shard is updated with every reroute
For example: if a node left the cluster and an async store fetch was triggered. In that time no shard is marked as delayed (and strictly speaking it's not yet delayed). This caused test for shard delays post node left to fail. see : http://build-us-00.elastic.co/job/es_core_master_windows-2012-r2/2074/testReport/

 To fix this, the delay update is now done by the Allocation Service, based of a fixed time stamp that is determined at the beginning of the reroute.

 Also, this commit fixes a bug where unassigned info instances were reused across shard routings, causing calculated delays to be leaked.

Closes #14890
2015-11-23 13:16:15 +01:00
Isabel Drost-Fromm 38e1ad5596 Switches from manual field parsing to ParseField
This switches query parsing from manual field parsing to using ParseField.

Also adds unit tests for each query that check original json can be parsed
into query builders.

Relates to #8964
2015-11-23 11:37:46 +01:00
Christoph Büscher e25f7042b9 Making CircleBuilder writable and adding equals/hashcode 2015-11-23 11:35:44 +01:00
Christoph Büscher 5ca9f3ff8f Geo: Make ShapeBuilders implement Writable
We recently refactored the queries to make them parsable on the
coordinating note and adding serialization and equals/hashCode
capability to them. So far ShapeBuilders nested inside queries
were still transported as a byte array that needs to be parsed
later on the shard receiving the query. To be able to also
serialize geo shapes this way, we also need to make all the
implementations of ShapeBuilder implement Writable.

This PR adds this to PointBuilder and also adds tests for
serialization, equality and hashCode.
2015-11-23 11:24:43 +01:00
Boaz Leskes 6e2e91cf2a Set an newly created IndexShard's ShardRouting before exposing it to operations
The work for #10708 requires tighter integration with the current shard routing of a shard. As such, we need to make sure it is set before the IndexService exposes the shard to external operations.

Closes #14918
2015-11-23 11:22:40 +01:00
Dominik Stadler 72cab4e9b6 Map log-level 'trace' to JDK-Level 'FINEST' to make it possible to actually enable the trace-log via JdkESLogger.setLevel() 2015-11-23 09:49:29 +01:00
Martijn van Groningen 48771f1a76 field stats: Added `min_value_as_string` and `max_value_as_string` response elements for all number based fields. The existing `min_value` and `max_value` will return the values as numbers instead.
Closes #14404
2015-11-23 08:48:28 +01:00
Martijn van Groningen 4b94b50eb1 field stats: Index constraints should remove indices in the response if the field to evaluate s empty
Index constraints should remove indices in the response if the field to evaluate if empty. Index constraints can't work with that and it is the same as if the field doesn't match.
2015-11-23 08:35:15 +01:00
Jason Tedor 5e2eb7d0de Clarify three possible outcomes in CacheTests#testCachePollution 2015-11-22 19:58:04 -05:00
Jason Tedor 14a635fe4b Use correct equality in CacheTests#testReplaceRecomputesSize.Value 2015-11-22 19:56:27 -05:00
Ryan Ernst b8e462b934 Build: Change project attachment into special extra-plugins dir
Currently we use the "gradle project attachment plugin" to support
building elasticsearch as part of another project. However, this plugin
has a number of issues, a large part of which is requiring consistent
use of the projectsPrefix.

This change removes projectsPrefix, and adds support for a special
extra-plugins directory in the root of elasticsearch. Any projects
checked out within this directory will be automatically added to
elasticsearch.
2015-11-22 08:44:33 -08:00
Robert Muir 5337686d8d Merge pull request #14914 from rmuir/lock_down_system_property_writes
Ban write access to system properties
2015-11-22 11:28:55 -05:00
Robert Muir 9e41d5a738 fix typo in comment 2015-11-21 23:13:05 -05:00
Robert Muir 30529c008d Ban write access to system properties
* Forbid System.setProperties & co in forbidden APIs.
* Ban property write access at runtime with security manager.

Plugins that need to modify system properties will need to request permission in their plugin-security.policy
2015-11-21 22:33:06 -05:00
Jason Tedor 5f5ffb7871 Remove unused imports from o.a.c.s.AbstractClient 2015-11-21 22:08:18 -05:00
Christoph Büscher be9dd035e2 Merge pull request #14887 from cbuescher/merge-base-shapebuilders
Merging BaseLineString and BasePolygonBuilder with subclass
2015-11-21 12:52:31 +01:00
Camilo Díaz Repka d683e10172 Return a better exception message when regexp query is used on a numeric field 2015-11-21 00:21:24 -03:00
Jason Tedor 22187113b6 Merge pull request #14909 from jasontedor/release-after-successful-acquire
Do not release unacquired semaphore
2015-11-20 18:20:22 -05:00
Jason Tedor d541be462b Do not release unacquired semaphore
This commit adds an acquired flag to BulkProcessor#execute that is set
only after successful acquisition of a permit on the semaphore
there. This flag is used to ensure that we do not release a permit on
the semaphore when we did not obtain a permit on the semaphore.

Closes #14908
2015-11-20 18:03:48 -05:00
Ryan Ernst 88ee2e23a6 Merge branch 'master' into javadocs 2015-11-20 12:20:59 -08:00
markharwood c3a50d7ca2 Search: shift SearchTimeoutTests into core tests minus the Groovy dependency and renamed with IT test suffix 2015-11-20 18:50:17 +00:00
markharwood 5a5f05a0e9 Aggregations: Shift significant terms score test back into core without the Groovy dependency 2015-11-20 18:32:04 +00:00
Colin Goodheart-Smithe 5a18f740ba Aggregations: Moves SumTests out of lang-groovy and back into core
Renames SumTests to SumIT and puts it in the same package as the other aggregation tests. Also updates the tests to not require Groovy
2015-11-20 12:42:08 +00:00
Daniel Mitterdorfer c95c142829 Merge remote-tracking branch 'danielmitterdorfer/fix/ValueCountTests' 2015-11-20 13:21:15 +01:00
Daniel Mitterdorfer 7d6b0fc481 Move ValueCountTests to core
This script moves ValueCountTests from plugins back to core
by using a mock script engine instead of Groovy.
2015-11-20 13:19:55 +01:00
Adrien Grand 4602b8177b Move AvgTests back to core.
This makes AvgTests use a mock plugin engine. I also removed the
textScriptExplicit* methods for the base class since they only make sense for
a groovy script, not a mock script.
2015-11-20 11:58:38 +01:00
Christoph Büscher a2dca2f6cb Geo: Merging BaseLineString and BasePolygonBuilder with subclass
After the removal of some internal shape builders in #14482 the
BaseLineStringBuilder has only one implementation, the LineStringBuilder.
Same for the BasePolygonBuilder. This PR removes the abstract classes
and merges them with their concrete implementation to simplify the
inheritance hierarchy.
2015-11-20 11:37:55 +01:00
William Bowling 18c9eba40a Allow CIDR notation in a query string query
Close #14773
2015-11-20 11:17:56 +01:00
Boaz Leskes 6d9e82311a Don't delete temp recovered checkpoint file if it was renamed
Closes #14872
2015-11-20 10:44:40 +01:00
Boaz Leskes b001e20a83 Disable RecoveryWhileUnderLoadIT for now
Working on a fix http://build-us-00.elastic.co/job/es_core_master_windows-2012-r2/2074/testReport/
2015-11-20 10:38:48 +01:00
Christoph Büscher aa1507d349 Merge pull request #14482 from cbuescher/remove-internal-linestring-builder
Remove InternalLineStringBuilder and InternalPolygonBuilder
2015-11-20 00:06:04 +01:00
Michael McCandless 201318a6f5 include root-cause exception when we fail to change shard's index buffer 2015-11-19 14:50:41 -05:00
Ryan Ernst c3cb1fd08c Merge branch 'master' into javadocs 2015-11-19 10:43:43 -08:00
Yannick Welsch 0a902bfe1c Fix concurrency issue when accessing field UnassignedInfo.lastComputedLeftDelayNanos from TransportClusterHealthAction 2015-11-19 13:12:44 +01:00
Christoph Büscher b5b3ff5eb0 Also removing Ring and replacing it by LineStringBuilder
The Ring subclass is just a LineStringBuilder that has an additional
close() method and keeps a reference to a parent shape builder so
builders can be chained. This PR removes it and replaces it by
using LineStringBuilder instead. The close() method is moved there
and tests are adapted.
2015-11-19 11:54:02 +01:00
Christoph Büscher 39f45496f2 Also removing InternalPolygonBuilder 2015-11-19 11:51:17 +01:00
Christoph Büscher 12941bd4b7 Geo: remove InternalLineStringBuilder
This is a first step in reducing the number of ShapeBuilders since
before we start making the remaining implement Writable for the
search request refactoring. This shape builder seems to have been
only used in tests, and those tests didn't do much to begin with,
so this removed them.

Relates to #14416
2015-11-19 11:51:17 +01:00
Isabel Drost-Fromm b71e72f246 Adds exception objects to log messages.
Also forwards exception objects through failure listeners eliminates another 17
calls to getMessage().

Relates to #10021
2015-11-19 11:37:03 +01:00
Yannick Welsch 6a2fa73fb5 Merge pull request #14808 from ywelsch/refactor/delayed-allocation
Simplify delayed shard allocation
2015-11-19 09:58:52 +01:00
Yannick Welsch 2084df825f Simplify delayed shard allocation
- moves calculation of the delay to a single place (ReplicaShardAllocator)
- reduces coupling between GatewayAllocator and RoutingService
- in master failover situations, elapsed delay time is forgotten

Closes #14808
2015-11-19 09:53:07 +01:00
Xu Zhang 2e6d72de27 Catch exception when reading corrupted snapshot.
Single corrupted snapshot file shouldn't prevent listing all other
snapshot in repository.
2015-11-18 21:43:46 -08:00
Nicholas Knize 0d349854d3 Add CONTAINS relation to geo_shape query
At the time of geo_shape query conception, CONTAINS was not yet a supported spatial operation in Lucene. Since it is now available this commit adds ShapeRelation.CONTAINS to GeoShapeQuery. Randomized testing is included and documentation is updated.
2015-11-18 14:15:35 -06:00
Nicholas Knize 9ed77afe1d Fix assertion precision for legacy GeoDistanceRangeQuery tests
This bug existed for GeoDistanceRangeQuery exclusion limits only (e.g., min/max included == false).

closes #14838
2015-11-18 14:09:27 -06:00
Jason Tedor 756f7876a9 Set ReplicationRequest.internalShardId to null
This commit sets ReplicationRequest.internalShardId to null when the
stream indicates that no ShardId is present in the stream.

Additionally, the use of StreamOutput#writeOptionalStreamable is
changed to be explicit for clarity since the use of
StreamInput#readOptionalStreamable is not possible due to the
no-argument constructor on ShardId being private.
2015-11-18 14:46:45 -05:00
Martijn van Groningen 1661ce5b8c field stats: Fix NPE when index constraint has been set on mapped field but the index is empty 2015-11-18 18:27:29 +01:00
Daniel Mitterdorfer a7bd9e8275 Merge remote-tracking branch 'danielmitterdorfer/fix/14798' 2015-11-18 17:13:06 +01:00
Daniel Mitterdorfer 96724e198a Restore thread interrupt flag after an InterruptedException
This commit replaces all occurrences of Thread.interrupted() with
Thread.currentThread().interrupt(). While the former checks and clears the current
thread's interrupt flag the latter sets it, which is actually intended.

Closes #14798
2015-11-18 17:11:53 +01:00
Christoph Büscher 3e483b5626 Merge pull request #14587 from cbuescher/move-datehistogramtests-core
Move DateHistogramTests back to core module
2015-11-18 16:35:59 +01:00
Jason Tedor d6c40fea20 Fix typo in TransportDeleteAction#shardOperationOnPrimary 2015-11-18 10:30:43 -05:00
Martijn van Groningen 8a454dae33 field stats: Added a `format` option to index constraint that allows to specify date index constraint values in a different format then the for specified in the mapping.
Closes #14804
2015-11-18 14:19:07 +01:00
Jason Tedor 3b2bba64ba Add unit test for LinkedHashMap serialization
This commit adds a unit test for LinkedHashMap serialization that tests
that the method of serialization writes the entries in the LinkedHashMap
in iteration order and that the reconstructed LinkedHashMap preserves
that order. This test is randomized and tests iteration order is
preserved whether the LinkedHashMap is ordered by insertion order or
access order.

Closes #14743
2015-11-17 18:52:28 -05:00
Jason Tedor c1ee90356a Add timeout mechanism for sending shard failures
This commit adds a timeout mechanism for sending shard failures. The
requesting thread can attach a listener to the timeout event so that
handling it is part of the event chain.

Relates #14252
2015-11-17 15:00:27 -05:00
Jason Tedor e08e137aeb Add license header to o.e.c.i.s.StreamTests.java 2015-11-17 14:02:12 -05:00
Jason Tedor 6872d545ac Add system CPU percent to OS stats
This commit adds the system CPU percent reflecting the recent CPU usage
for the whole system.
2015-11-17 13:48:46 -05:00
Jason Tedor 043319c482 Merge pull request #14806 from jasontedor/read-optional-streamable
Use Supplier for StreamInput#readOptionalStreamable
2015-11-17 13:41:35 -05:00
Jason Tedor 1cd4a29b6f Use Supplier for StreamInput#readOptionalStreamable
This commit changes the signature of StreamInput#readOptionalStreamable
to accept a Supplier to create new streamables rather than requiring
callers to construct new instances. This has the advantage of avoiding
an allocation in cases when the stream indicates the resulting
streamable is null
2015-11-17 13:40:22 -05:00
Nik Everett 0119caa4a6 Make Build work without git
If you build elasticsearch without a git repository it was creating a null
shortHash which was causing Elasticsearch not to be able to form transport
connections.

Closes #14748
2015-11-17 13:25:09 -05:00
Jason Tedor d2ffcba890 Merge pull request #14780 from jasontedor/variable-length-long
Add variable-length long encoding
2015-11-17 09:04:13 -06:00
Jason Tedor 9a11e75cf5 Add variable-length long encoding
This commit adds a method of encoding longs using a variable-length
representation. This encoding is an implementation of the zig-zag
encoding from protocol buffers. Numbers that have a small absolute value
will use a small number of bytes. This is achieved by zig-zagging
through the space of longs in order of increasing absolute value (0, -1,
1, -2, 2, …, Long.MAX_VALUE, Long.MIN_VALUE) -> (0, 1, 2, 3, 4, …, -2,
-1). The resulting values are then encoded as if they represent unsigned
numbers.
2015-11-17 09:01:54 -06:00
Daniel Mitterdorfer 2522645123 Fix race in ClusterStatsIT
ClusterStatsIT#testClusterStatus() contained a race where the
test cluster might still be initializing while test already checks
for a green health status.

With this commit the test waits until the cluster status changed and
checks health afterwards.

Checked with @bleskes.
2015-11-17 13:22:28 +01:00
Christoph Büscher d6a756fbe2 Merge pull request #14529 from cbuescher/introduce-shapebuilders
Geo: Moving static factory methods to ShapeBuilders
2015-11-17 11:08:35 +01:00
Ryan Ernst 4975422a35 Merge pull request #14783 from rjernst/type_doc_values
Make _type use doc values
2015-11-16 13:09:49 -08:00
Michael McCandless 01f12fc8d5 Merge pull request #14784 from mikemccand/upgrade_lucene_1714615
Upgrade Lucene to 5.4.0-snapshot-1714615
2015-11-16 16:07:46 -05:00
Nicholas Knize 7f5da1d6a3 Fix reproducible GeoDistanceRangeQueryTests.testToQuery error
This issue occurs if the center latitude of the GeoPointDistance query is set to one of the poles. Since this issue is set to be fixed in LUCENE-6897 this commit temporarily limits the random latitudinal location to not include the poles.
2015-11-16 14:45:02 -06:00
Michael McCandless e4cc94c11e remove confusing private default constant 2015-11-16 15:17:16 -05:00
Michael McCandless c3f7638f81 add comment about confusing constant value 2015-11-16 14:55:11 -05:00
Michael McCandless a0bf253d16 upgrade lucene 5.4 snapshot 2015-11-16 14:38:05 -05:00
Ryan Ernst baa3fd814e Mappings: Make _type use doc values
_type should have got doc values with the change to default doc values.
However, due to how metadata fields have separate builders and special
constructors, it was not picking it up. This change updates the field
type for _type to have doc values.

closes #14781
2015-11-16 11:32:52 -08:00
Simon Willnauer 1bdf29e263 Translog recovery can repeatedly fail if we run out of disk
If we run out of disk while recoverying the transaction log
we repeatedly fail since we expect the latest tranlog to be uncommitted.
This change adds 2 safety levels:

 * uncommitted checkpoints are first written to a temp file and then atomically
   renamed into a committed (recovered) checkpoint
 * if the latest uncommitted checkpoints generation is already recovered it has to be
   identical, if not the recovery fails

This allows to fail in between recovering the latest uncommitted checkpoint and moving
the checkpoint generation to N+1 which can for instance happen in a situation where
we can run out of disk. If we run out of disk while recovering the uncommitted checkpoint
either the temp file writing or the atomic rename will fail such that we never have a
half written or corrupted recovered checkpoint.

Close #14695
2015-11-16 18:47:19 +01:00
Christoph Büscher 0623684521 Geo: Moving static factory methods to ShapeBuilders
Currently the abstract ShapeBuilder class serves too many different
purposes, making it hard to refactor and maintain the code. In order
to reduce the size and responsibilities, this PR moved all the
static factory methods used as a shortcut to create new shape builders
out to a new ShapeBuilders class, similar to how QueryBuilders is
used already.
2015-11-16 18:01:23 +01:00
Tanguy Leroux d538f0dcf5 Add FilterPath implementation that uses Jackson 2.6 streaming filtering feature 2015-11-16 17:08:06 +01:00
Tanguy Leroux 8b961bc0e0 Remove unused AbstractXContentGenerator 2015-11-16 17:08:06 +01:00
Christoph Büscher a34b555fff Merge pull request #14775 from cbuescher/fix/14746
Add unique id to query names to avoid naming conflicts
2015-11-16 15:34:09 +01:00
Daniel Mitterdorfer 267f726a7f Merge remote-tracking branch 'danielmitterdorfer/feature/11657' 2015-11-16 15:15:41 +01:00
Daniel Mitterdorfer e31d66d137 Log cluster health status changes
With this commit the cluster health status changes are logged
on INFO level. The change is only logged on master and actively
triggered in AllocationService in order to minimize the impact of
constantly reevaluating ClusterState in a ClusterStateListener
although we know that no health-relevant change happened.

Closes #11657
2015-11-16 15:14:43 +01:00
Christoph Büscher 0a0bca6f5b Tests: Add unique id to query names to avoid naming conflicts
In AbstractQueryTestCase we randomly add the `_name` property to
some of the queries. While this generally works, there are exceptional
cases where we assign the same name to two queries in the setup which
leads to test failures later. This PR adds an increasing counter value
to the base tests that gets appended to all random query names to
avoid this name clashes.
2015-11-16 15:08:25 +01:00
Boaz Leskes fa55a940f7 Test: move SearchQueryIT back to core
There is no groovy dependency there (it's now in lang groovy)

Closes #14770
2015-11-16 14:43:36 +01:00
Simon Willnauer 044b3d59cd Take ingored unallocated shards into account when makeing allocation decision
ClusterRebalanceAllocationDecider did not take unassigned shards into account
that are temporarily marked as ingored. This can cause unexpected behavior
when gateway allocator is still fetching shards or has marked shareds as ignored
since their quorum is not met yet.

Closes #14670
Closes #14678
2015-11-16 14:25:02 +01:00
Yannick Welsch fb10e59578 Merge pull request #14765 from ywelsch/fix/delayed-allocation-race
Fix calculation of next delay for delayed shard allocation
2015-11-16 14:07:35 +01:00
Yannick Welsch 2206030fe6 Fix calculation of next delay for delayed shard allocation
Currently the next delay is calculated based on System.currentTimeMillis() but the actual shards to delay based on the last time the GatewayAllocator tried to assign/delay the shard.

This introduces an inconsistency for the case where shards should have been delay-allocated between the GatewayAllocator-based timestamp and System.currentTimeMillis().

Closes #14765
2015-11-16 14:06:34 +01:00
Christoph Büscher b377d81e1c Adding mock script engine for value script tests 2015-11-16 13:00:43 +01:00
Christoph Büscher 5108e6fd1f Adapting test by using mock script engine 2015-11-16 11:58:28 +01:00
Christoph Büscher e3f9824931 Tests: Move DateHistogramTests back to core module
DateHistogramTests had some dependency on groovy scripts and
were moved to the lang-groovy module. This PR moves it back
and replaces use of groovy scripts by a mock script engine.
Removing three test cases that were testing doing some date
manipulation using script, since these are more groovy script
tests than testing the DateHistogram aggregation.
2015-11-16 11:58:28 +01:00
Martijn van Groningen 293b2f6345 field data: Don't cache top level field data for fields that don't exist.
Just return an empty field data instance, like we do currently the same for segment level field data.
2015-11-16 06:27:48 +01:00
Boaz Leskes eff82fdbcd Tests: update the disco node id seed on node restart
To make sure the new node's id is changed.

Closes #14675
2015-11-15 21:49:54 +01:00
Jason Tedor 89172dfb02 Remove unused import in RecoverySourceHandler 2015-11-15 10:14:08 -05:00
Ryan Ernst 4b17492456 Build: Add javadocs jars
This change adds javadoc jars to core, test-framework and plugins. There
were a couple issues which javadoc found, but doclint did not already
find.
2015-11-15 01:44:42 -08:00
Jason Tedor 3acd7800cb Transport options should be immutable
This commit changes TransportRequestOptions and TransportResponseOptions
to be immutable. This is to address an issue where the empty options
were being mutated permanently altering their state. Making these
objects immutable is just good, clean coding.
2015-11-14 18:21:40 -05:00
Nicholas Knize 42470c252c Correcting maxRadius units in GeoDistanceRangeQueryTest 2015-11-13 11:18:43 -06:00
Nicholas Knize 9b3920b52c Removing pole restriction on GeoDistanceRangeQueryTests
This commit corrects the maxRadialDistance computation for points at the poles to half the distance of the meridian.
2015-11-13 09:32:31 -06:00
Colin Goodheart-Smithe 12bb1b79f6 Aggregations: Pass extended bounds into HistogramAggregator when creating an unmapped aggregator
This fixes an issue where if the field for the aggregation was unmapped the extended bounds would get dropped and the resulting buckets would not cover the extended bounds requested.

Closes #14735
2015-11-13 14:49:15 +00:00
Boaz Leskes e848dcfbca java docs fix 2015-11-13 15:05:38 +01:00
Boaz Leskes ac0da91bf7 Extend usage of IndexSetting class
I decided to leave external listeners (used by plugins) alone, for now.

Closes #14731
2015-11-13 14:30:23 +01:00
Yannick Welsch 9673ddc5d0 Throw MasterNotDiscoveredException whenever retry logic of TransportMasterNodeAction times out
Closes #14737
2015-11-13 14:24:00 +01:00
Colin Goodheart-Smithe be64bfb135 Revert "Aggregations: Pass extended bounds into HistogramAggregator when creating an unmapped aggregator"
This reverts commit e1bf9798bf.
2015-11-13 12:48:51 +00:00
Colin Goodheart-Smithe e1bf9798bf Aggregations: Pass extended bounds into HistogramAggregator when creating an unmapped aggregator
This fixes an issue where if the field for the aggregation was unmapped the extended bounds would get dropped and the resulting buckets would not cover the extended bounds requested.

Closes #14735
2015-11-13 12:33:06 +00:00
Jason Tedor a9ab35a487 Add sync_id to cat shards API
This commit adds the ability to get the sync_id from the cat shards API.

Closes #14705
2015-11-13 05:13:08 -05:00
Robert Muir 776bb288b5 fix gradle check under jigsaw
closes #14726

Squashed commit of the following:

commit 5b591e98570e3fa481b2816a44063b98bff36ddf
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Nov 13 00:54:08 2015 -0500

    add assumption for self-signing in PluginManagerTests

commit ed11e5371b6f71591dc41c6f60d033502cfcf029
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Nov 13 00:20:59 2015 -0500

    show error output from integ test startup

commit d8b187a10e95d89a0e775333dcbe1aaa903fb376
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Nov 12 22:14:11 2015 -0500

    fix gradle check under jigsaw
2015-11-13 00:57:12 -05:00
Nicholas Knize dc77815744 Enable GeoPointV2 with backward compatibility testing
This commit removes all noreleases and cuts over to Lucene 5.4 GeoPointField type. Included are randomized testing updates to unit and integration test suites for ensuring full backward compatability with existing geo_point indexes.
2015-11-12 23:01:51 -06:00
Robert Muir 720ebe347d Handle system policy correctly.
Just suck in the system policy, so its compatible with any version of java.
It means it also respects configuration (e.g. for monitoring agents)

Closes #14704
2015-11-12 18:08:13 -05:00
xuzha f33184329f Add extra validation into `cluster/stats`
The cluster health and cluster stats disagree on the status.
Add a extra validation step in `cluster/stats`.

closes #7390
2015-11-12 10:46:18 -08:00
Yannick Welsch 4729386347 Merge pull request #14494 from ywelsch/fix/delayed-allocation-reroute
Delayed allocation can miss a reroute
2015-11-12 17:58:08 +01:00
Yannick Welsch c32de1f72b [TEST] Use TransportService/TransportAddress instead of DiscoveryNode for disruption rules
The disruption rules are changed to work on all transport addresses that are bound by a node (not only publish address).
This is important as UnicastZenPing creates fake DiscoveryNode instances which match one of the bound addresses and not necessarily the publish address.

Closes #14625
Closes #14653
2015-11-12 17:20:07 +01:00
Jason Tedor fac472f90c Remove unused imports in o.e.c.a.s.ShardStateActionTests 2015-11-12 09:08:27 -05:00
Yannick Welsch 0220e45e2f Fix missing reroute in case of multiple delayed shards
After a delayed reroute of a shard, RoutingService misses to schedule a new delayed reroute of other delayed shards.

Closes #14494
Closes #14010
Closes #14445
2015-11-12 10:57:40 +01:00
Jason Tedor c2aec53b46 Move field to local variable 2015-11-11 12:50:21 -05:00
Jason Tedor 3a92012938 Use String#equals instead of reference equality 2015-11-11 12:50:21 -05:00
Jason Tedor e4e68c0463 Sanity check exception message in tests 2015-11-11 12:50:21 -05:00
Jason Tedor 06c07ce275 Remove unnecessary generic type parameters 2015-11-11 12:50:21 -05:00
Robert Muir 914cee213e Upgrade to aws 1.10.33
Security issues have been fixed. This removes our hacks.
2015-11-10 22:51:33 -05:00
Ryan Ernst 4b5f87cb7d Build: Remove transitive dependencies
Transitive dependencies can be confusing and hard to deal with when
conflicts arise between them. This change removes transitive
dependencies from elasticsearch, and forces any dependency conflicts to
be resolved manually, instead of automatically by gradle.

closes #14627
2015-11-10 15:01:41 -08:00
Tanguy Leroux 4063354dbe Fix Delete-by-Query with Shield
closes #14527
2015-11-10 20:39:38 +01:00
Lee Hinman bf3e32e8be Add breaker name to logging package
This allows different circuit breakers to have different logging levels.
It's useful when diagnosing problems (say for instance with the
fielddata breaker) and not seeing the enormous amount of logging from
the request breaker.

The log messages use the breaker name for logging, so example logging
will look like:

```
[2015-11-10 09:51:52,993][TRACE][indices.breaker.fielddata] [fielddata] Adding [27b][body] to used bytes [new used: [27b], limit: 623326003 [594.4mb], estimate: 27 [27b]]
[2015-11-10 09:51:53,000][TRACE][indices.breaker.fielddata] [fielddata] Adjusted breaker by [453] bytes, now [480]
[2015-11-10 09:51:53,016][TRACE][indices.breaker.request  ] [request] Adjusted breaker by [16440] bytes, now [16440]
[2015-11-10 09:51:53,018][TRACE][indices.breaker.request  ] [request] Adjusted breaker by [-16440] bytes, now [0]
```
2015-11-10 10:02:34 -07:00
Martijn van Groningen 973c094a5d Merge pull request #14654 from s1monw/fix_searcher_wrapper_interface
Fix IndexSearcherWrapper interface to not depend on the EngineConfig
2015-11-10 23:09:32 +07:00
Ryan Ernst 7bb4e10353 Merge pull request #14645 from rjernst/version_cleanup
Consolidate dependencies specified in multiple places
2015-11-10 07:49:10 -08:00
David Pilato a61fcfd8ca Remove AbstractLegacyBlobContainer
`AbstractLegacyBlobContainer` was kept for historical reasons (see #13434).
We can migrate Azure and S3 repositories to use the new methods added in #13434 so we can remove `AbstractLegacyBlobContainer` class.
2015-11-10 16:04:44 +01:00
Simon Willnauer 049430621d Fix IndexSearcherWrapper interface to not depend on the EngineConfig
We only passed the engine config since we had no chance to get the cache
etc. from the IndexSearcher. Now that we have these getters we can just pass the
searcher instead.
2015-11-10 14:53:32 +01:00
Simon Willnauer 17913c56fb Merge pull request #14591 from s1monw/issues/14387
Only allow rebalance operations to run if all shard store data is available
2015-11-10 12:06:31 +01:00
Simon Willnauer 84033f3f2e Use syntactic sugar for starting data only nodes 2015-11-10 12:00:29 +01:00
Ryan Ernst 7a6155e12f Build: Consolidate dependencies specified in multiple places
Some dependencies must be specified in a couple places in the build.
e.g. randomized runner is specified both in buildSrc (for the gradle
wrapper plugin), as well as in the test-framework.

This change creates buildSrc/versions.properties which acts similar to
the set of shared version properties we used to have in the maven parent
pom.
2015-11-09 23:59:56 -08:00
Nicholas Knize 3037970c33 Update GeoPoint FieldData for GeoPointV2
This commit adds an abstraction layer to GeoPoint FieldData for supporting new Lucene 5.4 GeoPointField and backwards compatibility.
2015-11-09 15:39:24 -06:00
Adrien Grand d0853e9253 Merge pull request #14619 from jpountz/upgrade/lucene-5.4.0-snapshot-1712973
Upgrade to lucene-5.4.0-snapshot-1712973.
2015-11-09 22:38:44 +01:00
Ryan Ernst 7034feeceb Merge pull request #14451 from andrejserafim/run
Add gradle run command to replace run.bat and run.sh
2015-11-09 11:15:53 -08:00
Lee Hinman 25f8db9aa9 Merge remote-tracking branch 'dakrone/node-limit-decider' 2015-11-09 11:03:19 -07:00
Lee Hinman 145374b762 Add cluster-wide setting for total shard limit
This adds the `cluster.routing.allocation.total_shards_per_node`
setting, which limits the total number of shards across all indices on
each node. It defaults to -1 and can be dynamically configured.

Resolves #14456
2015-11-09 11:03:07 -07:00
Nicholas Knize 36e294a03e Fix GeoDistanceRangeQueryTest tolerance
GeoDistanceRangeQueryTest was using an inconsistent tolerance when validating from=0 with include_lower/upper set to false. This commit changes the assertion such that validation is consistent with builder logic.
2015-11-09 11:03:22 -06:00
Yannick Welsch 33ca2ec597 Merge pull request #14222 from ywelsch/fix/refactor-transport-master-node-action
Refactor retry logic for TransportMasterNodeAction
2015-11-09 17:59:16 +01:00
Yannick Welsch 0575744a39 Refactor retry logic for TransportMasterNodeAction
- Same as for TransportInstanceSingleOperationAction and TransportReplicationAction, onClusterServiceClose consistently throws a NodeClosedException now.
- Added retry logic if master could not publish cluster state or stepped down before publishing (ZenDiscovery). The test IndexingMasterFailoverIT shows the issue.
- Simplified retry logic by moving bits from different places into shared retry method.
  - Removed boolean flag retrying that aborted retrying after a single master node change (now we retry until timeout).
  - Two existing predicates that deal with master node changes unified in a single predicate masterNodeChangedPredicate

Closes #14222
2015-11-09 17:56:18 +01:00
Adrien Grand d6d7af0a6c Upgrade to lucene-5.4.0-snapshot-1712973. 2015-11-09 15:53:27 +01:00
javanna 43b729f34c [TEST] test query and search source parsing from different XContent types
We used to test only json parsing as we relied on QueryBuilder#toString which uses the json format. This commit makes sure that we now output the randomly generated queries using a random format, and that we are always able to parse them correctly.

This revealed a couple of issues with binary objects that haven't been migrated yet to be structured Writeable objects. We used to keep them in the format they were sent while parsing, which led to problems when printing them out as we expected them to always be in json format. Also we can't compare different BytesReference objects that hold the same content but in different formats (unless we want to parse them as part of equal and hashcode, doesn't seem like a good idea) and verify that we have parsed the right objects if they can be different formats. The fix is to always keep binary objects in json format. Best fix would be not to have binary objects, which we'll get to once we are done with the search refactoring.

Closes #14415
2015-11-09 15:09:27 +01:00
javanna 10ddd691a3 Internal: move to lucene BoostQuery
Latest version of lucene deprecated Query#setBoost and Query#getBoost which made queries effectively immutable. Those methods need to be replaced with `BoostQuery` that wraps any query that needs boosting.

This commit replaces usages of setBoost with BoostQuery and adds it to forbidden-apis for prod code.

Usages of `getBoost` are only partially removed, as some will have to stay for backwards compatibility.

Closes #14264
2015-11-09 15:01:35 +01:00
Simon Willnauer 7b5e323ec0 add IT for #14387 2015-11-09 15:00:56 +01:00
Robert Muir 102e25413d Merge pull request #14605 from rmuir/migrate-mapper-attachments
Migrate mapper attachments plugin to main repository
2015-11-09 08:55:29 -05:00
Simon Willnauer 219fe8f30f apply @bleskes review changes 2015-11-09 12:58:11 +01:00
Daniel Mitterdorfer 1c5b0b3c5b Merge branch 'pr/14611' 2015-11-09 12:08:56 +01:00
Adrien Grand f400f58132 Merge pull request #14472 from jpountz/fix/simplify_xcontent_detection
Simplify XContent detection.
2015-11-09 12:02:39 +01:00