Commit Graph

2651 Commits

Author SHA1 Message Date
Isabel Drost-Fromm 38e1ad5596 Switches from manual field parsing to ParseField
This switches query parsing from manual field parsing to using ParseField.

Also adds unit tests for each query that check original json can be parsed
into query builders.

Relates to #8964
2015-11-23 11:37:46 +01:00
Christoph Büscher e25f7042b9 Making CircleBuilder writable and adding equals/hashcode 2015-11-23 11:35:44 +01:00
Christoph Büscher 5ca9f3ff8f Geo: Make ShapeBuilders implement Writable
We recently refactored the queries to make them parsable on the
coordinating note and adding serialization and equals/hashCode
capability to them. So far ShapeBuilders nested inside queries
were still transported as a byte array that needs to be parsed
later on the shard receiving the query. To be able to also
serialize geo shapes this way, we also need to make all the
implementations of ShapeBuilder implement Writable.

This PR adds this to PointBuilder and also adds tests for
serialization, equality and hashCode.
2015-11-23 11:24:43 +01:00
Boaz Leskes 6e2e91cf2a Set an newly created IndexShard's ShardRouting before exposing it to operations
The work for #10708 requires tighter integration with the current shard routing of a shard. As such, we need to make sure it is set before the IndexService exposes the shard to external operations.

Closes #14918
2015-11-23 11:22:40 +01:00
Dominik Stadler 72cab4e9b6 Map log-level 'trace' to JDK-Level 'FINEST' to make it possible to actually enable the trace-log via JdkESLogger.setLevel() 2015-11-23 09:49:29 +01:00
Martijn van Groningen 48771f1a76 field stats: Added `min_value_as_string` and `max_value_as_string` response elements for all number based fields. The existing `min_value` and `max_value` will return the values as numbers instead.
Closes #14404
2015-11-23 08:48:28 +01:00
Martijn van Groningen 4b94b50eb1 field stats: Index constraints should remove indices in the response if the field to evaluate s empty
Index constraints should remove indices in the response if the field to evaluate if empty. Index constraints can't work with that and it is the same as if the field doesn't match.
2015-11-23 08:35:15 +01:00
Jason Tedor 5e2eb7d0de Clarify three possible outcomes in CacheTests#testCachePollution 2015-11-22 19:58:04 -05:00
Jason Tedor 14a635fe4b Use correct equality in CacheTests#testReplaceRecomputesSize.Value 2015-11-22 19:56:27 -05:00
Ryan Ernst b8e462b934 Build: Change project attachment into special extra-plugins dir
Currently we use the "gradle project attachment plugin" to support
building elasticsearch as part of another project. However, this plugin
has a number of issues, a large part of which is requiring consistent
use of the projectsPrefix.

This change removes projectsPrefix, and adds support for a special
extra-plugins directory in the root of elasticsearch. Any projects
checked out within this directory will be automatically added to
elasticsearch.
2015-11-22 08:44:33 -08:00
Robert Muir 5337686d8d Merge pull request #14914 from rmuir/lock_down_system_property_writes
Ban write access to system properties
2015-11-22 11:28:55 -05:00
Robert Muir 9e41d5a738 fix typo in comment 2015-11-21 23:13:05 -05:00
Robert Muir 30529c008d Ban write access to system properties
* Forbid System.setProperties & co in forbidden APIs.
* Ban property write access at runtime with security manager.

Plugins that need to modify system properties will need to request permission in their plugin-security.policy
2015-11-21 22:33:06 -05:00
Jason Tedor 5f5ffb7871 Remove unused imports from o.a.c.s.AbstractClient 2015-11-21 22:08:18 -05:00
Christoph Büscher be9dd035e2 Merge pull request #14887 from cbuescher/merge-base-shapebuilders
Merging BaseLineString and BasePolygonBuilder with subclass
2015-11-21 12:52:31 +01:00
Camilo Díaz Repka d683e10172 Return a better exception message when regexp query is used on a numeric field 2015-11-21 00:21:24 -03:00
Jason Tedor 22187113b6 Merge pull request #14909 from jasontedor/release-after-successful-acquire
Do not release unacquired semaphore
2015-11-20 18:20:22 -05:00
Jason Tedor d541be462b Do not release unacquired semaphore
This commit adds an acquired flag to BulkProcessor#execute that is set
only after successful acquisition of a permit on the semaphore
there. This flag is used to ensure that we do not release a permit on
the semaphore when we did not obtain a permit on the semaphore.

Closes #14908
2015-11-20 18:03:48 -05:00
Ryan Ernst 88ee2e23a6 Merge branch 'master' into javadocs 2015-11-20 12:20:59 -08:00
markharwood c3a50d7ca2 Search: shift SearchTimeoutTests into core tests minus the Groovy dependency and renamed with IT test suffix 2015-11-20 18:50:17 +00:00
markharwood 5a5f05a0e9 Aggregations: Shift significant terms score test back into core without the Groovy dependency 2015-11-20 18:32:04 +00:00
Colin Goodheart-Smithe 5a18f740ba Aggregations: Moves SumTests out of lang-groovy and back into core
Renames SumTests to SumIT and puts it in the same package as the other aggregation tests. Also updates the tests to not require Groovy
2015-11-20 12:42:08 +00:00
Daniel Mitterdorfer c95c142829 Merge remote-tracking branch 'danielmitterdorfer/fix/ValueCountTests' 2015-11-20 13:21:15 +01:00
Daniel Mitterdorfer 7d6b0fc481 Move ValueCountTests to core
This script moves ValueCountTests from plugins back to core
by using a mock script engine instead of Groovy.
2015-11-20 13:19:55 +01:00
Adrien Grand 4602b8177b Move AvgTests back to core.
This makes AvgTests use a mock plugin engine. I also removed the
textScriptExplicit* methods for the base class since they only make sense for
a groovy script, not a mock script.
2015-11-20 11:58:38 +01:00
Christoph Büscher a2dca2f6cb Geo: Merging BaseLineString and BasePolygonBuilder with subclass
After the removal of some internal shape builders in #14482 the
BaseLineStringBuilder has only one implementation, the LineStringBuilder.
Same for the BasePolygonBuilder. This PR removes the abstract classes
and merges them with their concrete implementation to simplify the
inheritance hierarchy.
2015-11-20 11:37:55 +01:00
William Bowling 18c9eba40a Allow CIDR notation in a query string query
Close #14773
2015-11-20 11:17:56 +01:00
Boaz Leskes 6d9e82311a Don't delete temp recovered checkpoint file if it was renamed
Closes #14872
2015-11-20 10:44:40 +01:00
Boaz Leskes b001e20a83 Disable RecoveryWhileUnderLoadIT for now
Working on a fix http://build-us-00.elastic.co/job/es_core_master_windows-2012-r2/2074/testReport/
2015-11-20 10:38:48 +01:00
Christoph Büscher aa1507d349 Merge pull request #14482 from cbuescher/remove-internal-linestring-builder
Remove InternalLineStringBuilder and InternalPolygonBuilder
2015-11-20 00:06:04 +01:00
Michael McCandless 201318a6f5 include root-cause exception when we fail to change shard's index buffer 2015-11-19 14:50:41 -05:00
Ryan Ernst c3cb1fd08c Merge branch 'master' into javadocs 2015-11-19 10:43:43 -08:00
Yannick Welsch 0a902bfe1c Fix concurrency issue when accessing field UnassignedInfo.lastComputedLeftDelayNanos from TransportClusterHealthAction 2015-11-19 13:12:44 +01:00
Christoph Büscher b5b3ff5eb0 Also removing Ring and replacing it by LineStringBuilder
The Ring subclass is just a LineStringBuilder that has an additional
close() method and keeps a reference to a parent shape builder so
builders can be chained. This PR removes it and replaces it by
using LineStringBuilder instead. The close() method is moved there
and tests are adapted.
2015-11-19 11:54:02 +01:00
Christoph Büscher 39f45496f2 Also removing InternalPolygonBuilder 2015-11-19 11:51:17 +01:00
Christoph Büscher 12941bd4b7 Geo: remove InternalLineStringBuilder
This is a first step in reducing the number of ShapeBuilders since
before we start making the remaining implement Writable for the
search request refactoring. This shape builder seems to have been
only used in tests, and those tests didn't do much to begin with,
so this removed them.

Relates to #14416
2015-11-19 11:51:17 +01:00
Isabel Drost-Fromm b71e72f246 Adds exception objects to log messages.
Also forwards exception objects through failure listeners eliminates another 17
calls to getMessage().

Relates to #10021
2015-11-19 11:37:03 +01:00
Yannick Welsch 6a2fa73fb5 Merge pull request #14808 from ywelsch/refactor/delayed-allocation
Simplify delayed shard allocation
2015-11-19 09:58:52 +01:00
Yannick Welsch 2084df825f Simplify delayed shard allocation
- moves calculation of the delay to a single place (ReplicaShardAllocator)
- reduces coupling between GatewayAllocator and RoutingService
- in master failover situations, elapsed delay time is forgotten

Closes #14808
2015-11-19 09:53:07 +01:00
Xu Zhang 2e6d72de27 Catch exception when reading corrupted snapshot.
Single corrupted snapshot file shouldn't prevent listing all other
snapshot in repository.
2015-11-18 21:43:46 -08:00
Nicholas Knize 0d349854d3 Add CONTAINS relation to geo_shape query
At the time of geo_shape query conception, CONTAINS was not yet a supported spatial operation in Lucene. Since it is now available this commit adds ShapeRelation.CONTAINS to GeoShapeQuery. Randomized testing is included and documentation is updated.
2015-11-18 14:15:35 -06:00
Nicholas Knize 9ed77afe1d Fix assertion precision for legacy GeoDistanceRangeQuery tests
This bug existed for GeoDistanceRangeQuery exclusion limits only (e.g., min/max included == false).

closes #14838
2015-11-18 14:09:27 -06:00
Jason Tedor 756f7876a9 Set ReplicationRequest.internalShardId to null
This commit sets ReplicationRequest.internalShardId to null when the
stream indicates that no ShardId is present in the stream.

Additionally, the use of StreamOutput#writeOptionalStreamable is
changed to be explicit for clarity since the use of
StreamInput#readOptionalStreamable is not possible due to the
no-argument constructor on ShardId being private.
2015-11-18 14:46:45 -05:00
Martijn van Groningen 1661ce5b8c field stats: Fix NPE when index constraint has been set on mapped field but the index is empty 2015-11-18 18:27:29 +01:00
Daniel Mitterdorfer a7bd9e8275 Merge remote-tracking branch 'danielmitterdorfer/fix/14798' 2015-11-18 17:13:06 +01:00
Daniel Mitterdorfer 96724e198a Restore thread interrupt flag after an InterruptedException
This commit replaces all occurrences of Thread.interrupted() with
Thread.currentThread().interrupt(). While the former checks and clears the current
thread's interrupt flag the latter sets it, which is actually intended.

Closes #14798
2015-11-18 17:11:53 +01:00
Christoph Büscher 3e483b5626 Merge pull request #14587 from cbuescher/move-datehistogramtests-core
Move DateHistogramTests back to core module
2015-11-18 16:35:59 +01:00
Jason Tedor d6c40fea20 Fix typo in TransportDeleteAction#shardOperationOnPrimary 2015-11-18 10:30:43 -05:00
Martijn van Groningen 8a454dae33 field stats: Added a `format` option to index constraint that allows to specify date index constraint values in a different format then the for specified in the mapping.
Closes #14804
2015-11-18 14:19:07 +01:00
Jason Tedor 3b2bba64ba Add unit test for LinkedHashMap serialization
This commit adds a unit test for LinkedHashMap serialization that tests
that the method of serialization writes the entries in the LinkedHashMap
in iteration order and that the reconstructed LinkedHashMap preserves
that order. This test is randomized and tests iteration order is
preserved whether the LinkedHashMap is ordered by insertion order or
access order.

Closes #14743
2015-11-17 18:52:28 -05:00
Jason Tedor c1ee90356a Add timeout mechanism for sending shard failures
This commit adds a timeout mechanism for sending shard failures. The
requesting thread can attach a listener to the timeout event so that
handling it is part of the event chain.

Relates #14252
2015-11-17 15:00:27 -05:00
Jason Tedor e08e137aeb Add license header to o.e.c.i.s.StreamTests.java 2015-11-17 14:02:12 -05:00
Jason Tedor 6872d545ac Add system CPU percent to OS stats
This commit adds the system CPU percent reflecting the recent CPU usage
for the whole system.
2015-11-17 13:48:46 -05:00
Jason Tedor 043319c482 Merge pull request #14806 from jasontedor/read-optional-streamable
Use Supplier for StreamInput#readOptionalStreamable
2015-11-17 13:41:35 -05:00
Jason Tedor 1cd4a29b6f Use Supplier for StreamInput#readOptionalStreamable
This commit changes the signature of StreamInput#readOptionalStreamable
to accept a Supplier to create new streamables rather than requiring
callers to construct new instances. This has the advantage of avoiding
an allocation in cases when the stream indicates the resulting
streamable is null
2015-11-17 13:40:22 -05:00
Nik Everett 0119caa4a6 Make Build work without git
If you build elasticsearch without a git repository it was creating a null
shortHash which was causing Elasticsearch not to be able to form transport
connections.

Closes #14748
2015-11-17 13:25:09 -05:00
Jason Tedor d2ffcba890 Merge pull request #14780 from jasontedor/variable-length-long
Add variable-length long encoding
2015-11-17 09:04:13 -06:00
Jason Tedor 9a11e75cf5 Add variable-length long encoding
This commit adds a method of encoding longs using a variable-length
representation. This encoding is an implementation of the zig-zag
encoding from protocol buffers. Numbers that have a small absolute value
will use a small number of bytes. This is achieved by zig-zagging
through the space of longs in order of increasing absolute value (0, -1,
1, -2, 2, …, Long.MAX_VALUE, Long.MIN_VALUE) -> (0, 1, 2, 3, 4, …, -2,
-1). The resulting values are then encoded as if they represent unsigned
numbers.
2015-11-17 09:01:54 -06:00
Daniel Mitterdorfer 2522645123 Fix race in ClusterStatsIT
ClusterStatsIT#testClusterStatus() contained a race where the
test cluster might still be initializing while test already checks
for a green health status.

With this commit the test waits until the cluster status changed and
checks health afterwards.

Checked with @bleskes.
2015-11-17 13:22:28 +01:00
Christoph Büscher d6a756fbe2 Merge pull request #14529 from cbuescher/introduce-shapebuilders
Geo: Moving static factory methods to ShapeBuilders
2015-11-17 11:08:35 +01:00
Ryan Ernst 4975422a35 Merge pull request #14783 from rjernst/type_doc_values
Make _type use doc values
2015-11-16 13:09:49 -08:00
Michael McCandless 01f12fc8d5 Merge pull request #14784 from mikemccand/upgrade_lucene_1714615
Upgrade Lucene to 5.4.0-snapshot-1714615
2015-11-16 16:07:46 -05:00
Nicholas Knize 7f5da1d6a3 Fix reproducible GeoDistanceRangeQueryTests.testToQuery error
This issue occurs if the center latitude of the GeoPointDistance query is set to one of the poles. Since this issue is set to be fixed in LUCENE-6897 this commit temporarily limits the random latitudinal location to not include the poles.
2015-11-16 14:45:02 -06:00
Michael McCandless e4cc94c11e remove confusing private default constant 2015-11-16 15:17:16 -05:00
Michael McCandless c3f7638f81 add comment about confusing constant value 2015-11-16 14:55:11 -05:00
Michael McCandless a0bf253d16 upgrade lucene 5.4 snapshot 2015-11-16 14:38:05 -05:00
Ryan Ernst baa3fd814e Mappings: Make _type use doc values
_type should have got doc values with the change to default doc values.
However, due to how metadata fields have separate builders and special
constructors, it was not picking it up. This change updates the field
type for _type to have doc values.

closes #14781
2015-11-16 11:32:52 -08:00
Simon Willnauer 1bdf29e263 Translog recovery can repeatedly fail if we run out of disk
If we run out of disk while recoverying the transaction log
we repeatedly fail since we expect the latest tranlog to be uncommitted.
This change adds 2 safety levels:

 * uncommitted checkpoints are first written to a temp file and then atomically
   renamed into a committed (recovered) checkpoint
 * if the latest uncommitted checkpoints generation is already recovered it has to be
   identical, if not the recovery fails

This allows to fail in between recovering the latest uncommitted checkpoint and moving
the checkpoint generation to N+1 which can for instance happen in a situation where
we can run out of disk. If we run out of disk while recovering the uncommitted checkpoint
either the temp file writing or the atomic rename will fail such that we never have a
half written or corrupted recovered checkpoint.

Close #14695
2015-11-16 18:47:19 +01:00
Christoph Büscher 0623684521 Geo: Moving static factory methods to ShapeBuilders
Currently the abstract ShapeBuilder class serves too many different
purposes, making it hard to refactor and maintain the code. In order
to reduce the size and responsibilities, this PR moved all the
static factory methods used as a shortcut to create new shape builders
out to a new ShapeBuilders class, similar to how QueryBuilders is
used already.
2015-11-16 18:01:23 +01:00
Tanguy Leroux d538f0dcf5 Add FilterPath implementation that uses Jackson 2.6 streaming filtering feature 2015-11-16 17:08:06 +01:00
Tanguy Leroux 8b961bc0e0 Remove unused AbstractXContentGenerator 2015-11-16 17:08:06 +01:00
Christoph Büscher a34b555fff Merge pull request #14775 from cbuescher/fix/14746
Add unique id to query names to avoid naming conflicts
2015-11-16 15:34:09 +01:00
Daniel Mitterdorfer 267f726a7f Merge remote-tracking branch 'danielmitterdorfer/feature/11657' 2015-11-16 15:15:41 +01:00
Daniel Mitterdorfer e31d66d137 Log cluster health status changes
With this commit the cluster health status changes are logged
on INFO level. The change is only logged on master and actively
triggered in AllocationService in order to minimize the impact of
constantly reevaluating ClusterState in a ClusterStateListener
although we know that no health-relevant change happened.

Closes #11657
2015-11-16 15:14:43 +01:00
Christoph Büscher 0a0bca6f5b Tests: Add unique id to query names to avoid naming conflicts
In AbstractQueryTestCase we randomly add the `_name` property to
some of the queries. While this generally works, there are exceptional
cases where we assign the same name to two queries in the setup which
leads to test failures later. This PR adds an increasing counter value
to the base tests that gets appended to all random query names to
avoid this name clashes.
2015-11-16 15:08:25 +01:00
Boaz Leskes fa55a940f7 Test: move SearchQueryIT back to core
There is no groovy dependency there (it's now in lang groovy)

Closes #14770
2015-11-16 14:43:36 +01:00
Simon Willnauer 044b3d59cd Take ingored unallocated shards into account when makeing allocation decision
ClusterRebalanceAllocationDecider did not take unassigned shards into account
that are temporarily marked as ingored. This can cause unexpected behavior
when gateway allocator is still fetching shards or has marked shareds as ignored
since their quorum is not met yet.

Closes #14670
Closes #14678
2015-11-16 14:25:02 +01:00
Yannick Welsch fb10e59578 Merge pull request #14765 from ywelsch/fix/delayed-allocation-race
Fix calculation of next delay for delayed shard allocation
2015-11-16 14:07:35 +01:00
Yannick Welsch 2206030fe6 Fix calculation of next delay for delayed shard allocation
Currently the next delay is calculated based on System.currentTimeMillis() but the actual shards to delay based on the last time the GatewayAllocator tried to assign/delay the shard.

This introduces an inconsistency for the case where shards should have been delay-allocated between the GatewayAllocator-based timestamp and System.currentTimeMillis().

Closes #14765
2015-11-16 14:06:34 +01:00
Christoph Büscher b377d81e1c Adding mock script engine for value script tests 2015-11-16 13:00:43 +01:00
Christoph Büscher 5108e6fd1f Adapting test by using mock script engine 2015-11-16 11:58:28 +01:00
Christoph Büscher e3f9824931 Tests: Move DateHistogramTests back to core module
DateHistogramTests had some dependency on groovy scripts and
were moved to the lang-groovy module. This PR moves it back
and replaces use of groovy scripts by a mock script engine.
Removing three test cases that were testing doing some date
manipulation using script, since these are more groovy script
tests than testing the DateHistogram aggregation.
2015-11-16 11:58:28 +01:00
Martijn van Groningen 293b2f6345 field data: Don't cache top level field data for fields that don't exist.
Just return an empty field data instance, like we do currently the same for segment level field data.
2015-11-16 06:27:48 +01:00
Boaz Leskes eff82fdbcd Tests: update the disco node id seed on node restart
To make sure the new node's id is changed.

Closes #14675
2015-11-15 21:49:54 +01:00
Jason Tedor 89172dfb02 Remove unused import in RecoverySourceHandler 2015-11-15 10:14:08 -05:00
Ryan Ernst 4b17492456 Build: Add javadocs jars
This change adds javadoc jars to core, test-framework and plugins. There
were a couple issues which javadoc found, but doclint did not already
find.
2015-11-15 01:44:42 -08:00
Jason Tedor 3acd7800cb Transport options should be immutable
This commit changes TransportRequestOptions and TransportResponseOptions
to be immutable. This is to address an issue where the empty options
were being mutated permanently altering their state. Making these
objects immutable is just good, clean coding.
2015-11-14 18:21:40 -05:00
Nicholas Knize 42470c252c Correcting maxRadius units in GeoDistanceRangeQueryTest 2015-11-13 11:18:43 -06:00
Nicholas Knize 9b3920b52c Removing pole restriction on GeoDistanceRangeQueryTests
This commit corrects the maxRadialDistance computation for points at the poles to half the distance of the meridian.
2015-11-13 09:32:31 -06:00
Colin Goodheart-Smithe 12bb1b79f6 Aggregations: Pass extended bounds into HistogramAggregator when creating an unmapped aggregator
This fixes an issue where if the field for the aggregation was unmapped the extended bounds would get dropped and the resulting buckets would not cover the extended bounds requested.

Closes #14735
2015-11-13 14:49:15 +00:00
Boaz Leskes e848dcfbca java docs fix 2015-11-13 15:05:38 +01:00
Boaz Leskes ac0da91bf7 Extend usage of IndexSetting class
I decided to leave external listeners (used by plugins) alone, for now.

Closes #14731
2015-11-13 14:30:23 +01:00
Yannick Welsch 9673ddc5d0 Throw MasterNotDiscoveredException whenever retry logic of TransportMasterNodeAction times out
Closes #14737
2015-11-13 14:24:00 +01:00
Colin Goodheart-Smithe be64bfb135 Revert "Aggregations: Pass extended bounds into HistogramAggregator when creating an unmapped aggregator"
This reverts commit e1bf9798bf.
2015-11-13 12:48:51 +00:00
Colin Goodheart-Smithe e1bf9798bf Aggregations: Pass extended bounds into HistogramAggregator when creating an unmapped aggregator
This fixes an issue where if the field for the aggregation was unmapped the extended bounds would get dropped and the resulting buckets would not cover the extended bounds requested.

Closes #14735
2015-11-13 12:33:06 +00:00
Jason Tedor a9ab35a487 Add sync_id to cat shards API
This commit adds the ability to get the sync_id from the cat shards API.

Closes #14705
2015-11-13 05:13:08 -05:00
Robert Muir 776bb288b5 fix gradle check under jigsaw
closes #14726

Squashed commit of the following:

commit 5b591e98570e3fa481b2816a44063b98bff36ddf
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Nov 13 00:54:08 2015 -0500

    add assumption for self-signing in PluginManagerTests

commit ed11e5371b6f71591dc41c6f60d033502cfcf029
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Nov 13 00:20:59 2015 -0500

    show error output from integ test startup

commit d8b187a10e95d89a0e775333dcbe1aaa903fb376
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Nov 12 22:14:11 2015 -0500

    fix gradle check under jigsaw
2015-11-13 00:57:12 -05:00
Nicholas Knize dc77815744 Enable GeoPointV2 with backward compatibility testing
This commit removes all noreleases and cuts over to Lucene 5.4 GeoPointField type. Included are randomized testing updates to unit and integration test suites for ensuring full backward compatability with existing geo_point indexes.
2015-11-12 23:01:51 -06:00
Robert Muir 720ebe347d Handle system policy correctly.
Just suck in the system policy, so its compatible with any version of java.
It means it also respects configuration (e.g. for monitoring agents)

Closes #14704
2015-11-12 18:08:13 -05:00
xuzha f33184329f Add extra validation into `cluster/stats`
The cluster health and cluster stats disagree on the status.
Add a extra validation step in `cluster/stats`.

closes #7390
2015-11-12 10:46:18 -08:00
Yannick Welsch 4729386347 Merge pull request #14494 from ywelsch/fix/delayed-allocation-reroute
Delayed allocation can miss a reroute
2015-11-12 17:58:08 +01:00
Yannick Welsch c32de1f72b [TEST] Use TransportService/TransportAddress instead of DiscoveryNode for disruption rules
The disruption rules are changed to work on all transport addresses that are bound by a node (not only publish address).
This is important as UnicastZenPing creates fake DiscoveryNode instances which match one of the bound addresses and not necessarily the publish address.

Closes #14625
Closes #14653
2015-11-12 17:20:07 +01:00
Jason Tedor fac472f90c Remove unused imports in o.e.c.a.s.ShardStateActionTests 2015-11-12 09:08:27 -05:00
Yannick Welsch 0220e45e2f Fix missing reroute in case of multiple delayed shards
After a delayed reroute of a shard, RoutingService misses to schedule a new delayed reroute of other delayed shards.

Closes #14494
Closes #14010
Closes #14445
2015-11-12 10:57:40 +01:00
Jason Tedor c2aec53b46 Move field to local variable 2015-11-11 12:50:21 -05:00
Jason Tedor 3a92012938 Use String#equals instead of reference equality 2015-11-11 12:50:21 -05:00
Jason Tedor e4e68c0463 Sanity check exception message in tests 2015-11-11 12:50:21 -05:00
Jason Tedor 06c07ce275 Remove unnecessary generic type parameters 2015-11-11 12:50:21 -05:00
Robert Muir 914cee213e Upgrade to aws 1.10.33
Security issues have been fixed. This removes our hacks.
2015-11-10 22:51:33 -05:00
Ryan Ernst 4b5f87cb7d Build: Remove transitive dependencies
Transitive dependencies can be confusing and hard to deal with when
conflicts arise between them. This change removes transitive
dependencies from elasticsearch, and forces any dependency conflicts to
be resolved manually, instead of automatically by gradle.

closes #14627
2015-11-10 15:01:41 -08:00
Tanguy Leroux 4063354dbe Fix Delete-by-Query with Shield
closes #14527
2015-11-10 20:39:38 +01:00
Lee Hinman bf3e32e8be Add breaker name to logging package
This allows different circuit breakers to have different logging levels.
It's useful when diagnosing problems (say for instance with the
fielddata breaker) and not seeing the enormous amount of logging from
the request breaker.

The log messages use the breaker name for logging, so example logging
will look like:

```
[2015-11-10 09:51:52,993][TRACE][indices.breaker.fielddata] [fielddata] Adding [27b][body] to used bytes [new used: [27b], limit: 623326003 [594.4mb], estimate: 27 [27b]]
[2015-11-10 09:51:53,000][TRACE][indices.breaker.fielddata] [fielddata] Adjusted breaker by [453] bytes, now [480]
[2015-11-10 09:51:53,016][TRACE][indices.breaker.request  ] [request] Adjusted breaker by [16440] bytes, now [16440]
[2015-11-10 09:51:53,018][TRACE][indices.breaker.request  ] [request] Adjusted breaker by [-16440] bytes, now [0]
```
2015-11-10 10:02:34 -07:00
Martijn van Groningen 973c094a5d Merge pull request #14654 from s1monw/fix_searcher_wrapper_interface
Fix IndexSearcherWrapper interface to not depend on the EngineConfig
2015-11-10 23:09:32 +07:00
Ryan Ernst 7bb4e10353 Merge pull request #14645 from rjernst/version_cleanup
Consolidate dependencies specified in multiple places
2015-11-10 07:49:10 -08:00
David Pilato a61fcfd8ca Remove AbstractLegacyBlobContainer
`AbstractLegacyBlobContainer` was kept for historical reasons (see #13434).
We can migrate Azure and S3 repositories to use the new methods added in #13434 so we can remove `AbstractLegacyBlobContainer` class.
2015-11-10 16:04:44 +01:00
Simon Willnauer 049430621d Fix IndexSearcherWrapper interface to not depend on the EngineConfig
We only passed the engine config since we had no chance to get the cache
etc. from the IndexSearcher. Now that we have these getters we can just pass the
searcher instead.
2015-11-10 14:53:32 +01:00
Simon Willnauer 17913c56fb Merge pull request #14591 from s1monw/issues/14387
Only allow rebalance operations to run if all shard store data is available
2015-11-10 12:06:31 +01:00
Simon Willnauer 84033f3f2e Use syntactic sugar for starting data only nodes 2015-11-10 12:00:29 +01:00
Ryan Ernst 7a6155e12f Build: Consolidate dependencies specified in multiple places
Some dependencies must be specified in a couple places in the build.
e.g. randomized runner is specified both in buildSrc (for the gradle
wrapper plugin), as well as in the test-framework.

This change creates buildSrc/versions.properties which acts similar to
the set of shared version properties we used to have in the maven parent
pom.
2015-11-09 23:59:56 -08:00
Nicholas Knize 3037970c33 Update GeoPoint FieldData for GeoPointV2
This commit adds an abstraction layer to GeoPoint FieldData for supporting new Lucene 5.4 GeoPointField and backwards compatibility.
2015-11-09 15:39:24 -06:00
Adrien Grand d0853e9253 Merge pull request #14619 from jpountz/upgrade/lucene-5.4.0-snapshot-1712973
Upgrade to lucene-5.4.0-snapshot-1712973.
2015-11-09 22:38:44 +01:00
Ryan Ernst 7034feeceb Merge pull request #14451 from andrejserafim/run
Add gradle run command to replace run.bat and run.sh
2015-11-09 11:15:53 -08:00
Lee Hinman 25f8db9aa9 Merge remote-tracking branch 'dakrone/node-limit-decider' 2015-11-09 11:03:19 -07:00
Lee Hinman 145374b762 Add cluster-wide setting for total shard limit
This adds the `cluster.routing.allocation.total_shards_per_node`
setting, which limits the total number of shards across all indices on
each node. It defaults to -1 and can be dynamically configured.

Resolves #14456
2015-11-09 11:03:07 -07:00
Nicholas Knize 36e294a03e Fix GeoDistanceRangeQueryTest tolerance
GeoDistanceRangeQueryTest was using an inconsistent tolerance when validating from=0 with include_lower/upper set to false. This commit changes the assertion such that validation is consistent with builder logic.
2015-11-09 11:03:22 -06:00
Yannick Welsch 33ca2ec597 Merge pull request #14222 from ywelsch/fix/refactor-transport-master-node-action
Refactor retry logic for TransportMasterNodeAction
2015-11-09 17:59:16 +01:00
Yannick Welsch 0575744a39 Refactor retry logic for TransportMasterNodeAction
- Same as for TransportInstanceSingleOperationAction and TransportReplicationAction, onClusterServiceClose consistently throws a NodeClosedException now.
- Added retry logic if master could not publish cluster state or stepped down before publishing (ZenDiscovery). The test IndexingMasterFailoverIT shows the issue.
- Simplified retry logic by moving bits from different places into shared retry method.
  - Removed boolean flag retrying that aborted retrying after a single master node change (now we retry until timeout).
  - Two existing predicates that deal with master node changes unified in a single predicate masterNodeChangedPredicate

Closes #14222
2015-11-09 17:56:18 +01:00
Adrien Grand d6d7af0a6c Upgrade to lucene-5.4.0-snapshot-1712973. 2015-11-09 15:53:27 +01:00
javanna 43b729f34c [TEST] test query and search source parsing from different XContent types
We used to test only json parsing as we relied on QueryBuilder#toString which uses the json format. This commit makes sure that we now output the randomly generated queries using a random format, and that we are always able to parse them correctly.

This revealed a couple of issues with binary objects that haven't been migrated yet to be structured Writeable objects. We used to keep them in the format they were sent while parsing, which led to problems when printing them out as we expected them to always be in json format. Also we can't compare different BytesReference objects that hold the same content but in different formats (unless we want to parse them as part of equal and hashcode, doesn't seem like a good idea) and verify that we have parsed the right objects if they can be different formats. The fix is to always keep binary objects in json format. Best fix would be not to have binary objects, which we'll get to once we are done with the search refactoring.

Closes #14415
2015-11-09 15:09:27 +01:00
javanna 10ddd691a3 Internal: move to lucene BoostQuery
Latest version of lucene deprecated Query#setBoost and Query#getBoost which made queries effectively immutable. Those methods need to be replaced with `BoostQuery` that wraps any query that needs boosting.

This commit replaces usages of setBoost with BoostQuery and adds it to forbidden-apis for prod code.

Usages of `getBoost` are only partially removed, as some will have to stay for backwards compatibility.

Closes #14264
2015-11-09 15:01:35 +01:00
Simon Willnauer 7b5e323ec0 add IT for #14387 2015-11-09 15:00:56 +01:00
Robert Muir 102e25413d Merge pull request #14605 from rmuir/migrate-mapper-attachments
Migrate mapper attachments plugin to main repository
2015-11-09 08:55:29 -05:00
Simon Willnauer 219fe8f30f apply @bleskes review changes 2015-11-09 12:58:11 +01:00
Daniel Mitterdorfer 1c5b0b3c5b Merge branch 'pr/14611' 2015-11-09 12:08:56 +01:00
Adrien Grand f400f58132 Merge pull request #14472 from jpountz/fix/simplify_xcontent_detection
Simplify XContent detection.
2015-11-09 12:02:39 +01:00
Adrien Grand c350a3dfae Simplify XContent detection.
Currently we duplicate the logic for BytesReferences and InputStreams.
2015-11-09 12:00:03 +01:00
Martijn van Groningen 72e9bd8506 Merge pull request #14597 from wbowling/tophitsbuilder-addfield
Put method addField on TopHitsBuilder
2015-11-09 17:40:13 +07:00
Camilo Díaz Repka d795ce3b6e XContentFactory.xContentType: allow for possible UTF-8 BOM for JSON XContentType
Fixes #14442
2015-11-09 11:19:54 +01:00
Robert Muir 0de503b34b fixup issues with 32-bit jvm
If you run tests under a 32-bit jvm, you will get a test failure in IndexStoreTests,
the logic there is wrong in the case of 32-bit (its NIOFSDirectory on linux).

Also if mlockall fails, you'll see huge bogus values (because of use of `long` instead of `NativeLong`)

finally add seccomp support for 32 bit too, and clean up all its `long` usage as well.
2015-11-08 21:08:08 -05:00
andrejserafim 2bd353d72d replacing run.bat and run.sh with gradle run
run.sh and run.bat were calling out to the old maven build system.
This is no longer in place, so we've created new gradle tasks to
start an elasticsearch node from the current codebase.

fixed #14423
2015-11-08 17:07:19 +00:00
Simon Willnauer 47769cc4b4 add more logging to test for better readability 2015-11-08 12:43:24 +01:00
Simon Willnauer fa6a2932f5 Instead of relying on the in-flight number use a boolean flag on the
allocation to ensure that we skip rebalance if we have not processed
all pending shard / store fetches.
2015-11-08 12:40:05 +01:00
Robert Muir e88896e40e apply a few post-merge cleanups 2015-11-08 02:31:26 -05:00
Areek Zillur 6444194dae fix conflicts from merge 2015-11-07 18:22:28 -05:00
Areek Zillur d5198b326c fix comments and throw ElasticsearchException when we fail to report fst index size 2015-11-07 17:50:56 -05:00
Areek Zillur d37e011526 add query context builders 2015-11-07 17:50:56 -05:00
Areek Zillur 40022e2168 remove nocommit 2015-11-07 17:50:55 -05:00
Areek Zillur e87b4e00eb add back awaitfix tests 2015-11-07 17:50:55 -05:00
Simon Willnauer 6f9a486071 more parsers 2015-11-07 17:50:55 -05:00
Simon Willnauer f5560a3087 add parsers 2015-11-07 17:47:46 -05:00
Simon Willnauer 8911469e63 some basic cleanups 2015-11-07 17:46:28 -05:00
Areek Zillur 75d55e11ad cut over to using ObjectParser for context mappings 2015-11-07 17:46:27 -05:00
Areek Zillur 96c9849f5a added FuzzyOptionsBuilder javadocs and cleanups 2015-11-07 17:46:27 -05:00
Areek Zillur dd1c687ace Completion Suggester V2
The completion suggester provides auto-complete/search-as-you-type functionality.
This is a navigational feature to guide users to relevant results as they are typing, improving search precision.
It is not meant for spell correction or did-you-mean functionality like the term or phrase suggesters.

The completions are indexed as a weighted FST (finite state transducer) to provide fast Top N prefix-based
searches suitable for serving relevant results as a user types.

closes #10746
2015-11-07 17:46:27 -05:00
Robert Muir e06cae84f3 only allow code to bind to the user's configured port numbers/ranges
Random code shouldn't be listening on sockets elsewhere.

Today its the wild west, but we only need to grant access to what the user configured.

This means e.g. multicast plugin has to declare its intentions in its security.policy

Closes #14549
2015-11-07 12:22:12 -05:00
William Bowling 202a735d8f Put method addField on TopHitsBuilder 2015-11-07 17:15:02 +11:00
Robert Muir f03196193f Fix Build to correctly treat URLs and not leak a file handle.
Closes #14595

Squashed commit of the following:

commit d0b2b262e9dcdbc2aee163b9a84db082c8b5b96b
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Nov 6 22:36:54 2015 -0500

    Switch to JarInputStream, to contain suppressforbidden. Also add a test that fails if path is not accessible (regardless of whether its a jar)

commit f99c1d240db23ceb2a06987b3bd69eae0229550b
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Nov 6 22:16:16 2015 -0500

    remove leniency in i/o here

commit b160d4303ee81a8c9298729596ecbc893f5f8894
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Nov 6 21:58:21 2015 -0500

    Fix Build to correctly treat URLs and to not leak a file handle
2015-11-06 22:40:58 -05:00
Ryan Ernst 59a1cda9bb Merge pull request #14592 from rjernst/eclipse_cycles
Build: Fix eclipse generation to add a core-tests projects
2015-11-06 11:55:47 -08:00
Ryan Ernst 548430072f Build: Fix eclipse generation to add a core-tests projects
Eclipse does not have the ability to differentiate test dependencies
from main dependencies. This causes what looks like a circular
dependency through test-framework. This change sets up an additional
core-tests project for eclipse only, which removes this problem.
2015-11-06 11:41:18 -08:00
Simon Willnauer 01ca95a648 Only allow rebalance operations to run if all shard store data is available
This commit prevents running rebalance operations if the store allocator is
still fetching async shard / store data to prevent pre-mature rebalance decisions
which need to be reverted once shard store data is available. This is typically happening
on rolling restarts which can make those restarts extremely painful.

Closes #14387
2015-11-06 20:18:01 +01:00
Nicholas Knize afaf96732a Add GeoPointV2 Field Mapping
This commit adds the abstraction layer to GeoPointFieldMapper needed to cut over to Lucene 5.4's new GeoPointField type while maintaining backward compatibility with 'legacy' geo_point indexes.
2015-11-06 13:14:04 -06:00
Jason Tedor 699c701b59 Handle shards assigned to nodes that are not in the cluster state
This commit addresses an issue in TransportBroadcastByNodeAction.
Namely, if in between a master leaving the cluster and a new one being
assigned, a request that relies on TransportBroadcastByNodeAction
(e.g., an indices stats request) is issued,
TransportBroadcastByNodeAction might attempt to send a request to a
node that is no longer in the local node’s cluster state.

The exact circumstances that lead to this are as follows. When the
master leaves the cluster and another node’s master fault detection
detects this, the local node will update its local cluster state to no
longer include the master node. However, the routing table will not be
updated. This means that in the preparation for sending the requests in
TransportBroadcastByNodeAction, we need to check that not only is a
shard assigned, but also that it is assigned to a node that is still in
the local node’s cluster state.

This commit adds such a check to the constructor of
TransportBroadcastByNodeAction. A new unit test is added that checks
that no request is sent to the master node in such a situation; this
test fails with a NullPointerException without the fix. Additionally,
the unit test TransportBroadcastByNodeActionTests#testResultAggregation
is updated to also simulate a master failure. This updated test also
fails prior to the fix.

Closes #14584
2015-11-06 12:09:30 -05:00
Yannick Welsch c6b41bc704 Introduce inner class AsyncSingleAction to store state used across retry invocations 2015-11-06 16:38:33 +01:00
Yannick Welsch 1c31845b7a Merge pull request #14535 from ywelsch/fix/bound-vs-publish-port
Port of publishAddress should match port of corresponding boundAddress
2015-11-06 10:17:05 +01:00
Yannick Welsch 1902c66aaf Port of publishAddress should match port of corresponding boundAddress
Closes #14535
Closes #14503
Closes #14514
Closes #14513
Closes #14476
2015-11-06 10:12:10 +01:00
Simon Willnauer 1bff08b2b3 [TEST] Add test only API that allows to pass the inactive time in NS directly
if we set the inactive time for the shard via API the entire test if fully time
dependent and might fail if we concurrently check if the shard is inactive
while the document we are indexing is in-flight.
2015-11-06 09:18:23 +01:00
Jason Tedor 54d2b5f3b6 Merge pull request #14564 from jasontedor/restore-build-properties
Restore build properties
2015-11-05 21:34:07 -05:00
Jason Tedor 8278c9887e Restore build properties
This commit restores the build properties provided in
org.elasticsearch.Build. This class previously obtained the build hash
and timestamp from a resource es-build.properties that was included in
the jar and produced by the Maven resources plugin. After the switch to
Gradle, the production of this file was lost and these build properties
defaulted to “NA” in all instances.

The most important place that the build hash is used is in the plugin
manager to determine the URL of staging artifacts for plugins.

The build hash is also used in several responses including the /_nodes
response and the response to HTTP GET requests on the root path.

These properties can now be obtained from the jar manifest as they are
currently placed there by the gradle-info plugin. However, only the
short hash is provided. We now read the manifest for these properties
and no longer provide the full hash in responses to HTTP GET requests
on the root path.
2015-11-05 21:30:04 -05:00
Simon Willnauer 46bec2785e [TEST] mark test awaitsfix 2015-11-05 22:20:23 +01:00
Nicholas Knize 86ae08fa88 Add support for Lucene 5.4 GeoPoint queries
This commit adds query support for Lucene 5.4 GeoPointField type along with backward compatibility for (soon to be) "legacy" geo_point indexes.
2015-11-05 12:21:27 -06:00
Simon Willnauer 99b61ef91c [TEST] use TRACE Logging for syncFlush tests 2015-11-05 18:37:26 +01:00
Simon Willnauer 4a686c9750 mark shard as active before tripper synced flush in test 2015-11-05 16:50:10 +01:00
Simon Willnauer cf0edeaa4f Mark failing test as awaits fix 2015-11-05 16:39:11 +01:00
Simon Willnauer b328e26160 Merge pull request #14518 from s1monw/simplify_index_level_services
Remove guice from the index level
2015-11-05 15:06:37 +01:00
Robert Muir fffafbfd4d Merge pull request #14274 from rmuir/broken_code
Add workaround for JDK-8014008
2015-11-05 08:24:04 -05:00
Simon Willnauer 94bed42213 Simplify plugin API and fix IndexService internal allocation 2015-11-05 13:16:35 +01:00
Simon Willnauer cf0179bf12 apply @rjernst review comments 2015-11-05 11:18:12 +01:00
Simon Willnauer 0dac6bfa86 fix return value 2015-11-05 11:18:11 +01:00
Simon Willnauer a4a0390ff6 Move inactiveTime into IndexShard out of IndexingMemoryController 2015-11-05 11:18:11 +01:00
Simon Willnauer 7d77f182e2 fix tests 2015-11-05 11:18:11 +01:00
Simon Willnauer a311491c8e decouple IndexingMemoryController from IndexShard 2015-11-05 11:18:11 +01:00
Simon Willnauer 487af301ea Remove guice from the index level
This commit removes guice from the index level and adds a simple extension point
to add class instances with an index-lifecycle scope.
2015-11-05 11:18:11 +01:00
Simon Willnauer 45f7844948 Make CodecService a simple per shard registry. 2015-11-05 11:18:10 +01:00
Martijn van Groningen fc02b2bb0f test: don't run the second cluster with http enabled. 2015-11-05 14:08:54 +07:00
Robert Muir 1a0a910d1e Merge pull request #14526 from rjernst/standalone_test
Add qa/evil-tests
2015-11-04 13:49:55 -05:00
Boaz Leskes efde8e8a26 Gateway: failure to update the cluster state with the recovered state should make sure it will be recovered later
We currently fail to follow the right failure paths.

See http://build-us-00.elastic.co/job/es_core_master_window-2008/2477/

Closes #14485
2015-11-04 09:42:24 +00:00
Boaz Leskes 8eff4e211a Improve some logging around master election and cluster state
Tweaks done while debugging http://build-us-00.elastic.co/job/es_core_master_window-2008/2477/

Closes #14481
2015-11-04 09:39:56 +00:00
Yannick Welsch 825d0c64e6 Add duration field to /_cat/snapshots
Closes #14385
2015-11-04 10:34:00 +01:00
Daniel Mitterdorfer ae9fcd4f2b Merge remote-tracking branch 'danielmitterdorfer/feature/12596' 2015-11-04 09:31:12 +01:00
Daniel Mitterdorfer fea88bbc02 Add System#exit(), Runtime#exit() and Runtime#halt() to forbidden APIs
Closes #12596
2015-11-04 09:29:55 +01:00
Daniel Mitterdorfer 9ea25164b2 Merge remote-tracking branch 'danielmitterdorfer/fix/13132' 2015-11-04 09:06:35 +01:00
Daniel Mitterdorfer 7b62267008 Geo: Allow numeric parameters enclosed in quotes for 'geohash_grid' aggregation
Originally, only numeric values were allowed for parameters of the
'geohash_grid' aggregation in contrast to other places in the REST
API.

With this commit we also allow that parameters are enclosed in quotes (i.e.
as JSON strings). Additionally, with this commit the valid range for
'precision' is enforced for the Java API and the REST API (the latter was
previously missing the check).

Closes #13132
2015-11-04 09:04:22 +01:00
Robert Muir 3d8b4dae33 remove the ability for code to change file permissions, this was
only needed for pluginmanager's test, and pluginmanager doesn't
even run with securitymanager yet.
2015-11-04 00:31:15 -05:00
Robert Muir 6be9954d28 remove jimfs dep in tests 2015-11-03 23:29:14 -05:00
Robert Muir 602feac915 move tests never running in jenkins to new evil tests module 2015-11-03 21:42:22 -05:00
Jason Tedor 9d42b8b22b Merge pull request #14497 from jasontedor/java-9-type-inference-concatenated-iterator
Fix Java 9 type inference issue
2015-11-03 16:56:49 -05:00
Robert Muir 79467a138a Add property permissions so groovy scripts can serialize json
Allowing read to these properties is not really dangerous, even if
the code surrounding them is.

Closes #14488
2015-11-03 15:21:55 -05:00
Jason Tedor 2398f41db2 Fix Java 9 type inference issue
This commit fixes a compilation issue due to modified type inference in
the latest JDK 9 early access builds. We just have to lend a helping
hand to type inference by being explicit about the type.

Closes #14496
2015-11-03 15:15:40 -05:00
xuzha fb1d8bb149 Add os.allocated_processors
Current processors setting is not reflected in nodes info API
("os.available_processors"). Add os.allocated_processors to shows
actual number of processors that we are using.
2015-11-03 09:50:17 -08:00
Jason Tedor b2c098630e Merge pull request #14483 from jasontedor/cluster-block-to-string-dangling-comma
Fix dangling comma in ClusterBlock#toString
2015-11-03 11:55:01 -05:00
Jason Tedor 832267bcc1 Fix dangling comma in ClusterBlock#toString
This commit address some serious health issues that could arise from a
dangerous combination of OCD, string concatentation, and dangling
commas,
2015-11-03 11:52:40 -05:00
Simon Willnauer 759948f6c1 Merge branch 'master' into refactor_query_shard_context 2015-11-03 17:10:13 +01:00
Simon Willnauer 968561ad49 apply review comments 2015-11-03 17:09:32 +01:00
Robert Muir b6a2318563 upgrade rhino for plugins/lang-javascript
the current jar is over 3 years old, we should upgrade it for bugfixes.
the current integration could be more secure: set a global policy and enforce additional (compile-time) checks

closes #14466
2015-11-03 09:53:07 -05:00
David Tvaltchrelidze ea4e514385 Fix HTML response during redirection
It misses a quote...

Closes #11374
2015-11-03 13:59:57 +00:00
javanna ce6aa258a9 Remove support for query_binary and filter_binary
query_binary and filter_binary are unused at this point, as we only parse on the coordinating node and the java api only holds structured java objects for queries and filters, meaning they all implement Writeable and get natively serialized.

Relates to #14308
Closes #14433
2015-11-03 14:05:45 +01:00
Jason Tedor 43323c3541 Fix bug in cat thread pool
This commit fixes a bug in cat thread pool. This bug resulted from a
refactoring of the handling of thread pool types. To get the previously
displayed thread pool type from the ThreadPoolType object,
ThreadPoolType#getType needs to be called.
2015-11-03 07:40:49 -05:00
Simon Willnauer d76ae6718e Merge pull request #14432 from s1monw/add_shard_failure_as_suppressed
Deduplicate cause if already contained in shard failures
2015-11-03 12:06:23 +01:00
Christoph Büscher 2fe519c3e7 Revert fd3a46a because fix is to fragile
Reverting fix for #13884 because it was discussed to be too
fragile with respect to future changes in lucene simple query
string parsing. Undoes fix and removes test.
2015-11-03 11:48:14 +01:00
Adrien Grand a7bf06ee3a Fix test bug in UpdateThreadPoolSettingsTests. 2015-11-03 11:35:47 +01:00
Simon Willnauer 3f94a566ff Deduplicate cause if already contained in shard failures
If we have a shard failure on SearchPhaseExecutionException
we can deduplicate the original cause and use the more informative
ShardSearchFailure containing the shard ID etc. but we should deduplicate
the actual cause to prevent stack trace duplication.
2015-11-03 11:35:28 +01:00
Jason Tedor 48ee09fbd4 Add missing license file to ThreadPoolTypeSettingsValidatorTests.java 2015-11-02 22:14:28 -05:00
Jason Tedor e3b8dc7121 Forbid changing thread pool types
This commit forbids the changing of thread pool types for any thread
pool. The motivation here is that these are expert settings with
little practical advantage.

Closes #14294, relates #2509, relates #2858, relates #5152
2015-11-02 20:52:48 -05:00
Simon Willnauer 4176964358 Cleanup query parsing and remove IndexQueryParserService
IndexQueryParserService is only a factory for QueryShardContext instances
which are not even bound to a shard. The service only forwards dependencies and even
references node level service directly which makes dependency seperation on shard,
index and node level hard. This commit removes the service entirely, folds the creation
of QueryShardContext into IndexShard which is it's logical place and detaches the
ClusterService needed for index name matching during query parsing with a simple predicate
interface on IndexSettings.
2015-11-02 21:20:11 +01:00
Robert Muir f83800b1d2 don't discard original exception (suspiciously/possibly eclipse-generated) 2015-11-02 12:05:44 -05:00
Christoph Büscher 45dbcc4402 Merge pull request #14255 from cbuescher/feature/add-query-level-test
Parsers should throw exception on unknown objects
2015-11-02 11:48:17 +01:00
Christoph Büscher 72463768f5 Query DSL: Parsers should throw exception on unknown object
This PR adds a randomized test to the query test base class
that mutates an otherwise correct query by adding an additional
object into the query hierarchy. Doing so makes the query illegal
and should trigger some kind of exception. The new test revelead
that some query parsers quietly return queries when called with
such an illegal query. Those are also fixed here.

Relates to #10974
2015-11-02 11:39:22 +01:00
javanna b56bbf62dd Validate query api: move query parsing on the coordinating node
Similarly to what we did with the search api, we can now also move query parsing on the coordinating node for the validate query api. Given that the explain api is a single shard operation (compared to search which is instead a broadcast operation), this doesn't change a lot in how the api works internally. The main benefit is that we can simplify the java api by requiring a structured query object to be provided rather than a bytes array that will get parsed on the data node. Previously if you specified a QueryBuilder it would be serialized in json format and would get reparsed on the data node, while now it doesn't go through parsing anymore (as expected), given that after the query-refactoring we are able to properly stream queries natively. Note that the WrapperQueryBuilder can be used from the java api to provide a query as a string, in that case the actual parsing of the inner query will happen on the data node.

Relates to #10217
Closes #14384
2015-11-02 11:21:20 +01:00
Ryan Ernst b6dee6bd43 Merge pull request #14375 from rjernst/sweep_up_maven
Remove maven pom files and supporting ant files
2015-10-30 18:59:11 -07:00
Ryan Ernst 0a4fa7453b Merge pull request #14402 from rjernst/reorganize_rr_gradle_plugin
Move RR gradle plugin files to match external repo paths
2015-10-30 15:20:09 -07:00
Lee Hinman f726e11da6 Move logging for the amount of free disk to TRACE
Since this can potentially be logged for every `canRemain`, it's nicer
to put it in TRACE rather than DEBUG.

Resolves #12843
2015-10-30 14:56:11 -06:00
Ryan Ernst 197ed57ea4 Build: Move RR gradle plugin files to match external repo paths
The RR gradle plugin is at
https://github.com/randomizedtesting/gradle-randomized-testing-plugin.
However, we currently have a copy of this, since the plugin is still in
heavy development. This change moves the files around so they can be
copied directly from the elasticsearch fork to that repo, for ease of
syncing.
2015-10-30 13:22:59 -07:00
Areek Zillur 13b60e1b92 update to lucene-5.4.x-snapshot-1711508 2015-10-30 15:42:02 -04:00
Nik Everett c521b27e31 Merge pull request #13657 from nik9000/remove_transport
Remove transform
2015-10-30 12:34:08 -04:00
Nik Everett 28633fae21 [mapping] Remove transform
Removes the mapping transform feature which when used made debugging very
difficult. Users should transform their documents on the way into
Elasticsearch rather than having Elasticsearch do it.

Closes #12674
2015-10-30 11:46:54 -04:00
Christoph Büscher 5fcc648047 Tests: Check exception from query parsers for unknown field
Most query parsers throw a ParsingException when they trying
to parse a field with an unknown name. This adds a generic
check for this to the AbstractQueryTestCase so the behaviour
gets tested for all query parsers. The test works by first
making sure the test query has a `boost` field and then
changing this to an unknown field name and checking for an
error.

There are exceptions to this for WrapperQueryBuilder
and QueryFilterBuilder, because here the parser only expects
the wrapped `query` element. MatchNoneQueryBuilder and
MatchAllQueryBuilder so far had setters for boost() and
queryName() but didn't render them, which is added here for
consistency.

GeoDistance, GeoDistanceRange and GeoHashCellQuery so far
treat unknown field names in the json as the target field name
of the center point of the query, which so far was handled by
overwriting points previously specified in the query. This
is changed here so that an attempt to use two different field names
to specify the central point of the query throws a
ParsingException

Relates to #10974
2015-10-30 16:36:00 +01:00
Simon Willnauer b17d8c2c97 Merge pull request #14355 from s1monw/rewrite_analysis_registry
Simplify Analysis registration and configuration
2015-10-30 12:01:31 +01:00
Simon Willnauer aa38d053d7 Simplify Analysis registration and configuration
This change moves all the analysis component registration to the node level
and removes the significant API overhead to register tokenfilter, tokenizer,
charfilter and analyzer. All registration is done without guice interaction such
that real factories via functional interfaces are passed instead of class objects
that are instantiated at runtime.

This change also hides the internal analyzer caching that was done previously in the
IndicesAnalysisService entirely and decouples all analysis registration and creation
from dependency injection.
2015-10-30 11:40:18 +01:00
Yannick Welsch 017af34971 Merge pull request #14360 from ywelsch/fix/rest-indices-action
Properly set indices and indicesOptions on subrequest made by /_cat/indices
2015-10-30 11:34:04 +01:00
Ryan Ernst 542522531a Build: Remove maven pom files and supporting ant files
This change removes the leftover pom files. A couple files were left for
reference, namely in qa tests that have not yet been migrated (vagrant
and multinode). The deb and rpm assemblies also still exist for
reference when finishing their setup in gradle.

See #13930
2015-10-29 23:53:49 -07:00
Ryan Ernst 63f6c6db85 Build: Move test framework files to their new location
The test jar was previously built in maven by copying class files. With
gradle we now have a proper test framework artifact. This change moves
the classes used by the test framework into the test-framework module.

See #13930
2015-10-29 23:25:24 -07:00
Jason Tedor c7897a7524 Make throw statement consistent with others in same class 2015-10-29 21:53:33 -04:00
Lee Hinman 3b5058017e Merge branch 'remove-optimize-rest' 2015-10-29 15:18:03 -06:00
Ryan Ernst c86100f636 Switch build system to Gradle
See #13930
2015-10-29 11:40:19 -07:00
Yannick Welsch 15bd182668 Properly set indices and indicesOptions on subrequest made by /_cat/indices 2015-10-29 19:15:56 +01:00
Robert Muir 1194cd0bdc Add threadgroup isolation.
Closes #14353

Squashed commit of the following:

commit edae0729f71ea3d3f9fa9c0d27c9effc042eb5a9
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Oct 29 14:13:42 2015 -0400

    update sha1 and simplify test

commit 635c4f245d66ad353a16267c810e02b725553fad
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Oct 29 07:01:26 2015 -0400

    Add threadgroup isolation.

    Code with `modifyThread` and `modifyThreadGroup` may only modify
    its own threadgroup (or an ancestor of that). This enforces
    what is intended by the ThreadGroup class.

    This has two immediate implications:
    1. Code without these permissions (scripts) may not create or mess with threads
    2. ES application threads cannot mess with Java system threads

    ES puts all application threads in one single group today, but in the future
    this can be organized better, and we will have more isolation in the system.
2015-10-29 14:14:27 -04:00
Simon Willnauer 84fda97bb6 Merge pull request #14321 from s1monw/use_nano_time
Use System.nanoTime() to initialize Engine.lastWriteNanos
2015-10-29 16:48:36 +01:00
Boaz Leskes 5380322947 Recovery: mark shard as recovering on the cluster state thread
Closes #14115
Closes #14276
2015-10-29 11:41:23 +01:00
Clinton Gormley 085f9e032b Added versions 2.0.0 and 2.0.1-SNAPSHOT and bwc indices for 2.0.0 2015-10-28 21:21:06 +01:00
Robert Muir 0d0a4b3ff7 Upgrade to randomizedtesting 2.2.0
Closes #14342
2015-10-28 15:28:30 -04:00
Jason Tedor 06853209aa Fix issues with failed cache loads
This commit fixes two issues that could arise when a loader throws an
exception during a load in Cache#computeIfAbsent.

The underlying issue is that if the loader throws an exception,
Cache#computeIfAbsent would attempt to remove the polluted entry from
the cache. However, this cleanup was performed outside of the segment
lock. This means another thread could race and expire the polluted
entry (leading to NPEs) or get a polluted entry out of the cache before
the loading thread had a chance to cleanup (leading to ISEs).

The solution to the initial problem of correctly handling failed cached
loads is to check for failed loads in all places where entries are
retrieved from the map backing the segment. In such cases, we treat it
as if there was no entry in the cache, and we clean up the cache on a
best-effort basis. All of this is done outside of the segment lock to
avoid reintroducing the deadlock that was initially a problem when
loads were executed under a segment lock.
2015-10-28 14:22:46 -04:00
xuzha 97ecd7bf5a Expose pending cluster state queue size in node stats
Add 3 stats about the queue: total queue size, number of committed cluster
states, and number of pending cluster states.
2015-10-28 10:59:15 -07:00
Nicholas Knize 8c535e0f6e Refactor Geo utilities to Lucene 5.4
Remove local lucene/XGeo* classes and refactor to Lucene 5.4 Geo Utility classes.
2015-10-28 11:41:56 -05:00
Jason Tedor 1390e68465 Fix test bug in CacheTests#testDependentKeyDeadlock 2015-10-28 11:05:00 -04:00
Christoph Büscher 813f494f12 Tests: Fixed missing import 2015-10-28 15:57:11 +01:00
Christoph Büscher e7294322af Merge pull request #14286 from cbuescher/add-loop-querytests
Tests: run base query tests for more than one random query
2015-10-28 15:40:21 +01:00
Jason Tedor 90528ce5af Merge pull request #14334 from jasontedor/cache-deadlock-test
Add cache deadlock test
2015-10-28 10:30:25 -04:00
Jason Tedor baf361f1f9 Add cache deadlock test
This commit adds a unit test for a deadlock issue that existed prior to
commit 1d0b93f766. While commit
1d0b93f766 seems to have addressed the
deadlock issue it would be more robust to have a unit test for it and a
unit test will reduce the risk that future maintenance on Cache will
reintroduce the deadlock issue. This test reliably fails prior to but
passes after commit 1d0b93f766.
2015-10-28 10:29:53 -04:00
Adrien Grand 2ed02f9eb3 Merge pull request #14320 from jpountz/upgrade/lucene-5.4-snapshot-1710880
Upgrade to lucene-5.4-snapshot-1710880.
2015-10-28 14:45:45 +01:00
Christoph Büscher fdcfc7e817 Fix CI failure for recent commit in SimpleQueryStringBuilder 2015-10-28 13:58:36 +01:00