Commit Graph

62 Commits

Author SHA1 Message Date
Simon Willnauer 90ff1ad5fb Merge pull request #15235 from s1monw/remove_recovery_settings
Restore chunksize of 512kb on recovery and remove configurability
2015-12-11 15:27:20 +01:00
Adrien Grand d408313974 Make XContentGenerator.writeRaw* safer.
This method currently allows to write arbitrary bytes in an xcontent stream.
I changed it so that it can only write data to the same stream as the xcontent
(the bos parameter is removed) and that it yells at you if you try to write
raw bytes that can't be recognized as xcontent. Also the logic to copy the
structure instead of appending the bytes directly if the source and target
are of a different xcontent type have been moved to the low-level
XContentGenerator.
2015-12-11 11:13:57 +01:00
Ryan Ernst 93de1ed606 Merge pull request #15354 from rjernst/just_node
Remove NodeBuilder
2015-12-10 19:19:55 -08:00
Robert Muir 2741888498 Remove RuntimePermission("accessDeclaredMembers")
Upgrades lucene to 5.5.0-1719088, randomizedtesting to 2.3.2, and securemock to 1.2
2015-12-10 14:26:55 -05:00
Ryan Ernst 6f166dd0ff Remove NodeBuilder
The NodeBuilder is currently used to construct a Node. However, this is
really just yet-another-builder that wraps around a Settings.Builder
witha couple convenience methods. But there are very few uses of these
convenience methods.  This change removes NodeBuilder, in favor of just
using the Node constructor.
2015-12-10 00:01:44 -08:00
Robert Muir a6fe9a4212 move test back now that its fixed 2015-12-08 23:42:18 -05:00
Ryan Ernst 907d3cf878 move template query builder tests back to core (using mockscript) 2015-12-08 20:26:45 -08:00
Robert Muir 2e299860e5 refactor mustache to lang-mustache plugin. No rest tests yet. 2015-12-08 22:45:21 -05:00
Ryan Ernst 6f996e9e26 Merge pull request #15300 from rjernst/tribe_cleanup
Fix tribe node to load config file for internal client nodes
2015-12-08 08:07:26 -08:00
Britta Weber e0aa481bf5 Merge pull request #15213 from brwe/copy-to-in-multi-fields-exception
throw exception if a copy_to is within a multi field

Copy to within multi field is ignored from 2.0 on, see #10802.
Instead of just ignoring it, we should throw an exception if this
is found in the mapping when a mapping is added. For already
existing indices we should at least log a warning.
We remove the copy_to in any case.

related to #14946
2015-12-08 14:41:07 +01:00
Simon Willnauer 414c04eb66 Restore chunksize of 512kb on recovery and remove configurability
This commit restores the chunk size of 512kb lost in a previous but unreleased
refactoring. At the same time it removes the configurability of:
 * `indices.recovery.file_chunk_size` - now fixed to 512kb
 * `indices.recovery.translog_ops` - removed without replacement
 * `indices.recovery.translog_size` - now fixed to 512kb
 * `indices.recovery.compress` - file chunks are not compressed due to lucene's compression but translog operations are.

The compress option is gone entirely and compression is used where it makes sense. On sending files of the index
we don't compress as we rely on the lucene compression for stored fields etc.

Relates to #15161
2015-12-08 11:20:55 +01:00
Ryan Ernst 25d60e152f Tribe: Fix tribe node to load config file for internal client nodes
The tribe node creates one local client node for each cluster it
connects to. Refactorings in #13383 broke this so that each local client
node now tries to load the full elasticsearch.yml that the real tribe
node uses.

This change fixes the problem by adding a TribeClientNode which is a
subclass of Node. The Environment the node uses is now passed in (in
place of Settings), and the TribeClientNode simply does not use
InternalSettingsPreparer.prepareEnvironment.

The tests around tribe nodes are not great. The existing tests pass, but
I also manually tested by creating 2 local clusters, and configuring and
starting a tribe node. With this I was able to see in the logs the tribe
node connecting to each cluster.

closes #13383
2015-12-07 20:18:06 -08:00
Ryan Ernst 3d9d8bd45a Build: Remove duplicate runs of packaged rest tests
We currently use the full suite of packaged rest tests for each
distribution. We also used to run rest tests within core integ tests,
but this stopped working when we split out the test-framework, since the
test files are in there.

This change simplifies the code to run packaged rest tests just once,
for the integ-test-zip, and removes the unused rest tests from
test-framework. Distributions rest tests now check that all modules
were loaded.
2015-12-04 13:43:40 -08:00
Robert Muir 699b140d69 update lists, rename stuff to be clearer, remove public method that is unused 2015-12-04 01:43:07 -05:00
Robert Muir 7160c5ec15 list modules separately in pluginservice 2015-12-04 01:13:17 -05:00
Ryan Ernst f8026ed8fd Build: Fix dependency licenses check to correctly skip projects without dependencies 2015-12-02 11:03:56 -08:00
Jason Tedor 05430a788a Remove and forbid use of the type-unsafe empty Collections fields
This commit removes and now forbids all uses of the type-unsafe empty
Collections fields Collections#EMPTY_LIST, Collections#EMPTY_MAP, and
Collections#EMPTY_SET. The type-safe methods Collections#emptyList,
Collections#emptyMap, and Collections#emptySet should be used instead.
2015-12-02 10:41:59 -05:00
Jason Tedor 72e18ec681 Add builder to create cluster state executor results 2015-11-30 08:59:24 -05:00
Jason Tedor 7caee2fa4d Explicitly correspond cluster state tasks and execution results 2015-11-30 08:59:23 -05:00
Boaz Leskes f0f89e708d Split cluster state update tasks into roles
This commit splits cluster state update tasks into roles. Those roles
are:
 - task info
 - task configuration
 - task executor
 - task listener

All tasks that have the same executor will be executed in batches. This
removes the need for local batching as was previously in
MetaDataMappingService.

Additionally, this commit reintroduces batching on mapping update calls.

Relates #13627
2015-11-30 08:59:23 -05:00
Jason Spafford c7112bd866 Changed some of the language in documentation
Closes #14920
2015-11-28 16:00:29 +01:00
Lee Hinman a25b407aeb Add support for headers in REST tests
This adds support for arbitrary headers sent with each REST request, it
will allow us to test things like different xcontent-encoding (see
50_with_headers.yaml for what this looks like).

Headers are specified at the same level as `catch`, so a request would
look like:

```yaml
- do:
    headers:
      Content-Type: application/yaml
    get:
      index: test_1
      type:  _all
      id:    1
```
2015-11-24 08:25:02 -07:00
Boaz Leskes c1264b926f Improve trace logging in TransportReplicationAction and error reporting at RecoveryWhileUnderLoadIT
Things that helped me traced down an issue.

Closes #14931
2015-11-23 13:52:32 +01:00
Boaz Leskes 4a6f3c7840 Make sure the remaining delay of unassigned shard is updated with every reroute
For example: if a node left the cluster and an async store fetch was triggered. In that time no shard is marked as delayed (and strictly speaking it's not yet delayed). This caused test for shard delays post node left to fail. see : http://build-us-00.elastic.co/job/es_core_master_windows-2012-r2/2074/testReport/

 To fix this, the delay update is now done by the Allocation Service, based of a fixed time stamp that is determined at the beginning of the reroute.

 Also, this commit fixes a bug where unassigned info instances were reused across shard routings, causing calculated delays to be leaked.

Closes #14890
2015-11-23 13:16:15 +01:00
Robert Muir 30529c008d Ban write access to system properties
* Forbid System.setProperties & co in forbidden APIs.
* Ban property write access at runtime with security manager.

Plugins that need to modify system properties will need to request permission in their plugin-security.policy
2015-11-21 22:33:06 -05:00
Ryan Ernst 88ee2e23a6 Merge branch 'master' into javadocs 2015-11-20 12:20:59 -08:00
Adrien Grand 4602b8177b Move AvgTests back to core.
This makes AvgTests use a mock plugin engine. I also removed the
textScriptExplicit* methods for the base class since they only make sense for
a groovy script, not a mock script.
2015-11-20 11:58:38 +01:00
Ryan Ernst c3cb1fd08c Merge branch 'master' into javadocs 2015-11-19 10:43:43 -08:00
Daniel Mitterdorfer aeecce8072 Use a specific matcher implementation for REST test blacklists
With this commit we replace the previously used PathMatcher
from the JDK with a specific matcher that is implemented for
this purpose and supports only simple globbing patterns
(i.e. *).

Closes #11391
2015-11-19 15:29:05 +01:00
Yannick Welsch 6a2fa73fb5 Merge pull request #14808 from ywelsch/refactor/delayed-allocation
Simplify delayed shard allocation
2015-11-19 09:58:52 +01:00
Yannick Welsch 2084df825f Simplify delayed shard allocation
- moves calculation of the delay to a single place (ReplicaShardAllocator)
- reduces coupling between GatewayAllocator and RoutingService
- in master failover situations, elapsed delay time is forgotten

Closes #14808
2015-11-19 09:53:07 +01:00
Daniel Mitterdorfer 96724e198a Restore thread interrupt flag after an InterruptedException
This commit replaces all occurrences of Thread.interrupted() with
Thread.currentThread().interrupt(). While the former checks and clears the current
thread's interrupt flag the latter sets it, which is actually intended.

Closes #14798
2015-11-18 17:11:53 +01:00
Daniel Mitterdorfer 267f726a7f Merge remote-tracking branch 'danielmitterdorfer/feature/11657' 2015-11-16 15:15:41 +01:00
Daniel Mitterdorfer e31d66d137 Log cluster health status changes
With this commit the cluster health status changes are logged
on INFO level. The change is only logged on master and actively
triggered in AllocationService in order to minimize the impact of
constantly reevaluating ClusterState in a ClusterStateListener
although we know that no health-relevant change happened.

Closes #11657
2015-11-16 15:14:43 +01:00
Nik Everett b9c3516d9a Merge pull request #14756 from nik9000/leaking_clients
Don't leak clients from ExternalTestCluster
2015-11-16 08:54:06 -05:00
Yannick Welsch 56f543734e [TEST] Specify usage restrictions for @SuiteScopeTestCase, @SuppressLocalMode and @SuppressNetworkMode
Closes #14477
2015-11-16 14:34:13 +01:00
Martijn van Groningen 293b2f6345 field data: Don't cache top level field data for fields that don't exist.
Just return an empty field data instance, like we do currently the same for segment level field data.
2015-11-16 06:27:48 +01:00
Boaz Leskes eff82fdbcd Tests: update the disco node id seed on node restart
To make sure the new node's id is changed.

Closes #14675
2015-11-15 21:49:54 +01:00
Ryan Ernst 4b17492456 Build: Add javadocs jars
This change adds javadoc jars to core, test-framework and plugins. There
were a couple issues which javadoc found, but doclint did not already
find.
2015-11-15 01:44:42 -08:00
Nik Everett 1870820208 Don't leak clients from ExternalTestCluster
When the ExternalTestCluster can't fully build it was leaking a client.
These clients created several threads each, causing "fun" thread starvation
issues.
2015-11-13 17:20:08 -05:00
Boaz Leskes ac0da91bf7 Extend usage of IndexSetting class
I decided to leave external listeners (used by plugins) alone, for now.

Closes #14731
2015-11-13 14:30:23 +01:00
Robert Muir 776bb288b5 fix gradle check under jigsaw
closes #14726

Squashed commit of the following:

commit 5b591e98570e3fa481b2816a44063b98bff36ddf
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Nov 13 00:54:08 2015 -0500

    add assumption for self-signing in PluginManagerTests

commit ed11e5371b6f71591dc41c6f60d033502cfcf029
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Nov 13 00:20:59 2015 -0500

    show error output from integ test startup

commit d8b187a10e95d89a0e775333dcbe1aaa903fb376
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Nov 12 22:14:11 2015 -0500

    fix gradle check under jigsaw
2015-11-13 00:57:12 -05:00
Robert Muir 720ebe347d Handle system policy correctly.
Just suck in the system policy, so its compatible with any version of java.
It means it also respects configuration (e.g. for monitoring agents)

Closes #14704
2015-11-12 18:08:13 -05:00
Yannick Welsch 4729386347 Merge pull request #14494 from ywelsch/fix/delayed-allocation-reroute
Delayed allocation can miss a reroute
2015-11-12 17:58:08 +01:00
Yannick Welsch c32de1f72b [TEST] Use TransportService/TransportAddress instead of DiscoveryNode for disruption rules
The disruption rules are changed to work on all transport addresses that are bound by a node (not only publish address).
This is important as UnicastZenPing creates fake DiscoveryNode instances which match one of the bound addresses and not necessarily the publish address.

Closes #14625
Closes #14653
2015-11-12 17:20:07 +01:00
Yannick Welsch 0220e45e2f Fix missing reroute in case of multiple delayed shards
After a delayed reroute of a shard, RoutingService misses to schedule a new delayed reroute of other delayed shards.

Closes #14494
Closes #14010
Closes #14445
2015-11-12 10:57:40 +01:00
Ryan Ernst 4b5f87cb7d Build: Remove transitive dependencies
Transitive dependencies can be confusing and hard to deal with when
conflicts arise between them. This change removes transitive
dependencies from elasticsearch, and forces any dependency conflicts to
be resolved manually, instead of automatically by gradle.

closes #14627
2015-11-10 15:01:41 -08:00
Ryan Ernst 7bb4e10353 Merge pull request #14645 from rjernst/version_cleanup
Consolidate dependencies specified in multiple places
2015-11-10 07:49:10 -08:00
Simon Willnauer 17913c56fb Merge pull request #14591 from s1monw/issues/14387
Only allow rebalance operations to run if all shard store data is available
2015-11-10 12:06:31 +01:00
Ryan Ernst 7a6155e12f Build: Consolidate dependencies specified in multiple places
Some dependencies must be specified in a couple places in the build.
e.g. randomized runner is specified both in buildSrc (for the gradle
wrapper plugin), as well as in the test-framework.

This change creates buildSrc/versions.properties which acts similar to
the set of shared version properties we used to have in the maven parent
pom.
2015-11-09 23:59:56 -08:00