Commit Graph

77 Commits

Author SHA1 Message Date
Yannick Welsch 3a442db9bd Allocate primary shards based on allocation ids
Closes #15281
2015-12-17 15:55:50 +01:00
Robert Muir a7cc91e868 Merge pull request #15501 from rmuir/sheisty_classes
thirdPartyAudit round 2
2015-12-17 03:44:27 -05:00
Robert Muir 6692e42d9a thirdPartyAudit round 2
This fixes the `lenient` parameter to be `missingClasses`. I will remove this boolean and we can handle them via the normal whitelist.
It also adds a check for sheisty classes (jar hell with the jdk).
This is inspired by the lucene "sheisty" classes check, but it has false positives. This check is more evil, it validates every class file against the extension classloader as a resource, to see if it exists there. If so: jar hell.

This jar hell is a problem for several reasons:

1. causes insanely-hard-to-debug problems (like bugs in forbidden-apis)
2. hides problems (like internal api access)
3. the code you think is executing, is not really executing
4. security permissions are not what you think they are
5. brings in unnecessary dependencies
6. its jar hell

The more difficult problems are stuff like jython, where these classes are simply 'uberjared' directly in, so you cant just fix them by removing a bogus dependency. And there is a legit reason for them to do that, they want to support java 1.4.
2015-12-17 02:35:00 -05:00
Jack Conradson 4523eaec88 Added plumbing for compile time script parameters.
Closes #15464
2015-12-16 18:29:21 -08:00
Ryan Ernst a1312a5350 Merge pull request #15495 from rjernst/disable_mock_plugins
Allow integ tests to exclude mock plugins
2015-12-16 18:22:04 -08:00
Ryan Ernst 17a806631d Test: Allow integ tests to exclude mock plugins
We currently randomly add a set of mock plugins to integ tests.
Sometimes it is necessary to omit this mock plugins, but other times you
may just want to suppress a particular mock plugin. For example, if you
have your own transport, you want to omit the asserting local transport
mock, since they would both try to set the transport.type.
2015-12-16 17:59:43 -08:00
Robert Muir 4f9d4103f2 Merge pull request #15491 from rmuir/forbidden_third_party
Add gradle thirdPartyAudit to precommit tasks
2015-12-16 18:56:50 -05:00
Robert Muir ee79d46583 Add gradle thirdPartyAudit to precommit tasks 2015-12-16 16:38:16 -05:00
Ryan Ernst a2b8f4b90a Merge pull request #15434 from rjernst/http_type
Expose http.type setting, and collapse al(most all) modules relating to transport/http
2015-12-16 11:54:30 -08:00
Britta Weber db357f078a Merge pull request #15396 from brwe/java-api-for-synced-flush
Add java API for synced flush

closes #12812
2015-12-16 17:18:18 +01:00
Lee Hinman 91b55ded6a Trace log REST test headers 2015-12-15 08:48:43 -07:00
Ryan Ernst 60d35c81af Plugins: Expose http.type setting, and collapse al(most all) modules relating to transport/http
This change adds back the http.type setting. It also cleans up all the
transport related guice code to be consolidated within the
NetworkModule (as transport and http related stuff is what and how ES
exposes over the network). The setter methods previously used by some
plugins to override eg the TransportService or HttpServerTransport are
removed, and those plugins should now register a custom implementation
of the class with a name and set that using the appropriate config
setting. Note that I think ActionModule should also be moved into here,
to sit along side the rest actions, but I left that for a followup.

closes #14148
2015-12-14 22:01:04 -08:00
Ryan Ernst a0c69fe7f9 Remove forbidden suppressions for InetSocketAddress 2015-12-11 18:20:09 -08:00
Ryan Ernst 5c8a0da1fd Build: Change rest integ tests to not have hardcoded ports
This change removes hardcoded ports from cluster formation. It passes
port 0 for http and transport, and then uses a special property to have
the node log the ports used for http and transport (just for tests).
This does not yet work for multi node tests. This brings us one step
closer to working with --parallel.
2015-12-11 17:36:08 -08:00
Jason Tedor 3383c24be0 Remove and forbid use of Collections#shuffle(List) and Random#<init>()
This commit removes and now forbids all uses of
Collections#shuffle(List) and Random#<init>() across the codebase. The
rationale for removing and forbidding these methods is to increase test
reproducibility. As these methods use non-reproducible seeds, production
code and tests that rely on these methods contribute to
non-reproducbility of tests.

Instead of Collections#shuffle(List) the method
Collections#shuffle(List, Random) can be used. All that is required then
is a reproducible source of randomness. Consequently, the utility class
Randomness has been added to assist in creating reproducible sources of
randomness.

Instead of Random#<init>(), Random#<init>(long) with a reproducible seed
or the aforementioned Randomess class can be used.

Closes #15287
2015-12-11 11:16:38 -05:00
Simon Willnauer 90ff1ad5fb Merge pull request #15235 from s1monw/remove_recovery_settings
Restore chunksize of 512kb on recovery and remove configurability
2015-12-11 15:27:20 +01:00
Adrien Grand d408313974 Make XContentGenerator.writeRaw* safer.
This method currently allows to write arbitrary bytes in an xcontent stream.
I changed it so that it can only write data to the same stream as the xcontent
(the bos parameter is removed) and that it yells at you if you try to write
raw bytes that can't be recognized as xcontent. Also the logic to copy the
structure instead of appending the bytes directly if the source and target
are of a different xcontent type have been moved to the low-level
XContentGenerator.
2015-12-11 11:13:57 +01:00
Ryan Ernst 93de1ed606 Merge pull request #15354 from rjernst/just_node
Remove NodeBuilder
2015-12-10 19:19:55 -08:00
Robert Muir 2741888498 Remove RuntimePermission("accessDeclaredMembers")
Upgrades lucene to 5.5.0-1719088, randomizedtesting to 2.3.2, and securemock to 1.2
2015-12-10 14:26:55 -05:00
Ryan Ernst 6f166dd0ff Remove NodeBuilder
The NodeBuilder is currently used to construct a Node. However, this is
really just yet-another-builder that wraps around a Settings.Builder
witha couple convenience methods. But there are very few uses of these
convenience methods.  This change removes NodeBuilder, in favor of just
using the Node constructor.
2015-12-10 00:01:44 -08:00
Robert Muir a6fe9a4212 move test back now that its fixed 2015-12-08 23:42:18 -05:00
Ryan Ernst 907d3cf878 move template query builder tests back to core (using mockscript) 2015-12-08 20:26:45 -08:00
Robert Muir 2e299860e5 refactor mustache to lang-mustache plugin. No rest tests yet. 2015-12-08 22:45:21 -05:00
Ryan Ernst 6f996e9e26 Merge pull request #15300 from rjernst/tribe_cleanup
Fix tribe node to load config file for internal client nodes
2015-12-08 08:07:26 -08:00
Britta Weber e0aa481bf5 Merge pull request #15213 from brwe/copy-to-in-multi-fields-exception
throw exception if a copy_to is within a multi field

Copy to within multi field is ignored from 2.0 on, see #10802.
Instead of just ignoring it, we should throw an exception if this
is found in the mapping when a mapping is added. For already
existing indices we should at least log a warning.
We remove the copy_to in any case.

related to #14946
2015-12-08 14:41:07 +01:00
Simon Willnauer 414c04eb66 Restore chunksize of 512kb on recovery and remove configurability
This commit restores the chunk size of 512kb lost in a previous but unreleased
refactoring. At the same time it removes the configurability of:
 * `indices.recovery.file_chunk_size` - now fixed to 512kb
 * `indices.recovery.translog_ops` - removed without replacement
 * `indices.recovery.translog_size` - now fixed to 512kb
 * `indices.recovery.compress` - file chunks are not compressed due to lucene's compression but translog operations are.

The compress option is gone entirely and compression is used where it makes sense. On sending files of the index
we don't compress as we rely on the lucene compression for stored fields etc.

Relates to #15161
2015-12-08 11:20:55 +01:00
Ryan Ernst 25d60e152f Tribe: Fix tribe node to load config file for internal client nodes
The tribe node creates one local client node for each cluster it
connects to. Refactorings in #13383 broke this so that each local client
node now tries to load the full elasticsearch.yml that the real tribe
node uses.

This change fixes the problem by adding a TribeClientNode which is a
subclass of Node. The Environment the node uses is now passed in (in
place of Settings), and the TribeClientNode simply does not use
InternalSettingsPreparer.prepareEnvironment.

The tests around tribe nodes are not great. The existing tests pass, but
I also manually tested by creating 2 local clusters, and configuring and
starting a tribe node. With this I was able to see in the logs the tribe
node connecting to each cluster.

closes #13383
2015-12-07 20:18:06 -08:00
Ryan Ernst 3d9d8bd45a Build: Remove duplicate runs of packaged rest tests
We currently use the full suite of packaged rest tests for each
distribution. We also used to run rest tests within core integ tests,
but this stopped working when we split out the test-framework, since the
test files are in there.

This change simplifies the code to run packaged rest tests just once,
for the integ-test-zip, and removes the unused rest tests from
test-framework. Distributions rest tests now check that all modules
were loaded.
2015-12-04 13:43:40 -08:00
Robert Muir 699b140d69 update lists, rename stuff to be clearer, remove public method that is unused 2015-12-04 01:43:07 -05:00
Robert Muir 7160c5ec15 list modules separately in pluginservice 2015-12-04 01:13:17 -05:00
Ryan Ernst f8026ed8fd Build: Fix dependency licenses check to correctly skip projects without dependencies 2015-12-02 11:03:56 -08:00
Jason Tedor 05430a788a Remove and forbid use of the type-unsafe empty Collections fields
This commit removes and now forbids all uses of the type-unsafe empty
Collections fields Collections#EMPTY_LIST, Collections#EMPTY_MAP, and
Collections#EMPTY_SET. The type-safe methods Collections#emptyList,
Collections#emptyMap, and Collections#emptySet should be used instead.
2015-12-02 10:41:59 -05:00
Jason Tedor 72e18ec681 Add builder to create cluster state executor results 2015-11-30 08:59:24 -05:00
Jason Tedor 7caee2fa4d Explicitly correspond cluster state tasks and execution results 2015-11-30 08:59:23 -05:00
Boaz Leskes f0f89e708d Split cluster state update tasks into roles
This commit splits cluster state update tasks into roles. Those roles
are:
 - task info
 - task configuration
 - task executor
 - task listener

All tasks that have the same executor will be executed in batches. This
removes the need for local batching as was previously in
MetaDataMappingService.

Additionally, this commit reintroduces batching on mapping update calls.

Relates #13627
2015-11-30 08:59:23 -05:00
Jason Spafford c7112bd866 Changed some of the language in documentation
Closes #14920
2015-11-28 16:00:29 +01:00
Lee Hinman a25b407aeb Add support for headers in REST tests
This adds support for arbitrary headers sent with each REST request, it
will allow us to test things like different xcontent-encoding (see
50_with_headers.yaml for what this looks like).

Headers are specified at the same level as `catch`, so a request would
look like:

```yaml
- do:
    headers:
      Content-Type: application/yaml
    get:
      index: test_1
      type:  _all
      id:    1
```
2015-11-24 08:25:02 -07:00
Boaz Leskes c1264b926f Improve trace logging in TransportReplicationAction and error reporting at RecoveryWhileUnderLoadIT
Things that helped me traced down an issue.

Closes #14931
2015-11-23 13:52:32 +01:00
Boaz Leskes 4a6f3c7840 Make sure the remaining delay of unassigned shard is updated with every reroute
For example: if a node left the cluster and an async store fetch was triggered. In that time no shard is marked as delayed (and strictly speaking it's not yet delayed). This caused test for shard delays post node left to fail. see : http://build-us-00.elastic.co/job/es_core_master_windows-2012-r2/2074/testReport/

 To fix this, the delay update is now done by the Allocation Service, based of a fixed time stamp that is determined at the beginning of the reroute.

 Also, this commit fixes a bug where unassigned info instances were reused across shard routings, causing calculated delays to be leaked.

Closes #14890
2015-11-23 13:16:15 +01:00
Robert Muir 30529c008d Ban write access to system properties
* Forbid System.setProperties & co in forbidden APIs.
* Ban property write access at runtime with security manager.

Plugins that need to modify system properties will need to request permission in their plugin-security.policy
2015-11-21 22:33:06 -05:00
Ryan Ernst 88ee2e23a6 Merge branch 'master' into javadocs 2015-11-20 12:20:59 -08:00
Adrien Grand 4602b8177b Move AvgTests back to core.
This makes AvgTests use a mock plugin engine. I also removed the
textScriptExplicit* methods for the base class since they only make sense for
a groovy script, not a mock script.
2015-11-20 11:58:38 +01:00
Ryan Ernst c3cb1fd08c Merge branch 'master' into javadocs 2015-11-19 10:43:43 -08:00
Daniel Mitterdorfer aeecce8072 Use a specific matcher implementation for REST test blacklists
With this commit we replace the previously used PathMatcher
from the JDK with a specific matcher that is implemented for
this purpose and supports only simple globbing patterns
(i.e. *).

Closes #11391
2015-11-19 15:29:05 +01:00
Yannick Welsch 6a2fa73fb5 Merge pull request #14808 from ywelsch/refactor/delayed-allocation
Simplify delayed shard allocation
2015-11-19 09:58:52 +01:00
Yannick Welsch 2084df825f Simplify delayed shard allocation
- moves calculation of the delay to a single place (ReplicaShardAllocator)
- reduces coupling between GatewayAllocator and RoutingService
- in master failover situations, elapsed delay time is forgotten

Closes #14808
2015-11-19 09:53:07 +01:00
Daniel Mitterdorfer 96724e198a Restore thread interrupt flag after an InterruptedException
This commit replaces all occurrences of Thread.interrupted() with
Thread.currentThread().interrupt(). While the former checks and clears the current
thread's interrupt flag the latter sets it, which is actually intended.

Closes #14798
2015-11-18 17:11:53 +01:00
Daniel Mitterdorfer 267f726a7f Merge remote-tracking branch 'danielmitterdorfer/feature/11657' 2015-11-16 15:15:41 +01:00
Daniel Mitterdorfer e31d66d137 Log cluster health status changes
With this commit the cluster health status changes are logged
on INFO level. The change is only logged on master and actively
triggered in AllocationService in order to minimize the impact of
constantly reevaluating ClusterState in a ClusterStateListener
although we know that no health-relevant change happened.

Closes #11657
2015-11-16 15:14:43 +01:00
Nik Everett b9c3516d9a Merge pull request #14756 from nik9000/leaking_clients
Don't leak clients from ExternalTestCluster
2015-11-16 08:54:06 -05:00