Commit Graph

1740 Commits

Author SHA1 Message Date
Martijn van Groningen eace065931 Merge pull request from martijnvg/fix_some_messy_tests
fixed some messy tests
2015-09-30 22:13:50 +02:00
Martijn van Groningen dc858d2008 test: fixed some messy tests 2015-09-30 21:48:17 +02:00
Simon Willnauer 5915309939 Fix part size calculation if all parts are of the same size
Relates to 
2015-09-30 21:29:00 +02:00
Christoph Büscher 97db3d5ef6 Tests: fix test by making copy of geopoints in assertLuceneQuery 2015-09-30 20:47:39 +02:00
Simon Willnauer 4886562d78 Don't get IndexShard instance in the executor but use already availale instance
We try to get the index shard instance again from the index service on a different
threads while that shard might have already been closed or removed which can cause a NPE
instead of another expected expecption.
2015-09-30 20:43:22 +02:00
Simon Willnauer 9d12c38f44 [TEST] Don't corrupt extraFS files 2015-09-30 20:02:55 +02:00
Simon Willnauer f2ae29a1cb [TEST] don't run check index on an intentionally corrupted index 2015-09-30 20:00:53 +02:00
Simon Willnauer e38f4cf01e Start making RecoverySourceHandler unittestable
This commit shuffels and rewrites some code in RecoverySourceHandler to make it
simpler and more unittestable. This commit doesn't change all parts of this class
neither is it fully tested yet. It's an important part of the infrastrucutre so I started
to make it better tested but I don't want to change everything in one go since it makes
review simpler and more detailed. Future commits will continue cleaning up the class and
add more tests.
2015-09-30 17:45:29 +02:00
Simon Willnauer f0a8c10c87 [TEST] only trim for the comparison 2015-09-30 17:00:39 +02:00
Boaz Leskes 169d06cf9e Internal: Remove the disabled autogenerated id optimization from InternalEngine
If a document is indexed into ES with no id, ES will generate one for it. We used to have an optimization for this case where the engine will not try to resolve the ids of these request in the existing index but immediately try to index them. This optimization has proven to be the source of brittle bugs (solved!) and we disabled it in 1.5, preparing for it to be removed if no performance degradation was found. Since we haven't seen any such degradation we can remove it.

Along with the removal of the optmization, we can remove the autogenerate id flag on indexing requests and the can have duplicate flag. The only downside of the removal of the canHaveDuplicate flag is that we can't make sure any more that when we retry an autogenerated id create operation we will ignore any document already exists exception (See  for background and discussion). To work around this, we don't set the operation to CREATE any more when we generate an id, so the resulting request will never fail when it finds an existing doc but do return a version of 2. I think that's acceptable.

Closes 
2015-09-30 16:16:51 +02:00
Simon Willnauer c185a12ef8 Verify actually written checksum in VerifyingIndexOutput
today we don't verify that the actual checksum written to VerifyingIndexOutput
is the actual checksum we are expecting.
2015-09-30 15:33:05 +02:00
Simon Willnauer 6fff824402 Remove ClusterSerivce and IndexSettingsService dependency from IndexShard
We have two unneded heavy dependencies on IndexShard that are unneeded and only cause
trouble if you try to mock index shard. This commit removes IndexSettingsService as well as
ClusterSerivce from IndexShard to simplify future mocking and construction.
2015-09-30 14:19:26 +02:00
Tanguy Leroux d715dfd16c Fix blob size in writeBlob() method 2015-09-30 12:33:14 +02:00
javanna a3abfab865 Query refactoring: set has_parent & has_child types context properly
While refactoring has_child and has_parent query we lost an important detail around types. The types that the inner query gets executed against shouldn't be the main types of the search request but the parent or child type set to the parent query. We used to use QueryParseContext#setTypesWithPrevious as part of XContentStructure class which has been deleted, without taking care though of setting the types and restoring them as part of the innerQuery#toQuery call.

Meanwhile also we make sure that the original context types are restored in PercolatorQueriesRegistry

Closes 
2015-09-30 12:07:26 +02:00
Robert Muir b8b8bdac40 Add 2.2.0 version to master 2015-09-29 19:50:13 -04:00
Robert Muir 6d8c035f70 Add SpecialPermission to guard exceptions to security policy.
Closes 

Squashed commit of the following:

commit 42c1166efc55adda0d13fed77de583c0973e44b3
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Sep 29 11:59:43 2015 -0400

    Add paranoia

    Groovy holds on to a classloader, so check it before compilation too.
    I have not reviewed yet what Rhino is doing, but just be safe.

commit b58668a81428e964dd5ffa712872c0a34897fc91
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Sep 29 11:46:06 2015 -0400

    Add SpecialPermission to guard exceptions to security policy.

    In some cases (e.g. buggy cloud libraries, scripting engines), we must
    grant dangerous permissions to contained cases. Those AccessController blocks
    are dangerous, since they truncate the stack, and can allow privilege escalation.

    This PR adds a simple permission to check before each one, so that unprivileged code
    like groovy scripts, can't do anything they shouldn't be allowed to do otherwise.
2015-09-29 17:32:56 -04:00
Robert Muir ad6bc5b94c Throw exception on unsupported OS, so we don't falsely report 'enabled' 2015-09-29 17:16:00 -04:00
Simon Willnauer bdc21078ca [TEST] don't copy write locks or extraFS files 2015-09-29 17:01:46 +02:00
Jason Tedor 464d48cda6 Merge pull request from jasontedor/workaround-jdk-8056984
Workaround JDK-8056984
2015-09-29 16:49:44 +02:00
Jason Tedor ba0643da2b Workaround JDK-8056984
This commit works around JDK bug JDK-8056984 in the javac compiler.
This bug is impacting CI compilations on JDK 8u25.
2015-09-29 16:45:57 +02:00
Robert Muir d54dd63825 Merge pull request from rmuir/lock_down_classloader_and_reflection
Clean up scripting permissions.
2015-09-29 10:35:52 -04:00
Simon Willnauer 3b5ed08d49 Refactor StoreRecoveryService to be a simple package private util class
StoreRecoveryService used to be a pretty heavy class with lots of dependencies.
This class was basically not testable in isolation and had an async API with a listener.
This commit refactors this class to be a simple utility classs with a sync API hidden behind
the IndexShard interface. It includes single node tests and moves all the async properities to
the caller side.
Note, this change also removes the mapping update on master from the store recovery code since
it's not needed anymore in 3.0 because all stores have been subject to sync mapping updates such
that the master already has all the mappings for documents that made it into the transaction log.

Closes 
2015-09-29 15:59:21 +02:00
Robert Muir eeeb42abef Clean up scripting permissions.
Now that groovy is factored out, we contain this dangerous stuff there.

TODO: look into those test hacks inspecting class protection domains, maybe we can
clean that one up too.

TODO: generalize the GroovyCodeSourcePermission to something all script engines check,
before entering accesscontrollerblocks. this way e.g. groovy script cannot coerce
python engine into creating something with more privs if it gets ahold of it... we
should probably protect the aws/gce hacks in the same way.
2015-09-29 09:54:12 -04:00
Jason Tedor 882fe0784c Merge pull request from jasontedor/workaround-jdk-8056014
Workaround JDK-8056014
2015-09-29 15:43:25 +02:00
Nik Everett 8a7f9edf02 Merge pull request from nik9000/immutable_set_be_gone_2
Finish banning ImmutableSet
2015-09-29 15:34:34 +02:00
Jason Tedor ad090296fc Workaround JDK-8056014
This commit works around JDK bug JDK-8056014 in the javac compiler.
This bug is impacting CI compilations on JDK 8u11 and 8u25.
2015-09-29 15:03:32 +02:00
Robert Muir 94972bb39d Merge pull request from rmuir/add_seatbelt
improve seccomp syscall filtering
2015-09-29 08:54:06 -04:00
Colin Goodheart-Smithe 8c7c7652f6 Make sure equivalent geohashCellQueries are equal after toQuery called
Previous to this change if to equal geohash cell query builders were created and then toQuery was called on one, they would no longer be equal.

This change also adds a test to AbstractQueryTestCase to make sure calling toQuery on any query builder does not affect the query builder's equality
2015-09-29 14:34:58 +02:00
Simon Willnauer e814102bfb Ensure XContent is consistent across platforms
Today we generate XContent with platform dependent linefeeds. This
commit makes the pretty-printed json etc. consistent with \n across all
platforms.
2015-09-29 14:22:33 +02:00
Simon Willnauer e3b38e5af1 Remove IndexService dep. from IndexShard
there is no reason for the index shard to hold on to it's corresponding
index service. This dependency is unnecessary.
2015-09-29 14:07:58 +02:00
Robert Muir cbf894a2e7 Add abstract TestScript to the test framework jar 2015-09-29 06:37:33 -04:00
Robert Muir 20a203b1a2 Merge pull request from rmuir/groovy_factor_out
Factor groovy out of core into lang-groovy
2015-09-29 05:58:21 -04:00
Tanguy Leroux 9cb27ad068 Update to Jackson 2.6.2 2015-09-29 10:20:54 +02:00
Boaz Leskes c1ab2dc244 Test: make sure TranslogTests.randomNonTranslogPatternString returns a reference to a file
It can return a directory now (like /  or ..)
2015-09-29 07:53:31 +02:00
Robert Muir e0d42739dd Factor groovy out of core into lang-groovy 2015-09-28 20:17:45 -04:00
Robert Muir d7a087a41b improve seccomp syscall filtering
* Add OS X support via "seatbelt" mechanism. This gives consistency across dev and prod, since many devs use OS X.
* block execveat system call: it may be new, but we should not allow it.
2015-09-28 11:35:07 -04:00
Nik Everett 85b99d2011 Finish banning ImmutableSet
Ban ImmutableSet$Builder because that let you sneak some `ImmutableSet`s in.

Remove all remaining imports of ImmutableSet.
2015-09-28 01:49:46 +02:00
Boaz Leskes 148265bd16 Internal: an inactive shard is temporarily activated by triggered synced flush
When a shard becomes in active we trigger a sync flush in order to speed up future recoveries. The sync flush causes a new translog generation to be made, which in turn confuses the IndexingMemoryController making it think that the shard is active. If no documents comes along in the next 5m, the shard is made inactive again , triggering a sync flush and so forth.

To avoid this, the IndexingMemoryController is changed to ignore empty translogs when checking if a shard became active. This comes with the price of potentially missing indexing operations which are followed by a flush. This is acceptable as if no more index operation come in, it's OK to leave the shard in active.

A new unit test is introduced and comparable integration tests are removed.

Closes 
2015-09-27 07:16:54 +02:00
Simon Willnauer ebe02ec54a [TEST] Make test pass on windows - jackson uses platform line.separator which messes up comparisons 2015-09-26 21:25:49 +02:00
Nik Everett 892d6b3b89 Merge branch 'master' into immutable_set_be_gone 2015-09-25 21:05:51 -04:00
Jason Tedor f40ae25352 Another sync of ElasticsearchException ids with 2.0 2015-09-25 16:43:01 -04:00
Jason Tedor 5571c327c5 Sync ElasticsearchException ids with 2.0 2015-09-25 16:24:49 -04:00
Jason Tedor 65577816de Merge pull request from s1monw/remove_reflection_hacks_from_es_exception
Remove reflection hacks from ElasticsearchException
2015-09-25 15:02:48 -04:00
Robert Muir aa8d1740e4 Don't log multi-megabyte guice exceptions.
Instead just log the same thing we print to the startup console for that case (magic logic),
it sucks to do this, but guice exceptions are too much.

All other non-guice exceptions will still be fully logged.

Closes 
2015-09-25 15:02:16 -04:00
Jason Tedor 75ecc54f33 Add clarifying comment on removing an ElasticsearchException 2015-09-25 14:59:10 -04:00
Jason Tedor ab19bfbe9b Removing and reordering ElasticsearchExceptions should be okay 2015-09-25 14:43:40 -04:00
Jason Tedor 752b4798d1 Use explicit ids for ElasticsearchExceptions
This commit adds explicit ids for managing ElasticsearchException
serialization. By adding explicit ids and unit tests for them, the ids
are less brittle and breakage can be more clearly detected.
2015-09-25 14:32:44 -04:00
Boaz Leskes 00d8a84475 Test: reduce size in ByteSizeValueTests.testEquality to avoid double rounding issues 2015-09-25 20:24:11 +02:00
Boaz Leskes 5a0f2fd1ac ByteSizeValue.equals should normalize units
currently ByteSizeValue.parse("1GB") is not equal to ByteSizeValue.parse("1024MB")

Closes 
2015-09-25 20:05:13 +02:00
Simon Willnauer f4ff6647ad Remove reflection hacks from ElasticsearchException
Today we use reflection where it's not needed anymore since java8 can
pass ctors around. This commit replaces runtime checks with compile time
checks which is always preferrable.
2015-09-25 16:42:42 +02:00