Commit Graph

16799 Commits

Author SHA1 Message Date
Robert Muir 9c109cae0d Fix SeccompTests bug on older kernels
This test failed only on a feature branch, because that feature branch has
a different build randomization script and is the only place
randomizing tests.security.manager (this test cannot run with it enabled).

On old kernels without TSYNC support, the test fails because (surprise to me) the thread that
runs the test is not the same thread that runs static initializers:
7571489190/randomized-runner/src/main/java/com/carrotsearch/randomizedtesting/RandomizedRunner.java (L573-L574)

To fix this test (its not an issue in practice, since we do this before creating threadpools),
we just record for testing purposes that we couldn't TSYNC, and re-run the whole thing for the test thread
in setUp(), failing if something goes wrong.

Also add a bunch of additional paranoia and narrow our defensive checks better here after reading
through more chrome bug reports: they don't impact us but those linux distros are too cowboy
with the backports and the spirit of the checks makes me feel better.
2015-10-15 01:40:32 -04:00
Jason Tedor 68561ce55c Merge pull request #14119 from jasontedor/index-meta-data-clean-up
Cleanup IndexMetaData
2015-10-14 19:29:08 -04:00
Nicholas Knize ceefe2e91a remove centroid calculation from GeoHashGridAggregation 2015-10-14 16:19:28 -05:00
Nicholas Knize b31d3ddd3e Adds geo_centroid metric aggregator
This commit adds a new metric aggregator for computing the geo_centroid over a set of geo_point fields. This can be combined with other aggregators (e.g., geohash_grid, significant_terms) for computing the geospatial centroid based on the document sets from other aggregation results.
2015-10-14 16:19:09 -05:00
Jason Tedor 88dd3256fb Cleanup IndexMetaData
This commit cleans up IndexMetaData. In particular, all duplicate
getters (X and getX) have been collapsed into one (getX). Further, the
number of shards and number of replicas settings are now parsed once
and saved off as fields.
2015-10-14 14:59:35 -04:00
Robert Muir 5d001d1578 Decentralize plugin security
* Add ability for plugins to declare additional permissions with a custom plugin-security.policy file and corresponding AccessController logic. See the plugin author's guide for more information.
* Add warning messages to users for extra plugin permissions in bin/plugin.
* When bin/plugin is run interactively (stdin is a controlling terminal and -b/--batch not supplied), require user confirmation.
* Improve unit test and IDE support for plugins with additional permissions by exposing plugin's metadata as a maven test resource.

Closes #14108

Squashed commit of the following:

commit cf8ace65a7397aaccd356bf55f95d6fbb8bb571c
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Oct 14 13:36:05 2015 -0400

    fix new unit test from master merge

commit 9be3c5aa38f2d9ae50f3d54924a30ad9cddeeb65
Merge: 2f168b8 7368231
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Oct 14 12:58:31 2015 -0400

    Merge branch 'master' into off_my_back

commit 2f168b8038e32672f01ad0279fb5db77ba902ae8
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Oct 14 12:56:04 2015 -0400

    improve plugin author documentation

commit 6e6c2bfda68a418d92733ac22a58eec35508b2d0
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Oct 14 12:52:14 2015 -0400

    move security confirmation after 'plugin already installed' check, to prevent user from answering unnecessary questions.

commit 08233a2972554afef2a6a7521990283102e20d92
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Oct 14 05:36:42 2015 -0400

    Add documentation and pluginmanager support

commit 05dad86c51488ba43ccbd749f0164f3fbd3aee62
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Oct 14 02:22:24 2015 -0400

    Decentralize plugin permissions (modulo docs and pluginmanager work)
2015-10-14 14:46:45 -04:00
javanna 736823163f Plugins: plugin script to set proper plugin bin dir attributes
This commit makes sure that the plugin script looks at user, group and permissions of the elasticsearch bin dir and copies them over to the plugin bin subdirectory, whatever they are, so that they get properly setup depending on how elasticsearch was installed. We also make sure that execute permissions are added for files (we already did this before).

Relates to #11016
Closes #14088
2015-10-14 14:59:38 +02:00
Britta Weber f07853b56c use lambdas 2015-10-14 11:36:02 +02:00
Britta Weber 4f536945de remove unneeded Exception and handling thereof 2015-10-14 11:32:57 +02:00
Simon Willnauer 461ce98ff5 Merge pull request #14107 from s1monw/release_searcher_on_wrapper_failure
Ensure searcher is release if wrapping fails
2015-10-14 10:57:35 +02:00
Simon Willnauer da12290528 Merge pull request #14105 from s1monw/upgrade_netty_3_10_5_final
Upgrade Netty to 3.10.5.final
2015-10-14 10:57:05 +02:00
Simon Willnauer f4eab500aa Upgrade Netty to 3.10.5.final
This fixes a critical bug in SSL handling.

See http://netty.io/news/2015/10/13/3-10-5-Final.html
2015-10-14 10:56:38 +02:00
Simon Willnauer c133bec4bd Ensure searcher is release if wrapping fails
Today we leak an index searcher if we fail to wrap the seacher in
IndexShard. This commit ensures that the seacher is released if the wrapper
throws an exception.

This commit also restructures some test to be more atomic and only test a single
feature / attribute of the wrapper.
2015-10-14 10:44:57 +02:00
Clinton Gormley 9fa4cc92e5 Merge pull request #14097 from timschlechter/patch-4
Added link to Swagger for Elasticsearch
2015-10-14 10:02:49 +02:00
Simon Willnauer 5828796848 Merge pull request #14084 from s1monw/close_the_wrapper_only
Streamline top level reader close listeners and forbid general usage
2015-10-14 09:39:30 +02:00
Simon Willnauer db710c576f add javadocs 2015-10-14 09:28:05 +02:00
Simon Willnauer 7999027bf5 apply review comments 2015-10-14 09:19:02 +02:00
Jason Tedor 2e445d3ede Do not pollute Cache with failed futures 2015-10-13 22:14:13 -04:00
Jason Tedor 1d0b93f766 Merge pull request #14091 from jasontedor/no-deadlocks-in-compute-if-absent
Avoid deadlocks in Cache#computeIfAbsent
2015-10-13 21:16:23 -04:00
Jason Tedor b154c84306 Avoid deadlocks in Cache#computeIfAbsent
This commit changes the behavior of Cache#computeIfAbsent to not invoke
load for a key under the segment lock. Instead, the synchronization
mechanism to ensure that load is invoked at most once per key is
through the use of a future. Under the segment lock, we put a future in
the cache and through this ensure that load is invoked at most once per
key. This will not lead to the same deadlock situation as before
because a dependent key load on the same thread can not be triggered
while the segment lock is held.

Closes #14090
2015-10-13 21:15:28 -04:00
Simon Willnauer cac073dafa enforce that wrappers delegate core cache key and ban getCombinedCoreAndDeletesKey() entirely 2015-10-13 23:31:25 +02:00
Tim Schlechter 7e2f558887 Added link to Swagger for Elasticsearch 2015-10-13 21:29:53 +02:00
Clinton Gormley 3e7201ef63 Merge pull request #14096 from speedplane/patch-2
Fixed a typo ("when when")
2015-10-13 21:17:09 +02:00
Simon Willnauer ec60018e34 add comment why and when we have a leaf reader in the warmer 2015-10-13 21:06:20 +02:00
Clinton Gormley 14b6314766 Update fuzzy-query.asciidoc
Closes #14092
2015-10-13 21:03:46 +02:00
Simon Willnauer 0ead0faa1b fix typo 2015-10-13 20:51:43 +02:00
Simon Willnauer 1dca0e8f9b Add simple tests to ensure we can load and unload global ordinal with a wrapped reader 2015-10-13 20:51:10 +02:00
Britta Weber f7693b694b fix access denied for shard deletion with WindowsFS
We randomly use the WindowsFS mock file system to simulate that
windows does not delete files if they are opened by some other
process and instead gives you java.io.IOException: access denied.
In the tests we also check if the shard was deleted while it is
being deleted. This check loads the mata data of the index
and therefore might hold on to a file while the node
is trying to delete it and deletion will fail then.

Instead we should just check if the directory was removed.

closes #13758
2015-10-13 18:33:53 +02:00
Simon Willnauer e3f00e302c Add more pickyness to index warming and searcher wrappping
this commit also fixes a bug where we wramed a leaf reader in a top level context
which caused atomic segment readers to be used in our top level caches.
2015-10-13 17:21:46 +02:00
Simon Willnauer dac1799305 Cast DirectoryReader only when really requested 2015-10-13 17:21:46 +02:00
Simon Willnauer ba8de12429 remove unnecessary cast 2015-10-13 17:21:46 +02:00
Simon Willnauer bd5ac9ce1a enforce DirectoryReader when globals are loaded 2015-10-13 17:21:46 +02:00
Simon Willnauer d3436ff592 Streamline top level reader close listeners and forbid general usage
IndexReader#addReaderCloseListener is very error prone when it comes to
caching and reader wrapping. The listeners are not delegated to the sub readers
nor can it's implementation change since it's final in the base class. This commit
only allows installing close listeners on the top level ElasticsearchDirecotryReader
which is known to work an has a defined lifetime which corresponds to its subreader.
This ensure that cachesa re cleared once the reader goes out of scope.
2015-10-13 17:21:18 +02:00
jaymode 077a401c28 [TEST] do not wipe excluded templates
In #13606, the ability to exclude a template from being wiped was added but a call to
wipeTemplates() was not removed, so all index templates were still being removed.

This change removes the wipeTemplates() call in the wipe method so that only the
non-excluded templates are removed.
2015-10-13 09:59:32 -04:00
javanna 4d7d29f65b Plugins: plugin script to set proper plugin config dir attributes
Depending on how elasticsearch is installed, we have two scenarios to take into account that relate to user, group and permissions assigned to the config directory:

1) deb/rpm package: /etc/elasticsearch is root:elasticsearch 750 and the plugin script is run from root user
2) tar/zip archive: es config dir is most likely elasticsearch:elasticsearch and the plugin script is most likely run from elasticsearch user

When the plugin script copies over the plugin config dir within the es config dir, it should take care of setting the proper user, group and permissions, which vary depending on how elasticsearch was installed in the first place. Should be root:elasticsearch 750 if installed from a package, or elasticsearch:elasticsearch if installed from an archive.

This commit makes sure that the plugin script looks at user, group and permissions of the config dir and copies them over to the plugin config subdirectory, whatever they are, so that they get properly setup depending on how elasticsearch was installed in the first place. We also make sure that execute permissions are left untouched for files.

Relates to #11016
Closes #14048
2015-10-13 15:56:26 +02:00
Britta Weber 1634c8364e Merge pull request #14085 from brwe/fix-score-accuracy
fix numerical issue in function score query
2015-10-13 15:21:04 +02:00
Jason Tedor b7e09582ce Merge pull request #14087 from jasontedor/vle-integral-arrays
Add methods for variable-length encoding integral arrays
2015-10-13 09:12:10 -04:00
Robert Muir ceb969a13e Make PerThreadIDAndVersionLookup per-segment
Closes #14070

Squashed commit of the following:

commit ad5829e590cf6150763c31886e97d69976b7f02d
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Oct 13 06:50:25 2015 -0400

    Add more tests for version map caching

commit 617333eefd6d4ddc99db491d0843827b1b5936b2
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Oct 12 13:25:51 2015 -0400

    fix javadocs ref

commit 9626dd02e01a2676538e1071e302d882fc870837
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Oct 12 13:23:16 2015 -0400

    make the version map cache per-segment
2015-10-13 09:00:04 -04:00
Yannick Welsch a550ebf381 Merge pull request #13998 from ywelsch/tests/transportmasternodeaction
Tests for TransportMasterNodeAction
2015-10-13 05:48:40 -07:00
Jason Tedor 2162172f30 Add methods for variable-length encoding integral arrays
This commit adds methods to serialize the elements of int and long
arrays using variable-length encodings. This can be useful for
serializing int and long arrays containing mostly non-negative “not
large” values in a compressed form.
2015-10-13 08:47:42 -04:00
Yannick Welsch c2df88a21d [TEST] Add tests for TransportMasterNodeAction
Closes #13998
2015-10-13 14:46:59 +02:00
Britta Weber d021b1bfeb assert that doc was actually found 2015-10-13 14:35:32 +02:00
Yannick Welsch 57770d4d6c Fix import in SharedClusterSnapshotRestoreIT 2015-10-13 13:56:46 +02:00
Yannick Welsch 889edd00f0 Merge pull request #13931 from ywelsch/fix/index-settings-restore
Snapshot restore and index creates should keep index settings and cluster blocks in sync
2015-10-13 04:36:02 -07:00
Yannick Welsch 63366add7d Parse Java system properties in plugin.bat
Closes #13616
2015-10-13 13:33:34 +02:00
Britta Weber 318dfba464 use same score computation for actual scoring and explanation
FiltersFunctionScoreQuery sums up scores and weights and scores as double but when
we explain we cannot get the double scores from the explanation of score
functions. as a result we cannot compute the exact score from the explanations
of the functions alone.
this commit makes the explanation more accurate but also causes the score to be
computed one additional time.
2015-10-13 13:30:53 +02:00
Robert Muir a68c163959 Merge pull request #14064 from rmuir/blinded_forever
Allow tests to run from eclipse on windows
2015-10-13 06:58:19 -04:00
Britta Weber 17ce5d5242 fix numerical issue in function score query
we should sum the weights as double to not lose precision. also,
the tests should simulate exactly what function score does and then test
for equality of scores.
2015-10-13 11:54:37 +02:00
Simon Willnauer b9ba5fb765 Merge pull request #14072 from s1monw/remove_clear_index_reader
Remove unused clear(IndexReader) method from IndexFieldData
2015-10-13 11:48:15 +02:00
Simon Willnauer 6120f26c40 Remove unused clear(IndexReader) method from IndexFieldData
This method is unused can can simply be removed. It's rather confusing
instead since it's another way of invalidating a cache entry but not through
the close listener.
2015-10-13 11:32:31 +02:00