This test failed only on a feature branch, because that feature branch has
a different build randomization script and is the only place
randomizing tests.security.manager (this test cannot run with it enabled).
On old kernels without TSYNC support, the test fails because (surprise to me) the thread that
runs the test is not the same thread that runs static initializers:
7571489190/randomized-runner/src/main/java/com/carrotsearch/randomizedtesting/RandomizedRunner.java (L573-L574)
To fix this test (its not an issue in practice, since we do this before creating threadpools),
we just record for testing purposes that we couldn't TSYNC, and re-run the whole thing for the test thread
in setUp(), failing if something goes wrong.
Also add a bunch of additional paranoia and narrow our defensive checks better here after reading
through more chrome bug reports: they don't impact us but those linux distros are too cowboy
with the backports and the spirit of the checks makes me feel better.
This commit adds a new metric aggregator for computing the geo_centroid over a set of geo_point fields. This can be combined with other aggregators (e.g., geohash_grid, significant_terms) for computing the geospatial centroid based on the document sets from other aggregation results.
This commit cleans up IndexMetaData. In particular, all duplicate
getters (X and getX) have been collapsed into one (getX). Further, the
number of shards and number of replicas settings are now parsed once
and saved off as fields.
* Add ability for plugins to declare additional permissions with a custom plugin-security.policy file and corresponding AccessController logic. See the plugin author's guide for more information.
* Add warning messages to users for extra plugin permissions in bin/plugin.
* When bin/plugin is run interactively (stdin is a controlling terminal and -b/--batch not supplied), require user confirmation.
* Improve unit test and IDE support for plugins with additional permissions by exposing plugin's metadata as a maven test resource.
Closes#14108
Squashed commit of the following:
commit cf8ace65a7397aaccd356bf55f95d6fbb8bb571c
Author: Robert Muir <rmuir@apache.org>
Date: Wed Oct 14 13:36:05 2015 -0400
fix new unit test from master merge
commit 9be3c5aa38f2d9ae50f3d54924a30ad9cddeeb65
Merge: 2f168b8 7368231
Author: Robert Muir <rmuir@apache.org>
Date: Wed Oct 14 12:58:31 2015 -0400
Merge branch 'master' into off_my_back
commit 2f168b8038e32672f01ad0279fb5db77ba902ae8
Author: Robert Muir <rmuir@apache.org>
Date: Wed Oct 14 12:56:04 2015 -0400
improve plugin author documentation
commit 6e6c2bfda68a418d92733ac22a58eec35508b2d0
Author: Robert Muir <rmuir@apache.org>
Date: Wed Oct 14 12:52:14 2015 -0400
move security confirmation after 'plugin already installed' check, to prevent user from answering unnecessary questions.
commit 08233a2972554afef2a6a7521990283102e20d92
Author: Robert Muir <rmuir@apache.org>
Date: Wed Oct 14 05:36:42 2015 -0400
Add documentation and pluginmanager support
commit 05dad86c51488ba43ccbd749f0164f3fbd3aee62
Author: Robert Muir <rmuir@apache.org>
Date: Wed Oct 14 02:22:24 2015 -0400
Decentralize plugin permissions (modulo docs and pluginmanager work)
This commit makes sure that the plugin script looks at user, group and permissions of the elasticsearch bin dir and copies them over to the plugin bin subdirectory, whatever they are, so that they get properly setup depending on how elasticsearch was installed. We also make sure that execute permissions are added for files (we already did this before).
Relates to #11016Closes#14088
Today we leak an index searcher if we fail to wrap the seacher in
IndexShard. This commit ensures that the seacher is released if the wrapper
throws an exception.
This commit also restructures some test to be more atomic and only test a single
feature / attribute of the wrapper.
This commit changes the behavior of Cache#computeIfAbsent to not invoke
load for a key under the segment lock. Instead, the synchronization
mechanism to ensure that load is invoked at most once per key is
through the use of a future. Under the segment lock, we put a future in
the cache and through this ensure that load is invoked at most once per
key. This will not lead to the same deadlock situation as before
because a dependent key load on the same thread can not be triggered
while the segment lock is held.
Closes#14090
We randomly use the WindowsFS mock file system to simulate that
windows does not delete files if they are opened by some other
process and instead gives you java.io.IOException: access denied.
In the tests we also check if the shard was deleted while it is
being deleted. This check loads the mata data of the index
and therefore might hold on to a file while the node
is trying to delete it and deletion will fail then.
Instead we should just check if the directory was removed.
closes#13758
this commit also fixes a bug where we wramed a leaf reader in a top level context
which caused atomic segment readers to be used in our top level caches.
IndexReader#addReaderCloseListener is very error prone when it comes to
caching and reader wrapping. The listeners are not delegated to the sub readers
nor can it's implementation change since it's final in the base class. This commit
only allows installing close listeners on the top level ElasticsearchDirecotryReader
which is known to work an has a defined lifetime which corresponds to its subreader.
This ensure that cachesa re cleared once the reader goes out of scope.
In #13606, the ability to exclude a template from being wiped was added but a call to
wipeTemplates() was not removed, so all index templates were still being removed.
This change removes the wipeTemplates() call in the wipe method so that only the
non-excluded templates are removed.
Depending on how elasticsearch is installed, we have two scenarios to take into account that relate to user, group and permissions assigned to the config directory:
1) deb/rpm package: /etc/elasticsearch is root:elasticsearch 750 and the plugin script is run from root user
2) tar/zip archive: es config dir is most likely elasticsearch:elasticsearch and the plugin script is most likely run from elasticsearch user
When the plugin script copies over the plugin config dir within the es config dir, it should take care of setting the proper user, group and permissions, which vary depending on how elasticsearch was installed in the first place. Should be root:elasticsearch 750 if installed from a package, or elasticsearch:elasticsearch if installed from an archive.
This commit makes sure that the plugin script looks at user, group and permissions of the config dir and copies them over to the plugin config subdirectory, whatever they are, so that they get properly setup depending on how elasticsearch was installed in the first place. We also make sure that execute permissions are left untouched for files.
Relates to #11016Closes#14048
Closes#14070
Squashed commit of the following:
commit ad5829e590cf6150763c31886e97d69976b7f02d
Author: Robert Muir <rmuir@apache.org>
Date: Tue Oct 13 06:50:25 2015 -0400
Add more tests for version map caching
commit 617333eefd6d4ddc99db491d0843827b1b5936b2
Author: Robert Muir <rmuir@apache.org>
Date: Mon Oct 12 13:25:51 2015 -0400
fix javadocs ref
commit 9626dd02e01a2676538e1071e302d882fc870837
Author: Robert Muir <rmuir@apache.org>
Date: Mon Oct 12 13:23:16 2015 -0400
make the version map cache per-segment
This commit adds methods to serialize the elements of int and long
arrays using variable-length encodings. This can be useful for
serializing int and long arrays containing mostly non-negative “not
large” values in a compressed form.
FiltersFunctionScoreQuery sums up scores and weights and scores as double but when
we explain we cannot get the double scores from the explanation of score
functions. as a result we cannot compute the exact score from the explanations
of the functions alone.
this commit makes the explanation more accurate but also causes the score to be
computed one additional time.
we should sum the weights as double to not lose precision. also,
the tests should simulate exactly what function score does and then test
for equality of scores.
This method is unused can can simply be removed. It's rather confusing
instead since it's another way of invalidating a cache entry but not through
the close listener.
Restoring an index from a snapshot or creating a new index can bring the index settings index.blocks.read_only, index.blocks.read, index.blocks.write and index.blocks.metadata out-of-sync with the corresponding cluster blocks.
Closes#13931
The IndexAliasesService abstraction only adds unnecessary code and classes
and can be removed. This commit folds the rather simple methods in this
class into IndexService where the IndexAliasesService was obtained from in the past.
RoutingTable.Builder#build can only be called once but didn't have any
checks to make sure it _was_ only called once. And the error message it
threw when called more than once was a NullPointerException. This makes
it throw IllegalStateException when you try to reuse the builder in any way.
- removes Lucene.count(IndexSearcher,Query) in favor of IndexSearcher.count(Query)
- removes EarlyTerminatingCollector.reset(): reusing Collector objects does not
help given that query execution needs to allocate objects (weights, scorers)
anyway
- adds unit tests to Lucene.exists
When deleting an individual scroll ID, ES does produce a 200 in the
header if successful and a 404 if the scroll ID wasn't found, but
returns empty response body. It will be more user friendly to provide
some information on whether the scroll deletion is successful.
This commit removes all index level compatibilty and upgrade paths for
pre 2.0 indices. This includes:
* Remove leftover from delete_by_query to replay translog records,
since in 3.x all pending delelte_by_query instances are applied on the
upgrade to 2.x we can remove the bwc layer now.
* Remove Elasticsearch090PostingsFormat - we maintained our own posting format
until 2.0 this is now removed since folks need to upgrade to 2.x first before going
to 3.0
* Remove BloomFilterPostingFormat - this was only used for ID fields in the Elasticsearch090PostingsFormat
* Remove upgrade methods to pre 2.0 translogs without checkpoints
In #13971, the RenderSearchTemplateAction was made a cluster level action but the client methods
were not moved from the IndicesAdminClient. This is an inconsistency and this change cleans up the
inconsistency so that the client methods are now part of the ClusterAdminClient since the action is now
considered a cluster level action.
This test had to be moved to lang-groovy when groovy has been made a plugin.
I refactored it a bit to use mock plugins instead so that groovy is not
necessary anymore and it can come back to core.
The RenderSearchTemplateAction is currently a "indices" action, but the action does not really apply to
indices, it is more of a general action that is used to validate the search template. With our current
categorization of actions, `cluster:` and `indices:`, `cluster:` is a more appropriate type as this action
is not associated with indices. The classes are also moved to a cluster package.
GeoPoint now has native support in StreamOutput/StreamInput
impementing Writable is not necessary. This also adds tests
for XContentBuilder rendering GeoPoint
This fixes license checks to apply to all java files under src/ as opposed to
only those in the org.elasticsearch package. It found some license headers that
had to be reformatted. I also added a missing license header to Nullability.java
however this has not be caught by the license checker since it ignores guice
files.
Relates to #13703
If the plugin manager cannot successfully install a plugin, ensure
that every directory is cleaned up again. This includes
plugins/foo
config/foo
bin/foo
Closes#12749
When shard becomes active again, immediately increase its indexing buffer instead of waiting for up to 30 seconds while indexing with a tiny (500 KB) indexing buffer.
This commit removes and now forbids all uses of
com.google.common.collect.EvictingQueue across the codebase. This is
one of the few remaining steps in the eventual removal of Guava as a
dependency.
Relates #13224
Long running TransportNodesAction requests can retain old cluster states in memory for much longer than needed. This can cause nodes with frequent cluster state updates and long running requests to run out of memory.
This commit removes and now forbids all uses of
com.google.common.net.InetAddresses across the codebase. This is one of
the few remaining steps in the eventual removal of Guava as a
dependency.
Relates #13224
Currently it's not possible to specify a timeout for nodes operations (such as node info, node stats, cluster stats and hot threads) via REST-based APIs.
When running in GCE platform, an instance has access to:
http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/ip
Which gives back the private IP address, for example `10.240.0.2`.
http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/externalIp
Gives back the public Ip address, for example `130.211.108.21`.
As we have for `ec2`, we can support new network host settings:
* `_gce:privateIp:X_`: The private IP address of the machine for a given network interface.
* `_gce:hostname_`: The hostname of the machine.
* `_gce_`: Same as `_gce:privateIp:0_` (recommended).
Closes#13605.
Closes#13590.
BTW resolveIfPossible now throws IOException so code is also updated for ec2 discovery and
some basic tests have been added.
We previously removed the ability to specify metadata fields inside
documents in #11074, but the backcompat left leniency that allowed this
to still occur. This change locks down parsing so any metadata field
found while parsing a document results in an exception. This only
affects 2.0+ indexes; backcompat is maintained.
closes#13740
ShadowEngine doesn't have a translog but instead throws an
UOE when it's requested. ShadowIndexShard should not try to pull
stats for the translog either and should return null instead.
Closes#12730
SimilarityModule was binding two different interfaces SimilaritySerivce and SimilarityLookupSerice.
Both used a class Similarities which was holding some default impls etc. Thit commit folds
all the logic into a rather simplified SimilarityService which has not construction time dependency to
any other service in the system anymore. It's soely constructued from custom similarities, the index name
and index settings and SimilarityModule is just trivial glue code with out much logic.
This also adds a simple unittest for basic test coverage of the service.
This commit removes and now forbids all uses of
com.google.common.collect.Iterators across the codebase. This is one of
the final steps in the eventual removal of Guava as a dependency.
Relates #13224
The geoPoint gets normalized while calling toQuery. That should happen on a copy of the point though, the state of the request should never change as part of toQuery execution. Also updated corresponding test to support point normalization.
Closes#13984
The `_create` API is handy way to specify an index operation should only be done if the document doesn't exist. This is currently implemented in explicit code paths all the way down to the engine. However, conceptually this is no different than any other versioned operation - instead of requiring a document is on a specific version, we require it to be deleted (or non-existent). This PR removes Engine.Create in favor of a slight extension in the VersionType logic.
There are however a couple of side effects:
- DocumentAlreadyExistsException is removed and VersionConflictException is used instead (with an improved error message)
- Update will reject version parameters if the upsert option is used (it doesn't compute anyway).
- Translog.Create is also removed infavor of Translog.Index (that's OK because their binary format was the same, so we can just read Translog.Index of the translog file)
Closes#13955
It is rarely used and was not consistently handled by different distributions anyway.
This commit also adds a test for specifying CONF_DIR when installing plugins and
starting elasticsearch.
relates to #12712 and #12954closes#5329closes#13715
Mappings and many modules are not needed here compared to AbstractQueryTestCase, as we will never call toQuery in this other test. Parsing is independent from indices and types as well.
For minScore and terminateAfter we can just rely on defaults set to SearchSourceBuilder. Also simplified toString representation of CountRequestBuilder
Plugin cli tools configures logging with whatever is in the logging.yml.
If a file appender is configured for any of the logs this will cause creation
of an empty log file. If a plugin was for example installed as root it will
create empty logs at es.home/logs.
This is problematic when for example plugins are installed as root and es is run
as service. Logs will then be created in /usr/share/elasticsearch/logs
and can later not be removed by for example dpkg -r or -purge.
To avoid this, configure the logger to use an appender that writes to the same
output that plugin cli tool does. This allows other components that are called
from Plugin cli tool to write to the same terminal that plugin cli tool writes to
by using the logging mechanism already in place.
The logging conf is not read at all pb plugin cli tool.
As a side effect, the loging level for components that are called
from the plugin command such as the jar hell check can now be configured
with -Des.logger.level which makes it easier to debug the jar hell check.
The field is optional everywhere else but in the serialization methods, which causes problems. Also expanded tests so that they can catch this type of problem.
Closes#13963
These abstractions don't really do anything nor can they be extended.
We can just fold them into IndexModule for now. There are more but they
are tricky due to some test dependencies which I need to resolve first.
Types are still optional, but if you do provide them, they can't be null. Split the existing constructor that accepted nnull into two, one that accepts no arguments, and another one that accepts the types argument, which must be not null.
Also trimmed down different ways of setting ids, some were misleading as they would always add the ids to the existing ones and not set them, the add prefix makes that clear. Left `addIds` method that accepts a varargs argument. Added check for ids not be null.
I'm not 100% sure we should remove it as part of the pull request to drop
ImmutableMap. It might be more prudent to change its return type to map
and its implementation to an unmodifiable copy of the map being built
and then remove all consumers after banning ImmutableMap.
Today we use a hirachical injector on the shard level for each shard
created. This commit removes the shard level injetor and replaces
it with good old constructor calls. This also removes all shard level plugin
facilities such that plugins can only have node or index level modules.
For plugins that need to track shard lifecycles they should use the relevant
callback from the lifecycle we already provide.
This commit removes and now forbids all uses of
com.google.common.hash.HashCode, com.google.common.hash.HashFunction,
and com.google.common.hash.Hashing across the codebase. This is one of
the few remaining steps in the eventual removal of Guava as a
dependency.
Relates #13224
The fix in #13848 has an off by one issue where the first byte of the checksum
was never written. Unfortunately most tests shadowed the problem and the first
byte of the checksum seems to be very likely a 0 which causes only very rare
failures.
Relates to #13896
Relates to #13848
Right now, we allow allocation if there is only a single node in the
cluster. it would be nice to fail open when there is only one data node
(instead of only one node total).
closes#9391
Lucene's RateLimiter can do too much sleeping on small values (see also #6018).
The issue here is that calls to "pause" are not properly guarded in "restoreFile".
Instead of simply adding the guard, this commit uses the RateLimitingInputStream similar as for "snapshotFile".
Closes#13828
Today we are relying on calling Store.verify on the closed stream to validate the
checksum. This is still necessary to catch file truncation but for an actually corrupted
file or checksum we can fail early and check the checksum against the actual metadata
once it's been fully written to the VerifyingIndexOutput.
This commit removes and now forbids all uses of
com.google.common.collect.ImmutableCollection across the codebase. This
is one of the final steps in the eventual removal of Guava as a
dependency.
Relates #13224
This commit removes and now forbids all uses of
com.google.common.io.Resources across the codebase. This is one of the
few remaining steps in the eventual removal of Guava as a dependency.
Relates #13224
Mostly favoring unmodifiableMap and making sure to only wrap maps that aren't
otherwise returned and so cannot be copied.
MapMaker#immutableMap has to go next.
We're concerned that unmodifiableMap uses significantly more memory than
ImmutableMap did - especially in cluster state - so we ban it there outright
and move to ImmutableOpenMap.
Removes ClusterState$Builder#routingTable(RoutingTable$Builder) because that
method had the side effect of building the routing table which can only be
done once per RoutingTable$Builder now that it uses ImmutableOpenMap.
Closes#13880
Squashed commit of the following:
commit 316a328e5032e580ba840db993d907631334aac0
Author: Robert Muir <rmuir@apache.org>
Date: Wed Sep 30 16:57:47 2015 -0400
windows is terrible
commit 0406b560c58bf833f8d77af9c7cf3386771dd9c5
Author: Robert Muir <rmuir@apache.org>
Date: Wed Sep 30 16:43:09 2015 -0400
Nuke ES_CLASSPATH appending
Out of box, ES expects its stuff to be in particular places. We should not be appending to ES_CLASSPATH, allowing users to specify stuff there, like we do in elasticsearch.bin.sh
If the user sets it, its not going to work out of box.
Closes#13812
commit 415d8972df28eddec322bb6d70100a1993fa95f6
Author: Robert Muir <rmuir@apache.org>
Date: Wed Sep 30 16:26:35 2015 -0400
Fail hard on empty classpath elements.
This can happen easily, if somehow old 1.x shellscripts survive and try to launch 2.x code.
I have the feeling this happens maybe because of packaging upgrades or something.
Either way: we can just fail hard and clear in this situation, rather than the current situation
where CWD might be /, and we might traverse the entire filesystem until we hit an error...
Relates to #13864
When the plugin manager does not find in `plugin-descriptor.properties` the exact same elasticsearch version it was built on
as the current elasticsearch version, it fails with a message like:
```
ERROR: Elasticsearch version [2.0.0-beta1] is too old for plugin [elasticsearch-mapper-attachments]
```
Actually, the message should be:
```
Plugin [elasticsearch-mapper-attachments] is incompatible with Elasticsearch [2.0.0.beta2]. Was designed for version [2.0.0.beta1].
```
The opposite is true. If you try to install a version of a plugin which was built with a newer version of elasticsearch, it will fail the same way:
```
Plugin [elasticsearch-mapper-attachments] is incompatible with Elasticsearch [2.0.0.beta1]. Was designed for version [2.0.0.beta2].
```
This commit adds supports for expiration after writes to Cache. This
enables entries to expire after they were initially placed in the cache
without prolonging their life on retrieval. Replacements are considered
new writes.
This commit removes and now forbids all uses of
com.google.common.cache.Cache, com.google.common.cache.CacheBuilder,
com.google.common.cache.RemovalListener,
com.google.common.cache.RemovalNotification,
com.google.common.cache.Weigher across the codebase. This is a major
step in the eventual removal of Guava as a dependency.
Relates #13224
We try to get the index shard instance again from the index service on a different
threads while that shard might have already been closed or removed which can cause a NPE
instead of another expected expecption.
This commit shuffels and rewrites some code in RecoverySourceHandler to make it
simpler and more unittestable. This commit doesn't change all parts of this class
neither is it fully tested yet. It's an important part of the infrastrucutre so I started
to make it better tested but I don't want to change everything in one go since it makes
review simpler and more detailed. Future commits will continue cleaning up the class and
add more tests.
If a document is indexed into ES with no id, ES will generate one for it. We used to have an optimization for this case where the engine will not try to resolve the ids of these request in the existing index but immediately try to index them. This optimization has proven to be the source of brittle bugs (solved!) and we disabled it in 1.5, preparing for it to be removed if no performance degradation was found. Since we haven't seen any such degradation we can remove it.
Along with the removal of the optmization, we can remove the autogenerate id flag on indexing requests and the can have duplicate flag. The only downside of the removal of the canHaveDuplicate flag is that we can't make sure any more that when we retry an autogenerated id create operation we will ignore any document already exists exception (See #9125 for background and discussion). To work around this, we don't set the operation to CREATE any more when we generate an id, so the resulting request will never fail when it finds an existing doc but do return a version of 2. I think that's acceptable.
Closes#13857
We have two unneded heavy dependencies on IndexShard that are unneeded and only cause
trouble if you try to mock index shard. This commit removes IndexSettingsService as well as
ClusterSerivce from IndexShard to simplify future mocking and construction.
While refactoring has_child and has_parent query we lost an important detail around types. The types that the inner query gets executed against shouldn't be the main types of the search request but the parent or child type set to the parent query. We used to use QueryParseContext#setTypesWithPrevious as part of XContentStructure class which has been deleted, without taking care though of setting the types and restoring them as part of the innerQuery#toQuery call.
Meanwhile also we make sure that the original context types are restored in PercolatorQueriesRegistry
Closes#13863
Closes#13854
Squashed commit of the following:
commit 42c1166efc55adda0d13fed77de583c0973e44b3
Author: Robert Muir <rmuir@apache.org>
Date: Tue Sep 29 11:59:43 2015 -0400
Add paranoia
Groovy holds on to a classloader, so check it before compilation too.
I have not reviewed yet what Rhino is doing, but just be safe.
commit b58668a81428e964dd5ffa712872c0a34897fc91
Author: Robert Muir <rmuir@apache.org>
Date: Tue Sep 29 11:46:06 2015 -0400
Add SpecialPermission to guard exceptions to security policy.
In some cases (e.g. buggy cloud libraries, scripting engines), we must
grant dangerous permissions to contained cases. Those AccessController blocks
are dangerous, since they truncate the stack, and can allow privilege escalation.
This PR adds a simple permission to check before each one, so that unprivileged code
like groovy scripts, can't do anything they shouldn't be allowed to do otherwise.
StoreRecoveryService used to be a pretty heavy class with lots of dependencies.
This class was basically not testable in isolation and had an async API with a listener.
This commit refactors this class to be a simple utility classs with a sync API hidden behind
the IndexShard interface. It includes single node tests and moves all the async properities to
the caller side.
Note, this change also removes the mapping update on master from the store recovery code since
it's not needed anymore in 3.0 because all stores have been subject to sync mapping updates such
that the master already has all the mappings for documents that made it into the transaction log.
Closes#13766
Now that groovy is factored out, we contain this dangerous stuff there.
TODO: look into those test hacks inspecting class protection domains, maybe we can
clean that one up too.
TODO: generalize the GroovyCodeSourcePermission to something all script engines check,
before entering accesscontrollerblocks. this way e.g. groovy script cannot coerce
python engine into creating something with more privs if it gets ahold of it... we
should probably protect the aws/gce hacks in the same way.
Previous to this change if to equal geohash cell query builders were created and then toQuery was called on one, they would no longer be equal.
This change also adds a test to AbstractQueryTestCase to make sure calling toQuery on any query builder does not affect the query builder's equality
* Add OS X support via "seatbelt" mechanism. This gives consistency across dev and prod, since many devs use OS X.
* block execveat system call: it may be new, but we should not allow it.
When a shard becomes in active we trigger a sync flush in order to speed up future recoveries. The sync flush causes a new translog generation to be made, which in turn confuses the IndexingMemoryController making it think that the shard is active. If no documents comes along in the next 5m, the shard is made inactive again , triggering a sync flush and so forth.
To avoid this, the IndexingMemoryController is changed to ignore empty translogs when checking if a shard became active. This comes with the price of potentially missing indexing operations which are followed by a flush. This is acceptable as if no more index operation come in, it's OK to leave the shard in active.
A new unit test is introduced and comparable integration tests are removed.
Closes#13802
Instead just log the same thing we print to the startup console for that case (magic logic),
it sucks to do this, but guice exceptions are too much.
All other non-guice exceptions will still be fully logged.
Closes#13782
This commit adds explicit ids for managing ElasticsearchException
serialization. By adding explicit ids and unit tests for them, the ids
are less brittle and breakage can be more clearly detected.
Today we use reflection where it's not needed anymore since java8 can
pass ctors around. This commit replaces runtime checks with compile time
checks which is always preferrable.
These classes are really duplicates and are just here for historical reasons.
We don't need these anymore since the same classes exist in lucene today.
This also removes the guice injection for DeletionPolicy and make them shard private.
TermsLookupQueryBuilder was left around only for bw comp reasons, but TermsQueryBuilder is its replacement. We can remove it now that it is clear query refactoring goes in master (3.0).
There is no need to have term vectors service on the shard level where it's
created for every shard. This commit moves it to a higher level which makes
shard creation slightly simpler and reduces the number of long living objects.
Before the refactoring we didn't check any invalid settings for strategy and relation
in the GeoShapeQueryBuilder. However, using SpatialStrategy.TERM and ShapeRelation.INTERSECTS
together is invalid and we tried to protect against that in the validate() method.
This PR moves these checks to setter for strategy and relation and adds tests for the new
behaviour.
Relates to #10217
Block execve(), fork(), and vfork() system calls, returning EACCES instead,
on kernels that support seccomp-bpf: either via seccomp() or falling back
to prctl().
Only linux/amd64 is supported. This feature can be disabled (in case
of problems) with bootstrap.seccomp=false.
Closes#13753
Squashed commit of the following:
commit 92cee05c72b49e532d41be7b16709e1c9f919fa9
Author: Robert Muir <rmuir@apache.org>
Date: Thu Sep 24 10:12:51 2015 -0400
Add a note about why we don't parse uname() or anything
commit b427971f45cbda4d0b964ddc4a55fae638880335
Author: Robert Muir <rmuir@apache.org>
Date: Thu Sep 24 09:44:31 2015 -0400
style only: we already pull errno into a local, use it for catch-all case
commit ddf93305525ed1546baf91f7902148a8f5b1ad06
Author: Robert Muir <rmuir@apache.org>
Date: Thu Sep 24 08:36:01 2015 -0400
add TODO
commit f29d1b7b809a9d4c1fcf15f6064e43f7d1b24696
Author: Robert Muir <rmuir@apache.org>
Date: Thu Sep 24 08:33:28 2015 -0400
Add full stacktrace at debug level always
commit a3c991ff8b0b16dc5e128af8fb3dfa6346c6d6f1
Author: Robert Muir <rmuir@apache.org>
Date: Thu Sep 24 00:08:19 2015 -0400
Add missing check just in case.
commit 628ed9c77603699aa9c67890fe7632b0e662a911
Author: Robert Muir <rmuir@apache.org>
Date: Wed Sep 23 22:47:16 2015 -0400
Add public getter, for stats or whatever if they need to know this
commit 3e2265b5f89d42043d9a07d4525ce42e2cb1c727
Author: Robert Muir <rmuir@apache.org>
Date: Wed Sep 23 22:43:06 2015 -0400
Enable use of seccomp(2) on Linux 3.17+ which provides more protection.
Add nice errors.
Add all kinds of checks and paranoia.
Add documentation.
Add boolean switch.
commit 0e421f7fa2d5236c8fa2cd073bcb616f5bcd2d23
Author: Robert Muir <rmuir@apache.org>
Date: Wed Sep 23 21:36:32 2015 -0400
Add defensive checks and nice error messages
commit 6231c3b7c96a81af8460cde30135e077f60a3f39
Author: Robert Muir <rmuir@apache.org>
Date: Wed Sep 23 20:52:40 2015 -0400
clean up JNA and BPF. block fork and vfork too.
commit bb31e8a6ef03ceeb1d5137c84d50378c270af85a
Author: Robert Muir <rmuir@apache.org>
Date: Wed Sep 23 19:00:32 2015 -0400
order is LE already for the JNA buffer, but be explicit about it
commit 10456d2f08f12ddc3d60989acb86b37be6a4b12b
Author: Robert Muir <rmuir@apache.org>
Date: Wed Sep 23 17:47:07 2015 -0400
block process execution with seccomp on linux/amd64
Some methods had default implementation while queries were going to be refactored, now that they are all refactored all those methods can be made abstract.
Remove ParseFieldMatcher duplication query in both contexts. QueryParseContext is still contained in QueryShardContext, as parsing still happens in the shards here and there. Most of the norelease comments have been removed simply because the scope of the refactoring has become smaller. Some could only be removed once everything, the whole search request, gets parsed on the coordinating node. We will get there eventually.
SimpleIndexQueryParserTests was the main responsible: deleted lots of duplicated tests, moved the ones that made sense to keep to their corresponding unit tests (note they were ESSingleNode tests before while are now converted to unit tests).
Closes#13750
After all queries now have a `toQuery` method and the parsers all
support `fromXContent` it is possible to remove the following
workarounds and deprecated methods we kept around while doing the
refactoring:
* remove the BaseQueryParser and BaseQueryParserTemp. All parsers
implement QueryParser directly now
* remove deprecated methods in QueryParseContext that either returned
a Query or a Filter.
* remove the temporary QueryWrapperQueryBuilder
Relates to #10217
The IndexingMemoryController checks periodically if there is any indexing activity on the shard. If no activity is sean for 5m (default) the shard is marked as inactive allowing it's indexing buffer quota to given to other active shards.
Sadly the current check is bad as it checks for 0 translog operation. This makes the inactive wait for a flush to happen - which used to take 30m and since #13707 doesn't happen at all (as we rely on the synced flush triggered by inactivity). This commit fixes the check so it will work with any translog size.
Closes#13759
This commit fixes ping timeout settings inconsistencies in
ZenDiscovery. In particular, the documentation refers to the ping
timeout setting as discovery.zen.ping_timeout but the code was
ultimately using discovery.zen.ping.timeout if this was set.
This commit also changes all instances of the raw string
“discovery.zen.ping_timeout” to the constant
o.e.d.z.ZenDiscovery.SETTING_PING_TIMEOUT.
Finally, this commit removes the legacy setting
"discovery.zen.initial_ping_timeout".
Closes#6579, #9581, #9908
The current MoreLikeThisQueryBuilder validation checks for existence of at
least one `like` text or item. This is hard to check in setters, so this PR
tries to change the construction of the query so that we can do these checks
already at construction time.
Changing to using arrays for fieldnames, likeTexts, likeItems, unlikeTexts
and unlikeItems. `likeTexts` and/or `likeItems` need to be specified at
construction time to validate we have at least one item there.
Relates to #10217
Banning `ImmutableSet` outright is too much to do all at once - this starts
the process by banning `ImmutableMap#entrySet` - one of the more common ways
that `ImmutableSet`s come up. It then starts to remove calls to
`ImmutableMap#entrySet` by changing declarations from `ImmutableMap` to `Map`.
Unfortunately this process is like pulling on a long, windy string and one
declaration change requires another which requires 5 more which in turn
require another few. So this change is rather large.
As such, to keep the changes manageable they only remove `ImmutableMap` from
the signatures that are needed for `entrySet` and make little effort to stop
using `ImmutableMap` internally. Removing the usages of `ImmutableMap`
complicates immutability guarantees and will be done separately.
In #12942, the NettyTransport and NettyHttpServerTransport were updated to allow for binding
to multiple addresses. However, the BoundTransportAddress holder only exposed the first address
that the transport was bound to and this object is used to populate the values returned to the user
via our APIs.
This change exposes all of the bound addresses in the BoundTransportAddress holder, which allows
for an accurate representation of all interfaces that elasticsearch is bound to and listening on.
Not the neatest implementation in the world. Maybe we should consider changing the builders so it is a single builder for sort and a single builder for rescore instead of a list of builders for each?
This commit addresses a confusing error message that arises when a
property parameter (e.g. -D) is after a double-dash parameter. The
current error message reports to the user that the parameter does not
start with “--". Adding the second dash as the error message suggests
causes the parameter to be silently ignored. This is confusing for the
user. With this commit, the user is now informed that the parameter
order is violated.
Relates e27ede48ce
These exceptions are useless and unused, since we are on a major verison we should remove
them. This commit also makes it easier to remove excepitons in the future.
In the past ClusterStateUpdateTask was an interface and we had various derived marker interfaces to control behavior. Since then we moved ClusterStateUpdateTask to be an abstract class but we kept the old hierarchy of implementations. All of those (but the AckedClusterStateUpdateTask) can be folded into ClusterStateUpdateTask, adding correct default behavior.
Closes#13735
This commit moves the size and ops based flush into a synchronous API into
IndexShard and removes the time-based flush alltogether since it' basically
covered by the inactive async flush API we have today. The functionality doesn't
need to be covered by scheduled task and async APIs while we can actually make all
the decisions in a sync manner which is way easier to control and to test.
Closes#13707
This commit removes all the opaque bytes for extra_source and template_source.
Instead source and extra_source etc. are represented as SearchSourceBuilder which can
in-place be modified and is updated with the content of the request parameters.
Template Source is parsed and evaluated which in-turn replaces the actual source.
Refactor the function_score query so it can be parsed on the coordinating node, split parse into fromXContent and toQuery, make FunctionScoreQueryBuilder Writeable.
Closes#13653
Given that we are moving to parsing queries on the coordinating node, the index name is not relevant anymore in QueryParseContext, as the parsing phase cannot be related to any specific index. On the contrary, the QueryShardContext is the one that holds mappings etc. and the index name too, as the lucene query creation happens on the data node and can still be related with the index that it happens against.
Changes are mainly around tests that were expecting the index name, moved to using QueryShardException in some of them, removed the index name elsewhere.
Closes#13631
Moving validation from validate() to constructors and setters for the
following query builders:
* GeoDistanceQueryBuilder
* GeoDistanceRangeQueryBuilder
* GeoPolygonQueryBuilder
* GeoShapeQueryBuilder
* GeohashCellQuery
* TermsQueryBuilder
Relates to #10217
This parser prototype allows to decleratively define parsers for XContent
instead of writing messy and error prone while loops. It encapsulates all the error handling logic
and only even tries to parse if the token types match the declaration.
I want to refactor scripting engines so we can contain dangerous "God-like" permissions
like createClassloader/sun.reflect. These are used for dynamic class generation (scripts, mocks).
This will mean some refactoring to ES core.
But first lets get the plugins in order first. I removed those permissions globally, and
fixed grants for lang-javascript, lang-python, securemock so that everything works.
lang-javascript needs no code changes, because rhino is properly written :)
lang-python needs accesscontroller blocks. securemock was already working as of 1.1
This is just a baby step, to try to do some of this incrementally! It doesn't yet provide
us anything.
Currently the tribe node version always stays 0, which can cause issues for the services that rely on cluster state version. For example, ClusterStateObserver doesn't revalidate the cluster state after change, which leads to cluster health check with wait flags to take much longer then actually needed.
Until now we had a cloud-azure plugin which is providing 3 distinct features:
* discovery on Azure
* snapshot/restore on Aure
* SMB store
This commit splits the plugin by feature so people can use either one or the other or both features.
Doc is updated accordingly.
Robert Muir
javanna
Colin Goodheart-Smithe
Alexander Reelsen
Simon Willnauer
Robert Muir
Jason Tedor
Nicholas Knize
Britta Weber
Simon Willnauer
jaymode
Britta Weber
Yannick Welsch
Andreas Kohn
Adrien Grand
Christoph Büscher
Nik Everett
xuzha
Jay Modi
Martijn van Groningen
Patrick Kaufmann
Michael McCandless
David Pilato
Igor Motov
Itamar Syn-Hershko
Ryan Ernst
Michael McCandless
Clinton Gormley
Boaz Leskes
Xu Zhang
André Carvalho
Tanguy Leroux
Alex Chow
Luca Cavanna
Isabel Drost-Fromm
Chris Earle
Camilo Díaz Repka