Today we leak an index searcher if we fail to wrap the seacher in
IndexShard. This commit ensures that the seacher is released if the wrapper
throws an exception.
This commit also restructures some test to be more atomic and only test a single
feature / attribute of the wrapper.
This commit changes the behavior of Cache#computeIfAbsent to not invoke
load for a key under the segment lock. Instead, the synchronization
mechanism to ensure that load is invoked at most once per key is
through the use of a future. Under the segment lock, we put a future in
the cache and through this ensure that load is invoked at most once per
key. This will not lead to the same deadlock situation as before
because a dependent key load on the same thread can not be triggered
while the segment lock is held.
Closes#14090
We randomly use the WindowsFS mock file system to simulate that
windows does not delete files if they are opened by some other
process and instead gives you java.io.IOException: access denied.
In the tests we also check if the shard was deleted while it is
being deleted. This check loads the mata data of the index
and therefore might hold on to a file while the node
is trying to delete it and deletion will fail then.
Instead we should just check if the directory was removed.
closes#13758
this commit also fixes a bug where we wramed a leaf reader in a top level context
which caused atomic segment readers to be used in our top level caches.
IndexReader#addReaderCloseListener is very error prone when it comes to
caching and reader wrapping. The listeners are not delegated to the sub readers
nor can it's implementation change since it's final in the base class. This commit
only allows installing close listeners on the top level ElasticsearchDirecotryReader
which is known to work an has a defined lifetime which corresponds to its subreader.
This ensure that cachesa re cleared once the reader goes out of scope.
In #13606, the ability to exclude a template from being wiped was added but a call to
wipeTemplates() was not removed, so all index templates were still being removed.
This change removes the wipeTemplates() call in the wipe method so that only the
non-excluded templates are removed.
Depending on how elasticsearch is installed, we have two scenarios to take into account that relate to user, group and permissions assigned to the config directory:
1) deb/rpm package: /etc/elasticsearch is root:elasticsearch 750 and the plugin script is run from root user
2) tar/zip archive: es config dir is most likely elasticsearch:elasticsearch and the plugin script is most likely run from elasticsearch user
When the plugin script copies over the plugin config dir within the es config dir, it should take care of setting the proper user, group and permissions, which vary depending on how elasticsearch was installed in the first place. Should be root:elasticsearch 750 if installed from a package, or elasticsearch:elasticsearch if installed from an archive.
This commit makes sure that the plugin script looks at user, group and permissions of the config dir and copies them over to the plugin config subdirectory, whatever they are, so that they get properly setup depending on how elasticsearch was installed in the first place. We also make sure that execute permissions are left untouched for files.
Relates to #11016Closes#14048
Closes#14070
Squashed commit of the following:
commit ad5829e590cf6150763c31886e97d69976b7f02d
Author: Robert Muir <rmuir@apache.org>
Date: Tue Oct 13 06:50:25 2015 -0400
Add more tests for version map caching
commit 617333eefd6d4ddc99db491d0843827b1b5936b2
Author: Robert Muir <rmuir@apache.org>
Date: Mon Oct 12 13:25:51 2015 -0400
fix javadocs ref
commit 9626dd02e01a2676538e1071e302d882fc870837
Author: Robert Muir <rmuir@apache.org>
Date: Mon Oct 12 13:23:16 2015 -0400
make the version map cache per-segment
This commit adds methods to serialize the elements of int and long
arrays using variable-length encodings. This can be useful for
serializing int and long arrays containing mostly non-negative “not
large” values in a compressed form.
FiltersFunctionScoreQuery sums up scores and weights and scores as double but when
we explain we cannot get the double scores from the explanation of score
functions. as a result we cannot compute the exact score from the explanations
of the functions alone.
this commit makes the explanation more accurate but also causes the score to be
computed one additional time.
we should sum the weights as double to not lose precision. also,
the tests should simulate exactly what function score does and then test
for equality of scores.
This method is unused can can simply be removed. It's rather confusing
instead since it's another way of invalidating a cache entry but not through
the close listener.
Restoring an index from a snapshot or creating a new index can bring the index settings index.blocks.read_only, index.blocks.read, index.blocks.write and index.blocks.metadata out-of-sync with the corresponding cluster blocks.
Closes#13931
The IndexAliasesService abstraction only adds unnecessary code and classes
and can be removed. This commit folds the rather simple methods in this
class into IndexService where the IndexAliasesService was obtained from in the past.
RoutingTable.Builder#build can only be called once but didn't have any
checks to make sure it _was_ only called once. And the error message it
threw when called more than once was a NullPointerException. This makes
it throw IllegalStateException when you try to reuse the builder in any way.
- removes Lucene.count(IndexSearcher,Query) in favor of IndexSearcher.count(Query)
- removes EarlyTerminatingCollector.reset(): reusing Collector objects does not
help given that query execution needs to allocate objects (weights, scorers)
anyway
- adds unit tests to Lucene.exists
When deleting an individual scroll ID, ES does produce a 200 in the
header if successful and a 404 if the scroll ID wasn't found, but
returns empty response body. It will be more user friendly to provide
some information on whether the scroll deletion is successful.
This commit removes all index level compatibilty and upgrade paths for
pre 2.0 indices. This includes:
* Remove leftover from delete_by_query to replay translog records,
since in 3.x all pending delelte_by_query instances are applied on the
upgrade to 2.x we can remove the bwc layer now.
* Remove Elasticsearch090PostingsFormat - we maintained our own posting format
until 2.0 this is now removed since folks need to upgrade to 2.x first before going
to 3.0
* Remove BloomFilterPostingFormat - this was only used for ID fields in the Elasticsearch090PostingsFormat
* Remove upgrade methods to pre 2.0 translogs without checkpoints
In #13971, the RenderSearchTemplateAction was made a cluster level action but the client methods
were not moved from the IndicesAdminClient. This is an inconsistency and this change cleans up the
inconsistency so that the client methods are now part of the ClusterAdminClient since the action is now
considered a cluster level action.
This test had to be moved to lang-groovy when groovy has been made a plugin.
I refactored it a bit to use mock plugins instead so that groovy is not
necessary anymore and it can come back to core.
The RenderSearchTemplateAction is currently a "indices" action, but the action does not really apply to
indices, it is more of a general action that is used to validate the search template. With our current
categorization of actions, `cluster:` and `indices:`, `cluster:` is a more appropriate type as this action
is not associated with indices. The classes are also moved to a cluster package.
GeoPoint now has native support in StreamOutput/StreamInput
impementing Writable is not necessary. This also adds tests
for XContentBuilder rendering GeoPoint
This fixes license checks to apply to all java files under src/ as opposed to
only those in the org.elasticsearch package. It found some license headers that
had to be reformatted. I also added a missing license header to Nullability.java
however this has not be caught by the license checker since it ignores guice
files.
Relates to #13703
If the plugin manager cannot successfully install a plugin, ensure
that every directory is cleaned up again. This includes
plugins/foo
config/foo
bin/foo
Closes#12749
When shard becomes active again, immediately increase its indexing buffer instead of waiting for up to 30 seconds while indexing with a tiny (500 KB) indexing buffer.
This commit removes and now forbids all uses of
com.google.common.collect.EvictingQueue across the codebase. This is
one of the few remaining steps in the eventual removal of Guava as a
dependency.
Relates #13224
Long running TransportNodesAction requests can retain old cluster states in memory for much longer than needed. This can cause nodes with frequent cluster state updates and long running requests to run out of memory.
This commit removes and now forbids all uses of
com.google.common.net.InetAddresses across the codebase. This is one of
the few remaining steps in the eventual removal of Guava as a
dependency.
Relates #13224
Currently it's not possible to specify a timeout for nodes operations (such as node info, node stats, cluster stats and hot threads) via REST-based APIs.
When running in GCE platform, an instance has access to:
http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/ip
Which gives back the private IP address, for example `10.240.0.2`.
http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/externalIp
Gives back the public Ip address, for example `130.211.108.21`.
As we have for `ec2`, we can support new network host settings:
* `_gce:privateIp:X_`: The private IP address of the machine for a given network interface.
* `_gce:hostname_`: The hostname of the machine.
* `_gce_`: Same as `_gce:privateIp:0_` (recommended).
Closes#13605.
Closes#13590.
BTW resolveIfPossible now throws IOException so code is also updated for ec2 discovery and
some basic tests have been added.
We previously removed the ability to specify metadata fields inside
documents in #11074, but the backcompat left leniency that allowed this
to still occur. This change locks down parsing so any metadata field
found while parsing a document results in an exception. This only
affects 2.0+ indexes; backcompat is maintained.
closes#13740
ShadowEngine doesn't have a translog but instead throws an
UOE when it's requested. ShadowIndexShard should not try to pull
stats for the translog either and should return null instead.
Closes#12730
SimilarityModule was binding two different interfaces SimilaritySerivce and SimilarityLookupSerice.
Both used a class Similarities which was holding some default impls etc. Thit commit folds
all the logic into a rather simplified SimilarityService which has not construction time dependency to
any other service in the system anymore. It's soely constructued from custom similarities, the index name
and index settings and SimilarityModule is just trivial glue code with out much logic.
This also adds a simple unittest for basic test coverage of the service.
This commit removes and now forbids all uses of
com.google.common.collect.Iterators across the codebase. This is one of
the final steps in the eventual removal of Guava as a dependency.
Relates #13224
The geoPoint gets normalized while calling toQuery. That should happen on a copy of the point though, the state of the request should never change as part of toQuery execution. Also updated corresponding test to support point normalization.
Closes#13984
The `_create` API is handy way to specify an index operation should only be done if the document doesn't exist. This is currently implemented in explicit code paths all the way down to the engine. However, conceptually this is no different than any other versioned operation - instead of requiring a document is on a specific version, we require it to be deleted (or non-existent). This PR removes Engine.Create in favor of a slight extension in the VersionType logic.
There are however a couple of side effects:
- DocumentAlreadyExistsException is removed and VersionConflictException is used instead (with an improved error message)
- Update will reject version parameters if the upsert option is used (it doesn't compute anyway).
- Translog.Create is also removed infavor of Translog.Index (that's OK because their binary format was the same, so we can just read Translog.Index of the translog file)
Closes#13955
It is rarely used and was not consistently handled by different distributions anyway.
This commit also adds a test for specifying CONF_DIR when installing plugins and
starting elasticsearch.
relates to #12712 and #12954closes#5329closes#13715
Mappings and many modules are not needed here compared to AbstractQueryTestCase, as we will never call toQuery in this other test. Parsing is independent from indices and types as well.
For minScore and terminateAfter we can just rely on defaults set to SearchSourceBuilder. Also simplified toString representation of CountRequestBuilder
Plugin cli tools configures logging with whatever is in the logging.yml.
If a file appender is configured for any of the logs this will cause creation
of an empty log file. If a plugin was for example installed as root it will
create empty logs at es.home/logs.
This is problematic when for example plugins are installed as root and es is run
as service. Logs will then be created in /usr/share/elasticsearch/logs
and can later not be removed by for example dpkg -r or -purge.
To avoid this, configure the logger to use an appender that writes to the same
output that plugin cli tool does. This allows other components that are called
from Plugin cli tool to write to the same terminal that plugin cli tool writes to
by using the logging mechanism already in place.
The logging conf is not read at all pb plugin cli tool.
As a side effect, the loging level for components that are called
from the plugin command such as the jar hell check can now be configured
with -Des.logger.level which makes it easier to debug the jar hell check.
The field is optional everywhere else but in the serialization methods, which causes problems. Also expanded tests so that they can catch this type of problem.
Closes#13963
These abstractions don't really do anything nor can they be extended.
We can just fold them into IndexModule for now. There are more but they
are tricky due to some test dependencies which I need to resolve first.
Types are still optional, but if you do provide them, they can't be null. Split the existing constructor that accepted nnull into two, one that accepts no arguments, and another one that accepts the types argument, which must be not null.
Also trimmed down different ways of setting ids, some were misleading as they would always add the ids to the existing ones and not set them, the add prefix makes that clear. Left `addIds` method that accepts a varargs argument. Added check for ids not be null.
I'm not 100% sure we should remove it as part of the pull request to drop
ImmutableMap. It might be more prudent to change its return type to map
and its implementation to an unmodifiable copy of the map being built
and then remove all consumers after banning ImmutableMap.
Today we use a hirachical injector on the shard level for each shard
created. This commit removes the shard level injetor and replaces
it with good old constructor calls. This also removes all shard level plugin
facilities such that plugins can only have node or index level modules.
For plugins that need to track shard lifecycles they should use the relevant
callback from the lifecycle we already provide.
This commit removes and now forbids all uses of
com.google.common.hash.HashCode, com.google.common.hash.HashFunction,
and com.google.common.hash.Hashing across the codebase. This is one of
the few remaining steps in the eventual removal of Guava as a
dependency.
Relates #13224
The fix in #13848 has an off by one issue where the first byte of the checksum
was never written. Unfortunately most tests shadowed the problem and the first
byte of the checksum seems to be very likely a 0 which causes only very rare
failures.
Relates to #13896
Relates to #13848
Right now, we allow allocation if there is only a single node in the
cluster. it would be nice to fail open when there is only one data node
(instead of only one node total).
closes#9391
Lucene's RateLimiter can do too much sleeping on small values (see also #6018).
The issue here is that calls to "pause" are not properly guarded in "restoreFile".
Instead of simply adding the guard, this commit uses the RateLimitingInputStream similar as for "snapshotFile".
Closes#13828
Today we are relying on calling Store.verify on the closed stream to validate the
checksum. This is still necessary to catch file truncation but for an actually corrupted
file or checksum we can fail early and check the checksum against the actual metadata
once it's been fully written to the VerifyingIndexOutput.
This commit removes and now forbids all uses of
com.google.common.collect.ImmutableCollection across the codebase. This
is one of the final steps in the eventual removal of Guava as a
dependency.
Relates #13224
This commit removes and now forbids all uses of
com.google.common.io.Resources across the codebase. This is one of the
few remaining steps in the eventual removal of Guava as a dependency.
Relates #13224
Mostly favoring unmodifiableMap and making sure to only wrap maps that aren't
otherwise returned and so cannot be copied.
MapMaker#immutableMap has to go next.
We're concerned that unmodifiableMap uses significantly more memory than
ImmutableMap did - especially in cluster state - so we ban it there outright
and move to ImmutableOpenMap.
Removes ClusterState$Builder#routingTable(RoutingTable$Builder) because that
method had the side effect of building the routing table which can only be
done once per RoutingTable$Builder now that it uses ImmutableOpenMap.
Closes#13880
Squashed commit of the following:
commit 316a328e5032e580ba840db993d907631334aac0
Author: Robert Muir <rmuir@apache.org>
Date: Wed Sep 30 16:57:47 2015 -0400
windows is terrible
commit 0406b560c58bf833f8d77af9c7cf3386771dd9c5
Author: Robert Muir <rmuir@apache.org>
Date: Wed Sep 30 16:43:09 2015 -0400
Nuke ES_CLASSPATH appending
Out of box, ES expects its stuff to be in particular places. We should not be appending to ES_CLASSPATH, allowing users to specify stuff there, like we do in elasticsearch.bin.sh
If the user sets it, its not going to work out of box.
Closes#13812
commit 415d8972df28eddec322bb6d70100a1993fa95f6
Author: Robert Muir <rmuir@apache.org>
Date: Wed Sep 30 16:26:35 2015 -0400
Fail hard on empty classpath elements.
This can happen easily, if somehow old 1.x shellscripts survive and try to launch 2.x code.
I have the feeling this happens maybe because of packaging upgrades or something.
Either way: we can just fail hard and clear in this situation, rather than the current situation
where CWD might be /, and we might traverse the entire filesystem until we hit an error...
Relates to #13864
When the plugin manager does not find in `plugin-descriptor.properties` the exact same elasticsearch version it was built on
as the current elasticsearch version, it fails with a message like:
```
ERROR: Elasticsearch version [2.0.0-beta1] is too old for plugin [elasticsearch-mapper-attachments]
```
Actually, the message should be:
```
Plugin [elasticsearch-mapper-attachments] is incompatible with Elasticsearch [2.0.0.beta2]. Was designed for version [2.0.0.beta1].
```
The opposite is true. If you try to install a version of a plugin which was built with a newer version of elasticsearch, it will fail the same way:
```
Plugin [elasticsearch-mapper-attachments] is incompatible with Elasticsearch [2.0.0.beta1]. Was designed for version [2.0.0.beta2].
```
This commit adds supports for expiration after writes to Cache. This
enables entries to expire after they were initially placed in the cache
without prolonging their life on retrieval. Replacements are considered
new writes.
This commit removes and now forbids all uses of
com.google.common.cache.Cache, com.google.common.cache.CacheBuilder,
com.google.common.cache.RemovalListener,
com.google.common.cache.RemovalNotification,
com.google.common.cache.Weigher across the codebase. This is a major
step in the eventual removal of Guava as a dependency.
Relates #13224
Britta Weber
Simon Willnauer
Simon Willnauer
Jason Tedor
jaymode
javanna
Britta Weber
Robert Muir
Yannick Welsch
Robert Muir
Andreas Kohn
Colin Goodheart-Smithe
Adrien Grand
Christoph Büscher
Nik Everett
xuzha
Jay Modi
Martijn van Groningen
Patrick Kaufmann
Alexander Reelsen
Michael McCandless
David Pilato
Igor Motov
Itamar Syn-Hershko
Ryan Ernst
Michael McCandless
Clinton Gormley
Boaz Leskes
Xu Zhang
André Carvalho