This change unifies the way scripts and templates are specified for all instances in the codebase. It builds on the Script class added previously and adds request building and parsing support as well as the ability to transfer script objects between nodes. It also adds a Template class which aims to provide the same functionality for template APIs
Closes#11091
Mappers are currently used at both index and query time for deciding
how to "use" a field. For #8871, we need the index wide view of
mappings to have a unified set of settings for each field of a given
name within the index.
This change moves all the current settings (and methods defining
query time behavior) into subclasses of FieldType. In a future
PR, this will allow storing the field type at the index level,
instead of mappers (which can still have settings that differ
per document type).
The change is quite large (I'm sorry). I could not see a way to
migrate to this in a more piecemeal way. I did leave out cutting
over callers of the query methods to using the field type, as
that can be done in a follow up.
LZF only stays for backward-compatibility reasons and can only read, not write.
DEFLATE is configured to use level=3, which is a nice trade-off between speed
and compression ratio and is the same as we use for Lucene's high compression
codec.
IndexRequestBuilder#setSource as well as CreateIndexRequestBuilder#setSettings and
CreateIndexRequestBuilder#setSouce() will not work with Map<String, String> argument
although the API looks like it should. This PR fixes the problem introducing correct
wildcard parameters and adds tests.
Closes#10825
We have a lot of module classes that don't contain any actual logic,
only declarative bind actions. These classes are unnecessary and can
be consolidated into the already existings IndexShardModule
We have a compression framework that we use internally, mainly to compress some
xcontent bytes. However it is quite lenient: for instance it relies on the
assumption that detection of the compression format can only be called on either
some compressed xcontent bytes or some raw xcontent bytes, but nothing checks
this. By the way, we are misusing it in BinaryFieldMapper so that if someone
indexes a binary field which happens to have the same header as a LZF stream,
then at read time, we will try to decompress it.
It also simplifies the API by removing block compression (only streaming) and
some code duplication caused by some methods accepting a byte[] and other
methods a BytesReference.
We recently run into two issues where mapping weren't serialized in a consistent manner (#10302 and #10318). We rely on this consistency to do a byte level checl that mappings we get from the master are indentical to the one we have locally. Mistakes here can cause endless refresh mapping loops.
This commit adds an assert that verifies this upon every update from the master.
Setting the number of replicas on a closed index can leave the index
in an unopenable state since we might not be able to recover a quorum.
This commit simply prevents updating this setting on a closed index.
Closes#9566
The ShardFilterCache relies on the fact that it's
closed once the last reader on the shard is closed.
This is only guaranteed once the Store and all its
references are closed. This commit moves the closing
into the internal callback mechanism we use for deleting
shard data etc. to close the cache once we have all
searchers released.
fails
In the event that reading from the latest commit fails, we should fall
back to reading from the `Store` using the traditional
`Directory.listAll()`
Related to #11361
In #9474, we exported the hostname in the bin/elasticsearch scripts so that it
could be used as a variable in the elasticsearch.yml file but did not do the same
for plugin manager. When using the hostname variable in elasticsearch.yml and
trying to use the plugin manager, initialization will fail because the property could
not be resolved. This change will allow the hostname to be resolved in the same
manner as the service scripts.
Closes#10902
In #11072 we are adding a check that will prevent opening of old indices. However, this check doesn't take into consideration the fact that indices can be made compatible with the current version through upgrade API. In order to make compatibility check aware of the upgrade, the upgrade API should write a new setting `index.version.minimum_compatible` that will indicate the minimum compatible version of lucene this index is compatible with and `index.version.upgraded` that will indicate the version of elasticsearch that performed the upgrade.
Closes#11095
This commit adds a shell script which:
* move current elasticsearch core source in `/core`
* fetch `elasticsearch-parent` project in `/`
* fetch plugins in `/plugins`
* change `groupId` for all plugins to `org.elasticsearch.plugin` so versions won't conflict in maven central
* remove plugins/name/dev-tools dir which is not needed anymore
* remove plugins/name/.git dir
* remove plugins/name/LICENSE and plugins/name/CONTRIBUTING files
* clean `core/pom.xml` of useless settings that are inherited from parent project.
* `core/pom.xml` is adapted to change location of rest tests definition (`../`)
* change core name to `Elasticsearch Core`
* remove `plugins` dir from `.gitignore`
Plugins added:
* Analysis
* analysis-kuromoji
* analysis-smartcn
* analysis-stempel
* analysis-phonetic
* analysis-icu
* Mapper
* mapper-attachments
* Language
* lang-python
* lang-mvel
* lang-javascript
* Cloud
* cloud-gce
* cloud-azure
* cloud-aws
River plugins are ignored but might be added if we want to.
Todo:
* check and adapt our release tool. It now has to upload all submodules as well.
* adapt Jenkins jobs
The wildcard cat API REST tests relied on bulk.max and bulk.min in
the thread_pool response. However due to the thread pool types being
randomized in InternalTestCluster, the min/max values were not guaranteed
to exist (the cached thread pool type is unbounded and thus does not have a
max value).
In order to prevent this, the test has been removed and now the cat
nodes test is used for wildcard testing, which always returns stats
about the heap.
The Directory#makeLock API is trappy and can easily lead to unexpected
lock release if native locks are used. see LUCENE-6507 for details.
This commit consolidates the lock lock into one place and only returns
the lock instance if we actually acquired it.
Aggregations like Sampler and TopHits that require access to scores did not work if the query has size param set to zero. The assumption was that the Lucene query scoring logic was not required in these cases.
Added a Junit test to demonstrate the issue and a fix which relies on earlier creation of Collector wrappers so that Collector.needsScores() calls work for all search operations.
Closes#11119
On Windows, colons ':' are illegal in file names and since we use a Path to
check if the test is blacklisted, tests with a colon in the test section name
will fail. This change simply removes the colon from the name when matching
against the blacklist.
This catches UnsatifiedLinkError when attempting to load the JNA Native
class, in cases where there are error loading the native libraries that JNA
needs to function.
Today, JNA is a optional dependency in the build but when running tests or running
with mlockall set to true, JNA must be on the classpath for Windows systems since
we always try to load JNA classes when using mlockall.
The old Natives class was renamed to JNANatives, and a new Natives class is
introduced without any direct imports on JNA classes. The Natives class checks to
see if JNA classes are available at startup. If the classes are available the Natives
class will delegate to the JNANatives class. If the classes are not available the
Natives class will not use the JNANatives class, which results in no additional attempts
to load JNA classes.
Additionally, all of the JNA classes were moved to the bootstrap package and made
package private as this is the only place they should be called from.
Closes#11360
Colin Goodheart-Smithe
Adrien Grand
Ryan Ernst
Christoph Büscher
Simon Willnauer
Britta Weber
Spyros Kapnissis
Simon Willnauer
Clinton Gormley
Boaz Leskes
Michael McCandless
Alexander Reelsen
Igor Motov
Lee Hinman
jaymode
David Pilato
markharwood
javanna
Robert Muir
Zachary Tong
Colin Goodheart-Smithe