Many scripts are used to start/stop and install/uninstall elasticsearch. These scripts share a lot of configuration properties like directory paths, max value for a setting, default user etc. Most of the values are identical but some of them are different depending of the platform (Debian-based or Redhat-based OS), depending of the way elasticsearch is started (shell script, systemd, sysv-init...) or the way it is installed (zip, rpm, deb...). Today the values are duplicated in multiple places, making it difficult to maintain the scripts or to update a value.
This pull request make this more uniform: values used in scripts must be defined in a common packaging.properties file. Each value can be overridden in another specific packaging.properties file for Debian or Redhat. All startup and installation scripts are filtered with the common then the custom packaging.properties files before being packaged as a zip/tar.gz/rpm/dpkf archive.
Follow-up of #9135. We initially decreased the stack size because it would end
up a lot of memory when there are many threads. But we also have some thread
pools that may be oversized, in particular the search thread pool.
This commit proposes to decrease the default search thread pool size from
`3 * num_procs` to `3 * num_procs / 2 + 1`. This is large enough to be sure
that we can use all the machine resources even with a search-only work load
but not too large in order to not consume too much memory because of the stack
size and thread locals.
This pull request makes boolean handled like dates and ipv4 addresses: things
are stored as as numerics under the hood and aggregations add some special
formatting logic in order to return true/false in addition to 1/0.
For example, here is an output of a terms aggregation on a boolean field:
```
"aggregations": {
"top_f": {
"doc_count_error_upper_bound": 0,
"buckets": [
{
"key": 0,
"key_as_string": "false",
"doc_count": 2
},
{
"key": 1,
"key_as_string": "true",
"doc_count": 1
}
]
}
}
```
Sorted numeric doc values are used under the hood.
Close#4678Close#7851
A shard recovery response might serialize a shard state at the same time that it is
modified by the recovery process. The test
RelocationTests.testMoveShardsWhileRelocation
failed because of this with a ConcurrentModificationException.
closes#10381
RoutingTables activePrimaryShardsGrouped(), allActiveShardsGrouped() and
allAssignedShardsGrouped() methods treated empty index array input
parameters as meaning "all" indices and expanded to the routing maps
keyset. However, the expansion of index names is now already done in
MetaData#concreteIndices(). Returning an empty index name list here
when a wildcard pattern didn't match any index name could lead to
problems like #9081 because the RoutingTable still expanded this
list of names to "_all". In case of e.g. the recovery endpoint this
could lead to problems.
Closes#9081Closes#10148
This fixes an issue where this was logged:
```
[node_t1] [test][0] flush with org.elasticsearch.action.admin.indices.flush.FlushRequest@65f6f1e
```
by adding a .toString() method to FlushRequest.
It also changes:
```
creating Index [test], shards [1]/[2]
```
to:
```
creating Index [test], shards [1]/[2s]
```
If shadow replicas are being used.
Today we reuse the UUID of the source index on restore. This can create conflicts
with existing shard state on disk. This also causes multiple indices with the same
UUID. This commit preserves the UUID of an existing index or creates a new UUID for
a newly created index.
For quite some time now, our networking layer makes sure to create safe messages as in not using the shared buffers. This is great, and we should remove the old support for "unsafe" notion in our codebase.
closes#10360
Prevents a current edge case resolving concrete aliases or index names in cluster MetaData
that could potentialy lead to NullPointerException when the IndicesOptions don't allow
wildcard expansion and the method is called with aliasesOrIndices argument null or emtpy list.
This change adds a check for that and introduces randomized test that catches this.
Closes#10342Closes#10339
For optimization pruposes a function score query with an empty function
will just result in the original sub query. However, sometimes one might
want to use function_score query to actually filter out docs within for example
bool clauses by using the min_score functionallity.
Therefore the sub query should only be used without wrapping inside
a function_score query if min_score was also not set.
closes#10253closes#10326
When deleting a shard th node that deletes th shard first checks if all shard copies are
started on other nodes. A message is sent to each node tand each node checks locally for
STARTED or RELOCATED.
However, it might happen that the shard is still in state POST_RECOVERY, like this:
shard is relocating from node1 to node2
1. relocated shard on node2 goes in POST_RECOVERY and node2 sends shard started to master
2. master updates routing table and sends new cluster state to node1 and node2
3. node1 processes the cluster state and asks node2 if it has the active shard
before node2 processes the new cluster state (which would cause it to set the shard to started)
4. node2 sends back it does not have the shard started and so node1 does not delete it
This can be avoided by waiting until cluster state that sets the shard to started is actually processed.
closes#10018
Today there is a chance that the state version for shard, index or cluster
state goes backwards or is reset on a full restart etc. depending on
several factors not related to the state. To prevent any collisions
with already existing state files and to maintain write-once properties
this change introductes an incremental state ID instead of using the plain
state version. This also fixes a bug when the previous legacy state had a
greater version than the current state which causes an exception on node
startup or if left-over files are present.
Closes#10316
Now that fine-grained script settings are supported (#10116) we can remove support for the script.disable_dynamic setting.
Same result as `script.disable_dynamic: false` can be obtained as follows:
```
script.inline: on
script.indexed: on
```
An exception is thrown at startup when the old setting is set, so we make sure we tell users they have to change it rather than ignoring the setting.
Closes#10286
The query cache is disabled on dfs_query_then_fetch so we need to enforce
query_then_fetch instead of relying on the randomized search type set by the
test framework.
This commit brings the benefits of the `count` search type to search requests
that have a `size` of 0:
- a single round-trip to shards (no fetch phase)
- ability to use the query cache
Since `count` now provides no benefits over `query_then_fetch`, it has been
deprecated.
Close#7630
Even if there is a background thread that periodically closes search contexts
that seem unused (every minute by default), it is important to close search
contexts as soon as possible in order to not keep unnecessary open files or
to prevent segments from being deleted.
This check would help ensure that refactorings of the SearchContext management
like #9296 are correct.
This adds the exec-maven-plugin that allows a developer to run:
```
mvn exec:exec
```
To launch the `Bootstrap` process similar to the way that a Java IDE
would. All the logs go to logs/elasticsearch.log (or wherever
configured)
Adds a getter for the actual netty channel in NettyTransportChannel. The
channel can be used by plugins that need access into netty when processing
requests.
FakeRestRequest is used by a few tests and can also be leveraged by
tests outside of elasticsearch. Moving the package will mean the class
gets exported as part of the test jar.
We already force a refresh in index/create ops, to clear version map
when it's using too much RAM, but we were failing to do this for
deletes, so an app that does tons of deletes with no indexing, and has
set refresh_interval to -1, would have version map using unbounded
RAM.
Closes#10312
After processing mapping updates from the master, we compare the resulting binary representation of them and compare it the one cluster state has. If different, we send a refresh mapping request to master, asking it to reparse the mapping and serialize them again. This mechanism is used to update the mapping after a format change caused by a version upgrade.
The very same process can also be triggered when an old master leaves the cluster, triggering a local cluster state update. If that update contains old mapping format, the local node will again signal the need to refresh, but this time there is no master to accept the request. Instead of failing (which we now do because of #10283, we should just skip the notification and wait for the next elected master to publish a new mapping (triggering another refresh if needed).
Closes#10311
Tanguy Leroux
Adrien Grand
Martijn van Groningen
Britta Weber
Christoph Büscher
Simon Willnauer
wittyameta
Lee Hinman
Simon Willnauer
Ryan Schneider
Reuben Sutton
Shay Banon
Lucas Chi
Boaz Leskes
David Wittman
javanna
Patrick Peschlow
Igor Motov
Michael McCandless
jaymode