We have a custom logger implementation known as a prefix logger that is
used to write every message by the logger with a given prefix. This is
useful for node-level, index-level, and shard-level messages where we
want to log the node name, index name, and shard ID, respectively, if
possible. The mechanism that we employ is that of a marker. Log4j has a
built-in facility for managing these markers, but its effectively a
memory leak because these markers are held in a map and can never be
released. This is problematic for us since indices and shards do not
necessarily have infinite life spans and so on a node where there are
many indices being creted and destroyed, this infinite lifespan can be a
problem indeed. To solve this, we use our own cache of markers. This is
necessary to prevent too many instances of the marker for the same
prefix from being created (just think of all the shard-level components
that exist in the system), and to workaround the effective leak in
Log4j. These markers are stored as weak references in a weak hash
map. It is these weak references that are unneeded. When a key is
removed from a weak hash map, the corresponding entry is placed on a
reference queue that is eventually cleared. This commit simplifies
prefix logger by removing this unnecessary weak reference wrapper.
Relates #22460
This commit changes the task type of the checkoutBwcBranch task to Exec
from LoggedExec so that the output of the checkout command is
shown. This enables us to see the SHA used for the checkout which can be
useful when debugging a BWC break.
Relates #25166
When the cluster state is updated with Shard Started entries, it simply adds "shard-started" as the source of the change.
This adds the index name and shard ID so that we can see who/what is spamming the changes when the index creation step has already left the cluster state.
Commit bf007e8d936d82ed2d28c70da1a7366cfa862342 was a forward port of logic needed in 5.x to get
the correct bwc branch. However, other changes on master meant that this forward port was not
needed and actually broke the bwc tests. This change removes the incorrect if statement.
Relates #25134
There are a few places where arrays are output in messages yet the
output would merely use the default toString implementation rather than
actually putting the content of the array in the message. This commit
fixes the issue.
Relates #24340
This change extends the tests and parsing of SearchResponse to make sure we can
skip additional fields the parser doesn't know for forward compatibility
reasons.
This commit adds back "id" as the key within a script to specify a
stored script (which with file scripts now gone is no longer ambiguous).
It also adds "source" as a replacement for "code". This is in an attempt
to normalize how scripts are specified across both put stored scripts and script usages, including search template requests. This also deprecates the old inline/stored keys.
When `index.mapping.single_type` is `true` the `_uid` field is not used and instead `_id` field is used.
Prior to this change nested documents would in this case still use the `_uid` field to mark to what root
document they belong to. In case of deleting documents this could lead to only the root Lucene document
to be deleted and not the nested Lucene documents. This broke the docid block ordering the block join
relies on in order to work correctly and thus causing the `nested` query, `nested` aggregation, nested sorting
and nested inner hits to either fail or yield incorrect results.
This bug only manifests in 6.0.0-ALPHA2 release and snaphots (5.5.0-SNAPSHOT, 5.6.0-SNAPSHOT, 6.0.0-SNAPSHOT).
This was introduced in #24460: the constructor of `Translog.Delete` that takes
a `StreamInput` does not set the type and id. To make it a bit more robust, I
made fields final so that forgetting to set them would make the compiler
complain.
This change removes the `postings` highlighter. This highlighter has been removed from Lucene master (7.x) because it behaves
exactly like the `unified` highlighter when index_options is set to `offsets`:
https://issues.apache.org/jira/browse/LUCENE-7815
It also makes the `unified` highlighter the default choice for highlighting a field (if `type` is not provided).
The strategy used internally by this highlighter remain the same as before, it checks `term_vectors` first, then `postings` and ultimately it re-analyzes the text.
Ultimately it rewrites the docs so that the options that the `unified` highlighter cannot handle are clearly marked as such.
There are few features that the `unified` highlighter is not able to handle which is why the other highlighters (`plain` and `fvh`) are still available.
I'll open separate issues for these features and we'll deprecate the `fvh` and `plain` highlighters when full support for these features have been added to the `unified`.
This change extends the tests and parsing of SearchShardFailure to make sure we
can skip fields the parser doesn't know for forward compatibility reasons.
When parsing responses we should be ignoring any new unknown fields or inner
objects in most cases to be forward compatible with changes in core on the
client side. This change adds test for this for Suggestions and its various
subclasses to check if we are able to ignore new fields and objects in the
xContent.
This PR enables Ingest plugins to leverage processor-scoped REST
endpoints. First of which being the Grok endpoint that retrieves
Grok Patterns for users to retrieve all the built-in patterns.
Example usage: Kibana Grok Autocomplete!
When we disabled `_all` by default for indices created in 6.0, we missed adding
a layer that would handle the situation where `_all` was not enabled in 5.x and
then the cluster was updated to 6.0, this means that when the cluster was
updated the `_all` field would be disabled for 5.x indices and field values
would not be added to the `_all` field.
This adds a compatibility layer for 5.x indices where we treat the default
enabled value for the `_all` field to be `true` if unset on 5.x indices.
Resolves#25068
The Log4j dependency is separated into two artifacts, the API and the
core implementation. This is to enable replacing Log4j on the backend
through the SLF4J bridge with another logging implementation. For this
reason, the dependencies are marked as optional. This causes confusion
amongst users as to use the bridge, the API should be non-optional since
it is needed for the bridge to function correctly. While they could pull
it into their application directly, it would be clearer if we simply
marked this depdendency as non-optional. Note that this does not mean
that users have to use Log4j for logging in their application, so we are
not marking core as required, it only clarifies what they need to be
able to plug in a different logging implementation.
Relates #25136
When testing against the previous 5.x release, the bwc project incorrectly would checkout the 5.x
branch instead of the 5.5 branch as it still had the logic that applies for major versions bwc. This change adds
a check to compare the major version when making the decision on the branch to use.
Previously this would output:
```
GET /test-1/_mappings
{ }
```
And after this change:
```
GET /test-1/_mappings
{
"test-1": {
"mappings": {}
}
}
```
To bring parity back to the REST output after #24723.
Relates to #25090
Previously in #24723 we changed the `_alias` API to not go through the
`RestGetIndicesAction` endpoint, instead creating a `RestGetAliasesAction` that
did the same thing.
This changes the formatting so that it matches the old formatting of the
endpoint, before:
```
GET /test-1/_alias
{ }
```
And after this change:
```
GET /test-1/_alias
{
"test-1": {
"aliases": {}
}
}
```
This is related to #25090
GeoUtils#isValidLongitude is inconsistent with GeoUtils#isValidLatitude.
Neither technically need the isInfinite() check because they then compare
against min and max values.
When parsing resonses we should be ignoring any new unknown fields or inner
objects in most cases to be forward compatible with changes in core on the
client side. This change adds test for this for QueryProfileShardResult and
nested substructures and changes the parsing code where necessary to be able to
ignore new fields and objects in the xContent.
Test: randomVersionBetween works with unreleased
Modifies randomVersionBetween so that it works with unreleased
versions. This should make switching a version from unreleased
to released much simpler.
Added common base class for ScriptDocValues.Strings and ScriptDocValues.BytesRefs now that these classes are very similar.
Also cleaned up the BinaryDVFieldDataTests:
* Use junit assertions instead of hamcrest
* Use BytesRef directly instead of byte[]
Closes#24785
The FVH fails with an NPE when a match phrase prefix is rewritten in an empty phrase query.
This change makes sure that the multi match query rewrites to a MatchNoDocsQuery (instead of an empty phrase query) when there is
a single term and that term does not expand to any term in the index.
Fixes#25088
This commit refactors the query phase in order to be able
to automatically detect queries that can be early terminated.
If the index sort matches the query sort, the top docs collection is early terminated
on each segment and the computing of the total number of hits that match the query is delegated to a simple TotalHitCountCollector.
This change also adds a new parameter to the search request called `track_total_hits`.
It indicates if the total number of hits that match the query should be tracked.
If false, queries sorted by the index sort will not try to compute this information and
and will limit the collection to the first N documents per segment.
Aggregations are not impacted and will continue to see every document
even when the index sort matches the query sort and `track_total_hits` is false.
Relates #6720
This commit modifies query_string, simple_query_string and multi_match queries to always use a DisjunctionMaxQuery when a disjunction over multiple fields is built. The tiebreaker is set to 1 in order to behave like the boolean query in terms of scoring.
The removal of the coord factor in Lucene 7 made this change mandatory to correctly handle minimum_should_match.
Closes#23966
This change extracts the main logic from `TransportClearScrollAction`
into a new class `ClearScrollController` and adds a corresponding unit test.
Relates to #25094
