In the codebase there is the non-written convention that classes that extend `MappedFieldType` are generally called `*FieldType`. With this commit we adopt the same convention for runtime field types which allows us to shorten their names by removing the `Mapped` portion which is implicit.
We would only check for a null value and not for an empty string so
that meant that we were not actually enforcing this mandatory
setting. This commits ensures we check for both and fail
accordingly if necessary, on startup
7.x client can pass media type with a version which will return a 7.x
version of the api in ES 8.
In ES server 7 this media type shoulld be accepted but it serve the same
version of the API (7x)
relates #61427
Introduce : operator for doing case insensitive string comparisons.
Recognizes "*" for wildcard matches in string literals.
Restricted only to string types.
Relates #62941
(cherry picked from commit 201e577e65f26a9b958a6197fe6c7268da39de29)
Backport for #62985 that includes the related changes, but
not the actual plugin for yamlRestCompatTest. The plugin
is not necessary in 7.x, and back porting relevant changes to
help keep 7.x code inline with master.
* Add System Indices check to AutoCreateIndex
By default, Elasticsearch auto-creates indices when a document is
submitted to a non-existent index. There is a setting that allows users
to disable this behavior. However, this setting should not apply to
system indices, so that Elasticsearch modules and plugins are able to
use auto-create behavior whether or not it is exposed to users.
This commit constructs the AutoCreateIndex object with a reference to
the SystemIndices object so that we bypass the check for the user-facing
autocreate setting when it's a system index that is being autocreated.
We also modify the logic in TransportBulkAction to make sure that if a
system index is included in a bulk request, we don't skip the
autocreation step.
Since `fork` is not used, is undocumented in Python EQL and
there is no plan at the moment to implement it in the future,
removing it from the grammar. User will get parsing exceptions
instead of higher level messages about unsupported features
which can lead to wrong expectations.
(cherry picked from commit f6a0f8f01c1b1893bab86629d1de73e9f9dae8dc)
Currently we read in 64KB blocks from the network. When TLS is not
enabled, these bytes are normally passed all the way to the application
layer (some exceptions: compression). For the HTTP layer this means that
these bytes can live throughout the entire lifecycle of an indexing
request.
The problem is that if the reads from the socket are small, this means
that 64KB buffers can be consumed by 1KB or smaller reads. If the socket
buffer or TCP buffer sizes are small, the leads to massive memory
waste. It has been identified as a major source of OOMs on coordinating
nodes as Elasticsearch easily exhausts the heap for these network bytes.
This commit resolves the problem by placing a handler after the TLS
handler to copy these bytes to a more appropriate buffer size as
necessary. This comes after TLS, because TLS is a framing layer which
often resolves this problem for us (the 64KB buffer will be decoded
into a more appropriate buffer size). However, this extra handler will
solve it for the non-TLS pipelines.
This adds the network property from the MaxMind Geo ASN database.
This enables analysis of IP data based on the subnets that MaxMind have
previously identified for ASN networks.
closes#60942
Co-authored-by: Peter Ansell <p_ansell@yahoo.com>
This test accesses system indices for 2 reasons.
First, it creates a filter that has a different type. This was done
to assert that filter is not returned from the APIs. However,
now that access to the `.ml-meta` index is restricted,
it is not really a concern.
Second, it creates a `.ml-meta` index without mappings to test
the get API does not fail due to lack of mappings on a sorted field,
namely the `filter_id`. Once again, this test is less useful once
system indices have restricted access.
Relates #62501
Backport of #63111
Bouncy Castle's BC-FJA-1.0.2 has been certified for a while now
but we had noticed that it seems to be rather entropy hungry and
ES would start very slowly ( and tests would take forever )
because of blocking calls to /dev/random.
We verified that this is resolved when enabling hw RNG or a
software one like haveged. While rng-tools should be suggested for
production uses, our ephemeral workers have haveged installed
which should work just fine for CI.
Backport of 63099
This fixes a bug introduced when moving from mget to ids query. While
mget returns all the ids given, id query is a search query and thus by
default returns only 10 documents.
The fix correctly sets the expected size so all the information is
returned inside the response.
Fix#63030
(cherry picked from commit 09ba85548a0142a1fe8376efea9cc4e7764a207c)
It extracts the query capabilities from AbstractGeometryFieldType into two new interfaces, GeoshapeQueryable and ShapeQueryable. Those interfaces are implemented by the final mappers.
Bwc tests can consume much time to build and to run so it's nice to be
able to skip them when running the `check` task on the SQL module.
Introduce a new task `checkNoBwc` so one can use:
```
./gradlew -p x-pack/plugin/sql checkNoBwc
```
to skip them.
(cherry picked from commit a52e1846f338f6869273181c6f248579581fa68c)
Previously, backquote couldn't not be used inside an escaped identifier,
e.g.:
```
`my`identifier` = "some_value"
```
was not allowed. Introduce escaping of the backquote by using a
double backquote:
```
`my``identifier` = "some_value"
```
(cherry picked from commit 49514121486f42a58674b3e5901de4021fda5c15)
Unmute DeleteByQueryConcurrentTests
testConcurrentDeleteByQueriesOnDifferentDocs test.
LUCENE-9449 introduced a bug in sorting on _doc,
which resulted in failure of this test. As Lucene bug
has been fixed, this reenables the test.
Closes#62609
When an opened anomaly detection job is updated with a detection
rule that references a filter, apart from updating the c++ process
with the rule, we also need to update it with the referenced filter.
This commit fixes a bug which led to the job not applying such updates
on-the-fly.
Fixes#62948
Backport of #63057
Remove Count class and related artifacts since that functionality is not
(yet) available.
Update parser name for better error reporting.
Fix#62131
(cherry picked from commit 060f500346788c4c5d0b3b9c045facec5d677d3d)
Test testSortDifferentFormatsShouldFail was occasionally failing for
2 reasons:
1) Documents on "idx2" were not available for search before a
search request started
2) Running a test multiple times was causing
occasional ResourceAlreadyExistsException for idx2,
as idx2 was not deleted for a test.
This patch makes the following fixes:
1) Sets up immediate refresh policy for docs in the index"idx2"
2) Creates an index idx2 only once per cluster
Closes: #62997
This is a very minor optimization but trivial to implement, so might as well.
```
Benchmark (nGramStrs) Mode Cnt Score Error Units
NGramProcessorBenchmark.ngramInnerLoop 1,2,3 avgt 20 4415092.443 ± 31302.115 ns/op
NGramProcessorBenchmark.ngramOuterLoop 1,2,3 avgt 20 4235550.340 ± 103393.465 ns/op
```
This measurement is in nanoseconds, consequently, the overall performance of inference is dominated by other factors (i.e. map#put). But, this optimization adds up overtime and is simple.
This commit adds bwc testing for data frame analytics.
The bwc tests only go back to the 7.9.0.
Meaning, initially only rolling upgrades from 7.9.x -> 7.10.0 are tested.
Since the feature was experimental in < 7.9.0, this is acceptable.
Data frame analytics results format changed in version `7.10.0`.
If existing jobs that were not completed are restarted, it is
possible the destination index had already been created. That index's
mappings are not suitable for the new results format.
This commit checks the version of the destination index and deletes
it when the version is outdated. The job will then continue by
recreating the destination index and reindexing.
Backport of #62960
Today InternalClusterInfoService ignores hidden indices when
retrieving shard stats of the cluster. This can lead to suboptimal
shard allocation decisions as the size of shards are taken into
account when allocating new shards or rebalancing existing shards.
Benjamin Trent
Luca Cavanna
Ioannis Kakavas
David Kyle
Rafi Estrada
Christoph Büscher
Przemyslaw Gomulka
Costin Leau
Jake Landis
William Brafford
Igor Motov
Marios Trivyzas
Lee Hinman
Tim Brooks
Dan Hermann
Dimitris Athanasiou
Hendrik Muhs
Ignacio Vera
James Rodewig
Howard
Alan Woodward
Lisa Cawley
István Zoltán Szabó
Mayya Sharipova
Yang Wang
Przemysław Witek
Tanguy Leroux