This fixes pipeline aggregations used in cross cluster search from an older
version of Elasticsearch to a newer version of Elasticsearch. I broke
this in #53730 when I was too aggressive in shutting off serialization
of pipeline aggs. In particular, this comes up when the coordinating
node is pre-7.8.0 and the gateway node is on or after 7.8.0.
The fix is another step down the line to remove pipeline aggregators
from the aggregation tree. Sort of. It create a new
`List<PipelineAggregator>` member in `InternalAggregation` *but* it is
only used for bwc serialization and it is fed by the mechanism
established in #53730 to read the pipelines from the
These methods are not needed, we were only following a pattern in the
rest of the codebase, but it's legacy from the HLRC sharing
request/response objects with the server.
When one of ML's normalize processes fails to connect to the JVM
quickly enough and another normalize process for the same job
starts shortly afterwards it is possible that their named pipes
can get mixed up.
This change avoids the risk of that by adding an incrementing
counter value into the named pipe names used for normalize
processes.
Backport of #54636
* [ML] add num_matches and preferred_to_categories to category defintion objects (#54214)
This adds two new fields to category definitions.
- `num_matches` indicating how many documents have been seen by this category
- `preferred_to_categories` indicating which other categories this particular category supersedes when messages are categorized.
These fields are only guaranteed to be up to date after a `_flush` or `_close`
native change: https://github.com/elastic/ml-cpp/pull/1062
* adjusting for backport
Right now you can't tell from the task description whether or not the
search is a scroll. This adds that information to the description which
is super useful if you are trying to debug a cluster that is running out
of scroll contexts.
Today the usage service can let in some issues, such as handlers that do
not have a name, where the errors do not manifest until later (calling
the usage API), or conflicting handlers with the same name. This commit
addresses this by adding some validation to the usage service.
* Rename cat.transform => cat.transforms
To match the url.
We typically prefer singular url nouns but _cat tends to use plural and
this API does in fact uses `/_cat/transforms`
* also rename the api in the spec and tests
(cherry picked from commit c495d220ac8fedba7f70f82387cd6d6a672b8b14)
SamlIdentityProviderTests is failing with 409 conflicts that have not
been reproducible outside of CI.
This change turn on additional logging in this test to determine why
these conflict occur.
Relates: #54423
Backport of: #54475
This commit updates the rest API specs to validate against a
JSON schema for the specifications. Most updates are to add
a description, whilst others fix typos and unify conventions
e.g. deprecations, descriptions, urls starting with /. The schema
conforms to draft-07 JSON schema.
(cherry picked from commit da37e01d32f9764c3937736ef0c7d3ab40af9a77)
Adds tests for supported ValuesSourceTypes, unmapped fields, scripting,
and the missing param. The tests for unmapped fields and scripting are
migrated from the StatsIT integration test
This commit corrects the description for the request URI index for the Multi Get (mget) API.
The index can only be a single index name (multiple or wildcard expressions not supported),
and acts as the index to use when "ids" are specified, or a document in the "docs" array does
not specify an index.
(cherry picked from commit aa4926ed7f91dfbf7973a01b1e4682e91dda11a9)
This commit migrates the RestIntegTestTask from groovy to Java.
No changes to logic should be included, however the following changes
are needed:
* Move Fixture interface to Java (Java can not depend on Groovy classes)
* Support lazy evaluation of non-input System parameters (can not use Groovy strings)
* Use constants for system property names
* Remove dead System property pass through code (the build plugin does this already)
This adds tests for supported ValuesSourceTypes, unmapped fields,
scripting, and the missing param. The tests for unmapped fields and
scripting are migrated from the SumIT integration test
Tests for unmapped fields, the missing parameter, scripting, and correct
ValuesSource types in MissingAggregatorTests. Basic yaml tests for the
missing agg
For #42949
* Refactor nodes stats request builders to match requests (#54363)
* Remove hard-coded setters from NodesInfoRequestBuilder
* Remove hard-coded setters from NodesStatsRequest
* Use static imports to reduce clutter
* Remove uses of old info APIs
The test name says it is testing the put autoscaling decision API, but
that is not right, since no such API exists (nor will exist). This
commit corrects the name of this test to reflect the fact that the test
is about the put autoscaling policy API.
Because -1 is technically a valid TimeValue (as a sentinel value), that is now
explicitly checked for when validating gc_thresholds. The tests are also
adjusted to test this case separately from other negative values.
This commit adds a top-level link to the autoscaling API reference page
to the API docs. Additionally, we add a conditional guard on the API
pages to only include them in development builds of the docs.
Refactor SearchHit to have separate document and meta fields.
This is a part of bigger refactoring of issue #24422 to remove
dependency on MapperService to check if a field is metafield.
Relates to PR: #38373
Relates to issue #24422
Co-authored-by: sandmannn <bohdanpukalskyi@gmail.com>
We do not have access to JDK 9 collection convenience methods in 7.x
because we are compatible with JDK 8 there. Yet, we have recently added
a substitute for these convenience methods that even delegate to the
right places when running on JDK 9, to make backporting easier. This
commit utilizes these new methods in the autoscaling codebase.
Use the same ES cluster as both an SP and an IDP and perform
IDP initiated and SP initiated SSO. The REST client plays the role
of both the Cloud UI and Kibana in these flows
Backport of #54215
* fix compilation issues
EQL functions are an easy way for users to transform indexed data
at search time. However, using multiple functions can make
queries difficult to write and slows search speeds.
Users can circumvent this by indexing fields containing the transformed
data, but that usually slows index speeds.
This adds a related tip and example covering these tradeoffs.
This commit is the first in a series of commits that introduces
autoscaling policies, and APIs for working with them. For now, we
introduce the basic infrastructure, and a single API for putting an
autoscaling policy. We will follow in rapid succession with APIs for
getting, and deleting autoscaling policies.
When the SAML authentication is not successful, we return a SAML
Response with a status that indicates a failure. This commit adds
an error message in the REST API response along with the SAML
Response XML string so that the caller of the API can identify
that this is an unsuccessful response without needing to parse the
XML.
`AsyncSearchActionTests` currently fails quite often. That is since the introduction of `RestSubmitAsyncSearchActionTests` which indirectly manipulates the channels being tracked in `RestCancellableNodeClient`. There are channels left in the map after `RestSubmitAsyncSearchActionTests` is run, and later `AsyncSearchActionTests` checks that there are no channels in the map which makes each test method fail. This is particularily hard to reproduce as the order in which tests are run appears to be platform dependent.
The test cluster assertion that there are no channels in the map only makes sense in the context of internal cluster tests, while there may be collisions with unit tests that register http channels as part of their testing.
This can be solved by renaming `AsyncSearchActionTests` to `AsyncSearchActionIT`. This way it won't be run as part of unit tests but rather within another JVM where the number of channels is `0` and such assumption holds, because there are no expected manual manipulation of the channels.
Relates to #54180
Nik Everett
Jason Tedor
Christoph Büscher
David Roberts
qiye
Benjamin Trent
Aleh Zasypkin
Martijn Laarman
István Zoltán Szabó
Tim Vernum
Russ Cam
Andy Bristol
Jake Landis
lcawl
William Brafford
Gordon Brown
Mayya Sharipova
James Rodewig
Ioannis Kakavas
Dan Hermann
jimczi
bellengao
David Turner
Przemysław Witek
Luca Cavanna