This change adds a library to ML that can be used to deduce a log
file's structure given only a sample of the log file.
Eventually this will be used to add an endpoint to ML to make the
functionality available to end users, but this will follow in a
separate change.
The functionality is split into a library so that it can also be
used by a command line tool without requiring the command line
tool to include all server code.
* Implement Version in java
- This allows to move all all .java files from .groovy.
- Will prevent eclipse from tangling up in this setup
- make it possible to use Version from Java
* PR review comments
* Cluster formation plugin with reference counting
```
> Task :plugins:ingest-user-agent:listElasticSearchClusters
Starting cluster: myTestCluster
* myTestCluster: /home/alpar/work/elastic/elasticsearch/plugins/ingest-user-agent/foo
Asked to unClaimAndStop myTestCluster, since cluster still has 1 claim it will not be stopped
> Task :plugins:ingest-user-agent:testme UP-TO-DATE
Stopping myTestCluster, since no of claims is 0
```
- Meant to auto manage the clusters lifecycle
- Add integration test for cluster formation
* Fix rebase
* Change to `useCluster` method on task
This commit introduces the ability for global checkpoint listeners to be
registered at the shard level. These listeners are notified when the
global checkpoint is updated, and also when the shard closes. To
encapsulate these listeners, we introduce a shard-level component that
handles synchronization of notification and modifications to the
collection of listeners.
This is related to #31835. It moves the default connection profile into
the ConnectionManager class. The will allow us to have different
connection managers with different profiles.
This removes custom Response classes that extend `AcknowledgedResponse` and do nothing, these classes are not needed and we can directly use the non-abstract super-class instead.
While this appears to be a large PR, no code has actually changed, only class names have been changed and entire classes removed.
[ML] Removing old per-partition normalization code
Per-partition normalization is an old, undocumented feature that was
never used by clients. It has been superseded by per-partition maximum
scoring.
To maintain communication compatibility with nodes prior to 6.5 it is
necessary to maintain/cope with the old wire format
This test is superfluous - it was added to address #32770 but it later turned out there was an existing test that just required a fix to provide the missing test coverage.
Closes#32855
The dissect library will be used for the ingest node as an alternative
to Grok to split a string based on a pattern. Dissect differs from
Grok such that regular expressions are not used to split the string.
Note - Regular expressions are used during construction of the
objects, but not in the hot path.
A dissect pattern takes the form of: '%{a} %{b},%{c}' which is
composed of 3 keys (a,b,c) and two delimiters (space and comma).
This dissect pattern will match a string of the form: 'foo bar,baz'
and will result a key/value pairing of 'a=foo, b=bar, and c=baz'.
See the comments in DissectParser for a full explanation.
This commit does not include the ingest node processor that will consume
it. However, the consumption should be a trivial mapping between the
key/value pairing returned by the parser and the key/value pairing
needed for the IngestDocument.
This change removes the PasswordHashingBootstrapCheck and replaces it
with validation on the setting itself. This ensures we always get a
valid value from the setting when it is used.
This change moves the validation for values of usernames and passwords
from the request to the transport action. This is done to prevent
the need to move more classes into protocol once we add this API to the
high level rest client. Additionally, this resolves an issue where
validation depends on settings and we always pass empty settings
instead of the actual settings.
Relates #32332
This commit fixes existing uses of forbidden apis in the test framework
and re-enables the forbidden apis check. It was previously completely
disabled and had missed a rename of the forbidden apis signatures files.
closes#32772
This changes the whitelist parameter fqn_only to no_import when specifying that a
whitelisted class must have the fully-qualified-name instead of a shortcut name. This more
closely correlates with Java imports, hence the rename.
The HipChatMessage#render is no longer used, and instead the
HipChatAccount#render is used in the ExecutableHipChatAction. Only a
test that validated the HttpProxy used this render method still. This
commit cleans it up.
The auth.basic package was an example of a single implementation
interface that leaked into many different classes. In order to clean
this up, the HttpAuth interface, factories, and Registries all were
removed and the single implementation, BasicAuth, was substituted in all
cases. This removes some dependenies between Auth and the Templates,
which can now use static methods on BasicAuth. BasicAuth was also moved
into the http package and all of the other classes were removed.
The PagerDuty v1 API is EOL and will stop accepting new accounts
shortly. This commit swaps out the watcher use of the v1 API with the
new v2 API. It does not change anything about the existing watcher
API.
Closes#32243
Minio does not support dynamic ports. The workaround here is to scan for a free port first. This is
not foolproof, but as we don't expect too many of these builds to run at once on the same machine,
this should do the trick.
Closes#32701Closes#32208
This commit adds a java time version of the existing rounding classes, which features the same test suite and a small test class to check if serialization works as expected.
Significance score doubles were being parsed as long. Existing tests did not catch this because SignificantLongTermsTests and SignificantStringTermsTests did not set the score. Fixed these and also added integration test.
Thanks for the report/fix, Blakko
Closes#32770
* INGEST: Create Index Before Pipeline Execute
* Ensures that indices are created before the default pipeline setting is read to correcly handle the case of an index template containing a default pipeline (without the fix the first document does not get the pipeline applied as explained in #32758)
* closes#32758
#31821 introduced an unreleased bug where NOOP updates were incorrectly mutating the bulk
shard request, inserting null item to be replicated, which would result in NullPointerExceptions when
serializing the request to be shipped to the replicas.
Closes#32808
This is related to #31835. This commit adds a connection manager that
manages client connections to other nodes. This means that the
TcpTransport no longer maintains a map of nodes that it is connected
to.
All Unit tests in this module are muted in FIPS 140 JVMs and
as such the CI run fails. This commit disables test task for the
module in a FIPS JVM and reverts adding a dummy test in
4cbcc1.
WildflyIT test fails in a FIPS JVM due to the amount of output in stderr. The excessive stderr output is due to https://bugs.openjdk.java.net/browse/JDK-8202893 and is not an indication of a failure that should be tracked.
This commit adjusts the limit to something more lenient that would allow the test to succeed.
Reverts #32543
We only upgrade the ID when the state is saved in one of four scenarios:
- when we reach a checkpoint (every 50 pages)
- when we run out of data
- when explicitly stopped
- on failure
The test was relying on the pre-upgrade to finish, save state and then
the post-upgrade to start, hit the end of data and upgrade ID. THEN
get the new doc and apply the new ID.
But I think this is vulnerable to timing issues. If the pre-upgrade
portion shutdown before it saved the state, when restarting we would run
through all the data from the beginning with the old ID, meaning both
docs would still have the old scheme.
This change makes the pre-upgrade wait for the job to go back to STARTED
so that we know it persisted the end point. Post-upgrade, it stops and
restarts the job to ensure the state was persisted and the ID upgraded.
That _should_ rule out the above timing issue.
Closes#32773
Added infrastructure to push through the 'person name field value' to
the normalizer process. This is required by the normalizer to retrieve
the maximum scores for individual partitions.
The request and response classes have been extracted from `IndexUpgradeInfoAction` into top-level classes, and moved to the protocol jar. The `UpgradeActionRequired` enum is also moved.
Relates to #29827
* Add a task to run forbiddenapis using cli
Add a task that offers an equivalent check to the forbidden APIs plugin,
but runs it using the forbiddenAPIs CLI instead.
This isn't wired into precommit first, and doesn't work for projects
that require specific signatures, etc. It's meant to show how this can
be used. The next step is to make a custom task type and configure it
based on the project extension from the pugin and make some minor
adjustments to some build scripts as we can't bee 100% compatible with
that at least due to how additional signatures are passed.
Notes:
- there's no `--target` for the CLI version so we have to pass in
specific bundled signature names
- the cli task already wires to `runtimeJavaHome`
- no equivalent for `failOnUnsupportedJava = false` but doesn't seem to
be a problem. Tested with Java 8 to 11
- there's no way to pass additional signatures as URL, these will have
to be on the file system, and can't be resources on the cp unless we
rely on how forbiddenapis is implemented and mimic them as boundled
signatures.
- the average of 3 runs is 4% slower using the CLI for :server.
( `./gradlew clean :server:forbiddenApis` vs `./gradlew clean
:server:forbiddenApisCli`)
- up-to-date checks don't work on the cli task yet, that will happen
with the custom task.
See also: #31715
This commit adds missing debug log statements for exceptions
that occur during ticket validation. I thought these
get logged somewhere else in authentication chain
but even after enabling trace logs I could not see them
logged. As the Kerberos exception messages are cryptic
adding full stack trace would help debugging faster.
With the move to java time, the default formatter used by toString on
ZonedDateTime uses optional components for least significant portions of
the date. This commit changes the cat indices api to use a strict date
time format, which will always output milliseconds, even if they are
zero.
closes#32466
* Clear Job#finished_time when it is opened (#32605)
* not returning failure when Job#finished_time is not reset
* Changing error log string and source string
Our rest testing framework has support for sniffing the host metadata on
startup and, before this change, it'd sniff that metadata before running
the first test. This prevents running these tests against
elasticsearch installations that won't support sniffing like Elastic
Cloud. This change allows tests to only sniff for metadata when they
encounter a test with a `node_selector`. These selectors are the things
that need the metadata anyway and they are super rare. Tests that use
these won't be able to run against installations that don't support
sniffing but we can just skip them. In the case of Elastic Cloud, these
tests were never going to work against Elastic Cloud anyway.
Benjamin Trent
David Roberts
Armin Braun
Alpar Torok
Jason Tedor
Tim Brooks
Ed Savage
Lee Hinman
Ed Savage
markharwood
Andy Bristol
Jake Landis
Jay Modi
Ryan Ernst
Jack Conradson
Michael Basnight
Yannick Welsch
Alexander Reelsen
Nhat Nguyen
Ioannis Kakavas
Zachary Tong
Dimitris Athanasiou
Morten Segelvik
Luca Cavanna
Yogesh Gaikwad
Nik Everett