This commit upgrades the checkstyle configuration from version 5.9 to
version 7.5, the latest version as of today. The main enhancement
obtained via this upgrade is better detection of redundant modifiers.
Relates elastic/elasticsearch#4810
Original commit: elastic/x-pack-elasticsearch@2c9b7d23dc
This commit reuses the automaton that defines the allowed fields in
`FieldSubsetReader` rather than resolving the list of all matching fields from
the mapping. As a side-effect this change solves a bug that unmapped fields
could previously not be read from the source. Moreover it avoids determinization
errors in the case that the number of matching fields is high.
It also uses `CharacterRunAutomaton` to evaluate automata against a given
string, which should be faster than naively stepping into the automaton since
`CharacterRunAutomaton` builds a lookup table of transitions.
Closeselastic/elasticsearch#4679
Original commit: elastic/x-pack-elasticsearch@a30913dbd5
The way we check for the triggered watches on start-up did not take into account
that an index could be closed and thus resulted in an NPE.
This commit adds a check to ensure that the watch index and triggered watches index
are open, before trying to check if all primary shards are active.
Original commit: elastic/x-pack-elasticsearch@ee05779963
This change adapts x-pack to pass on the parsed XContentType from rest requests to transport
requests and use this value in place of attempting to auto-detect the content type.
Original commit: elastic/x-pack-elasticsearch@57475fd403
Similarly to task status on normal tasks it's now possible to update task status on the persistent tasks. This should allow updating the state of the running tasks (such as loading, started, etc) as well as store intermediate state or progress.
Original commit: elastic/x-pack-elasticsearch@ed109cfa84
The ScrollDataExtractor needs to clear the scroll after
it is complete. Originally, it was thought that completing a scroll
leads to an automatic clearing of its context. That is not true,
thus manual clearing has to be requested.
- Also removes sorting in AggregationDataExtractor as it was redundant
Original commit: elastic/x-pack-elasticsearch@8f955da8ce
If `domainSplit(` is detected in an inline script, the function and params are injected into
the script.
The majority of this PR is actually test-related. Adds a unit test to check for the injected
script/params. Also adds another QA test which -- through a very round-about mechanism --
confirms that the injected script compiles and functions correctly. The QA test can
be simplified greatly once the Preview API is added.
Original commit: elastic/x-pack-elasticsearch@c7c35a982c
Changes are:
1. The detector validation endpoint is changed from /_xpack/ml/_validate/detector
to /_xpack/ml/anomaly_detectors/_validate/detector
2. A new endpoint is added for validating an entire job config:
/_xpack/ml/anomaly_detectors/_validate
Relates elastic/elasticsearch#630
Original commit: elastic/x-pack-elasticsearch@7b2031e746
* Store input fields for anomaly records and influencers
* Address review comments
* Remove DotNotationReverser
* Remove duplicated constants
* Can’t use the same date for all records as they will have equivalent Ids
Original commit: elastic/x-pack-elasticsearch@40796b5efc
This needs to be moved to the single-node-tests qa modules since integTests shouldn’t access modules.
Original commit: elastic/x-pack-elasticsearch@289b697eb8
A persistent action is a transport-like action that is using the cluster state instead of transport to start tasks. This allows persistent tasks to survive restart of executing nodes. A persistent action can be implemented by extending TransportPersistentAction. TransportPersistentAction will start the task by using PersistentActionService, which controls persistent tasks lifecycle. See TestPersistentActionPlugin for an example implementing a persistent action.
Original commit: elastic/x-pack-elasticsearch@8ef4103cd6
This used to be 60 seconds, dating back to the days when the controller
had to be started manually after starting Elasticsearch. However, now
Elasticsearch starts it automatically it should already be running when
we try to connect, so the timeout can be much lower. It just needs to
be long enough to give the C++ process time to create its named pipes.
2 seconds seems reasonable, and matches what we use for autodetect and
normalize.
Original commit: elastic/x-pack-elasticsearch@7300d68482
This contains the Painless-based DomainSplit function, generated static maps and basic tests. Due to cross-module complications, the tests are run by executing searches with script_fields and checking the response
Original commit: elastic/x-pack-elasticsearch@c6c2942e01
When source fields are not required, stored_fields can be disabled.
This can make the query faster as no stored fields have to be
decompressed. Note that this means no metadata (_id, _index, _type, etc.)
will be returned.
Original commit: elastic/x-pack-elasticsearch@b1ea526d83
This is related to elastic/elasticsearch#22116. X-pack opens socket connections for a number
of pieces of functionality (Active Directory support, ssl, email
notification, etc). As SocketPermissions are transitioned out of core,
x-pack will require connect permission. This pull request wraps
operations requiring these permissions in doPrivileged blocks.
Original commit: elastic/x-pack-elasticsearch@1bfee97550
Even though a search response may return a 200 status code, things could
still have gone wrong. A search response may report shard failures.
The datafeed extractors should check for that and report an extraction
error accordingly.
Closeselastic/elasticsearch#775
Original commit: elastic/x-pack-elasticsearch@5d6d899738
* Audit messages in .ml-audit
* Rename ml-int to .ml-meta
* Remove no release comment
* Fix compilation after classes moved to a different package
* Create the Audit, state and meta indices every time a job is created
* Revert change creating the audit index etc when the job is created
* Rename index .ml-audit -> .ml-notifications
Original commit: elastic/x-pack-elasticsearch@95168fa341
* Handle manual aggregations in datafeeds
Adds a DataExtractor implementation that runs aggregated searches.
The manual aggregations supported have the following limitations:
- each aggregation can hava 0 or 1 sub-aggregations
- the top aggregation has to be a histogram
- sub-aggregations have to be either terms aggregations or single value
metric aggregations.
The response is converted into flat JSON documents that contain only the
fields of interest and can be parsed without additional context from our
JSON parser. The fields in the JSON documents correspond to the names of the aggregations.
Closeselastic/elasticsearch#680
Original commit: elastic/x-pack-elasticsearch@7dfd2d31e6
The new constructor takes an Environment object. This is needed for migration to X-Pack since the environment instance is built by the XPackPlugin and then passed into the feature plugins.
Original commit: elastic/x-pack-elasticsearch@f25225bc6a
This adds a "doc_id" parameter to the index action itself, which can accept a single ID value. This also allows the payload to set _id (or _doc._id) for each document being indexed in order to support this with bulk index actions.
If doc_id and _id are used together, then this blocks the action.
Original commit: elastic/x-pack-elasticsearch@f186ccceb8
Most transforms will be replaced with Painless scripts.
The exception is the DateTransform, whose functionality is now simplified
to what existed before the other transforms were added.
The SINGLE_LINE format relied on transforms to extract fields, so has also
been removed, but this is reasonable as it strays into Logstash territory.
Relates elastic/elasticsearch#630Closeselastic/elasticsearch#39
Original commit: elastic/x-pack-elasticsearch@a593d3e0ad
The `Integers` class was only used in tests (but lurked around in the src) and is not needed.
Also replaced some lambda calls with their shorter equivalents.
Original commit: elastic/x-pack-elasticsearch@a81a5c33d3
In order to display a better UI page for the watch history, the state
of a throttled watch should be visualized. However, right now there
is no way to differentiate between a time throttled watch and a user
acknowledged watch (or action). This commit introduces a new type in a
throttled result, which in turn is used to set the execution state of
a watch.
Closeselastic/elasticsearch#4531
Original commit: elastic/x-pack-elasticsearch@b86e666e54
RequestContext is a leftover from when we had no thread context. This
commit removes the last place where it was used and uses the thread context
instead.
Original commit: elastic/x-pack-elasticsearch@50a2bff400
Today we wrap the checked IOException in an unchecked exception when
sending back a failure the security transport interceptor. Yet, if that failure handling
in-turn fails due to a broken response channel we should rather log the exception instead
of bubbling it up since it can have unforeseeable side-effects.
Relates to elastic/elasticsearch#4706
* fix line len
Original commit: elastic/x-pack-elasticsearch@03c3826112
This matches the way tests that need to run without an Elasticsearch
bootstrap are run in core Elasticsearch. This should make merging to
x-pack easier.
Note that the no bootstrap tests now run after the integration tests, but
this doesn't really matter.
Original commit: elastic/x-pack-elasticsearch@5547f457b6
Today we might get a rejection on a critical operation where `forceExecution=true` but
due to the fact that the forceExecution flag is not passed to the transport interceptor
interface we were not able to preserve this flag when forking off the request after authentication.
This causes serious issues if for instance a replication handler is rejected.
Closeselastic/elasticsearch#4704
Original commit: elastic/x-pack-elasticsearch@f0aad7dede
This commit adds a mechanism for defining known sensitive values in rest bodies so that
these can be filtered when auditing the request body.
Original commit: elastic/x-pack-elasticsearch@d138a6bff7
The bulk request needed resetting after it was executed otherwise stale documents are persisted repeatedly after they have been updated causing a versioning error
Original commit: elastic/x-pack-elasticsearch@263fa9d25d
Since some of the rest handlers depend on components constructed
in `createComponents` we use `SetOnce` to save a reference to
the components at the class level and reuse the reference in
`initRestHandlers`. This does require that `initRestHandlers`
is called after `createComponents` but I think that is fairly
reasonable.
Original commit: elastic/x-pack-elasticsearch@4fd87ad911
* Gets build to use elasticsearch-extras
Also adds ci script for building repo on CI servers
To use this change you need to:
1. Clone elasticsearch: `git@github.com:elastic/elasticsearch.git`
2. create a directory at the same level as elasticsearch called `elasticsearch-extra`
3. Clone this repository into the `elasticsearch-extra` directory
4. Run `gradle build` from the `elasticsearch-extra/prelert-legacy` directory or run `gradle :prelert-legacy:build` from the `elasticsearch directory
* Adds USE_SSH option to ci script
* iter
Original commit: elastic/x-pack-elasticsearch@ea127dfef0
Also removes unused method convertToObject from XContentUtils that does not close XContentParser either.
Original commit: elastic/x-pack-elasticsearch@99ce977c55
The job open api starts a task and ties that AutodetectCommunicator.
The job close api is a sugar api, that uses the list and cancel task api to close a AutodetectCommunicator instance.
The flush job and post data api redirect to the node holding the job task and then delegate the flush or data to the AutodetectCommunicator instance.
Also:
* Added basic multi node cluster test.
* Fixed cluster state diffs bugs, forgot to mark ml metadata diffs as named writeable.
* Moved waiting for open job logic into OpenJobAction.TransportAction and moved the logic that was original there to a new action named InternalOpenJobAction.
Original commit: elastic/x-pack-elasticsearch@194a058dd2
* removes upload pack task from build
This is preventing us from being an elasticsearch-extra project and we cannot have this task when we move to x-pack. Once we are in X-Pack the unified build will be uploading the final artifact so for now we will change the CI build to add a build step to upload the pack artifact.
* Removes OS specific stuff from the build
the CPP_LOCAL_DIST will now look for any `ml-cpp` artifacts for the same version in the specified directory.
* review corrections
Original commit: elastic/x-pack-elasticsearch@be15e55ddb
This commit contains some more of the endpoint changes Sophie and Steve
agreed with Clint:
1. get_jobs_stats renamed to get_job_stats
2. Revert snapshot must now be done using an ID - other options removed
3. Renamed "categorydefinitions" to "categories" in endpoints
4. get_jobs now has an implicit _all if no job ID/wildcard is specified
5. There is an option to retrieve a specific model snapshot by ID in
get_model_snapshots
Relates elastic/elasticsearch#630
Original commit: elastic/x-pack-elasticsearch@9dd71c64a8
This change prepares for elastic/elasticsearch/elastic/elasticsearch#22575, where we don't have ClusterService available in rest actions.
Original commit: elastic/x-pack-elasticsearch@87658c7fe8
This PR removes all leniency in the conversion of Strings to booleans: "true" is converted to the boolean value true, "false" is converted to the boolean value false. Everything else raises an error.
Relates elastic/elasticsearchelastic/elasticsearch#22200
Original commit: elastic/x-pack-elasticsearch@a505df1f5d
This commit performs the following improvements:
- the time field is always requested as doc_value. This makes
specifying a time format for scheduled jobs unnecessary.
- adds DataDescription as a param to the PostDataAction. When set,
it overrides the job's DataDescription. This allows the scheduler to
override the job's DataDescription since it knows the data format (JSON)
and the time format (epoch_ms). This is not exposed in the REST API to
discourage users from using it.
- by default, data extractor search now requests doc_values for analysis fields. This is
expected to result in increased performance.
- a `_source` field is added to the scheduler config. This needs to be
set to true when one or more of the analysis fields do not have
doc_values.
- the ELASTICSEARCH data format is removed as is now redundant.
- fixes the usage of `script_fields`. Previously, setting
`script_fields` would result to none of the source to be returned. Thus,
is the analysis fields were a mixture of script and non-script fields it
would not work.
- ensures nested fields are handled properly
Closeselastic/elasticsearch#679, Closeselastic/elasticsearch#267
Original commit: elastic/x-pack-elasticsearch@fed35ed354
David Kyle
Jason Tedor
Adrien Grand
Dimitrios Athanasiou
Alexander Reelsen
Jay Modi
Igor Motov
Martijn van Groningen
Zachary Tong
Ali Beyad
Jack Conradson
David Roberts
Colin Goodheart-Smithe
Simon Willnauer
Tim Brooks
Nik Everett
Suyog Rao
Chris Earle
Luca Cavanna
javanna
polyfractal
Yannick Welsch
Boaz Leskes
Daniel Mitterdorfer
Daniel Mitterdorfer