Adds the GET overall_buckets API.
The REST end point is: GET
/_xpack/ml/anomaly_detectors/job_id/results/overall_buckets
The API returns overall bucket results. An overall bucket
is a summarized bucket result over multiple jobs.
It has the `bucket_span` of the longest job's `bucket_span`.
It also has an `overall_score` that is the `top_n` average of the
max anomaly scores per job.
relates elastic/x-pack-elasticsearch#2693
Original commit: elastic/x-pack-elasticsearch@ba6061482d
The execution state is kind of a global indicator if a watch has been
running successfully and is used by the watcher UI.
However this field is only stored in the watch history but not part of
the watch status, thus it is not available everywhere. In order to
simplify the watcher UI this commit also adds the field to the
watch status which is stored together with the watch.
It is stored under the `status.execution_state` field as `status.state`
is already taken. This is also reflects with the name of the java class.
The WatchStatus class does not contain serialization checks, as this is
intended to be backported to 6.x, where those checks will be added.
Once the backport is done, the old execution state field can be fully
deleted from the master branch in another commit (syncing with Kibana
folks required).
relates elastic/x-pack-elasticsearch#2385
* fix doc tests
Original commit: elastic/x-pack-elasticsearch@26e8f99571
The path of a JIRA endpoint used to be fixed. This commit allows the
path to be dynamic, so that users can deploy their JIRA instance under
an arbitrary prefix.
Original commit: elastic/x-pack-elasticsearch@7702505114
* [DOCS] Format Watcher APIs
* [DOCS] Removed master_timeout from Watcher APIs
* [DOCS] Added authority info to watcher APIs
Original commit: elastic/x-pack-elasticsearch@1e6de3b036
This change removes `xpack.security.authc.token.passphrase` entirely since from
6.0 onwards we use randomly generated keys by the master there is no need for
this setting anymore. This setting will be deprecated from 6.0 onwards.
Original commit: elastic/x-pack-elasticsearch@37ba90359e
As there are no master node operations anymore.
* TransportActions are regular Actions now
* Watcher requests are now ActionRequests, no MasterNodeRequests anymore
* REST spec does not contain master node timeout parameters anymore
* WatcherLifeCycleService does not have a check anymore if watcher is able to run distributed, this will be a given in 7.0
* Some serialization BWC checks against version 5 have been removed
Original commit: elastic/x-pack-elasticsearch@4607dd538c
Right now we'd like to be able to build the SQL docs without
modifying core even if they aren't quite how we're going to keep
them in the end. This removes the need for a change to core and
leaves a note that we'll have to make undo this commit before
release.
Original commit: elastic/x-pack-elasticsearch@9ca6a0e467
`authc.token.enabled` is true unless `http.ssl.enabled` is `false` and `http.enabled` is `true`.
* TokenService default enabled if HTTP_ENABLED == false
* Fixed tests that need TokenService explicitly enabled
* [DOC] Default value for `xpack.security.authc.token.enabled`
Original commit: elastic/x-pack-elasticsearch@bd154d16eb
* [DOCS] Describe severity threshold and interval in anomaly table
* [DOCS] Describe aggregation in anomaly table
* [DOCS] Fixed capitalization in ML getting started
Original commit: elastic/x-pack-elasticsearch@d4224c9fe8
* [DOCS] Add job groups to ML create/update job APIs
* [DOCS] Fix ML update job API example
* [DOCS] Address feedback for ML create/update job APIs
Original commit: elastic/x-pack-elasticsearch@0e7bb47342
* [DOCS] Add custom URL instructions
* [DOCS] Update custom URL examples
* [DOCS] Add example for custom URL
* [DOCS] Add ML category terms custom URL example
* [DOCS] Add tip for pop-up ML custom URLs
* [DOCS] Address feedback about ML custom URLs
* [DOCS] Clarify impact of interval on custom URL
Original commit: elastic/x-pack-elasticsearch@eee192226f
This change makes 2 improvements to the max_running_jobs setting:
1. Namespaces it by adding the xpack.ml. prefix
2. Renames "running" to "open", because the "running" terminology
is not used elsewhere
The old max_running_jobs setting is used as a fallback if the new
xpack.ml.max_open_jobs setting is not specified. max_running_jobs
is deprecated and (to ease backporting in the short term) will be
removed from 7.0 in a different PR closer to release of 7.0.
Relates elastic/x-pack-elasticsearch#2185
Original commit: elastic/x-pack-elasticsearch@18c539f9bb
* [DOCS] Add user setup to X-Pack install info
* [DOCS] Add TSL steps to X-Pack install
* [DOCS] Clarify SSL settings in Xpack install
Original commit: elastic/x-pack-elasticsearch@eee37729ff
* [DOCS] Update APIs for multiple jobs or datafeeds
* [DOCS] Fix syntax diagrams for ML stop/close APIs
* [DOCS] Removed TBD authorization for ML APIs
Original commit: elastic/x-pack-elasticsearch@1a9137a5a7
This commit adds detection of support for AES 256 bit ciphers and enables their use when the JVM
supports them. For OpenJDK, this is often the case without any changes but for the Oracle JVM, the
unlimited policy file needs to be installed. In order to simplify the work a user would need to do
we can detect this support and automatically enable the AES 256 bit versions of the ciphers we
already enable.
Original commit: elastic/x-pack-elasticsearch@5f23b18a1e
The deprecated handlers should have been removed earlier, but are now
going to to away finally.
Also the watcher restart action has been removed, mainly because users
should not blindly restart, but always make sure, that watcher is
stopped correctly before restarting. This had been removed from the
transport action previously.
Original commit: elastic/x-pack-elasticsearch@78a5ec3c05
- Fix typo `trustsore` -> `truststore` in several places
- Clarify that enabling TLS requires full restart
Original commit: elastic/x-pack-elasticsearch@0f430a1bea
Add an option to the ssl certificate generation tool (certgen) that generates PKCSelastic/x-pack-elasticsearch#12 (.p12) files in addition to the certificate (.crt) and key (.key) files.
A PKCSelastic/x-pack-elasticsearch#12 store is a container format for storing multiple crypto objects in a single file, which means we can put the cert and key into the same file.
These format is particularly useful for .NET environments, where .NET Core requires a single into file for PKI authentication.
Also adds documentation for all the command-line options in certgen.
Original commit: elastic/x-pack-elasticsearch@d10f88f12d
The active directory user principal name format typically takes the form user@domain, which is what
the current implementation expects. However, active directory also allows the definition of other
suffixes that are not actual domains. A user can still authenticate using this user principal name
but the behavior of our realm would cause it to fail as it parsed the suffix as a domain and used it
as the search base for the user. Instead, we should use the default user search base and only look
for entries that have this exact user principal name. In a scenario where a realm is configured for
multiple domains in the same forest, the search base should be the base for the entire forest.
relates elastic/x-pack-elasticsearch#1744
Original commit: elastic/x-pack-elasticsearch@de00c4817e
This commit adds support for a bind user when using the active directory realm. The addition of a
bind user also enables support for the user lookup mechanism, which is necessary to support the run
as functionality that we provide.
relates elastic/x-pack-elasticsearch#179
Original commit: elastic/x-pack-elasticsearch@40b07b3422
This commit adds a setting to allow changing the user search filter. Previously the filter was a
simple equality filter that mapped a given attribute to the value of the username. The default
behavior remains the same with this change but provides additional flexibility to users to who may
need more advanced LDAP searches. The user attribute setting has been deprecated due to the overlap
with the new filter setting.
relates elastic/x-pack-elasticsearch#1861
Original commit: elastic/x-pack-elasticsearch@e9d797e81c
* [DOCS] Update model_memory_limit
* [DOCS] Clarify minimum model_memory_limit value
* [DOCS] More updates to model_memory_limit
* [DOCS] Address feedback in jobresource.asciidoc
Original commit: elastic/x-pack-elasticsearch@3c62719037
- Document refresh interval for role mapping files
- Fix obsolete shield reference in transport profile example
- Clarify that AD & PKI don't support run_as
- Fix logstash conf examples
- Clarify interaction of SSL settings and PKI realm settings
- Document PKI DN format, and recommend use of pki_dn metadata
- Provide more details about action.auto_create_index during setup
Original commit: elastic/x-pack-elasticsearch@49ddb12a7e
This is step 2 of elastic/x-pack-elasticsearch#1604
This change stores `model_memory_limit` as a string with `mb` unit.
I considered using the `toString` method of `ByteSizeValue` but it
can lead to accuracy loss. Adding the fixed `mb` unit maintains
the accuracy, while making clear what unit the value is in.
Original commit: elastic/x-pack-elasticsearch@4dc48f0ce8
There are multiple references to this section in different areas of the
documentation. This commit brings back this section to fix the build.
A more extensive PR updating the documentation for "no default
password" work will follow up.
Original commit: elastic/x-pack-elasticsearch@0378e78c8a
This section has been removed from setting-up-authentication. This
commit removes a reference to this section that no longer exists.
Original commit: elastic/x-pack-elasticsearch@43aa0077f9
This commit removes the system key from master and changes watcher to use a secure setting instead
for the encryption key.
Original commit: elastic/x-pack-elasticsearch@5ac95c60ef
This is related to elastic/x-pack-elasticsearch#1217. This PR removes the default password of
"changeme" from the reserved users.
This PR adds special behavior for authenticating the reserved users. No
ReservedRealm user can be authenticated until its password is set. The
one exception to this is the elastic user. The elastic user can be
authenticated with an empty password if the action is a rest request
originating from localhost. In this scenario where an elastic user is
authenticated with a default password, it will have metadata indicating
that it is in setup mode. An elastic user in setup mode is only
authorized to execute a change password request.
Original commit: elastic/x-pack-elasticsearch@e1e101a237
* [DOCS] Move security APIs to Elasticsearch Ref
* [DOCS] Update links to security APIs
* [DOCS] Fix link to security APIs
Original commit: elastic/x-pack-elasticsearch@d7a9d3f1ab
* [DOCS] Model plot updates
Add to job create.
Remove terms from job resource.
* [DOCS] Describing terms as experimental
Original commit: elastic/x-pack-elasticsearch@815fa0ec37
* [DOCS] Getting started with multi-metric jobs
* [DOCS] More work on ML getting started with multi-metric jobs
* [DOCS] Add ML getting started with multi-metric jobs screenshot
* [DOCS] Add ML getting started information about influencers
* [DOCS] Getting started with multi-metric jobs
* [DOCS] Fix ML getting started build error
* [DOCS] Add ML getting started multi-metric snapshots
* [DOCS] Add screenshots and next steps to ML Getting Started tutorial
* [DOCS] Clarified anomaly scores in ML Getting Started pages
* [DOCS] Addressed ML getting started feedback
* [DOCS] Fix ML getting started links
Original commit: elastic/x-pack-elasticsearch@a7e80cfabf
* [DOCS] Add ML info about script fields
* [DOCS] Add links to ML script fields page
* [DOCS] Add ML API examples to transforms.asciidoc
* [DOCS] Addressed feedback in ML script field examples
* [DOCS] Add preview to ML script fields example
* [DOCS] Expanded code snippets in ML transform examples
* [DOCS] Add output for ML scripted fields example
* [DOCS] Add output for more ML scripted field examples
* [DOCS] Add output for final ML scripted field examples
* [DOC] Add Kibana details for ML script fields
* [DOCS] Remove example from ML transforms
Original commit: elastic/x-pack-elasticsearch@51057b029f
* [DOCS] Add configuration information for population analysis
* [DOCS] Add ML population analysis examples
* [DOCS] Address feedback for population analysis
* [DOCS] More feedback on population analysis
Original commit: elastic/x-pack-elasticsearch@ffa2bfeed9
This changes the validation criteria we use for user and role
names in the file realm, native realm, and the
realm-agnostic code in x-pack security. The new criteria is:
A valid username's length must be at least 1 and no more than 1024
characters. It may not contain leading or trailing whitespace. All
characters in the name must be be alphanumeric (`a-z`, `A-Z`, `0-9`),
printable punctuation or symbols in the https://en.wikipedia.org/wiki/Basic_Latin_(Unicode_block)[Basic Latin (ASCII) block],
or the space character.
Original commit: elastic/x-pack-elasticsearch@f77640f269
This allows to configure a proxy for the reporting attachment
action. The proxy is used by the HTTP client.
Original commit: elastic/x-pack-elasticsearch@87b6ab1b68
* [DOCS] Update ML APIs for Elasticsearch Reference
* [DOCS] Add X-Pack icon for ML APIs
* [DOCS] Add role attribute to ML APIs
Original commit: elastic/x-pack-elasticsearch@997ea39759
* [DOCS] Call shared index files to build Elasticsearch Ref
* [DOCS] Fix path and edit URL for Elasticsearch build
Original commit: elastic/x-pack-elasticsearch@626d6d79d8
Prior to this change, if the persistent tasks framework noticed that a
job was running on a node that was isolated but has rejoined the cluster
then it would close that job. This was not ideal, because then the job
would persist state from the autodetect process that was isolated. This
commit changes the behaviour to kill the autodetect process associated
with such a job, so that it does not interfere with the autodetect process
that is running on the node where the persistent tasks framework thinks it
should be running.
In order to achieve this a change has also been made to the behaviour of
force-close. Previously this would result in the autodetect process being
gracefully shut down asynchronously to the force-close request. However,
the mechanism by which this happened was the same as the mechanism for
cancelling tasks that end up running on more than one node due to nodes
becoming isolated from the cluster. Therefore, force-close now also kills
the autodetect process rather than gracefully stopping it. The documentation
has been changed to reflect this. It should not be a problem as force-close
is supposed to be a last resort for when normal close fails.
relates elastic/x-pack-elasticsearch#1186
Original commit: elastic/x-pack-elasticsearch@578c944371
* [DOCS] Add Anomaly Explorer limitation re scripted fields
* [DOCS] Update Anomaly Explorer limitation re scripted fields
Original commit: elastic/x-pack-elasticsearch@7c9971ada0
This commit adds a new Logstash component to x-pack to support the config management work. Currently, the functionality in this component is really simple; all it does is upload a new index template for `.logstash` index. This index stores the actual LS configuration.
On this template is bootstrapped in ES, Kibana can write user-created LS configs which adhere to the mapping defined here. In the future, we're looking into adding more functionality on the ES side to handle config documents, but for now, this is simple.
relates elastic/x-pack-elasticsearch#1499, relates elastic/x-pack-elasticsearch#1471
Original commit: elastic/x-pack-elasticsearch@d7cc8675f7
* [DOCS] Add ML categorization of messages
* [DOCS] Describe ML categorization_examples_limit property
* [DOCS] Updated ML categorization of messages
* [DOCS] Add links to ML categorization
Original commit: elastic/x-pack-elasticsearch@6403f6ce84
When a user or client intend to delete a datafeed
and its job, there is benefit into ensuring the
datafeed has gracefully stopped (ie no data loss).
In constrast, the desired behaviour is to stop and
delete the datafeed as quickly as possible.
This change adds a force option to the delete
datafeed action. When the delete is forced,
the datafeed is isolated, its task removed and,
finally, the datafeed itself is removed from the
metadata.
relates elastic/x-pack-elasticsearch#1533
Original commit: elastic/x-pack-elasticsearch@5ae0168bf2
* Add force delete job option
* Can’t kill a process on a 5.4 node
* Address review comments
* Rename KillAutodetectAction -> KillProcessAction
* Review comments
* Cancelling task is superfluous after it has been killed
* Update docs
* Revert "Cancelling task is superfluous after it has been killed"
This reverts commit 576950e2e1ee095b38174d8b71de353c082ae953.
* Remove unnecessary TODOs and logic that doesn't alwasys force close
Original commit: elastic/x-pack-elasticsearch@f8c8b38217
Includes:
- Extensive changes to "mapping roles" section
- New section for role mapping API
- Updates to LDAP/AD/PKI realms to refer to API based role mapping
- Updates to LDAP/AD realms: `unmapped_groups_as_roles` only looks at file-based mappings
- Updates to LDAP/AD realms: new setting for "metadata"
Original commit: elastic/x-pack-elasticsearch@6349f665f5
* [DOCS] Clarify ML node settings re transport requests
* [DOCS] Clarify xpack.ml.enabled based on feedback
Original commit: elastic/x-pack-elasticsearch@3102d1e3f3
Detectors now have a field called detector_index. This is also now the
field that needs to be supplied when updating a detector. (Previously
it was simply index, which was confusing.)
When detectors are added to an analysis_config it will reassign
ascending detector_index values starting from 0. The intention is
never to allow deletion of detectors from an analysis_config, but
possibly to allow disabling them in the future. This ensures that
detector_index values in results will always tie up with detector_ids
in the detectors that created them.
relates elastic/x-pack-elasticsearch#1275
Original commit: elastic/x-pack-elasticsearch@20a660b07b
* [DOCS] Add details about ML count functions
* [DOCS] Address feedback in ML count functions
* [DOCS] Clarify ML population analysis in non-zero count functions
Original commit: elastic/x-pack-elasticsearch@24dbeba891
* Remove sequenceNum from anomaly records and influencers
* Generate unqiue IDs without sequence numbers
* Remove more instances of sequence_num
* Handle parsing sequnce_num from v5.4
Original commit: elastic/x-pack-elasticsearch@e60b206daf
* [DOCS] Add ML aggregations configuration scenario
* [DOCS] Refine ML configuration page
* [DOCS] Add ML aggregation details
* [DOCS] Add links to aggregations in Configuring ML
* [DOCS] Address feedback about ML aggregations
Original commit: elastic/x-pack-elasticsearch@8474144093
* Updated docs for cross-cluster search
Wildcards are now supported in both cluster alias and index name,
and the "view_index_metadata" privilege is automatically conferred
by the "read_cross_cluster" role.
* For now, best to allow users permissions to query all indices on all remote clusters
Original commit: elastic/x-pack-elasticsearch@31959f78f4
* [DOCS] Add ML geographic functions
* [DOCS] Add script_fields info to ML geo functions
* [DOCS] Remove summary count from ML geographic functions
* [DOCS] Added example title to geographic functions
* [DOCS] Remove list from ML geographic functions
Original commit: elastic/x-pack-elasticsearch@a8e495657f
* Add sort parameter for get buckets
* Add secondary sort by time
* Use default values from actions in rest requests
Original commit: elastic/x-pack-elasticsearch@a530c0bed6
* [DOCS] Add script_fields to ML datafeed APIs
* [DOCS] Add datafeedresource.asciidoc to build.gradle
* [DOCS] Addressed feedback in PR 1372
Original commit: elastic/x-pack-elasticsearch@3404ca7850
* [DOCS] Add ML analytical functions
* [DOCS] Add pages for ML analytical functions
* [DOCS] Add links to ML functions from API definitions
Original commit: elastic/x-pack-elasticsearch@ae50b431d3
As fields with underscores will be disallowed in master, and we have to
prepare the upgrade, this commit renames the _status field to status.
When the 5.x upgrade logic is in place in the 5.x we can remove all the
old style _status handling from the master branch.
Note: All the BWC compatibility tests, that load 5.x indices are now
faking a finished upgrade by adding the `status` field to the mapping
of the watches index.
Original commit: elastic/x-pack-elasticsearch@9d5cc9aaec
* [DOCS] Add property table for ML Update Jobs API
* [DOCS] Updates based on feedback for ML Update Jobs API
* [DOCS] Removed detector properties from ML Update Jobs API
* [DOCS] Fixes typos
Original commit: elastic/x-pack-elasticsearch@68d1b5598c
David Roberts
Dimitris Athanasiou
Alexander Reelsen
Costin Leau
lcawley
Russ Cam
Igor Motov
Tanguy Leroux
Nik Everett
David Kyle
jaymode
Jay Modi
Simon Willnauer
Deb Adair
Jim Ferenczi
Albert Zaharovits
Jason Tedor
Tim Vernum
Tal Levy
Tim Brooks
Drew Raines
Colin Goodheart-Smithe
Clinton Gormley
Sophie Chang
Andy Bristol
Boaz Leskes
Suyog Rao
Ryan Ernst
markharwood
David Pilato
Honza Král
Lee Hinman
Chee Wee