Adds docs for the REST API, translate API, the CLI, and JDBC.
Next we need to add more example queries and documentation for our
extensions.
Original commit: elastic/x-pack-elasticsearch@ed6d1360d2
* [DOCS] Enabled code snippet testing for put datafeed API
* [DOCS] Addressed gradle errors in put datafeed API
* [DOCS] Added job creation test to build.gradle
Original commit: elastic/x-pack-elasticsearch@3548d920c7
For the purpose of getting this API consumed by our UI, returning
overall buckets that match the job's largest `bucket_span` can
result in too much data. The UI only ever displays a few buckets
in the swimlane. Their span depends on the time range selected and
the screen resolution, but it will only ever be a relatively
low number.
This PR adds the ability to aggregate overall buckets in a user
specified `bucket_span`. That `bucket_span` may be equal or
greater to the largest job's `bucket_span`. The `overall_score`
of the result overall buckets is the max score of the
corresponding overall buckets with a span equal to the job's
largest `bucket_span`.
The implementation is now chunking the bucket requests
as otherwise the aggregation would fail when too many buckets
are matching.
Original commit: elastic/x-pack-elasticsearch@981f7a40e5
* [DOCS] Added transport client info for X-Pack
* [DOCS] Relocated X-Pack java client info
* [DOCS] Added transport client deprecation info
Original commit: elastic/x-pack-elasticsearch@416aab1d76
Adds the GET overall_buckets API.
The REST end point is: GET
/_xpack/ml/anomaly_detectors/job_id/results/overall_buckets
The API returns overall bucket results. An overall bucket
is a summarized bucket result over multiple jobs.
It has the `bucket_span` of the longest job's `bucket_span`.
It also has an `overall_score` that is the `top_n` average of the
max anomaly scores per job.
relates elastic/x-pack-elasticsearch#2693
Original commit: elastic/x-pack-elasticsearch@ba6061482d
The execution state is kind of a global indicator if a watch has been
running successfully and is used by the watcher UI.
However this field is only stored in the watch history but not part of
the watch status, thus it is not available everywhere. In order to
simplify the watcher UI this commit also adds the field to the
watch status which is stored together with the watch.
It is stored under the `status.execution_state` field as `status.state`
is already taken. This is also reflects with the name of the java class.
The WatchStatus class does not contain serialization checks, as this is
intended to be backported to 6.x, where those checks will be added.
Once the backport is done, the old execution state field can be fully
deleted from the master branch in another commit (syncing with Kibana
folks required).
relates elastic/x-pack-elasticsearch#2385
* fix doc tests
Original commit: elastic/x-pack-elasticsearch@26e8f99571
The path of a JIRA endpoint used to be fixed. This commit allows the
path to be dynamic, so that users can deploy their JIRA instance under
an arbitrary prefix.
Original commit: elastic/x-pack-elasticsearch@7702505114
* [DOCS] Format Watcher APIs
* [DOCS] Removed master_timeout from Watcher APIs
* [DOCS] Added authority info to watcher APIs
Original commit: elastic/x-pack-elasticsearch@1e6de3b036
This change removes `xpack.security.authc.token.passphrase` entirely since from
6.0 onwards we use randomly generated keys by the master there is no need for
this setting anymore. This setting will be deprecated from 6.0 onwards.
Original commit: elastic/x-pack-elasticsearch@37ba90359e
As there are no master node operations anymore.
* TransportActions are regular Actions now
* Watcher requests are now ActionRequests, no MasterNodeRequests anymore
* REST spec does not contain master node timeout parameters anymore
* WatcherLifeCycleService does not have a check anymore if watcher is able to run distributed, this will be a given in 7.0
* Some serialization BWC checks against version 5 have been removed
Original commit: elastic/x-pack-elasticsearch@4607dd538c
Right now we'd like to be able to build the SQL docs without
modifying core even if they aren't quite how we're going to keep
them in the end. This removes the need for a change to core and
leaves a note that we'll have to make undo this commit before
release.
Original commit: elastic/x-pack-elasticsearch@9ca6a0e467
`authc.token.enabled` is true unless `http.ssl.enabled` is `false` and `http.enabled` is `true`.
* TokenService default enabled if HTTP_ENABLED == false
* Fixed tests that need TokenService explicitly enabled
* [DOC] Default value for `xpack.security.authc.token.enabled`
Original commit: elastic/x-pack-elasticsearch@bd154d16eb
* [DOCS] Describe severity threshold and interval in anomaly table
* [DOCS] Describe aggregation in anomaly table
* [DOCS] Fixed capitalization in ML getting started
Original commit: elastic/x-pack-elasticsearch@d4224c9fe8
* [DOCS] Add job groups to ML create/update job APIs
* [DOCS] Fix ML update job API example
* [DOCS] Address feedback for ML create/update job APIs
Original commit: elastic/x-pack-elasticsearch@0e7bb47342
* [DOCS] Add custom URL instructions
* [DOCS] Update custom URL examples
* [DOCS] Add example for custom URL
* [DOCS] Add ML category terms custom URL example
* [DOCS] Add tip for pop-up ML custom URLs
* [DOCS] Address feedback about ML custom URLs
* [DOCS] Clarify impact of interval on custom URL
Original commit: elastic/x-pack-elasticsearch@eee192226f
This change makes 2 improvements to the max_running_jobs setting:
1. Namespaces it by adding the xpack.ml. prefix
2. Renames "running" to "open", because the "running" terminology
is not used elsewhere
The old max_running_jobs setting is used as a fallback if the new
xpack.ml.max_open_jobs setting is not specified. max_running_jobs
is deprecated and (to ease backporting in the short term) will be
removed from 7.0 in a different PR closer to release of 7.0.
Relates elastic/x-pack-elasticsearch#2185
Original commit: elastic/x-pack-elasticsearch@18c539f9bb
* [DOCS] Add user setup to X-Pack install info
* [DOCS] Add TSL steps to X-Pack install
* [DOCS] Clarify SSL settings in Xpack install
Original commit: elastic/x-pack-elasticsearch@eee37729ff
* [DOCS] Update APIs for multiple jobs or datafeeds
* [DOCS] Fix syntax diagrams for ML stop/close APIs
* [DOCS] Removed TBD authorization for ML APIs
Original commit: elastic/x-pack-elasticsearch@1a9137a5a7
In order to not restart watcher on every test, this checks
if this is a watcher test and only restarts watcher in that case.
In addition this also checks if watcher is not marked as started.
as otherwise restarting does not make sense. Lastly, this waits until
watcher is marked as started before proceeding.
Original commit: elastic/x-pack-elasticsearch@a8d72f3ebb
This commit adds detection of support for AES 256 bit ciphers and enables their use when the JVM
supports them. For OpenJDK, this is often the case without any changes but for the Oracle JVM, the
unlimited policy file needs to be installed. In order to simplify the work a user would need to do
we can detect this support and automatically enable the AES 256 bit versions of the ciphers we
already enable.
Original commit: elastic/x-pack-elasticsearch@5f23b18a1e
The deprecated handlers should have been removed earlier, but are now
going to to away finally.
Also the watcher restart action has been removed, mainly because users
should not blindly restart, but always make sure, that watcher is
stopped correctly before restarting. This had been removed from the
transport action previously.
Original commit: elastic/x-pack-elasticsearch@78a5ec3c05
- Fix typo `trustsore` -> `truststore` in several places
- Clarify that enabling TLS requires full restart
Original commit: elastic/x-pack-elasticsearch@0f430a1bea
Add an option to the ssl certificate generation tool (certgen) that generates PKCSelastic/x-pack-elasticsearch#12 (.p12) files in addition to the certificate (.crt) and key (.key) files.
A PKCSelastic/x-pack-elasticsearch#12 store is a container format for storing multiple crypto objects in a single file, which means we can put the cert and key into the same file.
These format is particularly useful for .NET environments, where .NET Core requires a single into file for PKI authentication.
Also adds documentation for all the command-line options in certgen.
Original commit: elastic/x-pack-elasticsearch@d10f88f12d
The active directory user principal name format typically takes the form user@domain, which is what
the current implementation expects. However, active directory also allows the definition of other
suffixes that are not actual domains. A user can still authenticate using this user principal name
but the behavior of our realm would cause it to fail as it parsed the suffix as a domain and used it
as the search base for the user. Instead, we should use the default user search base and only look
for entries that have this exact user principal name. In a scenario where a realm is configured for
multiple domains in the same forest, the search base should be the base for the entire forest.
relates elastic/x-pack-elasticsearch#1744
Original commit: elastic/x-pack-elasticsearch@de00c4817e
This commit adds support for a bind user when using the active directory realm. The addition of a
bind user also enables support for the user lookup mechanism, which is necessary to support the run
as functionality that we provide.
relates elastic/x-pack-elasticsearch#179
Original commit: elastic/x-pack-elasticsearch@40b07b3422
This commit adds a setting to allow changing the user search filter. Previously the filter was a
simple equality filter that mapped a given attribute to the value of the username. The default
behavior remains the same with this change but provides additional flexibility to users to who may
need more advanced LDAP searches. The user attribute setting has been deprecated due to the overlap
with the new filter setting.
relates elastic/x-pack-elasticsearch#1861
Original commit: elastic/x-pack-elasticsearch@e9d797e81c
* [DOCS] Update model_memory_limit
* [DOCS] Clarify minimum model_memory_limit value
* [DOCS] More updates to model_memory_limit
* [DOCS] Address feedback in jobresource.asciidoc
Original commit: elastic/x-pack-elasticsearch@3c62719037
- Document refresh interval for role mapping files
- Fix obsolete shield reference in transport profile example
- Clarify that AD & PKI don't support run_as
- Fix logstash conf examples
- Clarify interaction of SSL settings and PKI realm settings
- Document PKI DN format, and recommend use of pki_dn metadata
- Provide more details about action.auto_create_index during setup
Original commit: elastic/x-pack-elasticsearch@49ddb12a7e
This is step 2 of elastic/x-pack-elasticsearch#1604
This change stores `model_memory_limit` as a string with `mb` unit.
I considered using the `toString` method of `ByteSizeValue` but it
can lead to accuracy loss. Adding the fixed `mb` unit maintains
the accuracy, while making clear what unit the value is in.
Original commit: elastic/x-pack-elasticsearch@4dc48f0ce8
There are multiple references to this section in different areas of the
documentation. This commit brings back this section to fix the build.
A more extensive PR updating the documentation for "no default
password" work will follow up.
Original commit: elastic/x-pack-elasticsearch@0378e78c8a
This section has been removed from setting-up-authentication. This
commit removes a reference to this section that no longer exists.
Original commit: elastic/x-pack-elasticsearch@43aa0077f9
This commit removes the system key from master and changes watcher to use a secure setting instead
for the encryption key.
Original commit: elastic/x-pack-elasticsearch@5ac95c60ef
This is related to elastic/x-pack-elasticsearch#1217. This PR removes the default password of
"changeme" from the reserved users.
This PR adds special behavior for authenticating the reserved users. No
ReservedRealm user can be authenticated until its password is set. The
one exception to this is the elastic user. The elastic user can be
authenticated with an empty password if the action is a rest request
originating from localhost. In this scenario where an elastic user is
authenticated with a default password, it will have metadata indicating
that it is in setup mode. An elastic user in setup mode is only
authorized to execute a change password request.
Original commit: elastic/x-pack-elasticsearch@e1e101a237
* [DOCS] Move security APIs to Elasticsearch Ref
* [DOCS] Update links to security APIs
* [DOCS] Fix link to security APIs
Original commit: elastic/x-pack-elasticsearch@d7a9d3f1ab
* [DOCS] Model plot updates
Add to job create.
Remove terms from job resource.
* [DOCS] Describing terms as experimental
Original commit: elastic/x-pack-elasticsearch@815fa0ec37
* [DOCS] Getting started with multi-metric jobs
* [DOCS] More work on ML getting started with multi-metric jobs
* [DOCS] Add ML getting started with multi-metric jobs screenshot
* [DOCS] Add ML getting started information about influencers
* [DOCS] Getting started with multi-metric jobs
* [DOCS] Fix ML getting started build error
* [DOCS] Add ML getting started multi-metric snapshots
* [DOCS] Add screenshots and next steps to ML Getting Started tutorial
* [DOCS] Clarified anomaly scores in ML Getting Started pages
* [DOCS] Addressed ML getting started feedback
* [DOCS] Fix ML getting started links
Original commit: elastic/x-pack-elasticsearch@a7e80cfabf
* [DOCS] Add ML info about script fields
* [DOCS] Add links to ML script fields page
* [DOCS] Add ML API examples to transforms.asciidoc
* [DOCS] Addressed feedback in ML script field examples
* [DOCS] Add preview to ML script fields example
* [DOCS] Expanded code snippets in ML transform examples
* [DOCS] Add output for ML scripted fields example
* [DOCS] Add output for more ML scripted field examples
* [DOCS] Add output for final ML scripted field examples
* [DOC] Add Kibana details for ML script fields
* [DOCS] Remove example from ML transforms
Original commit: elastic/x-pack-elasticsearch@51057b029f
* [DOCS] Add configuration information for population analysis
* [DOCS] Add ML population analysis examples
* [DOCS] Address feedback for population analysis
* [DOCS] More feedback on population analysis
Original commit: elastic/x-pack-elasticsearch@ffa2bfeed9
This changes the validation criteria we use for user and role
names in the file realm, native realm, and the
realm-agnostic code in x-pack security. The new criteria is:
A valid username's length must be at least 1 and no more than 1024
characters. It may not contain leading or trailing whitespace. All
characters in the name must be be alphanumeric (`a-z`, `A-Z`, `0-9`),
printable punctuation or symbols in the https://en.wikipedia.org/wiki/Basic_Latin_(Unicode_block)[Basic Latin (ASCII) block],
or the space character.
Original commit: elastic/x-pack-elasticsearch@f77640f269
This allows to configure a proxy for the reporting attachment
action. The proxy is used by the HTTP client.
Original commit: elastic/x-pack-elasticsearch@87b6ab1b68
* [DOCS] Update ML APIs for Elasticsearch Reference
* [DOCS] Add X-Pack icon for ML APIs
* [DOCS] Add role attribute to ML APIs
Original commit: elastic/x-pack-elasticsearch@997ea39759
* [DOCS] Call shared index files to build Elasticsearch Ref
* [DOCS] Fix path and edit URL for Elasticsearch build
Original commit: elastic/x-pack-elasticsearch@626d6d79d8
Prior to this change, if the persistent tasks framework noticed that a
job was running on a node that was isolated but has rejoined the cluster
then it would close that job. This was not ideal, because then the job
would persist state from the autodetect process that was isolated. This
commit changes the behaviour to kill the autodetect process associated
with such a job, so that it does not interfere with the autodetect process
that is running on the node where the persistent tasks framework thinks it
should be running.
In order to achieve this a change has also been made to the behaviour of
force-close. Previously this would result in the autodetect process being
gracefully shut down asynchronously to the force-close request. However,
the mechanism by which this happened was the same as the mechanism for
cancelling tasks that end up running on more than one node due to nodes
becoming isolated from the cluster. Therefore, force-close now also kills
the autodetect process rather than gracefully stopping it. The documentation
has been changed to reflect this. It should not be a problem as force-close
is supposed to be a last resort for when normal close fails.
relates elastic/x-pack-elasticsearch#1186
Original commit: elastic/x-pack-elasticsearch@578c944371
* [DOCS] Add Anomaly Explorer limitation re scripted fields
* [DOCS] Update Anomaly Explorer limitation re scripted fields
Original commit: elastic/x-pack-elasticsearch@7c9971ada0
This commit adds a new Logstash component to x-pack to support the config management work. Currently, the functionality in this component is really simple; all it does is upload a new index template for `.logstash` index. This index stores the actual LS configuration.
On this template is bootstrapped in ES, Kibana can write user-created LS configs which adhere to the mapping defined here. In the future, we're looking into adding more functionality on the ES side to handle config documents, but for now, this is simple.
relates elastic/x-pack-elasticsearch#1499, relates elastic/x-pack-elasticsearch#1471
Original commit: elastic/x-pack-elasticsearch@d7cc8675f7
* [DOCS] Add ML categorization of messages
* [DOCS] Describe ML categorization_examples_limit property
* [DOCS] Updated ML categorization of messages
* [DOCS] Add links to ML categorization
Original commit: elastic/x-pack-elasticsearch@6403f6ce84
When a user or client intend to delete a datafeed
and its job, there is benefit into ensuring the
datafeed has gracefully stopped (ie no data loss).
In constrast, the desired behaviour is to stop and
delete the datafeed as quickly as possible.
This change adds a force option to the delete
datafeed action. When the delete is forced,
the datafeed is isolated, its task removed and,
finally, the datafeed itself is removed from the
metadata.
relates elastic/x-pack-elasticsearch#1533
Original commit: elastic/x-pack-elasticsearch@5ae0168bf2
* Add force delete job option
* Can’t kill a process on a 5.4 node
* Address review comments
* Rename KillAutodetectAction -> KillProcessAction
* Review comments
* Cancelling task is superfluous after it has been killed
* Update docs
* Revert "Cancelling task is superfluous after it has been killed"
This reverts commit 576950e2e1ee095b38174d8b71de353c082ae953.
* Remove unnecessary TODOs and logic that doesn't alwasys force close
Original commit: elastic/x-pack-elasticsearch@f8c8b38217
Includes:
- Extensive changes to "mapping roles" section
- New section for role mapping API
- Updates to LDAP/AD/PKI realms to refer to API based role mapping
- Updates to LDAP/AD realms: `unmapped_groups_as_roles` only looks at file-based mappings
- Updates to LDAP/AD realms: new setting for "metadata"
Original commit: elastic/x-pack-elasticsearch@6349f665f5
* [DOCS] Clarify ML node settings re transport requests
* [DOCS] Clarify xpack.ml.enabled based on feedback
Original commit: elastic/x-pack-elasticsearch@3102d1e3f3
Detectors now have a field called detector_index. This is also now the
field that needs to be supplied when updating a detector. (Previously
it was simply index, which was confusing.)
When detectors are added to an analysis_config it will reassign
ascending detector_index values starting from 0. The intention is
never to allow deletion of detectors from an analysis_config, but
possibly to allow disabling them in the future. This ensures that
detector_index values in results will always tie up with detector_ids
in the detectors that created them.
relates elastic/x-pack-elasticsearch#1275
Original commit: elastic/x-pack-elasticsearch@20a660b07b
* [DOCS] Add details about ML count functions
* [DOCS] Address feedback in ML count functions
* [DOCS] Clarify ML population analysis in non-zero count functions
Original commit: elastic/x-pack-elasticsearch@24dbeba891
* Remove sequenceNum from anomaly records and influencers
* Generate unqiue IDs without sequence numbers
* Remove more instances of sequence_num
* Handle parsing sequnce_num from v5.4
Original commit: elastic/x-pack-elasticsearch@e60b206daf
* [DOCS] Add ML aggregations configuration scenario
* [DOCS] Refine ML configuration page
* [DOCS] Add ML aggregation details
* [DOCS] Add links to aggregations in Configuring ML
* [DOCS] Address feedback about ML aggregations
Original commit: elastic/x-pack-elasticsearch@8474144093
* Updated docs for cross-cluster search
Wildcards are now supported in both cluster alias and index name,
and the "view_index_metadata" privilege is automatically conferred
by the "read_cross_cluster" role.
* For now, best to allow users permissions to query all indices on all remote clusters
Original commit: elastic/x-pack-elasticsearch@31959f78f4
* [DOCS] Add ML geographic functions
* [DOCS] Add script_fields info to ML geo functions
* [DOCS] Remove summary count from ML geographic functions
* [DOCS] Added example title to geographic functions
* [DOCS] Remove list from ML geographic functions
Original commit: elastic/x-pack-elasticsearch@a8e495657f
* Add sort parameter for get buckets
* Add secondary sort by time
* Use default values from actions in rest requests
Original commit: elastic/x-pack-elasticsearch@a530c0bed6
* [DOCS] Add script_fields to ML datafeed APIs
* [DOCS] Add datafeedresource.asciidoc to build.gradle
* [DOCS] Addressed feedback in PR 1372
Original commit: elastic/x-pack-elasticsearch@3404ca7850
* [DOCS] Add ML analytical functions
* [DOCS] Add pages for ML analytical functions
* [DOCS] Add links to ML functions from API definitions
Original commit: elastic/x-pack-elasticsearch@ae50b431d3
As fields with underscores will be disallowed in master, and we have to
prepare the upgrade, this commit renames the _status field to status.
When the 5.x upgrade logic is in place in the 5.x we can remove all the
old style _status handling from the master branch.
Note: All the BWC compatibility tests, that load 5.x indices are now
faking a finished upgrade by adding the `status` field to the mapping
of the watches index.
Original commit: elastic/x-pack-elasticsearch@9d5cc9aaec
* [DOCS] Add property table for ML Update Jobs API
* [DOCS] Updates based on feedback for ML Update Jobs API
* [DOCS] Removed detector properties from ML Update Jobs API
* [DOCS] Fixes typos
Original commit: elastic/x-pack-elasticsearch@68d1b5598c
The distribution of watches now happens on the node which holds the
watches index, instead of on the master node. This requires several
changes to the current implementation.
1. Running on shards and replicas
In order to run watches on the nodes with the watches index on its
primaries and replicas. To ensure that watches do not run twice, there is
a logic which checks the local shards, runs a murmurhash on the id and
runs modulo against the number of shards and replicas, this is the way to
find out, if a watch should run local. Reloading happens
2. Several master node actions moved to a HandledTransportAction, as they
are basically just aliases for indexing actions, among them the
put/delete/get watch actions, the acknowledgement action, the de/activate
actions
3. Stats action moved to a broadcast node action, because we potentially
have to query every node to get watcher statistics
4. Starting/Stopping watcher now is a master node action, which updates
the cluster state and then listeners acts on those. Because of this watches
can be running on two systems, if you those have different cluster state
versions, until the new watcher state is propagated
5. Watcher is started on all nodes now. With the exception of the ticker
schedule engine most classes do not need a lot of resources while running.
However they have to run, because of the execute watch API, which can hit
any node - it does not make sense to find the right shard for this watch
and only then execute (as this also has to work with a watch, that has not
been stored before)
6. By using a indexing operation listener, each storing of a watch now
parses the watch first and only stores on successful parsing
7. Execute watch API now uses the watcher threadpool for execution
8. Getting the number of watches for the stats now simply queries the
different execution engines, how many watches are scheduled, so this is
not doing a search anymore
There will be follow up commits on this one, mainly to ensure BWC compatibility.
Original commit: elastic/x-pack-elasticsearch@0adb46e658
* [DOCS] Add ML limitations
* [DOCS] Address feedback about ML limitations
* [DOCS] Change ML limitations capitalization
Original commit: elastic/x-pack-elasticsearch@41682d8d93
lcawley
David Kyle
Nik Everett
Martijn van Groningen
Dimitris Athanasiou
Alexander Reelsen
David Roberts
Costin Leau
Russ Cam
Igor Motov
Tanguy Leroux
jaymode
Jay Modi
Simon Willnauer
Deb Adair
Jim Ferenczi
Michael Basnight
Albert Zaharovits
Jason Tedor
Luca Cavanna
Tim Vernum
Tal Levy
Tim Brooks
Drew Raines
Colin Goodheart-Smithe
Clinton Gormley
Sophie Chang
Andy Bristol
Boaz Leskes
Suyog Rao
Ryan Ernst
markharwood
David Pilato
Jack Conradson
Honza Král
Lee Hinman
Chee Wee