Commit Graph

46861 Commits

Author SHA1 Message Date
Benjamin Trent 82c1ddc117
[7.x] [ML][Data Frame] Add deduced mappings to _preview response payload (#43742) (#43849)
* [ML][Data Frame] Add deduced mappings to _preview response payload (#43742)

* [ML][Data Frame] Add deduced mappings to _preview response payload

* updating preview docs

* fixing code for backport
2019-07-02 06:52:14 -05:00
Tanguy Leroux b977f019b8
Expose translog stats in ReadOnlyEngine (#43752) (#43823)
Backport of #43752 for 7.x.
2019-07-02 13:39:00 +02:00
Ioannis Kakavas c8ed271937 Use URLEncoder#encode(String, String)
as URLEncoder#encode(String, Charset) is only available since Java
10
2019-07-02 14:20:29 +03:00
Ioannis Kakavas 4ea17b76dc Fix credentials encoding for OIDC token request (#43808)
As defined in https://tools.ietf.org/html/rfc6749#section-2.3.1
both client id and client secret need to be encoded with the
application/x-www-form-urlencoded encoding algorithm when used as
credentials for HTTP Basic Authentication in requests to the OP.

Resolves #43709
2019-07-02 13:36:00 +03:00
Tomas Della Vedova 4cdb24bceb
Use explicit string keys in data_frame test (#43854) 2019-07-02 11:06:29 +02:00
Alexander Reelsen ac7e1476a0 Update docs to refer to 6.8 instead of 6.7 (#43685)
A few places in the documentation had mentioned 6.7 as the version to
upgrade from, when doing an upgrade to 7.0. While this is technically
possible, this commit will replace all those mentions to 6.8, as this is
the latest version with the latest bugfixes, deprecation checks and
ugprade assistant features - which should be the one used for upgrades.

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2019-07-02 09:35:04 +02:00
David Turner 1e8e85797d Rename and refactor RoutingService (#43827)
The `RoutingService` has a confusing name, since it doesn't really have
anything to do with routing. Its responsibility is submitting reroute commands
to the master.

This commit renames this class to `BatchedRerouteService`, and extracts the
`RerouteService` interface to avoid passing `BiConsumer`s everywhere. It also
removes that `BatchedRerouteService extends AbstractLifecycleComponent` since
this service has no meaningful lifecycle. Finally, it introduces a small
wrapper class to allow for lazy initialization to deal with the dependency loop
when constructing a `Node`.
2019-07-02 07:04:18 +01:00
Jack Conradson 81f60652d8 Add timezone documentation for Painless datetimes (#43768) 2019-07-01 21:31:13 -07:00
Albert Zaharovits 4eb89a6912
UserRoleMapper non-null groups and metadata (#43836)
This is an odd backport of #41774

UserRoleMapper.UserData is constructed by each realm and it is used to
"match" role mapping expressions that eventually supply the role names
of the principal.

This PR filters out `null` collection values (lists and maps), for the groups
and metadata, which get to take part in the role mapping, in preparation
for using Java 9 collection APIs. It filters them as soon as possible, during
the construction.
2019-07-02 00:10:15 +03:00
Mark Vieira b5d47e5643
Upgrade to Gradle 5.5 (#43788) (#43832) 2019-07-01 11:54:58 -07:00
Christoph Büscher fe3f9f0c6b Yet another `the the` cleanup (#43815) 2019-07-01 20:22:19 +02:00
Alpar Torok eb0c7196cb Disable Gradle running in parallel 2019-07-01 18:18:38 +03:00
Yogesh Gaikwad 031d5e96ac
HLRC changes for kerberos grant type (#43642) (#43822)
The TODO from last PR for kerbero grant type was missed.
This commit adds the changes for kerberos grant type in HLRC.
2019-07-02 00:55:02 +10:00
Zachary Tong 1e47ea5f18 Update rare_term version skips, fix SetBackedScalingCuckooFilter javadoc 2019-07-01 10:52:06 -04:00
Zachary Tong ea1794832f Add RareTerms aggregation (#35718)
This adds a `rare_terms` aggregation.  It is an aggregation designed
to identify the long-tail of keywords, e.g. terms that are "rare" or
have low doc counts.

This aggregation is designed to be more memory efficient than the
alternative, which is setting a terms aggregation to size: LONG_MAX
(or worse, ordering a terms agg by count ascending, which has
unbounded error).

This aggregation works by maintaining a map of terms that have
been seen. A counter associated with each value is incremented
when we see the term again.  If the counter surpasses a predefined
threshold, the term is removed from the map and inserted into a cuckoo
filter.  If a future term is found in the cuckoo filter we assume it
was previously removed from the map and is "common".

The map keys are the "rare" terms after collection is done.
2019-07-01 10:30:02 -04:00
Benjamin Trent 8108834534
[ML][Data Frame] account for delay in writing stats docs (#43703) (#43819) 2019-07-01 09:14:44 -05:00
Benjamin Trent 4c95c0c456
[ML][Data Frame] reduce audit frequency, change log msg, and level (#43771) (#43818) 2019-07-01 09:14:26 -05:00
Tanguy Leroux 7554420581 Update docs for Open/Close API (#43809)
Relates #43530
2019-07-01 15:20:36 +02:00
Nhat Nguyen 598e00a689 Make peer recovery send file info step async (#43792)
Relates #36195
2019-07-01 08:40:45 -04:00
Dimitrios Liappis 9aa6f7c434
Update TLS configuration in Docker docs (#43816)
Following the removal of the `unzip` package from the Elasticsearch 
Docker image in #39040, update setup instructions for TLS in Docker.

Also avoid cross-platform ownership+permission issues by not relying
on local bind mounts for storing generated certs and don't require 
`curl` locally installed.

Backport of #43748
2019-07-01 15:33:34 +03:00
Mark Vieira 13887c01cc Remove compile-time dependency on test fixtures (#43651) 2019-07-01 14:59:41 +03:00
David Turner 40d43e3f87 Avoid IP addresses for bootstrapping in setup docs (#43802)
Removes the suggestion to use IP addresses for `cluster.initial_master_nodes`
in the "important settings" discovery docs, leaving only the suggestion to use
node names.

Relates #41179, #41569
2019-07-01 12:39:54 +01:00
Julie Tibshirani ffa5919d7c
Add support for 'flattened object' fields. (#43762)
This commit merges the `object-fields` feature branch. The new 'flattened
object' field type allows an entire JSON object to be indexed into a field, and
provides limited search functionality over the field's contents.
2019-07-01 12:08:50 +03:00
Alpar Torok b4303290d9 Testclusters: improove timeout handling (#43440) 2019-07-01 11:39:53 +03:00
Hendrik Muhs a58d231f4d relax trigger count for transform stats test (#43753)
relax trigger count test as we can not guarantee it due to async behaviour
2019-07-01 10:30:40 +02:00
Alpar Torok 717d14a7e2
Backport: convert x pack qa (#43763)
* Revert "Revert "Test clusters: convert x-pack qa tests (#43283)" (#43549)"

This reverts commit 8d9a971259.

* Fix failing test
2019-07-01 10:38:56 +03:00
Dimitris Athanasiou 3bdb9d5f08
[7.x][ML] Correct df-analytics version introduced to 7.3.0 (#43784) (#43795) 2019-07-01 09:19:04 +03:00
weizijun 28358fdbed
show a full ingest example in the index page, to let user fast understand ingest node. (#43476) 2019-07-01 08:04:57 +02:00
Ryan Ernst 3a2c698ce0
Rename Action to ActionType (#43778)
Action is a class that encapsulates meta information about an action
that allows it to be called remotely, specifically the action name and
response type. With recent refactoring, the action class can now be
constructed as a static constant, instead of needing to create a
subclass. This makes the old pattern of creating a singleton INSTANCE
both misnamed and lacking a common placement.

This commit renames Action to ActionType, thus allowing the old INSTANCE
naming pattern to be TYPE on the transport action itself. ActionType
also conveys that this class is also not the action itself, although
this change does not rename any concrete classes as those will be
removed organically as they are converted to TYPE constants.

relates #34389
2019-06-30 22:00:17 -07:00
Dimitris Athanasiou 8f49d01113
[7.x][ML] Rename df-analytics `_id_copy` to `ml__id_copy` (#43754) (#43783)
Renames `_id_copy` to `ml__id_copy` as field names starting with
underscore are deprecated. The new field name `ml__id_copy` was
chosen as an obscure enough field that users won't have in their data.
Otherwise, this field is only intented to be used by df-analytics.
2019-06-30 19:37:00 +03:00
David Turner fca7a19713 Avoid parallel reroutes in DiskThresholdMonitor (#43381)
Today the `DiskThresholdMonitor` limits the frequency with which it submits
reroute tasks, but it might still submit these tasks faster than the master can
process them if, for instance, each reroute takes over 60 seconds. This causes
a problem since the reroute task runs with priority `IMMEDIATE` and is always
scheduled when there is a node over the high watermark, so this can starve any
other pending tasks on the master.

This change avoids further updates from the monitor while its last task(s) are
still in progress, and it measures the time of each update from the completion
time of the reroute task rather than its start time, to allow a larger window
for other tasks to run.

It also now makes use of the `RoutingService` to submit the reroute task, in
order to batch this task with any other pending reroutes. It enhances the
`RoutingService` to notify its listeners on completion.

Fixes #40174
Relates #42559
2019-06-30 16:54:16 +01:00
Nhat Nguyen 55b3ec8d7b Make peer recovery clean files step async (#43787)
Relates #36195
2019-06-29 18:30:51 -04:00
Albert Zaharovits 5e17bc5dcc
Consistent Secure Settings #40416
Introduces a new `ConsistentSecureSettingsValidatorService` service that exposes
a single public method, namely `allSecureSettingsConsistent`. The method returns
`true` if the local node's secure settings (inside the keystore) are equal to the
master's, and `false` otherwise. Technically, the local node has to have exactly
the same secure settings - setting names should not be missing or in surplus -
for all `SecureSetting` instances that are flagged with the newly introduced
`Property.Consistent`. It is worth highlighting that the `allSecureSettingsConsistent`
is not a consensus view across the cluster, but rather the local node's perspective
in relation to the master.
2019-06-29 23:26:17 +03:00
David Roberts b599c68d23 [ML] Assert that a no-op job creates no results nor state (#43681)
If a job is opened and then closed and does nothing in
between then it should not persist any results or state
documents. This change adapts the no-op job test to
assert no results in addition to no state, and to log
any documents that cause this assertion to fail.

Relates elastic/ml-cpp#512
Relates #43680
2019-06-29 14:57:49 +01:00
Ryan Ernst 28ab77a023
Add StreamableResponseAction to aid in deprecation of Streamable (#43770)
The Action base class currently works for both Streamable and Writeable
response types. This commit intorduces StreamableResponseAction, for
which only the legacy Action implementions which provide newResponse()
will extend. This eliminates the need for overriding newResponse() with
an UnsupportedOperationException.

relates #34389
2019-06-28 21:40:00 -07:00
David Roberts 7951c63b91 [ML] Mark ml-cpp dependency as regularly changing (#43760)
Since #41817 was merged the ml-cpp zip file for any
given version has been cached indefinitely by Gradle.
This is problematic, particularly in the case of the
master branch where the version 8.0.0-SNAPSHOT will
be in use for more than a year.

This change tells Gradle that the ml-cpp zip file is
a "changing" dependency, and to check whether it has
changed every two hours. Two hours is a compromise
between checking on every build and annoying developers
with slow internet connections and checking rarely
causing bug fixes in the ml-cpp code to take a long
time to propagate through to elasticsearch PRs that
rely on them.
2019-06-28 21:21:18 +01:00
Martijn van Groningen 9d5c66be41
Migrate watcher hlrc response tests to use AbstractResponseTestCase (#43478)
Relates to #43472
2019-06-28 21:38:44 +02:00
James Rodewig d8fe0f5c13 [DOCS] Rewrite `terms_set` query (#43060) 2019-06-28 12:57:22 -04:00
Benjamin Trent 67a3c656c3
[7.x] [ML][Data Frame] removing format support (#43659) (#43747)
* [ML][Data Frame] removing format support (#43659)

* Fixing conflicts
2019-06-28 10:02:37 -05:00
Tanguy Leroux f02cbe9e40 Trim translog for closed indices (#43156)
Today when an index is closed all its shards are forced flushed
but the translog files are left around. As explained in #42445
we'd like to trim the translog for closed indices in order to
consume less disk space. This commit reuses the existing
AsyncTrimTranslogTask task and reenables it for closed indices.

At the time the task is executed, we should have the guarantee
that nothing holds the translog files that are going to be removed.
It also leaves a short period of time (10 min) during which translog
files of a recently closed index are still present on disk. This could
 also help in some cases where the closed index is reopened
shortly after being closed (in order to update an index setting
for example).

Relates to #42445
2019-06-28 16:58:39 +02:00
Jim Ferenczi 7ca69db83f Refactor IndexSearcherWrapper to disallow the wrapping of IndexSearcher (#43645)
This change removes the ability to wrap an IndexSearcher in plugins. The IndexSearcherWrapper is replaced by an IndexReaderWrapper and allows to wrap the DirectoryReader only. This simplifies the creation of the context IndexSearcher that is used on a per request basis. This change also moves the optimization that was implemented in the security index searcher wrapper to the ContextIndexSearcher that now checks the live docs to determine how the search should be executed. If the underlying live docs is a sparse bit set the searcher will compute the intersection
betweeen the query and the live docs instead of checking the live docs on every document that match the query.
2019-06-28 16:28:02 +02:00
weizijun 377c4cfdc0 Fix threshold spelling errors (#43326)
Substitutes treshold by threshold
2019-06-28 15:47:57 +02:00
Alpar Torok 23f739b513 Testclusters: Convert additional projects (#43625)
* Testclusters: Convert additional projects

Found some more that were not using testclusters from elasticsearch-ci/1

* Allow IOException too

* Make the client more resilient
2019-06-28 16:44:42 +03:00
Alpar Torok d1a4d8866d Add missing dependencies so we can build in parallel (#43672) 2019-06-28 16:41:18 +03:00
Alan Woodward 81dbcfb268 Wildcard intervals (#43691)
This commit adds a wildcard intervals source, similar to the prefix. It
also changes the term parameter in prefix to read prefix, to bring it
in to line with the pattern parameter in wildcard.

Closes #43198
2019-06-28 14:04:03 +01:00
James Rodewig 74dd6e49fc [DOCS] Rewrite boosting query (#43647) 2019-06-28 08:35:55 -04:00
Henning Andersen 632da7f2c8 Enabled cannot be updated (#43701)
Removed the invalid tip that enabled can be updated for existing fields
and clarified instead that it cannot.

Related to #33566 and #33933
2019-06-28 12:59:00 +02:00
Dimitris Athanasiou 86c853a7c2
[7.x][ML] Rename outlier score setting to feature_influence_threshold (#43705) (#43734)
Renames outlier score setting `minimum_score_to_write_feature_influence`
to `feature_influence_threshold`.
2019-06-28 13:28:25 +03:00
Dimitris Athanasiou cab879118d
[7.x][ML] Support multiple source indices for df-analytics (#43702) (#43731)
This commit adds support for multiple source indices.
In order to deal with multiple indices having different mappings,
it attempts a best-effort approach to merge the mappings assuming
there are no conflicts. In case conflicts exists an error will be
returned.

To allow users creating custom mappings for special use cases,
the destination index is now allowed to exist before the analytics
job runs. In addition, settings are no longer copied except for
the `index.number_of_shards` and `index.number_of_replicas`.
2019-06-28 13:28:03 +03:00
Christoph Büscher 2cc7f5a744
Allow reloading of search time analyzers (#43313)
Currently changing resources (like dictionaries, synonym files etc...) of search
time analyzers is only possible by closing an index, changing the underlying
resource (e.g. synonym files) and then re-opening the index for the change to
take effect.

This PR adds a new API endpoint that allows triggering reloading of certain
analysis resources (currently token filters) that will then pick up changes in
underlying file resources. To achieve this we introduce a new type of custom
analyzer (ReloadableCustomAnalyzer) that uses a ReuseStrategy that allows
swapping out analysis components. Custom analyzers that contain filters that are
markes as "updateable" will automatically choose this implementation. This PR
also adds this capability to `synonym` token filters for use in search time
analyzers.

Relates to #29051
2019-06-28 09:55:40 +02:00