Commit Graph

48010 Commits

Author SHA1 Message Date
Yogesh Gaikwad cec2ff5ef4 Enhance docs for create api keys created when role descriptor not specified (#46897)
This commit adds the documentation to point the user that when one
creates API keys with no role descriptor specified then that API
key will have a point in time snapshot of user permissions.

Closes#46876
2019-09-30 12:15:29 +10:00
Rory Hunter 53a4d2176f
Convert most awaitBusy calls to assertBusy (#45794) (#47112)
Backport of #45794 to 7.x. Convert most `awaitBusy` calls to
`assertBusy`, and use asserts where possible. Follows on from #28548 by
@liketic.

There were a small number of places where it didn't make sense to me to
call `assertBusy`, so I kept the existing calls but renamed the method to
`waitUntil`. This was partly to better reflect its usage, and partly so
that anyone trying to add a new call to awaitBusy wouldn't be able to find
it.

I also didn't change the usage in `TransportStopRollupAction` as the
comments state that the local awaitBusy method is a temporary
copy-and-paste.

Other changes:

  * Rework `waitForDocs` to scale its timeout. Instead of calling
    `assertBusy` in a loop, work out a reasonable overall timeout and await
    just once.
  * Some tests failed after switching to `assertBusy` and had to be fixed.
  * Correct the expect templates in AbstractUpgradeTestCase.  The ES
    Security team confirmed that they don't use templates any more, so
    remove this from the expected templates. Also rewrite how the setup
    code checks for templates, in order to give more information.
  * Remove an expected ML template from XPackRestTestConstants The ML team
    advised that the ML tests shouldn't be waiting for any
    `.ml-notifications*` templates, since such checks should happen in the
    production code instead.
  * Also rework the template checking code in `XPackRestTestHelper` to give
    more helpful failure messages.
  * Fix issue in `DataFrameSurvivesUpgradeIT` when upgrading from < 7.4
2019-09-29 12:21:46 +01:00
Jason Tedor 98989f7b37
Use fallback settings in throttling decider (#47261)
This commit replaces some uses of Setting#getRaw in the throttling
allocation decider settings. Instead, these settings should be using
fallback settings.
2019-09-28 08:06:24 -04:00
Jason Tedor bd603b0a7b
Remove dead leniency in allow rebalance setting use (#47259)
This commit removes some leniency that exists in getting the allow
rebalance setting. Fortunately, that leniency is dead code, this can
never happen. The reason this can never happen is because the settings
infrastructure will not allow setting an invalid value for this
setting. If you try to set this in the elasticsearch.yml, then the node
will fail to start, since parsing the setting will fail. If you try to
set this via an update settings API call, then parsing the setting will
fail and the settings update will be rejected. Therefore, this leniency
can never be activated, so we remove it.

This commit is the first of a few in an attempt to remove the public
uses of Setting#getRaw.
2019-09-28 08:05:37 -04:00
lcawl ab871add36 [DOCS] Skip code snippet testing in monitoring example 2019-09-27 17:11:24 -07:00
Lisa Cawley 62baa85632 [DOCS] Adds production monitoring page (#47184) 2019-09-27 15:01:54 -07:00
James Rodewig b98c57ab21
[DOCS] Document missing query parms for cat recovery API (#47181) (#47235)
Documents the following query parameters for
the /_cat/recovery API:

* `active_only`
* `detailed`
* `index`
2019-09-27 16:31:10 -04:00
Tim Brooks e11c56760d
Fix bind failure logging for mock transport (#47150)
Currently the MockNioTransport uses a custom exception handler for
server channel exceptions. This means that bind failures are logged at
the warn level. This commit modifies the transport to use the common
TcpTransport exception handler which will log exceptions at the correct
level.
2019-09-27 13:53:48 -06:00
Jack Conradson d09965a6dd Add ClassWriter to Painless writing pass (#47140)
This the first part of a series to allow nodes to write all of their appropriate
pieces to the class. Currently, nodes must add their bindings, constants, and
functions to main SClass node for delayed writing. This instead adds a
Painless version of ClassWriter to the write pass. The Painless ClassWriter
contains an appropriate ClassVisitor that can be accessed in any node
during the process along with access to the clinit method, and finally a
shortcut for creating new MethodWriter. The next step will be removing the
delayed writing in SClass, and instead, delegate all writing responsibilities to
the nodes.
2019-09-27 11:04:15 -07:00
Nhat Nguyen 444b47ce88 Relax maxSeqNoOfUpdates assertion in FollowingEngine (#47188)
We disable MSU optimization if the local checkpoint is smaller than
max_seq_no_of_updates. Hence, we need to relax the MSU assertion in
FollowingEngine for that scenario. Suppose the leader has three
operations: index-0, delete-1, and index-2 for the same doc Id. MSU on
the leader is 1 as index-2 is an append. If the follower applies index-0
then index-2, then the assertion is violated.

Closes #47137
2019-09-27 14:00:20 -04:00
James Rodewig e01465eb88
[DOCS] Correct typo in ICU Analysis plugin description (#47175) (#47219) 2019-09-27 13:04:14 -04:00
James Rodewig b159305274
[DOCS] Add redirect for SLM API docs (#46838) (#46865) 2019-09-27 11:05:55 -04:00
David Roberts e943e27954 Spawn controller processes from a different directory on macOS (#47013)
This is the Java side of https://github.com/elastic/ml-cpp/pull/593
with a fallback so that ml-cpp bundles with either the
new or old directory structure work for the time being.
A few days after merging the C++ changes a followup to
this change will be made that removes the fallback.
2019-09-27 14:02:40 +01:00
Marios Trivyzas 01623f9f1c
SQL: Add alias DATETRUNC to DATE_TRUNC function (#47173)
To be on the safe side in terms of use cases also add the alias
DATETRUNC to the DATE_TRUNC function.

Follows: #46473

(cherry picked from commit 9ac223cb1fc66486f86e218fa785a32b61e9bacc)
2019-09-27 15:38:51 +03:00
Andrei Dan 4c909438dd
Fix OriginationDate parsing tests. (#47170) (#47200)
Drop the usage of `SimpleDateFormat` and use the `DateFormatter` instead

(cherry picked from commit 7cf509a7a11ecf6c40c44c18e8f03b8e81fcd1c2)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
2019-09-27 13:16:45 +01:00
Yannick Welsch 6fd3b4723f Remove write lock for Translog.getGeneration (#47036)
No need for the write lock, and currentFileGeneration is already protected by the read lock. 
Also removes the unused method "isCurrent".
2019-09-27 13:58:07 +02:00
Przemysław Witek 3fbd58d156
[7.x] Allow evaluation to consist of multiple steps. (#46653) (#47194) 2019-09-27 13:01:51 +02:00
Henning Andersen a1e2e208ce Mute Snapshot/Restore with repository-azure (#47204)
Relates #47201
2019-09-27 12:13:01 +02:00
István Zoltán Szabó 0ab7132c47 [DOCS] Reformats Profile API (#47168)
* [DOCS] Reformats Profile API.

* [DOCS] Fixes failing docs test.
2019-09-27 11:14:14 +02:00
Colin Goodheart-Smithe 2cd6bed9c8
Updates 7.4.0 release notes 2019-09-27 09:20:08 +01:00
Tanguy Leroux 42ae76ab7c Injected response errors in Azure repository tests should have a body (#47176)
The Azure SDK client expects server errors to have a body, 
something that looks like:

<?xml version="1.0" encoding="utf-8"?>  
<Error>  
  <Code>string-value</Code>  
  <Message>string-value</Message>  
</Error> 

I've forgot to add such errors in Azure tests and that triggers 
some NPE in the client like the one reported in #47120.

Closes #47120
2019-09-27 09:43:29 +02:00
Alpar Torok 813b130e08 Exclude the demo folder form the JDK (#47161)
The folder contains jars with source code that fail the lintian test on
debian (based) distributions.
2019-09-27 10:35:34 +03:00
Alpar Torok 944421627d Work around error building deb on Windows (#47011)
Relates to  #47007 . the `gradle-ospackage-plugin` plugin doesn't
properly support symlink on windows.

This PR changes the way we configure tasks to prevent building these
packages as part of a windows check.
2019-09-27 09:04:29 +03:00
Pius 9c60e15ebd [DOCS] Add 'remote clusters' requirement link to CCR docs (#47185) 2019-09-26 16:15:21 -04:00
Nicole Albee 4714f10650 [DOCS] Remove coming tag from 7.2.0 release highlights (#47183) 2019-09-26 15:48:54 -04:00
Costin Leau b29a2cb360 SQL: Check case where the pivot limit is reached (#47121)
In some cases, the fetch size affects the way the groups are returned
causing the last page to go beyond the limit. Add dedicated check to
prevent extra data from being returned.

Fix #47002

(cherry picked from commit f4c29646f097bbd29855300342823ef4cef61c05)
2019-09-26 22:32:42 +03:00
Jack Conradson 9b4f377474 Change Painless function node to use a block instead of raw statements (#46884)
This change improves the node structure of SFunction. SFunction now uses 
an SBlock instead of a List of AStatments reducing code duplication and 
gives a future target for symbol table scoping.
2019-09-26 10:33:35 -07:00
Igor Motov ae202fda21 SQL: Add support for shape type (#46464)
Enables support for Cartesian geometries shape type. We still need to
decide how to handle the distance function since it is currently using
the haversine distance formula and returns results in meters, which
doesn't make any sense for Cartesian geometries.

Closes #46412
Relates to #43644
2019-09-26 09:47:42 -04:00
James Rodewig 0c575dc1e8 [DOCS] Correct link to `index.store.preload` setting (#47145) 2019-09-26 08:57:09 -04:00
James Rodewig 3b626c2d56
[DOCS] Reformat get pipeline API (#47131) (#47163) 2019-09-26 08:51:12 -04:00
David Roberts 77cc6d5bad [TEST] Work around _cat/indices bug with security enabled (#47160)
When the ML native multi-node tests use _cat/indices/_all
and the request goes to a non-master node, _all is
translated to a list of concrete indices by the authz layer
on the coordinating node before the request is forwarded
to the master node. Then it is possible for the master
node to return an index_not_found_exception if one of
the concrete indices that was expanded on the
coordinating node has been deleted in the meantime.
(#47159 has been opened to track the underlying problem.)

It has been observed that the index that gets deleted when
the problem affects the ML native multi-node tests is
always the ML notifications index. The tests that fail are
only interested in the presence or absense of ML results
indices. Therefore the workaround is to only _cat indices
that match the ML results index pattern.

Fixes #45652
2019-09-26 13:29:40 +01:00
Dimitris Athanasiou 0765bd4bf7
[7.x][ML] Ensure data frame analytics task is only marked completed once (#47119) (#47157)
Closes #46907
2019-09-26 15:26:06 +03:00
Jim Ferenczi 73a09b34b8
Replace SearchContextException with SearchException (#47046)
This commit removes the SearchContextException in favor of a simpler
SearchException that doesn't leak the SearchContext.

Relates #46523
2019-09-26 14:21:23 +02:00
Tanguy Leroux 95e2ca741e
Remove unused private methods and fields (#47154)
This commit removes a bunch of unused private fields and unused
private methods from the code base.

Backport of (#47115)
2019-09-26 12:49:21 +02:00
Yogesh Gaikwad 9a64b7a888
[Backport] Validate `query` field when creating roles (#46275) (#47094)
In the current implementation, the validation of the role query
occurs at runtime when the query is being executed.

This commit adds validation for the role query when creating a role
but not for the template query as we do not have the runtime
information required for evaluating the template query (eg. authenticated user's
information). This is similar to the scripts that we
store but do not evaluate or parse if they are valid queries or not.

For validation, the query is evaluated (if not a template), parsed to build the
QueryBuilder and verify if the query type is allowed.

Closes #34252
2019-09-26 17:57:36 +10:00
jimczi 97d977f381 #47046 Fix serialization version check after backport 2019-09-26 09:56:24 +02:00
Jim Ferenczi 04972baffa
Merge ShardSearchTransportRequest and ShardSearchLocalRequest (#46996) (#47081)
This change merges the `ShardSearchTransportRequest` and `ShardSearchLocalRequest`
into a single `ShardSearchRequest` that can be used to create a SearchContext.

Relates #46523
2019-09-26 09:20:53 +02:00
Martijn van Groningen 429f23ea2f
Allow ingest processors to execute in a non blocking manner. (#47122)
Backport of #46241

This PR changes the ingest executing to be non blocking
by adding an additional method to the Processor interface
that accepts a BiConsumer as handler and changing
IngestService#executeBulkRequest(...) to ingest document
in a non blocking fashion iff a processor executes
in a non blocking fashion.

This is the second PR that merges changes made to server module from
the enrich branch (see #32789) into the master branch.

The plan is to merge changes made to the server module separately from
the pr that will merge enrich into master, so that these changes can
be reviewed in isolation.

This change originates from the enrich branch and was introduced there
in #43361.
2019-09-26 08:55:28 +02:00
David Turner 45c7783018
Warn on slow metadata persistence (#47130)
Today if metadata persistence is excessively slow on a master-ineligible node
then the `ClusterApplierService` emits a warning indicating that the
`GatewayMetaState` applier was slow, but gives no further details. If it is
excessively slow on a master-eligible node then we do not see any warning at
all, although we might see other consequences such as a lagging node or a
master failure.

With this commit we emit a warning if metadata persistence takes longer than a
configurable threshold, which defaults to `10s`. We also emit statistics that
record how much index metadata was persisted and how much was skipped since
this can help distinguish cases where IO was slow from cases where there are
simply too many indices involved.

Backport of #47005.
2019-09-26 07:40:54 +01:00
Benjamin Trent fcddaa90de
[7.x] [ML][Inference] adding tree model (#47044) (#47141)
* [ML][Inference] adding tree model (#47044)

* [ML][Inference] adding tree model

* renaming features for updated schema

* fixing 7.x compilation
2019-09-25 19:11:15 -04:00
Gordon Brown 7ac647c365
Add support for POST requests to SLM Execute API (#47061)
This commit adds support for POST requests to the SLM `_execute` API,
because POST is a more appropriate HTTP verb for this action as it is
not idempotent. The docs are also changed to favor POST over PUT,
although PUT is not removed or officially deprecated.
2019-09-25 16:15:10 -06:00
Tim Brooks 4f47e1f169
Extract proxy connection logic to specialized class (#47138)
Currently the logic to check if a connection to a remote discovery node
exists and otherwise create a proxy connection is mixed with the
collect nodes, cluster connection lifecycle, and other
RemoteClusterConnection logic. This commit introduces a specialized
RemoteConnectionManager class which handles the open connections.
Additionally, it reworks the "round-robin" proxy logic to create the list
of potential connections at connection open/close time, opposed to each
time a connection is requested.
2019-09-25 15:58:18 -06:00
Nhat Nguyen 7c5a088aa5 Increase ensureGreen timeout for testReplicaCorruption (#47136)
We can have a large number of shard copies in this test. For example,
the two recent failures have 24 and 27 copies respectively and all
replicas have to copy segment files as their stores are corrupted. Our
CI needs more than 30 seconds to start all these copies.

Note that in two recent failures, the cluster was green just after the
cluster health timed out.

Closes #41899
2019-09-25 17:04:08 -04:00
Andrei Dan 27520cac3b
ILM: parse origination date from index name (#46755) (#47124)
* ILM: parse origination date from index name (#46755)

Introduce the `index.lifecycle.parse_origination_date` setting that
indicates if the origination date should be parsed from the index name.
If set to true an index which doesn't match the expected format (namely
`indexName-{dateFormat}-optional_digits` will fail before being created.
The origination date will be parsed when initialising a lifecycle for an
index and it will be set as the `index.lifecycle.origination_date` for
that index.

A user set value for `index.lifecycle.origination_date` will always
override a possible parsable date from the index name.

(cherry picked from commit c363d27f0210733dad0c307d54fa224a92ddb569)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>

* Drop usage of Map.of to be java 8 compliant
2019-09-25 21:44:16 +01:00
Lee Hinman a267df30fa Wait for snapshot completion in SLM snapshot invocation (#47051)
* Wait for snapshot completion in SLM snapshot invocation

This changes the snapshots internally invoked by SLM to wait for
completion. This allows us to capture more snapshotting failure
scenarios.

For example, previously a snapshot would be created and then registered
as a "success", however, the snapshot may have been aborted, or it may
have had a subset of its shards fail. These cases are now handled by
inspecting the response to the `CreateSnapshotRequest` and ensuring that
there are no failures. If any failures are present, the history store
now stores the action as a failure instead of a success.

Relates to #38461 and #43663
2019-09-25 14:25:22 -06:00
Lisa Cawley 287d96d1a1 [DOCS] Fix links to transform pages (#47134) 2019-09-25 13:09:47 -07:00
Mark Vieira d0495b8f93
Track enabled test task candidate class files as task input (#47054) 2019-09-25 11:20:02 -07:00
Gordon Brown a46eef9634
Change SLM stats format (#46991)
Using arrays of objects with embedded IDs is preferred for new APIs over
using entity IDs as JSON keys.  This commit changes the SLM stats API to
use the preferred format.
2019-09-25 11:32:08 -06:00
Colin Goodheart-Smithe 35f056661f
Updates 7.4.0 release notes 2019-09-25 18:10:09 +01:00
James Rodewig 9ddc99de23 [DOCS] Reformat clone index API docs (#46762) 2019-09-25 12:37:53 -04:00