Commit Graph

42567 Commits

Author SHA1 Message Date
Benjamin Trent f7ada9b29b
Add delayed datacheck to the datafeed job runner (#35387)
* ML: Adding missing datacheck to datafeedjob

* Adding client side and docs

* Making adjustments to validations

* Making values default to on, having more sensible limits

* Intermittent commit, still need to figure out interval

* Adjusting delayed data check interval

* updating docs

* Making parameter Boolean, so it is nullable

* bumping bwc to 7 before backport

* changing to version current

* moving delayed data check config its own object

* Separation of duties for delayed data detection

* fixing checkstyles

* fixing checkstyles

* Adjusting default behavior so that null windows are allowed

* Mentioning the default value

* Fixing comments, syncing up validations
2018-11-15 13:32:45 -06:00
Jim Ferenczi c7a2c6d549
Handle IndexOrDocValuesQuery in composite aggregation (#35392)
The `composite` aggregation can optimize its execution when the query
is a `match_all` or a `range` over the field that is used in the first source
of the aggregation. However we only check for instances of `PointRangeQuery` whereas
the range query builder creates an  `IndexOrDocValuesQuery`. This means that
today the optimization does not apply to `range` query even if the code could handle it.
This change fixes this issue by extracting the index query inside `IndexOrDocValuesQuery`.
2018-11-15 17:52:06 +01:00
Lee Hinman 90b38d9b3d
[ILM] Avoid NullPointerException in CopyExecutionStateStep (#35568)
In the event that the target index does not exist when `CopyExecutionStateStep`
executes, this avoids a `NullPointerException` and provides a more helpful error
to the ILM user.

Resolves #35567
2018-11-15 09:29:26 -07:00
debadair 19001e71cc
[DOCS] Updated release state for 7.0.0-alpha1 (#35596)
* [DOCS] Updated release state for 7.0.0-alpha1

* Update Versions.asciidoc
2018-11-15 08:22:13 -08:00
Tim Vernum 1e60b282fc Grant .tasks access to kibana_system role (#35573)
Kibana now uses the tasks API to manage automatic reindexing of the
.kibana index during upgrades.

The implementation of the tasks API requires that
1. the user executing the task can create & write to the ".tasks" index
2. the user checking on the status of the task can read (Get) the
   relevant document from the ".tasks" index
2018-11-15 08:14:58 -07:00
Ed Savage 2d948a001e
[HLRC][ML] Add ML delete model snapshot API (#35537)
Relates to #29827
2018-11-15 14:57:17 +00:00
Armin Braun 68d9b56fcb
MINOR: Remove SNAPSHOT Suffixes in Gradle (#35577)
* MINOR: Remove SNAPSHOT Suffixes in Gradle

* The snapshot qualifiers here are redundant and 6.5.0 is released anyway
2018-11-15 14:24:04 +01:00
Armin Braun e6f4425e01
REST-Tests: Use Dedicated AntBuilders (#35576)
* REST-Tests: Use Dedicated AntBuilders

* Use dedicated AntBuilder everywhere since AntBuilder is not threadsafe
* Closes #33778
2018-11-15 14:23:28 +01:00
Andrei Stefan eaf010cd73
SQL: clear the cursor if nested inner hits are enough to fulfill the query required limits (#35398) 2018-11-15 14:33:47 +02:00
Christoph Büscher bc5b1afb55
HLRC: Add parameters to stopRollupJob API (#35545)
With #34811 the API for stopping rollup jobs got two new url parameters
"wait_for_completion" and "timeout". This change adds these to the HLRC APIs as
well.

Relates to #34811
2018-11-15 10:50:48 +01:00
Luca Cavanna 8e2c84ad8b
Clean up XPackInfoResponse class and related tests (#35547)
Response classes in Elasticsearch (and xpack) only need to implement ToXContent, which is needed to print their output put in the REST layer and return the response in json (or others) format. On the other hand, response classes that are added to the high-level REST client, need to do the opposite: parse xcontent and create a new object based on that.

This commit removes the parsing code from the XPackInfoResponse server variant, and the toXContent portion from the corresponding client variant. It also removes a client specific test class that looks redundant now that we have a single test class for both classes.
2018-11-15 09:59:27 +01:00
David Turner 928ab6a5b1
Suppress CachedTimeThread in hot threads output (#35558)
In #23175 we renamed `ThreadPool$EstimatedTimeThread` to
`ThreadPool$CachedTimeThread` but did not update the corresponding entry in
`HotThreads#isIdleThread`. This commit addresses this.
2018-11-15 08:25:09 +00:00
Tanguy Leroux c9b4ef0dfd
Use RunOnce when appropriate (#35553)
This pull request replaces some blocks of code that must be run once 
and that are currently based on AtomicBoolean by the convient RunOnce 
class added in #35489.
2018-11-15 09:24:40 +01:00
Clinton Gormley cb8bdeae68 Fixed bad link in ingest-geo-point 2018-11-15 09:20:04 +01:00
Tanguy Leroux 5cd9b518ac
Fix usages of XContentParserUtils.ensureExpectedToken() (#35530)
Some usages of XContentParserUtils.ensureExpectedToken() are inverting
the expected and actual tokens, resulting in wrong error messages.
2018-11-15 09:18:32 +01:00
Tal Levy dc1821c707
explain geo_point mapping in geoip-processor (#29114)
simple docs change to add missing mapping explanation. Users may not be aware this is
a prerequisite for doing geo-queries on this enriched data.
2018-11-14 20:05:45 -08:00
Tal Levy 95a09ab2d6
[ILM] TEST: skip checking _ilm/explain response b/c flaky timing (#35562)
The docs are not resilient to timing issues where the ILM metadata is not set on newly 
created indices, so we shouldn't be so strict on the returned response
2018-11-14 16:45:06 -08:00
Deb Adair 91a38b8fe0 [DOCS] Fixed bad cross doc link. 2018-11-14 16:09:18 -08:00
Jay Modi faa9523d19
Remove deprecated constructor from failure handler (#35565)
The DefaultAuthenticationFailureHandler has a deprecated constructor
that was present to prevent a breaking change to custom realm plugin
authors in 6.x. This commit removes the constructor and its uses.
2018-11-14 17:04:33 -07:00
debadair 38d6bb27f5
[DOCS] Added link to standalone ODBC driver doc. (#35526) 2018-11-14 15:23:28 -08:00
Tal Levy 5d0544d732
update move-to-step spec with proper docs and extra warning to clients (#35477) 2018-11-14 14:08:31 -08:00
Tal Levy f70d4470f7
[ILM] getting started documentation (#35379) 2018-11-14 13:58:08 -08:00
Lee Hinman 8ea999e489
Include stack trace with ILM error in explain output (#35512)
This changes the stacktrace to be included with the ILM explain error when the
index is an on ERROR step.

Before:

```json
{
  "indices" : {
    "foo" : {
      "index" : "foo",
      "managed" : true,
      "policy" : "bad",
      "lifecycle_date_millis" : 1542131670601,
      "phase" : "warm",
      "phase_time_millis" : 1542131676335,
      "action" : "shrink",
      "action_time_millis" : 1542131676335,
      "step" : "ERROR",
      "step_time_millis" : 1542131676451,
      "failed_step" : "shrink",
      "step_info" : {
        "type" : "illegal_argument_exception",
        "reason" : "the number of target shards [13] must be less that the number of source shards [2]"
      },
      "phase_execution" : {
        "policy" : "bad",
        "phase_definition" : {
          "min_age" : "5s",
          "actions" : {
            "shrink" : {
              "number_of_shards" : 13
            }
          }
        },
        "version" : 1,
        "modified_date_in_millis" : 1542131669839
      }
    }
  }
}
```

After

```
{
  "indices" : {
    "foo" : {
      "index" : "foo",
      "managed" : true,
      "policy" : "bad",
      "lifecycle_date_millis" : 1542131670601,
      "phase" : "warm",
      "phase_time_millis" : 1542131676335,
      "action" : "shrink",
      "action_time_millis" : 1542131676335,
      "step" : "ERROR",
      "step_time_millis" : 1542131676451,
      "failed_step" : "shrink",
      "step_info" : {
        "type" : "illegal_argument_exception",
        "reason" : "the number of target shards [13] must be less that the number of source shards [2]",
        "stack_trace" : "java.lang.IllegalArgumentException: the number of target shards [13] must be less that the number of source shards [2]\n\tat org.elasticsearch.cluster.metadata.IndexMetaData.selectShrinkShards(IndexMetaData.java:1509)\n\tat org.elasticsearch.action.admin.indices.shrink.TransportResizeAction.prepareCreateIndexRequest(TransportResizeAction.java:146)\n\tat org.elasticsearch.action.admin.indices.shrink.TransportResizeAction$1.onResponse(TransportResizeAction.java:104)\n\tat org.elasticsearch.action.admin.indices.shrink.TransportResizeAction$1.onResponse(TransportResizeAction.java:101)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:64)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:60)\n\tat org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.onCompletion(TransportBroadcastByNodeAction.java:383)\n\tat org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.onNodeResponse(TransportBroadcastByNodeAction.java:352)\n\tat org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction$1.handleResponse(TransportBroadcastByNodeAction.java:324)\n\tat org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction$1.handleResponse(TransportBroadcastByNodeAction.java:314)\n\tat org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1117)\n\tat org.elasticsearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:1198)\n\tat org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1178)\n\tat org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:54)\n\tat org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:417)\n\tat org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:391)\n\tat org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:251)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:309)\n\tat org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63)\n\tat org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:714)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:726)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:834)\n"
      },
      "phase_execution" : {
        "policy" : "bad",
        "phase_definition" : {
          "min_age" : "5s",
          "actions" : {
            "shrink" : {
              "number_of_shards" : 13
            }
          }
        },
        "version" : 1,
        "modified_date_in_millis" : 1542131669839
      }
    }
  }
}
```

Resolves #35498
2018-11-14 14:40:05 -07:00
Hendrik Muhs fc774a3776
add ES 6.5.1 (on master) (#35549)
add ES 6.5.1
2018-11-14 21:08:21 +01:00
Tal Levy 30a713cc3f
[ILM] add docs explaining high-level concepts (#35322)
add docs explaining high-level concepts
2018-11-14 09:23:50 -08:00
Benjamin Trent 803eccec11
HLRC: Adding ML Update Filter API (#35522)
* HLRC: Adding ml get filters api

* HLRC: Adding ML Update Filter API
2018-11-14 11:13:11 -06:00
Tanguy Leroux c8c8ce2374
Extract RunOnce into a dedicated class (#35489)
This commit extracts the static inner class RunOnce from 
WorkerBulkByScrollTaskState so that it can be reused at 
other places.
2018-11-14 17:33:04 +01:00
Thiago Souza 98e075227b [Doc] Correct jdbc url example (#35513)
Adds a missing `?` to separate the url parameters.
2018-11-14 17:30:23 +01:00
Alpar Torok a3cb74190c Fix flaky gradle integration test 2018-11-14 17:21:25 +02:00
Zachary Tong c7ea0343d2 [Rollup] Update versions after backport
Both serialization and yaml test skips
2018-11-14 09:56:45 -05:00
Tanguy Leroux 5b7446bb5f
Add Delete Privileges API to HLRC (#35454)
This commit adds the Delete Privileges API to the high level REST
client.

Related to #29827
2018-11-14 14:04:30 +01:00
Ed Savage e7b7d52a6a
[HLRC][ML] Add ML get model snapshots API (#35487)
Relates #29827
2018-11-14 13:03:04 +00:00
Yannick Welsch 4cfdb0609e
Adapt InternalCluster#fullRestart to call onNodeStopped when all nodes are stopped (#35494)
Refactors and simplifies the logic around stopping nodes, making sure that for a full cluster restart
onNodeStopped is only called after the nodes are actually all stopped (and in particular not while
starting up some nodes again). This change also ensures that a closed node client is not being used
anymore (which required a small change to a test).

Relates to #35049
2018-11-14 13:24:56 +01:00
Alpar Torok 212c202709 Remove hard coded version string in tests
- The current version was hard coded in the test, causing it to fail as
we removed the qualifier
- also applied the base plugin to the root project to get the `clean`
task to work as expected. This was preventing the failure from
reproducing locally.
2018-11-14 14:14:03 +02:00
Jim Ferenczi 72504c2512
Do not recommend to use the _id field in search_after docs (#35370)
The documentation of `search_after` recommends to use the `_id`
field as a tiebreaker for the sort without warning against
the additional memory required. This change changes the recommandation
to use a copy of the `_id` field with doc_values enabled.
2018-11-14 10:50:31 +01:00
Alpar Torok 0ce4649e88
Manage dependencies for test clusters (#35304)
* Manage dependencies for test clusters

Create a configuration and add the distribution to it automatically.
A task is created and added as a dependency to any task that uses a test
cluster.
The task extracts all the zip archives ( only zip support for now )
in the configuration.
We do this only once because most tests mostly use the same distribution
and thus we can avoid extracting it multiple times.
With this we will be able to start the node from the same files which
will most of the time live in OS caches or COW if the
configuration requires it.
2018-11-14 11:22:00 +02:00
Tanguy Leroux bbe50e7a86
Remove LoggingRunnable class (#35486)
This commit removes the unused LoggingRunnable class.
2018-11-14 10:12:25 +01:00
Alpar Torok dc0e05f35a
Fix Eclipse configuration (#35452) 2018-11-14 11:11:46 +02:00
Alpar Torok b49f461bf1
Swithc to jcenter and hope to have fewer network failures in builds (#35427)
* Swithc to jcenter

Jcenter suposedly operates on a CDN.
It should be faster and more reliable.
It's the default in Andorid Studio currently
( it switched from mavenCentral ).

This change is safe because jcenter is a superset of mavenCentral.

* update comment
2018-11-14 10:50:08 +02:00
Tanguy Leroux 31567cefb4
[RCI] Check blocks while having index shard permit in TransportReplicationAction (#35332)
Today, the TransportReplicationAction checks the global level blocks and 
the index level blocks before routing the operation to the primary, in the 
ReroutePhase, and it happens at the very beginning of the transport 
replication action execution. For the upcoming rework of the Close Index 
API and in order to deal with primary relocation, we'll need to also check 
for blocks before executing the operation on the primary (while holding a 
permit) but before routing to the new primary.

This pull request change the AsyncPrimaryAction so that it checks for 
replication action's blocks before executing the operation locally or before 
routing the primary action to the newly primary shard. The check is done 
while holding a PrimaryShardReference.

Related to #33888
2018-11-14 09:43:55 +01:00
Hendrik Muhs 5c84708ee5 test: expose error message on failure 2018-11-14 08:25:41 +01:00
Tim Vernum 231f6c1595
Formal support for "password_hash" in Put User (#35242)
For some time, the PutUser REST API has supported storing a pre-hashed
password for a user. The change adds validation and tests around that
feature so that it can be documented & officially supported.

It also prevents the request from containing both a "password" and a "password_hash".
2018-11-14 16:46:31 +11:00
Christoph Büscher d8b1c23e1d
Remove Comparable interface from ScoreAccessor (#35519)
The way ScoreAccessor implements `compareTo()` is problematic because it doesn't
completely follow the Comparable contract, specificaly symmetry (if x is a
ScoreAccessor and y any Number then x.comparTo(y) works, but y.compareTo(x)
generally does not even compile). Fortunately we don't seem to use the fact that
ScoreAccessor is a Comparable anywhere, so we can simply remove it.
2018-11-14 05:58:05 +01:00
Christoph Büscher 603d1a470f
Add stop rollup job support to HL REST Client (#34702)
This change adds support for stoping a rollup job to the High Level REST Client.

Relates to #29827
2018-11-14 05:40:42 +01:00
Zachary Tong c346a0f027
[Rollup] Add `wait_for_completion` option to StopRollupJob API (#34811)
This adds a `wait_for_completion` flag which allows the user to block 
the Stop API until the task has actually moved to a stopped state, 
instead of returning immediately.  If the flag is set, a `timeout` parameter
can be specified to determine how long (at max) to block the API
call.  If unspecified, the timeout is 30s.

If the timeout is exceeded before the job moves to STOPPED, a
timeout exception is thrown.  Note: this is just signifying that the API
call itself timed out.  The job will remain in STOPPING and evenutally
flip over to STOPPED in the background.

If the user asks the API to block, we move over the the generic
threadpool so that we don't hold up a networking thread.
2018-11-13 16:37:17 -05:00
jaymode 634bc9d4ea
mute TasksClientDocumentationIT testCancelTasks
This commit mutes this test, which has failed twice in CI today.

See #35514
2018-11-13 14:35:05 -07:00
Benjamin Trent b9eb5f7b63
HLRC: Adding ml get filters api (#35502)
* HLRC: Adding ml get filters api

* refactoring setId name
2018-11-13 14:53:32 -06:00
Colin Goodheart-Smithe c934fb087a
Adds HLRC docs for put lifecycle policy (#35457)
* Adds HLRC docs for put lifecycle policy

* Adds link to docs in client javadocs

* Fixes checkstyle

* Make the documentation use the right ack response
2018-11-13 20:40:53 +00:00
Vladimir Dolzhenko 9728119b82 [CI] AllocationIdIT testFailedRecoveryOnAllocateStalePrimaryRequiresAnotherAllocateStalePrimary failure
Closes #35504
2018-11-13 20:57:50 +01:00
Julie Tibshirani bc799e4a6f
Ignore warnings related to types deprecation in REST tests. (#35395) 2018-11-13 11:56:01 -08:00