Commit Graph

2004 Commits

Author SHA1 Message Date
Ryan Ernst 3c03975ce3
Watcher: Use joda method to get local TZ (#35608)
This commit switches from using java util's default timezone method to
using joda. The former can cause problems when the string representation
of the timezone is unknown to joda.

closes #35518
2018-11-16 15:46:27 -08:00
Arthur Gavlyukovskiy 022726011c Remove use of AbstractComponent in server (#35444)
Removed extending of AbstractComponent and changed logger usage to
explicit declaration. Abstract classes still have logger
declaration using this.getClass() in order to show implementation class
name in its logs.

See #34488
2018-11-16 16:10:32 -05:00
Julie Tibshirani c6a0904e0e
Deprecate types in count and msearch. (#35421)
* Deprecate types in count requests.
* Move RestCountAction to the 'search' package.
* Deprecate types in multi search requests.
* Add tests for types deprecation in the _search endpoint.
2018-11-16 13:04:43 -08:00
Benjamin Trent e360c600d0
ML: Adjusing BWC version post backport to 6.6 (#35605) 2018-11-16 06:50:12 -06:00
Gordon Brown 3883e9bf4c
Split RolloverStep into Wait and Action steps (#35524)
RolloverAction will now periodically check the rollover conditions using
the Rollover API with the dry_run option as an AsyncWaitStep, then run
the rollover itself by calling the Rollover API with no conditions,
which will always roll over, as an AsyncActionStep. This will resolve
race condition issues in policies using RolloverAction.
2018-11-15 17:11:31 -07:00
Benjamin Trent f7ada9b29b
Add delayed datacheck to the datafeed job runner (#35387)
* ML: Adding missing datacheck to datafeedjob

* Adding client side and docs

* Making adjustments to validations

* Making values default to on, having more sensible limits

* Intermittent commit, still need to figure out interval

* Adjusting delayed data check interval

* updating docs

* Making parameter Boolean, so it is nullable

* bumping bwc to 7 before backport

* changing to version current

* moving delayed data check config its own object

* Separation of duties for delayed data detection

* fixing checkstyles

* fixing checkstyles

* Adjusting default behavior so that null windows are allowed

* Mentioning the default value

* Fixing comments, syncing up validations
2018-11-15 13:32:45 -06:00
Lee Hinman 90b38d9b3d
[ILM] Avoid NullPointerException in CopyExecutionStateStep (#35568)
In the event that the target index does not exist when `CopyExecutionStateStep`
executes, this avoids a `NullPointerException` and provides a more helpful error
to the ILM user.

Resolves #35567
2018-11-15 09:29:26 -07:00
Tim Vernum 1e60b282fc Grant .tasks access to kibana_system role (#35573)
Kibana now uses the tasks API to manage automatic reindexing of the
.kibana index during upgrades.

The implementation of the tasks API requires that
1. the user executing the task can create & write to the ".tasks" index
2. the user checking on the status of the task can read (Get) the
   relevant document from the ".tasks" index
2018-11-15 08:14:58 -07:00
Andrei Stefan eaf010cd73
SQL: clear the cursor if nested inner hits are enough to fulfill the query required limits (#35398) 2018-11-15 14:33:47 +02:00
Luca Cavanna 8e2c84ad8b
Clean up XPackInfoResponse class and related tests (#35547)
Response classes in Elasticsearch (and xpack) only need to implement ToXContent, which is needed to print their output put in the REST layer and return the response in json (or others) format. On the other hand, response classes that are added to the high-level REST client, need to do the opposite: parse xcontent and create a new object based on that.

This commit removes the parsing code from the XPackInfoResponse server variant, and the toXContent portion from the corresponding client variant. It also removes a client specific test class that looks redundant now that we have a single test class for both classes.
2018-11-15 09:59:27 +01:00
Tanguy Leroux c9b4ef0dfd
Use RunOnce when appropriate (#35553)
This pull request replaces some blocks of code that must be run once 
and that are currently based on AtomicBoolean by the convient RunOnce 
class added in #35489.
2018-11-15 09:24:40 +01:00
Jay Modi faa9523d19
Remove deprecated constructor from failure handler (#35565)
The DefaultAuthenticationFailureHandler has a deprecated constructor
that was present to prevent a breaking change to custom realm plugin
authors in 6.x. This commit removes the constructor and its uses.
2018-11-14 17:04:33 -07:00
Tal Levy 5d0544d732
update move-to-step spec with proper docs and extra warning to clients (#35477) 2018-11-14 14:08:31 -08:00
Lee Hinman 8ea999e489
Include stack trace with ILM error in explain output (#35512)
This changes the stacktrace to be included with the ILM explain error when the
index is an on ERROR step.

Before:

```json
{
  "indices" : {
    "foo" : {
      "index" : "foo",
      "managed" : true,
      "policy" : "bad",
      "lifecycle_date_millis" : 1542131670601,
      "phase" : "warm",
      "phase_time_millis" : 1542131676335,
      "action" : "shrink",
      "action_time_millis" : 1542131676335,
      "step" : "ERROR",
      "step_time_millis" : 1542131676451,
      "failed_step" : "shrink",
      "step_info" : {
        "type" : "illegal_argument_exception",
        "reason" : "the number of target shards [13] must be less that the number of source shards [2]"
      },
      "phase_execution" : {
        "policy" : "bad",
        "phase_definition" : {
          "min_age" : "5s",
          "actions" : {
            "shrink" : {
              "number_of_shards" : 13
            }
          }
        },
        "version" : 1,
        "modified_date_in_millis" : 1542131669839
      }
    }
  }
}
```

After

```
{
  "indices" : {
    "foo" : {
      "index" : "foo",
      "managed" : true,
      "policy" : "bad",
      "lifecycle_date_millis" : 1542131670601,
      "phase" : "warm",
      "phase_time_millis" : 1542131676335,
      "action" : "shrink",
      "action_time_millis" : 1542131676335,
      "step" : "ERROR",
      "step_time_millis" : 1542131676451,
      "failed_step" : "shrink",
      "step_info" : {
        "type" : "illegal_argument_exception",
        "reason" : "the number of target shards [13] must be less that the number of source shards [2]",
        "stack_trace" : "java.lang.IllegalArgumentException: the number of target shards [13] must be less that the number of source shards [2]\n\tat org.elasticsearch.cluster.metadata.IndexMetaData.selectShrinkShards(IndexMetaData.java:1509)\n\tat org.elasticsearch.action.admin.indices.shrink.TransportResizeAction.prepareCreateIndexRequest(TransportResizeAction.java:146)\n\tat org.elasticsearch.action.admin.indices.shrink.TransportResizeAction$1.onResponse(TransportResizeAction.java:104)\n\tat org.elasticsearch.action.admin.indices.shrink.TransportResizeAction$1.onResponse(TransportResizeAction.java:101)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:64)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:60)\n\tat org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.onCompletion(TransportBroadcastByNodeAction.java:383)\n\tat org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.onNodeResponse(TransportBroadcastByNodeAction.java:352)\n\tat org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction$1.handleResponse(TransportBroadcastByNodeAction.java:324)\n\tat org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction$1.handleResponse(TransportBroadcastByNodeAction.java:314)\n\tat org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1117)\n\tat org.elasticsearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:1198)\n\tat org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1178)\n\tat org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:54)\n\tat org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:417)\n\tat org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:391)\n\tat org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:251)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:309)\n\tat org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63)\n\tat org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:714)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:726)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:834)\n"
      },
      "phase_execution" : {
        "policy" : "bad",
        "phase_definition" : {
          "min_age" : "5s",
          "actions" : {
            "shrink" : {
              "number_of_shards" : 13
            }
          }
        },
        "version" : 1,
        "modified_date_in_millis" : 1542131669839
      }
    }
  }
}
```

Resolves #35498
2018-11-14 14:40:05 -07:00
Zachary Tong c7ea0343d2 [Rollup] Update versions after backport
Both serialization and yaml test skips
2018-11-14 09:56:45 -05:00
Tim Vernum 231f6c1595
Formal support for "password_hash" in Put User (#35242)
For some time, the PutUser REST API has supported storing a pre-hashed
password for a user. The change adds validation and tests around that
feature so that it can be documented & officially supported.

It also prevents the request from containing both a "password" and a "password_hash".
2018-11-14 16:46:31 +11:00
Zachary Tong c346a0f027
[Rollup] Add `wait_for_completion` option to StopRollupJob API (#34811)
This adds a `wait_for_completion` flag which allows the user to block 
the Stop API until the task has actually moved to a stopped state, 
instead of returning immediately.  If the flag is set, a `timeout` parameter
can be specified to determine how long (at max) to block the API
call.  If unspecified, the timeout is 30s.

If the timeout is exceeded before the job moves to STOPPED, a
timeout exception is thrown.  Note: this is just signifying that the API
call itself timed out.  The job will remain in STOPPING and evenutally
flip over to STOPPED in the background.

If the user asks the API to block, we move over the the generic
threadpool so that we don't hold up a networking thread.
2018-11-13 16:37:17 -05:00
Luca Cavanna 0b7d18d733
HLRC: add support for get license basic/trial status API (#33176)
Relates to #29827
2018-11-13 15:52:59 +01:00
Martijn van Groningen 0487181d0f
[TEST] Force flush to ensure multiple segments.
Relates to #35333
2018-11-13 14:58:17 +01:00
Simon Willnauer 3229dfc4de
Allow efficient can_match phases on frozen indices (#35431)
This change adds a special caching reader that caches all relevant
values for a range query to rewrite correctly in a can_match phase
without actually opening the underlying directory reader. This
allows frozen indices to be filtered with can_match and in-turn
searched with wildcards in a efficient way since it allows us to
exclude shards that won't match based on their date-ranges without
opening their directory readers.

Relates to #34352
Depends on #34357
2018-11-13 14:53:55 +01:00
David Roberts 09965cb370
[ML] Fix find_file_structure NPE with should_trim_fields (#35465)
The NPE would occur if should_trim_field was overridden to
true and any field value was completely blank.  This change
defends against this situation.

Fixes #35462
2018-11-13 08:49:24 +00:00
Tal Levy 16cbbab7b7
[ILM] fix retry so it picks up latest policy and executes async action (#35406)
Before, moving to a failed step would only change the step info
to be that of the failed step. This means two things.

1. Async Steps would never be triggered to execute
2. If there are inherent problems with the action definition that can
be fixed with a policy update, these changes were not being reflected
by the new execution info.

Changes now

1. Async steps are executed after the move to the failed step in cluster state
2. the lifecycle execution info's phase definition is updated from the current
latest policy definition, even though the index isn't moving to a new phase.

Closes #35397.
2018-11-12 11:32:59 -08:00
Nick Knize 2591f66a33
upgrade to lucene-8.0.0-snapshot-6d9c714052 (#35428) 2018-11-12 10:48:27 -06:00
Jason Tedor 3859d21661
Fix the names of CCR stats endpoints in usage API (#35438)
This commit fixes the names of the CCR stats endpoints reported in the
usage API.
2018-11-12 10:27:12 -05:00
Marios Trivyzas 55ff96b43f
SQL: Fix query translation for scripted queries (#35408)
IsNull, IsNotNull and Not where generating wrong queries
as the check to generate ScriptQuery was missing.

Fixes: #35232
2018-11-12 10:42:27 +01:00
Martijn van Groningen ef10461caf
[TEST] Instead of ignoring the ccr downgrade to basic license qa test
avoid the assertions that check the log files, because that does not work on Windows.
The rest of the test is still useful and should work on Windows CI.

Currently on Windows CI this qa module fails because there is just one test and
that test si ignored if OS is Windows.
2018-11-12 10:17:33 +01:00
Christoph Büscher 09cac321e7
Upgrade to Joda 2.10.1 (#35410)
This version contains a bugfix that allows us to reenable one of our muted tests
in DateTimeUnitTests.

Closes #33749
2018-11-12 10:02:41 +01:00
Simon Willnauer 64df803af0
Add a java level freeze/unfreeze API (#35353)
This change adds a high level freeze API that allows to mark an
index as frozen and vice versa. Indices must be closed in order to
become frozen and an open but frozen index must be closed to be
defrosted. This change also adds a index.frozen setting to
mark frozen indices and integrates the frozen engine with the
SearchOperationListener that resets and releases the directory
reader after and before search phases.

Relates to #34352
Depends on #34357
2018-11-10 12:15:31 +01:00
Gordon Brown 67f9e8fa23
Enforce limitations on ILM policy names (#35104)
Enforces restrictions on ILM policy names to ensure we don't accept
policy names the system can't handle, or may reserve for future use.
2018-11-09 10:11:26 -07:00
Martijn van Groningen ae2af20ae5
[CCR] Validate remote cluster license as part of put auto follow pattern api call (#35364)
Validate remote cluster license as part of put auto follow pattern api call
in addition of validation that when auto follow coordinator starts auto
following indices in the leader cluster.

Also added qa module that tests what happens to ccr after downgrading to basic license.
Existing active follow indices should remain to follow,
but the auto follow feature should not pickup new leader indices.
2018-11-09 17:43:43 +01:00
Martijn van Groningen 807ce10f73
[TEST] Increased timeout for verifying ccr monitoring. 2018-11-09 15:40:15 +01:00
Martijn van Groningen fba811fa3a
[TEST] increased the number of index and delete ops to make it less likely that all ops exist as soft delete docs. 2018-11-09 15:31:51 +01:00
Martijn van Groningen 83152b3835
[CCR] Get all auto follow patterns and no auto follow metadata (#35381)
Return empty response when querying all auto follow patterns,
but there is no auto follow metadata.
2018-11-09 14:24:27 +01:00
Marios Trivyzas 36da6e1671
SQL: Introduce IsNull node to simplify expressions (#35206)
Add `IsNull` node in parser to simplify expressions so that `<value> IS NULL` is
no longer translated internally to `NOT(<value> IS NOT NULL)`

Replace `IsNotNullProcessor` with `CheckNullProcessor` to encapsulate both
isNull and isNotNull functionality.

Closes: #34876
Fixes: #35171
2018-11-09 11:32:38 +01:00
Armin Braun 529910a43c
DISCOVERY: Fix RollingUpgradeTests (#35375)
* DISCOVERY: Fix RollingUpgradeTests

* Don't manually manage min master nodes if not necessary
* Remove some dead code
* Allow for manually supplying list of seed nodes
* Closes #35178
2018-11-09 10:47:30 +01:00
Albert Zaharovits 617f91bb0f
DOCS Auditing search queries (#35301)
This documents how to include the search queries in the audit log.

There is a catch, that even if enabling `emit_request_body`, which should
output queries included in request bodies, search queries were not output
because, implicitly, no REST layer audit event type was included.

This folk knowledge is herein imprinted.
2018-11-09 11:38:45 +02:00
Tim Vernum a9641960e1
Refactor realm tests to use single settings (#35362)
Many realm tests were written to use separate setting objects for
"global settings" and "realm settings".
Since #30241 there is no distinction between these settings, so these
tests can be cleaned up to use a single Settings object.
2018-11-09 18:13:24 +11:00
Alpar Torok 66e3ca9724 Revert "Mute test for #35361"
This reverts commit f9154758e6.
2018-11-09 08:04:56 +02:00
Marios Trivyzas c9a84fe361
SQL: Handle null literal for AND and OR in `WHERE` (#35236)
Change `nullable()` logic of AND and OR to false since in the Optimizer
we cannot fold to null as we might be handling and expression in the
SELECT clause.

Introduce folding of null for AND and OR expressions in PruneFilter()
since we now know that we are in HAVING or WHERE clause and we
can fold `null` to `false`

Fixes: #35088

Co-authored-by: Costin Leau <costin.leau@gmail.com>
2018-11-08 23:32:59 +01:00
Tim Brooks 93c2c604e5
Move compression config to ConnectionProfile (#35357)
This is related to #34483. It introduces a namespaced setting for
compression that allows users to configure compression on a per remote
cluster basis. The transport.tcp.compress remains as a fallback
setting. If transport.tcp.compress is set to true, then all requests
and responses are compressed. If it is set to false, only requests to
clusters based on the cluster.remote.cluster_name.transport.compress
setting are compressed. However, after this change regardless of any
local settings, responses will be compressed if the request that is
received was compressed.
2018-11-08 10:37:59 -07:00
Jason Tedor 730ec1ddfb
Add more detailed OS name on Linux (#35352)
Today our OS information returned in node stats only returns a
high-level name of the OS (e.g., "Linux"). Yet, for some uses this is
too high-level and knowing at a finer level of granularity the
underlying OS can be useful. This commit extracts the pretty name on
Linux from /etc/os-release. This pretty name usually includes the Linux
vendor and the Linux vendor version number (e.g., Fedora 28).
2018-11-08 12:16:58 -05:00
Hendrik Muhs 433469d4d0
[Rollup] improve handling of failures on first search (#35269)
Improve error handling in the Indexer if an exception occurs during the very 1st retrieval (query execution)
2018-11-08 16:28:48 +01:00
Martijn van Groningen 07a69a528b
[CCR] Rename leaderClient variables and parameters to remoteClient (#35368) 2018-11-08 16:26:14 +01:00
Albert Zaharovits 025a0c82e5
Remove deprecated audit settings (#35205)
Removes `.prefix` deprecated settings for the logfile
auditing and also documents it in the migrate asciidoc.
2018-11-08 14:06:47 +02:00
David Kyle 9494e046e7
[ML] Prevent notifications on deletion of a non existent job (#35337) 2018-11-08 09:57:07 +00:00
Martijn van Groningen 8a85251da0
[CCR] Auto follow Coordinator fetch cluster state in system context (#35120)
Auto follow Coordinator should fetch the leader cluster state using system context.
2018-11-08 10:48:27 +01:00
Alpar Torok f9154758e6 Mute test for #35361 2018-11-08 11:31:51 +02:00
Tim Vernum 0b24c4f8e7 Test: Mute failing SSL test
Relates: https://github.com/elastic/elasticsearch/issues/35360
2018-11-08 19:37:29 +11:00
Martijn van Groningen 2f2090f562 [CCR] Adjust list of dynamic index settings that should be replicated (#35195)
Adjust list of dynamic index settings that should be replicated
and added a test that verifies whether builtin dynamic index settings
are classified as replicated or non replicated (whitelisted).
2018-11-07 21:59:58 -05:00
Jason Tedor 4f4fc3b8f8
Replicate index settings to followers (#35089)
This commit uses the index settings version so that a follower can
replicate index settings changes as needed from the leader.

Co-authored-by: Martijn van Groningen <martijn.v.groningen@gmail.com>
2018-11-07 21:20:51 -05:00
Tim Vernum 7d05257896
Rename RealmConfig.globalSettings() to settings() (#35330)
There is no longer a concept of non-global "realm settings". All realm
settings should be loaded from the node's settings using standard
Setting classes.

This change renames the "globalSettings" field and method to simply be
"settings".
2018-11-08 12:57:42 +11:00
Tim Vernum 566979cc31
[TEST] Cleanup FileUserPasswdStoreTests (#35329)
The file realm has not supported custom filenames/locations since at
least 5.0, but this test still tried to configure them.

Remove all configuration of file locations, and cleaned up a few other
warnings and deprecations
2018-11-08 12:57:22 +11:00
Jake Landis d7a4cef483
watcher: Fix integration tests to ensure correct start/stop of Watcher (#35271)
Ensure that Watcher is correctly started and stopped between tests for
SmokeTestWatcherWithSecurityIT,
SmokeTestWatcherWithSecurityClientYamlTestSuiteIT,
SmokeTestWatcherTestSuiteIT, WatcherRestIT,
XDocsClientYamlTestSuiteIT, and XPackRestIT

The change here is to throw an `AssertionError` instead of `break;` to
allow the `assertBusy()` to continue to busy wait until the desired
state is reached.

closes #33291, closes #29877, closes #34462, closes #30705, closes #34448
2018-11-07 15:08:03 -06:00
Lee Hinman b1655c468a
Remove ALL shard check in CheckShrinkReadyStep (#35346)
Since it's still possible to shrink an index when replicas are unassigned, we
should not check that all copies are available when performing the shrink, since
we set the allocation requirement for a single node.

Resolves #35321
2018-11-07 13:40:19 -07:00
Lee Hinman c51dcb1f5d
[ILM] Check shard and relocation status in AllocationRoutedStep (#35316)
* [ILM] Check shard and relocation status in AllocationRoutedStep

This is a follow-up from #35161 where we now check for started and relocating
state in `AllocationRoutedStep`.

Resolves #35258
2018-11-07 13:24:21 -07:00
Simon Willnauer 0cc0fd2d15
Add a frozen engine implementation (#34357)
This change adds a `frozen` engine that allows lazily open a directory reader
on a read-only shard. The engine wraps general purpose searchers in a LazyDirectoryReader
that also allows to release and reset the underlying index readers after any and before
secondary search phases.

Relates to #34352
2018-11-07 20:23:35 +01:00
Martijn van Groningen 314b9ca44c
[CCR] Enforce auto follow pattern name restrictions (#35197)
An auto follow pattern:
* cannot start with `_`
* cannot contain a `,`
* can be encoded in UTF-8
* the length of UTF-8 encoded bytes is no longer than 255 bytes
2018-11-07 20:16:26 +01:00
Tal Levy 4167c14d6c
[ILM] rolling upgrade tests (#35328)
Adds basic rolling upgrade tests to check that lifecycles are still recognizable between rolling cluster upgrades and managed indices stay managed.

This is a placeholder for discussing types of checks so they are ready once we backported

This is a re-boot of the previous PR against index-lifecycle that needed to be
reverted due to CI bwc issues. #32828
2018-11-07 10:50:55 -08:00
Benjamin Trent 2117f4f358
[ML] Add Missing data checking class (#35310)
* ML: Adding missing data check class

* reverting bad change

* Adding bucket + missing data object for returns

* reverting unnecessary change

* adding license header

* Make client calls synchronous, akin to DatafeedJob

* Fixing line length

* Renaming things, addressing PR comments
2018-11-07 12:48:15 -06:00
Martijn van Groningen e685cfe8f9
[CCR] Fail with a better error if leader index is red (#35298)
as part of fetching history uuids from leader index.
2018-11-07 13:23:30 +01:00
Alpar Torok 8a85b2eada
Remove build qualifier from server's Version (#35172)
With this change, `Version` no longer carries information about the qualifier,
we still need a way to show the "display version" that does have both
qualifier and snapshot. This is now stored  by the build and red from `META-INF`.
2018-11-07 14:01:05 +02:00
Marios Trivyzas 42dcdd0aa8
SQL: Improve CircuitBreaker logic for SqlParser (#35300)
Grammar's identifiers can be completely skipped from counting depths
as they just add another level to the tree and they are always children
of some other expression which gets counted.

Increased maximum depth from 100 to 200. After testing on production
configuration with -Xss1m, depths of at least 250 can be used, so being
conservative we put the limit lower.

Fixes: #35299
2018-11-07 12:08:32 +01:00
Alexander Reelsen 9f3effd690
Watcher: Ignore system locale/timezone in croneval CLI tool (#33215)
The elasticsearch-croneval CLI tool uses local dates to display when
something gets triggered the next time. This is very confusing.

This commit ensures, that UTC and local timezone times will be written
out.

The output looks like this and contains localized dates for each trigger
date as well as for `now`.

Now is [Tue, 28 Aug 2018 17:23:51 +0000] in UTC, local time is [ᏔᎵᏁ, 28 ᎦᎶ 2018 12:23:51 -0500]
Here are the next 10 times this cron expression will trigger:
1.	Mon, 2 Jan 2040 11:00:00 +0000
	ᏉᏅᎯ, 2 ᎤᏃ 2040 06:00:00 -0500
2.      ...

This also removes an old outstanding TODO to use the jopt parsing to
cast the count to an integer instead of doing it ourselves.
2018-11-07 09:03:35 +01:00
Tim Vernum b4173c8393
Remove deprecated RealmConfig constructor (#35327)
This removes an obsolete constructor that was still being called from
some tests.

Relates: #30241
2018-11-07 18:21:30 +11:00
Martijn van Groningen 2395e16d84
[CCR] Change resume follow api to be a master node action (#35249)
In order to start shard follow tasks, the resume follow api already
needs execute N requests to the elected master node.

The pause follow API is also a master node action, which would make
how both APIs execute more consistent.
2018-11-07 07:38:44 +01:00
Tim Brooks f395b1eace
Open node connections asynchronously (#35144)
This is related to #29023. Additionally at other points we have
discussed a preference for removing the need to unnecessarily block
threads for opening new node connections. This commit lays the groudwork
for this by opening connections asynchronously at the transport level.
We still block, however, this work will make it possible to eventually
remove all blocking on new connections out of the TransportService
and Transport.
2018-11-06 17:58:20 -07:00
Andrei Stefan 432c518266
Upgrade jline to 3.8.2 (#35288) 2018-11-06 18:42:57 +02:00
Andrei Stefan c3e4575f61
SQL: new SQL CLI logo (#35261)
Added new SQL CLI logo representing the Elastic logo "painted" with
"Elastic" words, "SQL" under the logo and version on the last line
2018-11-06 17:51:52 +02:00
Tal Levy a85b4f42ca
[ILM] change remove-policy-from-index http method from DELETE to POST (#35268)
The remove-ilm-from-index API was using the DELETE http method
to signify that something is being removed. Although, metadata
about ILM for the index is being deleted, no entity/resource
is being deleted during this operation. POST is more in line with
what this API is actually doing, it is modifying the metadata for
an index. As part of this change, `remove` is also appended to the path 
to be more explicit about its actions.
2018-11-06 07:46:25 -08:00
Martijn van Groningen a937d7f5f3
[CCR] Forgot missing return statement,
Error was thrown if leader index had no soft deletes enabled, but it then continued creating the follower index.

The test caught this bug, but very rarely due to timing issue.

Build failure instance:

```
1> [2018-11-05T20:29:38,597][INFO ][o.e.x.c.LocalIndexFollowingIT] [testDoNotCreateFollowerIfLeaderDoesNotHaveSoftDeletes] before test
  1> [2018-11-05T20:29:38,599][INFO ][o.e.c.s.ClusterSettings  ] [node_s_0] updating [cluster.remote.local.seeds] from [[]] to [["127.0.0.1:9300"]]
  1> [2018-11-05T20:29:38,599][INFO ][o.e.c.s.ClusterSettings  ] [node_s_0] updating [cluster.remote.local.seeds] from [[]] to [["127.0.0.1:9300"]]
  1> [2018-11-05T20:29:38,609][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s_0] [leader-index] creating index, cause [api], templates [random-soft-deletes-templat
e, one_shard_index_template], shards [2]/[0], mappings []
  1> [2018-11-05T20:29:38,628][INFO ][o.e.c.r.a.AllocationService] [node_s_0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[leader-
index][0]] ...]).
  1> [2018-11-05T20:29:38,660][INFO ][o.e.x.c.a.TransportPutFollowAction] [node_s_0] [follower-index] creating index, cause [ccr_create_and_follow], shards [2]/[0]
  1> [2018-11-05T20:29:38,675][INFO ][o.e.c.s.ClusterSettings  ] [node_s_0] updating [cluster.remote.local.seeds] from [["127.0.0.1:9300"]] to [[]]
  1> [2018-11-05T20:29:38,676][INFO ][o.e.c.s.ClusterSettings  ] [node_s_0] updating [cluster.remote.local.seeds] from [["127.0.0.1:9300"]] to [[]]
  1> [2018-11-05T20:29:38,678][INFO ][o.e.x.c.LocalIndexFollowingIT] [testDoNotCreateFollowerIfLeaderDoesNotHaveSoftDeletes] after test
  1> [2018-11-05T20:29:38,678][INFO ][o.e.x.c.LocalIndexFollowingIT] [testDoNotCreateFollowerIfLeaderDoesNotHaveSoftDeletes] [LocalIndexFollowingIT#testDoNotCreateFoll
owerIfLeaderDoesNotHaveSoftDeletes]: cleaning up after test
  1> [2018-11-05T20:29:38,678][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_s_0] [follower-index/TlWlXp0JSVasju2Kr_hksQ] deleting index
  1> [2018-11-05T20:29:38,678][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_s_0] [leader-index/FQ6EwIWcRAKD8qvOg2eS8g] deleting index
FAILURE 0.23s J0 | LocalIndexFollowingIT.testDoNotCreateFollowerIfLeaderDoesNotHaveSoftDeletes <<< FAILURES!
   > Throwable #1: java.lang.AssertionError:
   > Expected: <false>
   >      but: was <true>
   >    at __randomizedtesting.SeedInfo.seed([7A3C89DA3BCA17DD:65C26CBF6FEF0B39]:0)
   >    at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   >    at org.elasticsearch.xpack.ccr.LocalIndexFollowingIT.testDoNotCreateFollowerIfLeaderDoesNotHaveSoftDeletes(LocalIndexFollowingIT.java:83)
   >    at java.lang.Thread.run(Thread.java:748)
```

Build failure: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.5+intake/46/console
2018-11-06 16:05:35 +01:00
Marios Trivyzas 1be64a7b8b
SQL: Fix null handling for AND and OR in SELECT (#35277)
Override `process()` in `BinaryLogicProcessor` which doesn't immediately
return null if left or right argument is null, which is the behaviour of
`process()` of the parent class `BinaryProcessor`.

Also, add more tests for `AND` and `OR` in SELECT clause with literal.

Fixes: #35240
2018-11-06 15:40:16 +01:00
Tal Levy 2bf843e768 [TEST] Mute ChangePolicyForIndexIT#testChangePolicyForIndex 2018-11-06 06:09:49 -08:00
Costin Leau 75e9a639ee
SQL: Introduce Coalesce function (#35253)
Add Coalesce conditional for replacing null values

Fix #35060
2018-11-06 13:12:24 +02:00
Nick Knize a5e1f4d3a2 Upgrade to lucene-8.0.0-snapshot-31d7dfe6b1 (#35224) 2018-11-06 11:55:23 +01:00
Simon Willnauer 833e0f8ecf
Prevent throttled indices to be searched through wildcards by default (#34354)
Today if a wildcard, date-math expression or alias expands/resolves
to an index that is search-throttled we still search it. This is likely
not the desired behavior since it can unexpectedly slow down searches
significantly.

This change adds a new indices option that allows `search`, `count`
and `msearch` to ignore throttled indices by default. Users can
force expansion to throttled indices by using `ignore_throttled=true`
on the rest request to expand also to throttled indices.

Relates to #34352
2018-11-06 09:45:30 +01:00
Martijn van Groningen 46c238d792
[CCR] Improve error when operations are missing (#35179)
Improve error when operations are missing
2018-11-06 08:42:47 +01:00
Martijn van Groningen cac67f8bcc
[CCR] Add extra validation to unfollow api (#35245)
Validate whether the follow index actually exists and
whether the follow index actually has custom ccr metadata.
2018-11-06 08:00:34 +01:00
Tim Vernum 574ec6686e
Include realm type in Security Realm setting keys (#30241)
This moves all Realm settings to an Affix definition.
However, because different realm types define different settings
(potentially conflicting settings) this requires that the realm type
become part of the setting key.

Thus, we now need to define realm settings as:

    xpack.security.authc.realms:
      file.file1:
        order: 0

      native.native1:
        order: 1

- This is a breaking change to realm config
- This is also a breaking change to custom security realms (SecurityExtension)
2018-11-06 14:56:50 +11:00
Shaunak Kashyap 00e66bab36
[Elasticsearch Monitoring] Collect only display_name (for now) (#35265)
This is a forward port of some of the changes made in #8445, specifically the change mentioned in https://github.com/elastic/elasticsearch/pull/34023#issuecomment-433212636.

Currently, in master, the `cluster_stats` collector collects _all_ cluster metadata and indexes it into `.monitoring-es-*`. However, per the discussion linked to above, we decided to collect _only_ the `display_name` cluster metadata setting for now. This PR makes this change.
2018-11-05 16:27:05 -08:00
Marios Trivyzas 9ac7af6b3e
SQL: Introduce NotEquals node to simplify expressions (#35234)
Add NotEquals node in parser to simplify expressions so that <value1> != <value2> is
no longer translated internally to NOT(<value1> = <value2>)

Closes: #35210
Fixes: #35233
2018-11-05 22:23:07 +01:00
Nik Everett f72ef9b5fd
Build: Pull "skip assemble on qa" to common build (#35214)
Pull all of the logic that we use to skip the `assemble` and
`dependenciesInfo` tasks on `qa` projects into one spot in our root
build file.
2018-11-05 16:16:00 -05:00
Lee Hinman 3ee004c7b9
Add dedicated step for checking shrink allocation status (#35161)
This adds a new step for checking whether an index is allocated correctly based
on the rules added prior to running the shrink step. It also fixes a bug where
for shrink we are not allowed to have the shards relocating for the shrink step.

This also allows us to simplify AllocationRoutedStep and provide better
feedback in the step info for why either the allocation or the shrink checks
have failed.

Resolves #34938
2018-11-05 12:02:59 -07:00
Marios Trivyzas fd82813660
SQL: [tests] Fix select csv spec and enable it (#35239)
Also, replace `||` and `&&` with `OR` and `AND` as the former are not
SQL standard operators.
2018-11-05 18:45:58 +01:00
Jake Landis c2766b65cf
ingest: raise visibility of ingest plugin documentation (#35048)
* move the set security user processor to the main documentation
* link to plugin processors

part of #33188
2018-11-05 11:44:10 -06:00
Gordon Brown 0fbb8a16bc
Skip Rollover step if next index already exists (#35168)
If the Rollover step would fail due to the next index in sequence
already existing, just skip to the next step instead of going to the
Error step.

This prevents spurious `ResourceAlreadyExistsException`s created by
simultaneous RolloverStep executions from causing ILM to error out
unnecessarily.
2018-11-05 09:20:43 -07:00
Lee Hinman 3473217563
Remove Joda usage from ILM (#35220)
This commit removes the Joda time usage from ILM and the HLRC components of ILM.
It also fixes an issue where using the `?human=true` flag could have caused the
parser not to work. These millisecond fields now follow the standard we use
elsewhere in the code, with additional fields added iff the `human` flag is
specified.

This is a breaking change for ILM, but since ILM has not yet been released, no
compatibility shim is needed.
2018-11-05 08:17:15 -07:00
Alexander Reelsen 409050e8de
Refactor: Remove settings from transport action CTOR (#35208)
As settings are not used in the transport action constructor, this
removes the passing of the settings in all the transport actions.
2018-11-05 13:08:18 +01:00
Martijn van Groningen ddda2d419c
[CCR] Change max_read_request_size default (#35247)
This changes the max_read_request_size default from unlimited to 32MB.
2018-11-05 12:51:42 +01:00
David Kyle 85f8458f06 [ML] Add comment describing test behaviour 2018-11-05 11:21:59 +00:00
Nhat Nguyen 54e1231ebd
CCR/TEST: Limit indexing docs in FollowerFailOverIT (#35228)
The suite FollowerFailOverIT is failing because some documents are not
replicated to the follower. Maybe the FollowTask is not working as
expected or the background indexers eat all resources while the follower
cluster is trying to reform after a failover; then CI is not fast enough
to replicate all the indexed docs within 60 seconds (sometimes I see 80k
docs on the leader).

This commit limits the number of documents to be indexed into the leader
index by the background threads so that we can eliminate the latter
case. This change also replaces a docCount assertion with a docIds
assertion so we can have more information if these tests fail again.

Relates #33337
2018-11-03 10:00:54 -04:00
Jay Modi ec1c2c4a76
Remove AbstractComponent usage in x-pack core (#35187)
This change removes the use of AbstractComponent in the security
module. The classes now declare their own loggers.

Relates #34488
2018-11-02 14:30:43 -06:00
Gordon Brown b3da3eae08
[ILM] Fix race condition in test (#35143)
Previously, testRunStateChangePolicyWithNextStep asserted that the
ClusterState before and after running the steps were equal. The test
only passed due to a race condition: The latch would be triggered by the
step execution, but the cluster state update thread would continue
running before committing the change to the cluster state. This allowed
the test to read the old cluster state and pass the equality check about
99.99% of the time.

The test now waits for the new cluster state to be committed before
checking that it is _not_ equal to the old cluster state.
2018-11-02 11:09:48 -06:00
Jason Tedor 1e241190eb
Disable assemble task from ILM qa projects
This commit disables the assemble tasks from all ILM qa projects. These
projects do not have an assemble task to execute.
2018-11-02 11:16:34 -04:00
David Kyle 43cff56aec Mute FollowerFailOverIT.testFollowIndexAndCloseNode
Issue #33337
2018-11-02 15:13:09 +00:00
Alan Woodward e2af849f70
Move ObjectPath and XContentUtils to libs/x-content (#34803)
These are generally useful utility classes that do not need to live in the Watcher code
2018-11-02 15:12:09 +00:00
Jay Modi 6b08d5fc89
Remove use of AbstractComponent in security (#35186)
This change removes the use of AbstractComponent in the security
module. The classes now declare their own loggers.

Relates #34488
2018-11-02 08:33:26 -06:00
Benjamin Trent 7fb95be8bd
Moving xpack.plugin.core ml response to use new base class (#35157)
* Moving xpack.plugin.core ml response to use new base class

* changing back to streamable
2018-11-02 07:46:53 -05:00
Colin Goodheart-Smithe 0aa66b077a
Merge branch 'master' into index-lifecycle 2018-11-02 11:20:54 +00:00
Nhat Nguyen 4875d6fb0b
CCR: Add NodeClosedException to retryable list (#35191)
This change adds NodeClosedException to the retry-able exception list.
2018-11-02 07:01:46 -04:00
Colin Goodheart-Smithe fc6e1f7f3f
Merge branch 'master' into index-lifecycle 2018-11-02 10:56:35 +00:00
Marios Trivyzas 340363b9fa
SQL: Fix null handling for IN => painless script (#35124)
Include `null` literals when generating the painless script for `IN` expressions.
Previously, they were skipped, because of an issue that had been fixed with #35108.

Fixes: #35122
2018-11-02 10:37:54 +01:00
Alpar Torok b432cbdafe Fix upgrade tests involving snapshot
Fixes
```
Could not determine the dependencies of task ':x-pack:qa:full-cluster-restart:with-system-key:v5.6.13#oldClusterTestCluster#node1.copyBwcPlugins'.
```
2018-11-02 09:51:41 +02:00
Martijn van Groningen 19d6cf1b9e
[CCR] Change response classes to not use from Streamable. (#35085)
Only the response classes of get auto follow pattern, the follow and stats APIs
were moved away from Streamable.  The other APIs use `AcknowledgedResponse`
or `BaseTasksResponse` as response class and
moving that class away from Streamable is a bigger change.
2018-11-02 08:02:17 +01:00
Nhat Nguyen dbc7c9259c CCR/TEST: Add debug log to testFailOverOnFollower 2018-11-01 22:28:06 -04:00
Tal Levy b260406ecf
ILM: use node _id attribute when allocating to one node (#35061)
ILM's Shrink Action was using a nodes "_name" attribute to
allocate to prepare for the shrink step. Since the name is
configurable by a user and may use the same name for
multiple nodes on one machine, _id is safer since it is guaranteed
to be unique.

closes #35043.
2018-11-01 14:38:00 -07:00
Tal Levy 6b312a500d uninherit from AbstractComponent in IndexLifecycleService 2018-11-01 10:22:55 -07:00
Tal Levy c3cf7dd305 Merge remote-tracking branch 'upstream/master' into index-lifecycle 2018-11-01 10:13:02 -07:00
Alpar Torok f8378d91a6
Implement VersionCollection in Java (#34050) 2018-11-01 17:43:57 +02:00
Benjamin Trent 2fadec5c3d
ML: Add support for rollup Indexes in Datafeeds (#34654)
* Adding rollup support for datafeeds

* Fixing tests and adjusting formatting

* minor formatting chagne

* fixing some syntax and removing redundancies

* Refactoring and fixing failing test

* Refactoring, adding paranoid null check

* Moving rollup into the aggregation package

* making AggregationToJsonProcessor package private again

* Addressing test failure

* Fixing validations, chunking

* Addressing failing test

* rolling back RollupJobCaps changes

* Adding comment and cleaning up test

* Addressing review comments and test failures

* Moving builder logic into separate methods

* Addressing PR comments, adding test for rollup permissions

* Fixing test failure

* Adding rollup priv check on datafeed put

* Handling missing index when getting caps

* Fixing unused import
2018-11-01 10:02:24 -05:00
Igor Motov 7b13d0592f
Watcher: fix metric stats names (#34951)
* Watcher: fix metric stats names

The current watcher stats metric names doesn't match the current
documentation. This commit fixes the behavior of `queued_watches`
metric, deprecates `pending_watches` metric and adds `current_watches`
to match the documented behavior. It also fixes the documentation, which
introduced `executing_watches` metric that was never added.

Fixes #34865
2018-11-01 10:12:05 -04:00
Andrei Stefan b6659847c2
SQL: handle wildcard expansion on incorrect fields (#35134) 2018-11-01 11:30:05 +02:00
Marios Trivyzas 4f78e2b576
SQL: Improve painless script generated from `IN` (#35055)
Replace standard `||` and `==` painless operators with
new `in` method introduced in `InternalSqlScriptUtils`.
This allows the list of values to become a script variable
which is replaced each time with the list of  values provided
by the user.

Move In to the same package as InPipe & InProcessor

Follow up to #34750

Co-authored-by: Costin Leau <costin.leau@gmail.com>
2018-11-01 10:23:49 +01:00
Nik Everett e28509fbfe
Core: Less settings to AbstractComponent (#35140)
Stop passing `Settings` to `AbstractComponent`'s ctor. This allows us to
stop passing around `Settings` in a *ton* of places. While this change
touches many files, it touches them all in fairly small, mechanical
ways, doing a few things per file:
1. Drop the `super(settings);` line on everything that extends
`AbstractComponent`.
2. Drop the `settings` argument to the ctor if it is no longer used.
3. If the file doesn't use `logger` then drop `extends
AbstractComponent` from it.
4. Clean up all compilation failure caused by the `settings` removal
and drop any now unused `settings` isntances and method arguments.

I've intentionally *not* removed the `settings` argument from a few
files:
1. TransportAction
2. AbstractLifecycleComponent
3. BaseRestHandler

These files don't *need* `settings` either, but this change is large
enough as is.

Relates to #34488
2018-10-31 21:23:20 -04:00
Tal Levy f8e23f6400
update ILM integ test cluster poll interval to 1s (#35113) 2018-10-31 17:09:35 -07:00
Costin Leau f87a53451b
SQL: Register missing processors (#35121)
Add registration (and tests) for missing processors in the serialization
chain.

Fix #35119
2018-10-31 23:07:36 +02:00
David Turner 0072c90e2a
Pre-populate unicast hosts files (#35136)
Today when ESIntegTestCase starts some nodes it writes out the unicast hosts
files each time a node starts its transport service. This does mean that a
number of nodes can start and perform their first pinging round without any
unicast hosts which, if the timing is unlucky and a lot of nodes are all
started at the same time, can lead to a split brain as in #35052.

Prior to #33554 this was unlikely to happen since the MockUncasedHostsProvider
would always have yielded the existing hosts, so the timing would have to have
been implausibly unlucky. Since #33554, however, it's more likely because the
race occurs between the start of the first round of pinging and the writing of
the unicast hosts file. It is realistic that new nodes will be configured with
the existing nodes from startup, so this change reinstates that behaviour.

Closes #35052.
2018-10-31 19:21:24 +00:00
Tal Levy 5f4b23f8c1
cleanup ILM qa structure (#35110)
This commit does a few things

- moves ILM-specifc rest yaml tests into plugin/ilm/qa, and creates special
  :plugin:ilm:qa:rest module to test them
- removes the with-security tests of the yaml tests since they are covered in
  the rest tests now
- moves ChangePolicyforIndexIT into the qa/multi-node project since that test is
  not currently running in main ilm since integTest is disabled
2018-10-31 11:49:29 -07:00
Tal Levy a294a7c6b5 fix IndexLifecycleService setting member
the settings variable was previously created by
the AbstractComponent class inherited by IndexLifecycleService.
this is no more.
2018-10-31 11:17:16 -07:00
Tal Levy d5d28420b6 Merge remote-tracking branch 'upstream/master' into index-lifecycle 2018-10-31 10:47:07 -07:00
Vladimir Dolzhenko 28bb1a3ac0
XPack/HLRC request/response compatibility tests (#34547)
Relates #34451 , #29827
2018-10-31 13:48:35 +01:00
Luca Cavanna 674225aaa1
[TEST] Enforce skip headers when needed (#34735)
The java yaml test runner supports sending request headers, yet not all clients support headers. This commit makes sure that we enforce adding a skip section with feature "headers" whenever headers are used in a do section as part of a test. That decreases the chance for new tests to break client builds due to the missing skip section.

Closes #34650
2018-10-31 13:07:02 +01:00
Dimitris Athanasiou 00dc2ba36f
[ML] Enable reusing field extraction logic when no time field is required (#35100) 2018-10-31 10:55:11 +00:00
Marios Trivyzas bf6f521827
SQL: Fix NPE thrown if HAVING filter evals to null (#35108)
Surround script with `nullSafeFilter` also for `AggFilter` translation.
Fixed query translation tests to properly set `onAggs`.

Fixes: #35107
2018-10-31 10:32:16 +01:00
Tal Levy fdb0db30c4
throw exception when ilm-rollover-step runs on index without rollover alias (#35065)
The ILM Rollover Step can execute on the incorrect index if the rollover alias
exists on another valid index, but not the one the step is executing against. This
is a problem and is now guarded against
2018-10-30 16:42:12 -07:00
Tal Levy 5141084048
rename CRUD api REST path prefix _ilm to _ilm/policy (#35056)
This PR renames the CRUD APIS for ILM

GET _ilm/<policy>, _ilm -> _ilm/policy/<policy>, _ilm/policy
PUT _ilm/<policy> -> _ilm/policy/<policy>
DELETE _ilm/<policy> -> _ilm/policy/<policy>

closes #34929.
2018-10-30 16:19:05 -07:00
Marios Trivyzas 455906d431
SQL: [tests] Remove AbstractBuilderTestCase dep (#35099)
Extend directly from `EsTestCase` to speed up execution.

Fixes: #35098
2018-10-30 22:31:29 +01:00
Nik Everett 086ada4c08
Core: Drop settings member from AbstractComponent (#35083)
Drops the `Settings` member from `AbstractComponent`, moving it from the
base class on to the classes that use it. For the most part this is a
mechanical change that doesn't drop `Settings` accesses. The one
exception to this is naming threads where it switches from an invocation
that passes `Settings` and extracts the node name to one that explicitly
passes the node name.

This change doesn't drop the `Settings` argument from
`AbstractComponent`'s ctor because this change is big enough as is.
We'll do that in a follow up change.
2018-10-30 16:10:38 -04:00
Tal Levy f5323cd86e
rename ILM Phase "minimum_age" to "min_age" (#35058)
closes #34927.
2018-10-30 12:54:02 -07:00
Jay Modi 09a124e8d6
Native roles store uses mget to retrieve roles (#33531)
The native roles store previously would issue a search if attempting to
retrieve more than a single role. This is fine when we are attempting
to retrieve all of the roles to list them in the api, but could cause
issues when attempting to find roles for a user. The search is not
prioritized over other search requests, so heavy aggregations/searches
or overloaded nodes could cause roles to be cached as missing if the
search is rejected.

When attempting to load specific roles, we know the document id for the
role that we are trying to load, which allows us to use the multi-get
api for loading these roles. This change makes use of the multi-get api
when attempting to load more than one role by name. This api is also
constrained by a threadpool but the tasks in the GET threadpool should
be quicker than searches.

See #33205
2018-10-30 13:35:07 -06:00
Gordon Brown 6ecb8ff344
Move to Error step if ClusterState* steps throw (#35069)
Previously, if ClusterStateActionSteps or ClusterStateWaitSteps threw an
exception executing, the exception would only be caught and logged by
the generic ClusterStateUpdateTask machinery and the index would become
stuck on that step.

Now, exceptions thrown in these steps will be caught and the index will
be moved to the Error step.
2018-10-30 13:33:32 -06:00
Nik Everett 7de9eda391
Graph: Move qa into plugin (#34517)
Move `x-pack/qa/smoke-test-graph-with-security` to
`x-pack/plugin/graph/qa/security` which should make it easier to run all
of the tests with a single command. It also lines up the directories
more closely with newer projects like cross cluster replication.
2018-10-30 15:14:33 -04:00
Nik Everett 6857d30527
SQL: Move qa directory into sql directory (#34496)
Moves `x-pack/qa/sql/*` into `x-pack/plugin/sql/qa` to make it simpler
to run all of the sql tests. This lines up with how newer projects like
cross cluster replication are testing themselves.
2018-10-30 14:54:15 -04:00
Costin Leau 6abddd8ca6
SQL: Optimizer rule for folding nullable expressions (#35080)
Add optimization for folding nullable expressions with a NULL 
argument. This is a variant of folding for the NULL case.

Fix #34826
2018-10-30 20:17:20 +02:00
Ryan Ernst 512319cef7
Test: Filter out deprecated joda tzs in tests (#34868)
This commit filters out usage of deprecated tzs by tests. These are
tested separately and should not require checking for warnings on any
test using random timezones.

closes #34188
2018-10-30 11:15:34 -07:00
Zachary Tong f9dd33a0b9
[Rollup] Proactively resolve index patterns in RollupSearch endoint (#34930)
This changes the RollupSearch endpoint to proactively resolve index
patterns.  If the index pattern(s) match more than one rollup index,
an exception is throw as before.  But if the pattern only matches one
rollup index, execution is allowed to continue (unlike before where
it would assume all patterns were for raw data).

This also allows the search endpoint to resolve aliases that point to
a rollup index.

Also tweaks the documentation to make this clear.

Closes #34828
2018-10-30 13:50:50 -04:00
Tal Levy 18c72e86c5 Merge remote-tracking branch 'upstream/master' into index-lifecycle 2018-10-30 08:09:57 -07:00
Tal Levy 6cf7f1205a Revert "add ILM rolling upgrade tests (#32828)"
This reverts commit 5c88356ad6.
2018-10-30 08:06:53 -07:00
Andrei Stefan 7bd113d284
Test for https://github.com/elastic/elasticsearch/issues/30054 bug (#35050)
Added a test for a bug in Elasticsearch (https://github.com/elastic/elasticsearch/issues/32941) that had implications in SQL and activated another test that didn't work in the past
2018-10-30 16:18:52 +02:00
Andrei Stefan 253152172c
SQL: Proper handling of nested fields at the beginning of the columns list (#35068) 2018-10-30 15:52:05 +02:00
Martijn van Groningen 6da2fb7d5b
Change CCR API request classes to use Writeable serialization instead of Streamable (#34911)
Only the follow stats request couldn't be changed to use Writeable serialization,
because that requires changes in `TransportTasksAction` and `BaseTasksRequest` base classes.
2018-10-30 11:03:30 +01:00
Marios Trivyzas e0fef52058
SQL: Fix failing test due to random timezone (#35067)
Use null values for `languages` instead of `birth_date` to avoid
differences with H2 because of timezone randomization.

Fixes: #35042
2018-10-30 10:12:33 +01:00
David Turner f7760ddacb Increase discovery log level in NetworkDisruptionIT
Sometimes the cluster forming here will split-brain when it grows up to 5
nodes. This could be a timing issue or could be something going wrong in
discovery, so this asks for more logs. Relates #35052
2018-10-30 08:45:23 +00:00
Dimitris Athanasiou d85a654ebb
[ML] Refactor doc value format into ExtractedField (#35053)
This commit moves the knowledge of which doc value format
to be used down to the `ExtractedField` instead of being
in the data extractor.
2018-10-29 22:56:53 +00:00
Tal Levy c9e4d26a53 Merge remote-tracking branch 'upstream/master' into index-lifecycle 2018-10-29 14:03:55 -07:00
Pratik Sanglikar f1135ef0ce Core: Replace deprecated Loggers calls with LogManager. (#34691)
Replace deprecated Loggers calls with LogManager.

Relates to #32174
2018-10-29 15:52:30 -04:00
Gordon Brown f6ac0e4bbc
[ILM] Fix Move To Step API causing ILM to hang (#34618)
The Move To Step API now checks to see if the target step is an
AsyncActionStep, and if so, runs it.

Previously, AsyncActionSteps would only be run when they are entered by
executing the previous step, so if an AsyncActionStep was entered via
the Move To Step API, ILM would never touch that index again.
2018-10-29 11:18:12 -06:00
Marios Trivyzas 42b4f23a76
SQL: Add `CAST` and `CONVERT` to `SHOW FUNCTIONS` (#34940)
Fixes: #34939
2018-10-29 16:27:07 +01:00
Nik Everett b093116a1e
Logging: Drop another deprecated Loggers method (#34520)
Drop a method from `Loggers` that we deprecated because it just
delegated to `LogManager`.
2018-10-29 10:05:24 -04:00
Jason Tedor cc9894d78a
Fix name of CCR stats transport action
This class name should include an indication that it is for CCR. This
commit adds that to the name of this class.
2018-10-29 09:50:13 -04:00
Jason Tedor 26f5c509af
Fix CCR API specification (#34963)
This commit fixes two issues with the CCR API specification:
 - remove the CCR stats endpoint, it is not currently implemented
 - fix the documentation links
2018-10-29 09:37:13 -04:00