1620 Commits

Author SHA1 Message Date
Zachary Tong
b01daedfb5 [TEST] Ensure job does not trigger
By setting the cron to 2017, we ensure it won't trigger.  This makes it
easier to test because we know the job will _only_ be in STARTED,
and we can ignore INDEXING states due to transient triggers.

Closes #35779
2018-11-21 18:18:00 -05:00
Tal Levy
927b2425b6
[TEST] fix invalid ExplainLifecycleRequest in tests (#35804) 2018-11-21 14:59:08 -08:00
Marios Trivyzas
d95d885bae
SQL: Implement ISNULL(expr1, expr2) (#35793)
Add ISNULL as an alias of IFNULL as they have the
same behaviour. Add basic test and docs.

Closes: #35781
2018-11-21 23:15:10 +01:00
Gordon Brown
d9c6986b75
Ensure all Steps have Javadoc (#35757)
Adds or corrects Javadoc on subclasses of Step.
2018-11-21 10:06:57 -07:00
Marios Trivyzas
e179bd393d
SQL: Implement IFNULL variant of COALESCE (#35762)
IFNULL is a MySQL variant (also used in other DBs) which
takes only 2 arguments and returns the first one that is not null.

Closes: #35749
2018-11-21 17:07:07 +01:00
Costin Leau
03f003733d
SQL: Perform lazy evaluation of mismatched mappings (#35676)
Move away from performing eager, fail-fast validation of mismatched
 mapping to a lazy evaluation based on the fields actually used in the
 query. This allows queries to run on the parts of the indices that
 "work" which is not just convenient but also a necessity for large
 mappings (like logging) where alignment is hard/impossible to achieve.

Fix #35659
2018-11-21 17:35:15 +02:00
Albert Zaharovits
592a909ba7
manage_token privilege for kibana_system
Creates the manage_token cluster privilege and adds it to the
kibana_system role. This is required if kibana were to use the token
service for its authenticator process.
Because kibana_system already has manage_saml this effectively
only adds the privilege to create tokens.
2018-11-21 17:15:14 +02:00
Costin Leau
f0a3d32602
SQL: Introduce INTERVAL support (#35521)
Introduce INTERVAL as a DataType
Add INTERVAL to the grammar which supports the standard SQL declaration
 (without precision):
 > INTERVAL '1 23:45:01.123456789' DAY TO SECOND
 but also number for single unit intervals:
 > INTERVAL 1 YEAR
 as well as the plurals of the units:
 > INTERVAL 2 YEARS
Interval are internally supported as just another Literal being backed
 by java.time.Period and java.time.Duration
Move JDBC away from JDBCType enum to SQLType interface
Refactor DataType by moving it into server core and adding dedicated (and
 much simpler) JDBC driver type
Improve internal JDBC conversion by normalizing on the DataType
Rename JDBC columnInfo to JdbcColumnInfo to differentiate between it and
 the SQL ColumnInfo

Fix #29990
2018-11-21 15:45:59 +02:00
Christoph Büscher
6638708b56
Remove deprecated QueryStringQueryBuilder#splitOnWhiteSpace (#35763)
This parameter has been deprecated and was ignored since 6.0, so its Java API
methods can be removed.
2018-11-21 10:29:08 +01:00
Luca Cavanna
65d8fdf3da
Clean up PutLicenseResponse (#35689)
This commit removes the parsing code from the PutLicenseResponse server variant, and the toXContent portion from the corresponding client variant.

Relates to #35547
2018-11-21 10:21:45 +01:00
Luca Cavanna
778550a97d
Clean up StartBasicResponse (#35688)
This commit removes the parsing code from the PostStartBasicResponse server variant. It also makes the server response implement StatusToXContent which allows us to save a couple of lines of code in the corredponding REST action.

Relates to #35547
2018-11-21 10:21:10 +01:00
Tim Vernum
30c5422561
Move XContent generation to HasPrivilegesResponse (#35616)
The RestHasPrivilegesAction previously handled its own XContent
generation. This change moves that into HasPrivilegesResponse and
makes the response implement ToXContent.

This allows HasPrivilegesResponseTests to be used to test
compatibility between HLRC and X-Pack internals.

A serialization bug (cluster privs) was also fixed here.
2018-11-21 14:33:10 +11:00
Marios Trivyzas
b1818dbdce
SQL: [docs] Add documentation for COALESCE (#35740)
Follows: #35253
2018-11-21 01:43:05 +01:00
Armin Braun
89cf4a7397
NETWORKING: Fix IpFiltering Test (#35730)
* The port assigned to all loopback interfaces doesn't necessarily have to be the same  for ipv4 and ipv6
=> use actual address from profile instead of just port + loopback in test
* Closes #35584
2018-11-20 17:50:08 +01:00
Christoph Büscher
e91f404d16
Remove remains of 'auto_generate_phrase_queries' (#35735)
This parameter in the `query_string` query was deprecated in 6.0 and ignored
since then. Its API methods and remaining uses can be removed in the upcoming
major version.

Relates to #35734
2018-11-20 16:07:17 +01:00
Alpar Torok
8659af68e0
Auto skip license headers on no source (#35640)
* Unmute BuildExamplePluginsIT

* Skip licenseHeaders when there are no sources
2018-11-20 13:02:33 +02:00
Simon Willnauer
29ef442841
Add a _freeze / _unfreeze API (#35592)
This commit adds a rest endpoint for freezing and unfreezing an index.
Among other cleanups mainly fixing an issue accessing package private APIs
from a plugin that got caught by integration tests this change also adds
documentation for frozen indices.
Note: frozen indices are marked as `beta` and available as a basic feature.

Relates to #34352
2018-11-20 08:03:24 +01:00
Tim Brooks
a989b675b5
Remove NPE from IndexFollowingIT (#35717)
Currently there is a common NPE in the IndexFollowingIT that does not
indicate the test failing. This is when a cluster state listener is
called and certain index metadata is not yet available.

This commit checks that the metadata is not null before performing the
logic that depends on the metadata.
2018-11-19 20:38:49 -07:00
Tim Vernum
3bba9a56de
TEST: Enable skipped test (#35682)
This test was skipped due to infr complications during the work
to open x-pack, but wasn't enabled again when the issue was resolved.

Relates: #29890
2018-11-20 12:51:01 +11:00
Benjamin Trent
80bcf7c704
[ML]: adjusting post backport of (#35645) (#35696) 2018-11-19 12:49:07 -06:00
Gordon Brown
b2057138a7
Remove AbstractComponent from AbstractLifecycleComponent (#35560)
AbstractLifecycleComponent now no longer extends AbstractComponent. In
order to accomplish this, many, many classes now instantiate their own
logger.
2018-11-19 09:51:32 -07:00
Benjamin Trent
6d4a3f8fce
Removes two unused AnalysisConfig options (#35645)
* ML: Removing result_finalization_window && overlapping_buckets

* Reverting bad method deletions

* Setting to current before backport to try and get a green build

* fixing testBuildAutodetectCommand test

* disabling bwc tests for backport
2018-11-19 08:29:53 -06:00
Benjamin Trent
bc7dea4480
ML: changing automatic check_window calculation (#35643)
* ML: changing automatic check_window calculation

* adding docs on how we calculate the default
2018-11-19 08:03:34 -06:00
David Roberts
f70b7dc158 Mute DataTypeConversionTests.testConversionToDate
Due to https://github.com/elastic/elasticsearch/issues/35683
2018-11-19 12:55:33 +00:00
David Roberts
2ee25c858d Mute CronEvalToolTests.testEnsureDateIsShownInRootLocale
Due to https://github.com/elastic/elasticsearch/issues/35687
2018-11-19 09:35:29 +00:00
Costin Leau
4119409b6d
SQL: Improve validation of unsupported fields (#35675)
Fix bug in Analyzer that caused it to report unsupported fields only
 when declared in projections. The rule has been extended to all field
 declarations.

Fix #35673
2018-11-18 23:35:18 +02:00
Costin Leau
f8e333b117
SQL: Move internals from Joda to java.time (#35649)
Remove/Limit usage of Joda through-out the processors and functions
Use ZonedDateTime wherever possible instead of long/tzId

Fix #35633
2018-11-17 15:30:27 +02:00
Gordon Brown
cce9648f9d
Align RolloverStep's name with other step names (#35655)
RolloverStep previously had a name of "attempt_rollover", which was
inconsistent with all other step names due it its use of an underscore
instead of a dash.
2018-11-16 17:42:48 -07:00
Ryan Ernst
3c03975ce3
Watcher: Use joda method to get local TZ (#35608)
This commit switches from using java util's default timezone method to
using joda. The former can cause problems when the string representation
of the timezone is unknown to joda.

closes #35518
2018-11-16 15:46:27 -08:00
Arthur Gavlyukovskiy
022726011c Remove use of AbstractComponent in server (#35444)
Removed extending of AbstractComponent and changed logger usage to
explicit declaration. Abstract classes still have logger
declaration using this.getClass() in order to show implementation class
name in its logs.

See #34488
2018-11-16 16:10:32 -05:00
Benjamin Trent
e360c600d0
ML: Adjusing BWC version post backport to 6.6 (#35605) 2018-11-16 06:50:12 -06:00
Gordon Brown
3883e9bf4c
Split RolloverStep into Wait and Action steps (#35524)
RolloverAction will now periodically check the rollover conditions using
the Rollover API with the dry_run option as an AsyncWaitStep, then run
the rollover itself by calling the Rollover API with no conditions,
which will always roll over, as an AsyncActionStep. This will resolve
race condition issues in policies using RolloverAction.
2018-11-15 17:11:31 -07:00
Benjamin Trent
f7ada9b29b
Add delayed datacheck to the datafeed job runner (#35387)
* ML: Adding missing datacheck to datafeedjob

* Adding client side and docs

* Making adjustments to validations

* Making values default to on, having more sensible limits

* Intermittent commit, still need to figure out interval

* Adjusting delayed data check interval

* updating docs

* Making parameter Boolean, so it is nullable

* bumping bwc to 7 before backport

* changing to version current

* moving delayed data check config its own object

* Separation of duties for delayed data detection

* fixing checkstyles

* fixing checkstyles

* Adjusting default behavior so that null windows are allowed

* Mentioning the default value

* Fixing comments, syncing up validations
2018-11-15 13:32:45 -06:00
Lee Hinman
90b38d9b3d
[ILM] Avoid NullPointerException in CopyExecutionStateStep (#35568)
In the event that the target index does not exist when `CopyExecutionStateStep`
executes, this avoids a `NullPointerException` and provides a more helpful error
to the ILM user.

Resolves #35567
2018-11-15 09:29:26 -07:00
Tim Vernum
1e60b282fc Grant .tasks access to kibana_system role (#35573)
Kibana now uses the tasks API to manage automatic reindexing of the
.kibana index during upgrades.

The implementation of the tasks API requires that
1. the user executing the task can create & write to the ".tasks" index
2. the user checking on the status of the task can read (Get) the
   relevant document from the ".tasks" index
2018-11-15 08:14:58 -07:00
Andrei Stefan
eaf010cd73
SQL: clear the cursor if nested inner hits are enough to fulfill the query required limits (#35398) 2018-11-15 14:33:47 +02:00
Luca Cavanna
8e2c84ad8b
Clean up XPackInfoResponse class and related tests (#35547)
Response classes in Elasticsearch (and xpack) only need to implement ToXContent, which is needed to print their output put in the REST layer and return the response in json (or others) format. On the other hand, response classes that are added to the high-level REST client, need to do the opposite: parse xcontent and create a new object based on that.

This commit removes the parsing code from the XPackInfoResponse server variant, and the toXContent portion from the corresponding client variant. It also removes a client specific test class that looks redundant now that we have a single test class for both classes.
2018-11-15 09:59:27 +01:00
Tanguy Leroux
c9b4ef0dfd
Use RunOnce when appropriate (#35553)
This pull request replaces some blocks of code that must be run once 
and that are currently based on AtomicBoolean by the convient RunOnce 
class added in #35489.
2018-11-15 09:24:40 +01:00
Jay Modi
faa9523d19
Remove deprecated constructor from failure handler (#35565)
The DefaultAuthenticationFailureHandler has a deprecated constructor
that was present to prevent a breaking change to custom realm plugin
authors in 6.x. This commit removes the constructor and its uses.
2018-11-14 17:04:33 -07:00
Tal Levy
5d0544d732
update move-to-step spec with proper docs and extra warning to clients (#35477) 2018-11-14 14:08:31 -08:00
Lee Hinman
8ea999e489
Include stack trace with ILM error in explain output (#35512)
This changes the stacktrace to be included with the ILM explain error when the
index is an on ERROR step.

Before:

```json
{
  "indices" : {
    "foo" : {
      "index" : "foo",
      "managed" : true,
      "policy" : "bad",
      "lifecycle_date_millis" : 1542131670601,
      "phase" : "warm",
      "phase_time_millis" : 1542131676335,
      "action" : "shrink",
      "action_time_millis" : 1542131676335,
      "step" : "ERROR",
      "step_time_millis" : 1542131676451,
      "failed_step" : "shrink",
      "step_info" : {
        "type" : "illegal_argument_exception",
        "reason" : "the number of target shards [13] must be less that the number of source shards [2]"
      },
      "phase_execution" : {
        "policy" : "bad",
        "phase_definition" : {
          "min_age" : "5s",
          "actions" : {
            "shrink" : {
              "number_of_shards" : 13
            }
          }
        },
        "version" : 1,
        "modified_date_in_millis" : 1542131669839
      }
    }
  }
}
```

After

```
{
  "indices" : {
    "foo" : {
      "index" : "foo",
      "managed" : true,
      "policy" : "bad",
      "lifecycle_date_millis" : 1542131670601,
      "phase" : "warm",
      "phase_time_millis" : 1542131676335,
      "action" : "shrink",
      "action_time_millis" : 1542131676335,
      "step" : "ERROR",
      "step_time_millis" : 1542131676451,
      "failed_step" : "shrink",
      "step_info" : {
        "type" : "illegal_argument_exception",
        "reason" : "the number of target shards [13] must be less that the number of source shards [2]",
        "stack_trace" : "java.lang.IllegalArgumentException: the number of target shards [13] must be less that the number of source shards [2]\n\tat org.elasticsearch.cluster.metadata.IndexMetaData.selectShrinkShards(IndexMetaData.java:1509)\n\tat org.elasticsearch.action.admin.indices.shrink.TransportResizeAction.prepareCreateIndexRequest(TransportResizeAction.java:146)\n\tat org.elasticsearch.action.admin.indices.shrink.TransportResizeAction$1.onResponse(TransportResizeAction.java:104)\n\tat org.elasticsearch.action.admin.indices.shrink.TransportResizeAction$1.onResponse(TransportResizeAction.java:101)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:64)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:60)\n\tat org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.onCompletion(TransportBroadcastByNodeAction.java:383)\n\tat org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.onNodeResponse(TransportBroadcastByNodeAction.java:352)\n\tat org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction$1.handleResponse(TransportBroadcastByNodeAction.java:324)\n\tat org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction$1.handleResponse(TransportBroadcastByNodeAction.java:314)\n\tat org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1117)\n\tat org.elasticsearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:1198)\n\tat org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1178)\n\tat org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:54)\n\tat org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:417)\n\tat org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:391)\n\tat org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:251)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:309)\n\tat org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63)\n\tat org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:714)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:726)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:834)\n"
      },
      "phase_execution" : {
        "policy" : "bad",
        "phase_definition" : {
          "min_age" : "5s",
          "actions" : {
            "shrink" : {
              "number_of_shards" : 13
            }
          }
        },
        "version" : 1,
        "modified_date_in_millis" : 1542131669839
      }
    }
  }
}
```

Resolves #35498
2018-11-14 14:40:05 -07:00
Zachary Tong
c7ea0343d2 [Rollup] Update versions after backport
Both serialization and yaml test skips
2018-11-14 09:56:45 -05:00
Tim Vernum
231f6c1595
Formal support for "password_hash" in Put User (#35242)
For some time, the PutUser REST API has supported storing a pre-hashed
password for a user. The change adds validation and tests around that
feature so that it can be documented & officially supported.

It also prevents the request from containing both a "password" and a "password_hash".
2018-11-14 16:46:31 +11:00
Zachary Tong
c346a0f027
[Rollup] Add wait_for_completion option to StopRollupJob API (#34811)
This adds a `wait_for_completion` flag which allows the user to block 
the Stop API until the task has actually moved to a stopped state, 
instead of returning immediately.  If the flag is set, a `timeout` parameter
can be specified to determine how long (at max) to block the API
call.  If unspecified, the timeout is 30s.

If the timeout is exceeded before the job moves to STOPPED, a
timeout exception is thrown.  Note: this is just signifying that the API
call itself timed out.  The job will remain in STOPPING and evenutally
flip over to STOPPED in the background.

If the user asks the API to block, we move over the the generic
threadpool so that we don't hold up a networking thread.
2018-11-13 16:37:17 -05:00
Luca Cavanna
0b7d18d733
HLRC: add support for get license basic/trial status API (#33176)
Relates to #29827
2018-11-13 15:52:59 +01:00
Martijn van Groningen
0487181d0f
[TEST] Force flush to ensure multiple segments.
Relates to #35333
2018-11-13 14:58:17 +01:00
Simon Willnauer
3229dfc4de
Allow efficient can_match phases on frozen indices (#35431)
This change adds a special caching reader that caches all relevant
values for a range query to rewrite correctly in a can_match phase
without actually opening the underlying directory reader. This
allows frozen indices to be filtered with can_match and in-turn
searched with wildcards in a efficient way since it allows us to
exclude shards that won't match based on their date-ranges without
opening their directory readers.

Relates to #34352
Depends on #34357
2018-11-13 14:53:55 +01:00
David Roberts
09965cb370
[ML] Fix find_file_structure NPE with should_trim_fields (#35465)
The NPE would occur if should_trim_field was overridden to
true and any field value was completely blank.  This change
defends against this situation.

Fixes #35462
2018-11-13 08:49:24 +00:00
Tal Levy
16cbbab7b7
[ILM] fix retry so it picks up latest policy and executes async action (#35406)
Before, moving to a failed step would only change the step info
to be that of the failed step. This means two things.

1. Async Steps would never be triggered to execute
2. If there are inherent problems with the action definition that can
be fixed with a policy update, these changes were not being reflected
by the new execution info.

Changes now

1. Async steps are executed after the move to the failed step in cluster state
2. the lifecycle execution info's phase definition is updated from the current
latest policy definition, even though the index isn't moving to a new phase.

Closes #35397.
2018-11-12 11:32:59 -08:00
Nick Knize
2591f66a33
upgrade to lucene-8.0.0-snapshot-6d9c714052 (#35428) 2018-11-12 10:48:27 -06:00