Commit Graph

41664 Commits

Author SHA1 Message Date
Costin Leau 7570d69254
SQL: Introduce support for IP fields (#34758)
IP fields are recognized and can be used through-out the query

Close #32499
2018-10-24 22:03:11 +03:00
Marios Trivyzas 98cd7ca861
SQL: Fix queries with filter resulting in NO_MATCH (#34812)
Previously, `Mapper` was returning an incorrect plan which resulted in an
```
SQLFeatureNotSupportedException: Found 1 problem(s)
line 1:8: Unexecutable item
```

Queries with a `WHERE` and/or `HAVING` clause which results in NO_MATCH
are now handled correctly and return 0 rows.

Fixes: #34613

Co-authored-by: Costin Leau <costin.leau@gmail.com>
2018-10-24 20:36:30 +02:00
Mayya Sharipova bf4d90a5dc
HLRC API for _termvectors (#33447)
* HLRC API for _termvectors

relates to #27205
2018-10-24 14:27:22 -04:00
Costin Leau f19565c3e0
SQL: Verifier allows aliases aggregates for sorting (#34773)
Improve Verifier rule that prevented grouping with aliases inside aggregates
to not be accepted for ordering.

Close #34607
2018-10-24 19:31:26 +03:00
markharwood f7a6fb288f
Fix compilation error peculiar to eclipse2018-09 (#34798)
Fix compilation error peculiar to eclipse2018-09.
Added generics to make eclipse compiler happy
2018-10-24 16:58:31 +01:00
Nhat Nguyen d73768f812
CCR: Do not follow if leader does not have soft-deletes (#34767)
We should not create a follower index and abort a follow request if the
leader does not have soft-deletes. Moreover, we also should not
auto-follow an index if it does not have soft-deletes.
2018-10-24 11:19:39 -04:00
Armin Braun 0a85997cb0
TESTS: Use File Based Discovery in REST Tests (#34560)
* For `6.5+` use file based discovery in REST tests
* Relates #33675
2018-10-24 17:18:52 +02:00
Nik Everett e0d7808148 REST: No strict warning testing for head body test
Our tests that HEAD requests don't have a body was hitting a warning.
For now we'll run that test without "strict" warnings mode enabled.
2018-10-24 11:17:38 -04:00
Alpar Torok 4370b44ef0
Rename testcluster extension to align with plugin name (#34751) 2018-10-24 18:15:25 +03:00
Shaunak Kashyap 0f1a5ecf61
Adding stack_monitoring_agent role (#34369)
* Adding stack_monitoring_agent role

* Fixing checkstyle issues

* Adding tests for new role

* Tighten up privileges around index templates

* s/stack_monitoring_user/remote_monitoring_collector/ + remote_monitoring_user

* Fixing checkstyle violation

* Fix test

* Removing unused field

* Adding missed code

* Fixing data type

* Update Integration Test for new builtin user
2018-10-24 07:20:46 -07:00
Alpar Torok 795d57b4f9
Auto configure all test tasks (#34666)
With this change, we apply the common test config automatically to all
newly created tasks instead of opting in specifically.

For plugin authors using the plugin externally this means that the
configuration will be applied to their RandomizedTestingTasks as well.

The purpose of the task is to simplify setup and make it easier to
change projects that use the `test` task but actually run integration
tests to use a task called `integTest` for clarity, but also because
we may want to configure and run them differently.
E.x. using different levels of concurrency.
2018-10-24 16:05:50 +03:00
Marios Trivyzas 4c73854da7
SQL: Implement null handling for `IN(v1, v2, ...)` (#34750)
Implemented null handling for both the value tested but also for
values inside the list of values tested against.

The null handling is implemented for local processors, painless scripts
and Lucene Terms queries making it available for `IN` expressions occuring
in `SELECT`, `WHERE` and `HAVING` clauses.

Closes: #34582
2018-10-24 14:42:40 +02:00
lipsill d5ad3de42e [test] Introduce strict deprecation mode for REST tests (#34338)
#33708 introduced a strict deprecation mode that makes a REST request
fail if there is a warning header in the response returned by
Elasticsearch (usually a deprecation message signaling that a feature
or a field has been deprecated).

This change adds the strict deprecation mode into the REST integration
tests, and makes the tests fail if a deprecated feature is used. Also
any test using a deprecated feature has been modified to pass the build.

The YAML integration tests already analyzed HTTP warnings so they do
not use this mode, keeping their "expected vs actual" behavior.
2018-10-24 08:21:24 -04:00
Martijn van Groningen 76240e6bbe
[CCR] Renamed leader_cluster to remote_cluster (#34776)
and also some occurrences of clusterAlias to remoteCluster.

Closes #34682
2018-10-24 13:39:36 +02:00
Boaz Leskes be907516ad
Change ShardFollowTask defaults (#34793)
Per #31717 this commit changes the defaults to the following:

Batch size of 5120 ops.
Maximum of 12 concurrent read requests.
Maximum of 9 concurrent write requests.

This is not necessarily our final values but it's good to have these as defaults for the purposes of initial testing.
2018-10-24 13:32:48 +02:00
hanbing0715 fe54f73434 Persist generated public keys unencrypted (#34626)
The changes introduced in cca1a2a mean that we should
not encrypt the public keys that might be generated by
the key-pair-generator when storing the file, as the code 
that would consume them assumes that they are not encrypted
2018-10-24 13:55:21 +03:00
Martijn van Groningen 18e004c3ea
[CCR] Removed unused static parse field. 2018-10-24 09:27:08 +02:00
Martijn van Groningen 18007a29b2
[CCR] Made leader cluster required in shard follow task.
Left over from #34580
2018-10-24 08:38:25 +02:00
Martijn van Groningen abf8cb6706
[CCR] Cleanup pause follow action (#34183)
* Change the `TransportPauseFollowAction` to extend from `TransportMasterNodeAction`
  instead of `HandledAction`, this removes a sync cluster state api call.
* Introduced `ResponseHandler` that removes duplicated code in `TransportPauseFollowAction` and
  `TransportResumeFollowAction`.
* Changed `PauseFollowAction.Request` to not use `readFrom()`.
2018-10-24 08:12:39 +02:00
Nhat Nguyen 52266d8b11
TEST: Clone replicas list when compute replication targets (#34728)
In #34407, we supposed to clone the list of replicas of ReplicationGroup
when computing replication targets, but somehow we missed it. If we
don't clone the list, a WriteReplicationAction may use an old
ReplicationTargets which consists replicas which are removed from the
current list of replicas

Relates #34407
Closes #33457
2018-10-23 21:08:34 -04:00
Ryan Ernst 8da1c9626a
Scripting: Add back params._source access in scripted metric aggs (#34777)
Access to special variables _source and _fields were accidentally
removed in recent refactorings. This commit adds them back, along with a
test.

closes #33884
2018-10-23 18:07:53 -07:00
Ryan Ernst 596b5cf108
Test: Fix last reference to SearchScript (#34731)
This was accidentally left over when converting to FieldScript.

closes #34683
2018-10-23 17:26:17 -07:00
Gordon Brown da20dfd81c
Add cluster-wide shard limit warnings (#34021)
In a future major version, we will be introducing a soft limit on the
number of shards in a cluster based on the number of nodes in the
cluster. This limit will be configurable, and checked on operations
which create or open shards and issue a warning if the operation would
take the cluster over the limit.

There is an option to enable strict enforcement of the limit, which
turns the warnings into errors.  In a future release, the option will be
removed and strict enforcement will be the default (and only) behavior.
2018-10-23 16:35:10 -06:00
Julie Tibshirani c5a0739381 Mute SettingsBasedHostProviderIT to avoid future test flakes. 2018-10-23 15:26:39 -07:00
Julie Tibshirani 7580f59c9f Mute FullClusterRestartIT#testSqlFailsOnIndexWithTwoTypes while we await a fix. 2018-10-23 15:14:08 -07:00
Gordon Brown 17adfeb20f
HLRC: Standardize access in *RequestConverters (#34768)
With the move to separate RequestConverters classes for each client,
some of the access restrictions on the new classes are more open than
the prior RequestConverters classes. This standardizes the
*RequestConverters classes as package-private, final, and with a private
constructor so that no instances of the can be inadvertently created.
2018-10-23 15:59:11 -06:00
Martijn van Groningen 0efba0675e
[CCR] Add qa test library (#34611)
* Introduced test qa lib that all CCR qa modules depend on to avoid
test code duplication.
2018-10-23 23:24:32 +02:00
Julie Tibshirani 1f42024243 Mute NodeSubclassTests#testReplaceChildren while we await a fix. 2018-10-23 14:00:53 -07:00
Armin Braun f79bdec58a INGEST: Document Pipeline Processor (#33418)
* Added documentation for Pipeline Processor
* Relates #33188
2018-10-23 15:36:57 -05:00
Zachary Tong 299d044bfc
Collapse pipeline aggs into single package (#34658)
- Restrict visibility of Aggregators and Factories
- Move PipelineAggregatorBuilders up a level so it is consistent with
AggregatorBuilders
- Checkstyle line length fixes for a few classes
- Minor odds/ends (swapping to method references, formatting, etc)
2018-10-23 16:01:01 -04:00
Jake Landis 067d135bfd
ingest: document index.default_pipeline (#34500) 2018-10-23 13:49:25 -05:00
Julie Tibshirani 5a4866f67d Mute CharArraysTests#testCharsBeginsWith while we await a fix. 2018-10-23 11:37:54 -07:00
Jake Landis a8e1ee34ca
ingest: document fields that support templating (#34536)
This change also updates many of the examples to use ecs as the example.
Some additional minor improvements are also included.

Part of #33188
2018-10-23 13:28:44 -05:00
Nhat Nguyen e242fd2e42
CCR: Add TransportService closed to retryable errors (#34722)
Both testFollowIndexAndCloseNode and testFailOverOnFollower failed
because they responded to the FollowTask a TransportService closed
exception which is currently considered as a fatal error. This behavior
is not desirable since a closing node can throw that exception, and we
should retry in that case.

This change adds TransportService closed error to the list of retryable
errors.

Closes #34694
2018-10-23 14:23:29 -04:00
Julie Tibshirani 90fd15bb56 Mute RollupIndexerIndexingTests#testRandomizedDateHisto as we await a fix. 2018-10-23 11:14:43 -07:00
Martijn van Groningen ed817fb265
[CCR] Move leader_index and leader_cluster parameters from resume follow to put follow api (#34638)
As part of this change the leader index name and leader cluster name are
stored in the CCR metadata in the follow index. The resume follow api
will read that when a resume follow request is executed.
2018-10-23 19:37:45 +02:00
Jake Landis c447fc258a
ingest: documentation for the drop processor (#34570) 2018-10-23 12:30:23 -05:00
Emily S eaa05fe762
[TEST] Minor updates to rest api spec tests (#34551) 2018-10-23 19:09:09 +02:00
Jake Landis 89dc07bdd9
ingest: better support for conditionals with simulate?verbose (#34155)
This commit introduces two corrections to the way simulate?verbose
handles conditionals on processors.

1) Prior to this change when executing simulate?verbose for
processors with conditionals that evaluate to false, that processor
would still be displayed in the result set. What was displayed was
correct, such that no changes to the document occurred. However, if the
conditional evaluates to false, the processor should not even be
displayed.

2) Prior to this change when executing simulate?verbose for
pipeline processors with conditionals, the individual steps would no
longer be displayed. Commit e37e5df addressed the issue, but
failed account for a conditional on the pipeline processor. Since
a pipeline processor can introduce cycles and is effectively a
single processor that encapsulates multiple other processors that
are potentially guarded by a single conditional, special handling is
needed to for pipeline and conditional pipeline processors.
2018-10-23 11:33:48 -05:00
Zachary Tong 4dbf498721
[Rollup] Job deletion should be invoked on the allocated task (#34574)
We should delete a job by directly talking to the allocated 
task and telling it to shutdown. Today we shut down a job 
via the persistent task framework. This is not ideal because, 
while the job has been removed from the persistent task 
CS, the allocated task continues to live until it gets the 
shutdown message.

This means a user can delete a job, immediately delete 
the rollup index, and then see new documents appear in
 the just-deleted index. This happens because the indexer
 in the allocated task is still running and indexes a few 
more documents before getting the shutdown command.

In this PR, the transport action is changed to a TransportTasksAction, 
and we invoke onCancelled() directly on the matching job. 
The race condition still exists after this PR (albeit less likely), 
but this was a precursor to fixing the issue and a self-contained
chunk of code. A second PR will followup to fix the race itself.
2018-10-23 12:23:22 -04:00
Albert Zaharovits 46b49b01c4
[DOCS] .Security index is never auto created (#34589) 2018-10-23 18:51:59 +03:00
Nhat Nguyen 5923ea536e
CCR: Requires soft-deletes on the follower (#34725)
Since #34412 and #34474, a follower must have soft-deletes enabled 
to work correctly. This change requires soft-deletes on the follower.

Relates #34412
Relates #34474
2018-10-23 11:51:17 -04:00
Jake Landis cadd6731e7
re-enable bwc tests (#34743) 2018-10-23 10:50:49 -05:00
Albert Zaharovits 11881e7b50
Empty GetAliases authorization fix (#34444)
This fixes a bug about aliases authorization.
That is, a user might see aliases which he is not authorized to see.
This manifests when the user is not authorized to see any aliases
and the `GetAlias` request is empty which normally is a marking
that all aliases are requested. In this case, no aliases should be
returned, but due to this bug, all aliases will have been returned.
2018-10-23 18:50:20 +03:00
Armin Braun f0f732908e
INGEST: Document Processor Conditional (#33388)
* INGEST: Document Processor Conditional

Relates #33188
2018-10-23 17:37:30 +02:00
Martijn van Groningen e6d87cc09f
[CCR] Add total fetch time leader stat (#34577)
Add total fetch time leader stat, that
keeps track how much time was spent on fetches
from the leader cluster perspective.
2018-10-23 16:41:06 +02:00
Costin Leau ca6808e55d
SQL: Support pattern against compatible indices (#34718)
Extend querying support on multiple indices from being strictly
identical to being just compatible.
Use FieldCapabilities API (extended through #33803) for mapping merging.

Close #31837 #31611
2018-10-23 17:07:51 +03:00
Martijn van Groningen 36baf3823d
[CCR] Auto follow pattern APIs adjustments (#34518)
* Changed the resource id of auto follow patterns to be a user defined name
instead of being the leader cluster alias name.
* Fail when an unfollowed leader index matches with two or more auto follow patterns.
2018-10-23 15:48:51 +02:00
Christoph Büscher 583f2852f0
[Test] Remove dead code from ExceptionSerializationTests (#34713)
The `ignore` set contains entries of type Class<?>, but the check is performed
on Path objects. This always returns false so is useless currently. Looking at
the first commit of this test that already shows this behaviour this never
excluded anything, so it can be removed.
2018-10-23 15:44:47 +02:00
Przemyslaw Gomulka f91cc13c18
A small typo in migration-assistance doc (#34704)
typo - missing word "be"
2018-10-23 15:28:25 +02:00