1901 Commits

Author SHA1 Message Date
Marios Trivyzas
731e48beff
SQL: [Test] Fix and enable test with randomness (#34850)
Increase minimum number of elements for List<> ctor arguments
for specific classes that validate the size of the list.

Fixes: #34754
2018-10-29 12:02:28 +01:00
Marios Trivyzas
c9ae1928e0
SQL: Fix function args verification and error msgs (#34926)
Extract data type verification for function arguments to a single place
so that NULL type can be treated as RESOLVED for all functions. Moreover
this enables to have consistent verification error messages for all functions.

Fixes: #34752
Fixes: #33469
2018-10-29 11:32:17 +01:00
David Roberts
c455be7bc2
[ML] Rename the json file structure to ndjson (#34901)
The file structure finder endpoint can find the NDJSON
(newline-delimited JSON) file format, but called it
`json`.  This change renames the `format` for this file
structure to `ndjson`, which is more precise and will
hopefully avoid confusion.
2018-10-29 10:06:12 +01:00
Martijn van Groningen
1801518527
[CCR] Refactor stats APIs (#34912)
* Changed the auto follow stats to also include follow stats.
* Renamed the auto follow stats api to stats api and changed its url path
  from `/_ccr/auto_follow/stats` `/_ccr/stats`.
* Removed `/_ccr/stats` url path for the follow stats api, which makes
  the index parameter a required parameter.
* Fixed docs.
2018-10-29 07:45:27 +01:00
Martijn van Groningen
bad5972f62
[CCR] Fix request serialization bug (#34917)
and some parameters that were not set in tests.
2018-10-29 07:38:55 +01:00
Tim Vernum
bb5b59004e
[TEST] Manually trigger resource watching (#34893)
SSLTrustRestrictionsTests.testRestrictionsAreReloaded checks that the
SSL trust configuration is automatically updated reapplied if the
underlying "trust_restrictions.yml" file is modified.

Since the default resource watcher frequency is 5seconds, it could
take 10 second to run that test (as it waits for 2 reloaded).

Previously this test set that frequency to a very low value (3ms) so
that the elapsed time for the test would be reduced. However this
caused other problems, including that the resource watcher would
frequently run while the cluster was shutting down and files were
being cleaned up.

This change resets that watch frequency back to its default (5s) and
then manually calls the "notifyNow" method on the resource watcher
whenever the restrictions file is modified, so that the SSL trust
configuration is reloaded at exactly the right time.

Resolves: #34502
2018-10-29 11:23:36 +11:00
Costin Leau
14fb7edc1d
SQL: Fix incorrect AVG data type (#34948)
AVG now returns double (not the type of the target field)

Close #33773
2018-10-28 22:13:04 +02:00
Tal Levy
5c88356ad6
add ILM rolling upgrade tests (#32828) 2018-10-26 15:40:26 -07:00
Tal Levy
d8322ca069 Merge remote-tracking branch 'upstream/master' into index-lifecycle 2018-10-26 12:46:21 -07:00
Nik Everett
9f87fdc7ab
Drop deprecationLogger from AbstractComponent (#34859)
Drops the `deprecationLogger` from `AbstractComponent`, moving it to
places where we need it. This saves us from building a bunch of
`DeprecationLogger`s that we don't need.

Relates to #34488
2018-10-26 15:40:16 -04:00
Nik Everett
10295b306d
Core: Drop nodeName from AbstractComponent (#34487)
`AbstractComponent` is trouble because its name implies that
*everything* should extend from it. It *is* useful, but maybe too
broadly useful. The things it offers access too, the `Settings` instance
for the entire server and a logger are nice to have around, but not
really needed *everywhere*. The `Settings` instance especially adds a
fair bit of ceremony to testing without any value.

This removes the `nodeName` method from `AbstractComponent` so it is
more clear where we actually need the node name.
2018-10-26 15:26:14 -04:00
Tal Levy
e737ea7d4a
remove old doc placeholder and migrate ilm docs to top-level (#34615)
we are restructuring the docs, this migrates ILM docs outside of the x-pack doc structure.
2018-10-26 12:19:52 -07:00
Costin Leau
55d48e4cc1
SQL: Handle aggregation for null group (#34916)
When dealing with a null group, the associated metric aggs need to
return null as well

Fix #34896
2018-10-26 21:34:01 +03:00
Jay Modi
a0279bc069
Responses can use Writeable.Reader interface (#34655)
In order to remove Streamable from the codebase, Response objects need
to be read using the Writeable.Reader interface which this change
enables. This change enables the use of Writeable.Reader by adding the
`Action#getResponseReader` method. The default implementation simply
uses the existing `newResponse` method and the readFrom method. As
responses are migrated to the Writeable.Reader interface, Action
classes can be updated to throw an UnsupportedOperationException when
`newResponse` is called and override the `getResponseReader` method.

Relates #34389
2018-10-26 09:21:54 -06:00
Costin Leau
5a7b8c0e69
SQL: Provide null-safe scripts for Not and Neg (#34877)
Introduce null-safe Painless scripts for Not and Neg
Simplify script generation for Unary functions

Close #34848
2018-10-26 18:21:14 +03:00
Lee Hinman
6880e2ad79 [TEST] Fix RolloverActionTests.mutateInstance
In some cases it wasn't mutating and caused test failures
2018-10-26 09:10:55 -06:00
Jason Tedor
43f6ba1c63
Fix put/resume follow request parsing (#34913)
This commit adds some fields that were missing from put follow, and
fixes a bug in resume follow.
2018-10-26 11:09:55 -04:00
Dimitris Athanasiou
a39a67cd38
[ML] Extract common native process base class (#34856)
We currently have two different native processes:
autodetect & normalizer. There are plans for introducing
a new process. All these share many things in common.
This commit refactors the processes to extend an
`AbstractNativeProcess` class that encapsulates those
commonalities with the purpose of reusing the code
for new processes in the future.
2018-10-26 15:34:48 +01:00
Martijn van Groningen
306f1d78f8
[CCR] Retry when no index shard stats can be found (#34852)
Index shard stats for the follower shard are fetched, when a shard follow task is started.
This is needed in order to bootstap the shard follow task with the follower global checkpoint.

Sometimes index shard stats are not available (e.g. during a restart) and
we fail now, while it is very likely that these stats will be available some time later.
2018-10-26 15:14:24 +02:00
Albert Zaharovits
3f1fec1813
[Docs] audit logfile structured format (#34584)
Documents the new structured logfile format for auditing
that was introduced by #31931. Most changes herein
are for 6.x . In 7.0 the deprecated format is gone and a
follow-up PR is in order.
2018-10-26 15:19:35 +03:00
David Roberts
734088673e
[ML] Include message in field_stats for text log files (#34861)
This change ensures the `message` field is always included
in the `field_stats` for the semi-structured text log file
file structure.  Previously it was not, as it will almost
certainly contain all distinct values.  However, for
consistency in the UI it's useful to include it.
2018-10-26 07:45:02 +02:00
Tal Levy
e1fdd00420
Lowercase static final DeprecationLogger instance names (#34887)
After discussing on the team's FixItFriday, we concluded that
static final instance variables that are mutable should be lowercased.

Historically, DeprecationLogger was uppercased more frequently than lowercased.
2018-10-25 21:12:19 -07:00
Tal Levy
4c4afef9cc fix PolicyStatsTests: mutate kept same indicesManaged 2018-10-25 16:13:09 -07:00
Tal Levy
f6ce935444
fix GET _ilm response with uninitialized ILM metadata (#34881)
ILM would return a resource-not-found exception when requesting policies
while the IndexLifecycleMetaData is not initialized. The behavior here
should not be as extreme since it is not the user's fault.

This commit changes the behavior so that it succeeds and returns no policies
when no policy names are explicitely specified, otherwise keep the same behavior
of throwing an exception
2018-10-25 16:00:44 -07:00
Tal Levy
f76ced4e14
update updating-policy docs (#34661)
This contains updates to the ILM Docs explaining how updates to policies work and
affect policy execution on indices.
2018-10-25 15:24:25 -07:00
Tal Levy
810cd46a30 Merge remote-tracking branch 'upstream/master' into index-lifecycle 2018-10-25 14:35:33 -07:00
Marios Trivyzas
06d7431950
SQL: Return error with ORDER BY on non-grouped. (#34855)
Previously, for some queries the validation for ORDER BY
fields didn't kick in since a HAVING close or an ORDER BY
with scalar function would add `Filter` and `Project` plans
between the `OrderBy` and the `Aggregate`.

Now the LogicalPlan tree beneath `OrderBy` is traversed and
the ORDER BY fields are properly verified.

Fixes: #34590
2018-10-25 22:15:28 +02:00
Tim Brooks
cf9aff954e
Reduce channels in AbstractSimpleTransportTestCase (#34863)
This is related to #30876. The AbstractSimpleTransportTestCase initiates
many tcp connections. There are normally over 1,000 connections in
TIME_WAIT at the end of the test. This is because every test opens at
least two different transports that connect to each other with 13
channel connection profiles. This commit modifies the default
connection profile used by this test to 6. One connection for each
type, except for REG which gets 2 connections.
2018-10-25 13:37:49 -06:00
Lee Hinman
a4b634e60e Remove unnecessary "acknowledge: true" statements in REST tests 2018-10-25 11:53:31 -06:00
Lee Hinman
3e7042832a Merge remote-tracking branch 'origin/master' into index-lifecycle 2018-10-25 11:00:36 -06:00
Tal Levy
17fe37e5dd
ILM Docs: update start/stop documentation (#34787)
adds documentation explaining the different operation modes in the
context of starting and stopping ILM.
2018-10-25 07:30:53 -07:00
Tal Levy
41eaa586e8
remove index.lifecycle.skip setting (#34823)
With the introduction of _ilm/stop and _ilm/start APIs, the
use cases where one would only target a select group
of indices to start/stop has been reduced. Since there is no
strong use-case for skipping specific indices, it is best to
remove this functionality and only adding if later desired, with the
hopes of keeping things more simple.
2018-10-25 07:27:04 -07:00
Tal Levy
21b9b024c7
fix PolicyStatsTests mutateInstance (#34835)
through randomization, there is a chance that the mutateInstance
for PolicyStatsTests does not actually mutate the original object.
This PR aims to fix this
2018-10-25 07:24:54 -07:00
Marios Trivyzas
bd143334d3
SQL: Fix edge case: <field> IN (null) (#34802)
Handle the case when `null` is the only value in the list so that
it's translated to a `MatchNoDocsQuery`.

Followup to: #34750
2018-10-25 14:24:11 +02:00
Colin Goodheart-Smithe
0b26f8b14c
Fixes NPE in multi node qa testt 2018-10-25 10:45:30 +01:00
Costin Leau
b97546a5a9
SQL: Introduce ODBC mode, similar to JDBC (#34825)
Close #34720
2018-10-25 12:40:00 +03:00
Andrei Stefan
a7e08f462f
SQL: handle X-Pack or X-Pack SQL not being available in a more graceful way (#34736)
Throw a different error message for a http response code of 400, but also when the error itself is of a specific type.
2018-10-25 12:14:49 +03:00
Nhat Nguyen
ff49e79d40 CCR: Rename follow-task parameters and stats (#34836)
* CCR: Rename follow parameters and stats

This commit renames the follow-task parameters and its stats.
Below are the changes:

## Params
- remote_cluster (unchanged)
- leader_index (unchanged)
- max_read_request_operation_count -> max_read_request_operation_count
- max_batch_size -> max_read_request_size
- max_write_request_operation_count (new)
- max_write_request_size (new)
- max_concurrent_read_batches -> max_outstanding_read_requests
- max_concurrent_write_batches -> max_outstanding_write_requests
- max_write_buffer_size (unchanged)
- max_write_buffer_count (unchanged)
- max_retry_delay (unchanged)
- poll_timeout -> read_poll_timeout

## Stats
- remote_cluster (unchanged)
- leader_index (unchanged)
- follower_index (unchanged)
- shard_id (unchanged)
- leader_global_checkpoint (unchanged)
- leader_max_seq_no (unchanged)
- follower_global_checkpoint (unchanged)
- follower_max_seq_no (unchanged)
- last_requested_seq_no (unchanged)
- number_of_concurrent_reads -> outstanding_read_requests
- number_of_concurrent_writes -> outstanding_write_requests
- buffer_size_in_bytes -> write_buffer_size_in_bytes (new)
- number_of_queued_writes -> write_buffer_operation_count
- mapping_version -> follower_mapping_version
- total_fetch_time_millis -> total_read_time_millis
- total_fetch_remote_time_millis -> total_read_remote_exec_time_millis
- number_of_successful_fetches -> successful_read_requests
- number_of_failed_fetches -> failed_read_requests
- operation_received -> operations_read
- total_transferred_bytes -> bytes_read
- total_index_time_millis -> total_write_time_millis [?]
- number_of_successful_bulk_operations -> successful_write_requests
- number_of_failed_bulk_operations -> failed_write_requests
- number_of_operations_indexed -> operations_written
- fetch_exception -> read_exceptions
- time_since_last_read_millis -> time_since_last_read_millis

* add test for max_write_request_(operation_count|size)
2018-10-25 10:36:15 +02:00
Armin Braun
e7ced94a65
NETWORKING: Add SSL Handler before other Handlers (#34636)
* NETWORKING: Add SSL Handler before other Handlers

* The only way to run into the issue in #33998 is for `Netty4MessageChannelHandler`
to be in the pipeline while the SslHandler is not. Adding the SslHandler before any
other handlers should ensure correct ordering here even when we handle upstream events
in our own thread pool
* Ensure that channels that were closed concurrently don't trip the assertion
* Closes #33998
2018-10-25 07:51:14 +02:00
Alpar Torok
59536966c2
Add a new "contains" feature (#34738)
The contains syntax was added in #30874 but the skips were not properly
put in place.
The java runner has the feature so the tests will run as part of the
build, but language clients will be able to support it at their own
pace.
2018-10-25 08:50:50 +03:00
Ryan Ernst
687dc1eb11
Scripting: Remove SearchScript (#34730)
This commit removes the last non context based script class.
2018-10-24 15:03:38 -07:00
Martijn van Groningen
6fe0e62b7a
[CCR] Added write buffer size limit (#34797)
This limit is based on the size in bytes of the operations in the write buffer. If this limit is exceeded then no more read operations will be coordinated until the size in bytes of the write buffer has dropped below the configured write buffer size limit.

Renamed existing `max_write_buffer_size` to ``max_write_buffer_count` to indicate that limit is count based.

Closes #34705
2018-10-24 23:48:49 +02:00
Andrey Atapin
5f588180f9 Improve IndexNotFoundException's default error message (#34649)
This commit adds the index name to the error message when an index is not found.
2018-10-24 12:53:31 -07:00
Benjamin Trent
cd27b0b996
Revert "Rollup add default metrics to histo groups (#34534)" (#34815)
This reverts commit 4236358f5d53ab4f0d8457faacb19983b6b1ab82.
2018-10-24 14:25:10 -05:00
Costin Leau
7570d69254
SQL: Introduce support for IP fields (#34758)
IP fields are recognized and can be used through-out the query

Close #32499
2018-10-24 22:03:11 +03:00
Marios Trivyzas
98cd7ca861
SQL: Fix queries with filter resulting in NO_MATCH (#34812)
Previously, `Mapper` was returning an incorrect plan which resulted in an
```
SQLFeatureNotSupportedException: Found 1 problem(s)
line 1:8: Unexecutable item
```

Queries with a `WHERE` and/or `HAVING` clause which results in NO_MATCH
are now handled correctly and return 0 rows.

Fixes: #34613

Co-authored-by: Costin Leau <costin.leau@gmail.com>
2018-10-24 20:36:30 +02:00
Colin Goodheart-Smithe
e7fddb5c93
Adds usage data for ILM (#33377)
* Adds usage data for ILM

* Adds tests for IndexLifecycleFeatureSetUsage and friends

* Adds tests for IndexLifecycleFeatureSet

* Fixes merge errors

* Add number of indices managed to usage stats

Also adds more tests

* Addresses Review comments
2018-10-24 18:28:46 +01:00
Costin Leau
f19565c3e0
SQL: Verifier allows aliases aggregates for sorting (#34773)
Improve Verifier rule that prevented grouping with aliases inside aggregates
to not be accepted for ordering.

Close #34607
2018-10-24 19:31:26 +03:00
markharwood
f7a6fb288f
Fix compilation error peculiar to eclipse2018-09 (#34798)
Fix compilation error peculiar to eclipse2018-09.
Added generics to make eclipse compiler happy
2018-10-24 16:58:31 +01:00
Nhat Nguyen
d73768f812
CCR: Do not follow if leader does not have soft-deletes (#34767)
We should not create a follower index and abort a follow request if the
leader does not have soft-deletes. Moreover, we also should not
auto-follow an index if it does not have soft-deletes.
2018-10-24 11:19:39 -04:00