Commit Graph

44755 Commits

Author SHA1 Message Date
Michael Basnight 14c571532a
Fix ILM Lifecycle Policy to allow unknown fields (#38041)
A few of the ILM Lifecycle Policy and classes did not allow for unknown
fields. This commit sets those parsers and fixes the tests for the
parsers.

Relates #36938
2019-01-30 14:31:55 -06:00
Michael Basnight 945ad05d54
Update verify repository to allow unknown fields (#37619)
The subparser in verify repository allows for unknown fields. This
commit sets the value to true for the parser and modifies the test such
that it accurately tests it.

Relates #36938
2019-01-30 14:31:16 -06:00
David Roberts be788160ef
[ML] Datafeed deprecation checks (#38026)
Deprecation checks for the ML datafeed
query and aggregations.
2019-01-30 20:12:20 +00:00
David Turner 81c443c9de
Deprecate minimum_master_nodes (#37868)
Today we pass `discovery.zen.minimum_master_nodes` to nodes started up in
tests, but for 7.x nodes this setting is not required as it has no effect.
This commit removes this setting so that nodes are started with more realistic
configurations, and deprecates it.
2019-01-30 20:09:15 +00:00
Jake Landis 6a78b6a31c
Remove types from watcher docs (#38002)
Types have been deprecated and this commit removes the documentation
for specifying types in the index action, and search input/transform.

Relates #37594 #35190
2019-01-30 13:12:13 -06:00
Jack Conradson 36ee78d924
Add test coverage for Painless general casting of boolean and Boolean (#37780)
This adds test coverage for general casts in Painless between boolean and other types and Boolean and other types.
2019-01-30 11:01:45 -08:00
Martijn van Groningen 5433af28e3
Fixed test bug, lastFollowTime is null if there are no follower indices. 2019-01-30 19:33:16 +01:00
Lee Hinman cac6b8e06f
Add ECS schema for user-agent ingest processor (#37727) (#37984)
* Add ECS schema for user-agent ingest processor (#37727)

This switches the format of the user agent processor to use the schema from [ECS](https://github.com/elastic/ecs).
So rather than something like this:

```
{
  "patch" : "3538",
  "major" : "70",
  "minor" : "0",
  "os" : "Mac OS X 10.14.1",
  "os_minor" : "14",
  "os_major" : "10",
  "name" : "Chrome",
  "os_name" : "Mac OS X",
  "device" : "Other"
}
```

The structure is now like this:

```
{
  "name" : "Chrome",
  "original" : "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36",
  "os" : {
    "name" : "Mac OS X",
    "version" : "10.14.1",
    "full" : "Mac OS X 10.14.1"
  },
  "device" : "Other",
  "version" : "70.0.3538.102"
}
```

This is now the default for 7.0. The deprecated `ecs` setting in 6.x is not
supported.

Resolves #37329

* Remove `ecs` setting from docs
2019-01-30 11:24:18 -07:00
Armin Braun a070b8acc0
Extract TransportRequestDeduplication from ShardStateAction (#37870)
* Extracted the logic for master request duplication so it can be reused by the snapshotting logic
* Removed custom listener used by `ShardStateAction` to not leak these into future users of this class
* Changed semantics slightly to get rid of redundant instantiations of the composite listener
* Relates #37686
2019-01-30 19:21:09 +01:00
Jason Tedor 6500b0cbd7
Expose retention leases in shard stats (#37991)
This commit exposes retention leases via shard-level stats.
2019-01-30 13:20:40 -05:00
Jason Tedor c468b2f7ca
Make primary terms fields private in index shard (#38036)
This commit encapsulates the primary terms fields in index shard. This
is a precursor to pushing the operation primary term down to the
replication tracker.
2019-01-30 12:56:58 -05:00
Benjamin Trent 9782aaa1b8
ML: Add reason field in JobTaskState (#38029)
* ML: adding reason to job failure status

* marking reason as nullable

* Update AutodetectProcessManager.java
2019-01-30 11:56:24 -06:00
Nhat Nguyen ed460c2815 Log flush_stats and commit_stats in testMaybeFlush
This test failed a few times over the last several months. It seems that
we triggered a flush, but CI was too slow to finish it in several
seconds. I added the flush stats and commit stats and unmuted this test.
We should have a good clue if this test fails again.

Relates #37896
2019-01-30 12:54:31 -05:00
Michael Basnight 2cbc6888a2
HLRC: Fix strict setting exception handling (#37247)
The LLRC's exception handling for strict mode was previously throwing an
exception the HLRC assumed was an error response. This is not the case
if the result is valid in strict mode, as it will return the proper
response wrapped in an exception with warnings. This commit fixes the
HLRC such that it no longer spews if it encounters a strict LLRC
response.

Closes #37090
2019-01-30 11:31:59 -06:00
Nik Everett e97718245d
Test: Enable strict deprecation on all tests (#36558)
This drops the option for tests to disable strict deprecation mode in
the low level rest client in favor of configuring expected warnings on
any calls that should expect warnings. This behavior is
paranoid-by-default which is generally the right way to handle
deprecations and tests in general.
2019-01-30 11:48:34 -05:00
Colin Goodheart-Smithe 21e392e95e
Removes typed calls from YAML REST tests (#37611)
This PR attempts to remove all typed calls from our YAML REST tests. The PR adds include_type_name: false to create index requests that use a mapping and also to put mapping requests. It also removes _type from index requests where they haven't already been removed. The PR ignores tests named *_with_types.yml since this are specifically testing typed API behaviour.

The change also includes changing the test harness to add the type _doc to index, update, get and bulk requests that do not specify the document type when the test is running against a mixed 7.x/6.x cluster.
2019-01-30 16:32:58 +00:00
David Roberts 2f7776c8b7
Switch default time format for ingest from Joda to Java for v7 (#37934)
Date formats with and without the "8" prefix are now all treated
as Java time formats, so that ingest does the same as mappings
in this respect.
2019-01-30 16:26:28 +00:00
Christoph Büscher ecbaa38864
Remove deprecated Plugin#onModule extension points (#37866)
Removes some guice index level extension point marked as @Deprecated since at
least 6.0. They served as a signpost for plugin authors upgrading from 2.x but
this shouldn't be relevant in 7.0 anymore.
2019-01-30 17:17:54 +01:00
Igor Motov 23805fa41a
Geo: Fix Empty Geometry Collection Handling (#37978)
Fixes handling empty geometry collection and re-enables
testParseGeometryCollection test.

Fixes #37894
2019-01-30 09:20:30 -05:00
Albert Zaharovits 53e80e9814 Fix failure in test code ClusterPrivilegeTests
Closes #38030
2019-01-30 16:11:44 +02:00
Benjamin Trent 8280a20664
ML: Add upgrade mode docs, hlrc, and fix bug (#37942)
* ML: Add upgrade mode docs, hlrc, and fix bug

* [DOCS] Fixes build error and edits text

* adjusting docs

* Update docs/reference/ml/apis/set-upgrade-mode.asciidoc

Co-Authored-By: benwtrent <ben.w.trent@gmail.com>

* Update set-upgrade-mode.asciidoc

* Update set-upgrade-mode.asciidoc
2019-01-30 06:51:11 -06:00
Alpar Torok f3379940c6
Fix the packer cache script (#38023)
The script is used to create a cache on ephemeral CI workers.
Changes:
    - create and use a `pullFixture` task that always exists regardless
    of docker support
    - wire dependencies correctly so any pre fixture setup runs for pull
    as well
    - set up java env vars so bwc versions can build
2019-01-30 14:41:35 +02:00
Luca Cavanna b91d587275
Move SearchHit and SearchHits to Writeable (#37931)
This allowed to make SearchHits immutable, while quite a few fields in
SearchHit have to stay mutable unfortunately.

Relates to #34389
2019-01-30 12:05:54 +01:00
Jason Tedor ba285a56a7
Fix limit on retaining sequence number (#37992)
We only assign non-negative sequence numbers to operations, so the lower
limit on retaining sequence numbers should be that it is non-negative
only.
2019-01-30 05:25:17 -05:00
Martijn van Groningen 3865435a01
Docs test fix, wait for shards active.
(a restore needs to be complete, which happens in the background and
by default the ccr put follow api doesn't wait for this)

(this was a recent change and the pr that added this docs test,
did not include this change)

Relates to #37917
2019-01-30 11:02:16 +01:00
Martijn van Groningen e959dbaa99
Revert "Revert "Documented default values for index follow request parameters. (#37917)""
This reverts commit 7205833f92.
2019-01-30 10:49:42 +01:00
Adrien Grand 7205833f92 Revert "Documented default values for index follow request parameters. (#37917)"
This reverts commit 4da7a44648.
2019-01-30 10:48:01 +01:00
Alexander Reelsen 9ec4abc31e
Ensure date parsing BWC compatibility (#37929)
In order to retain BWC this changes the java date formatters to be able to
parse nanoseconds resolution, even if only milliseconds are supported.
This used to work on joda time as well so that a user could store a date
like `2018-10-03T14:42:44.613469+0000` and then just loose the precision
on anything lower than millisecond level.
2019-01-30 10:47:12 +01:00
Andrei Stefan 908c8def06
SQL: Skip the nested and object field types in case of an ODBC request (#37948) 2019-01-30 11:34:47 +02:00
Adrien Grand c8af0f4bfa
Use mappings to format doc-value fields by default. (#30831)
Doc-value fields now return a value that is based on the mappings rather than
the script implementation by default.

This deprecates the special `use_field_mapping` docvalue format which was added
in #29639 only to ease the transition to 7.x and it is not necessary anymore in
7.0.
2019-01-30 10:31:51 +01:00
Adrien Grand b63b50b945
Give precedence to index creation when mixing typed templates with typeless index creation and vice-versa. (#37871)
Currently if you mix typed templates and typeless index creation or typeless
templates and typed index creation then you will end up with an error because
Elasticsearch tries to create an index that has multiple types: `_doc` and
the explicit type name that you used.

This commit proposes to give precedence to the index creation call so that
the type from the template will be ignored if the index creation call is
typeless while the template is typed, and the type from the index creation
call will be used if there is a typeless template.

This is consistent with the fact that index creation already "wins" if a field
is defined differently in the index creation call and in a template: the
definition from the index creation call is used in such cases.

Closes #37773
2019-01-30 10:28:24 +01:00
Tim Vernum 4dee3f7418
Add classifier to tar.gz in docker compose (#38011)
The distribution now includes a platform specific classifier that the
docker build wasn't taking into account.

Relates: #37881
2019-01-30 20:13:48 +11:00
Martijn van Groningen 4da7a44648
Documented default values for index follow request parameters. (#37917) 2019-01-30 09:37:41 +01:00
Jim Ferenczi 2732bb5cf3
Fix fetch source option in expand search phase (#37908)
This change fixes the copy of the fetch source option into the
expand search request that is used to retrieve the documents of each
collapsed group.

Closes #23829
2019-01-30 08:46:14 +01:00
Jim Ferenczi 5dcc805dc9
Restore a noop _all metadata field for 6x indices (#37808)
This commit restores a noop version of the AllFieldMapper that is instanciated only
for indices created in 6x. We need this metadata field mapper to be present in this version
in order to allow the upgrade of indices that explicitly disable _all (enabled: false).
The mapping of these indices contains a reference to the _all field that we cannot remove
in 7 so we'll need to keep this metadata mapper in 7x. Since indices created in 6x will not
be compatible with 8, we'll remove this noop mapper in the next major version.

Closes #37429
2019-01-30 08:45:50 +01:00
Martijn van Groningen f51bc00fcf
Added ccr to xpack usage infrastructure (#37256)
* Added ccr to xpack usage infrastructure

Closes #37221
2019-01-30 07:58:26 +01:00
Tim Vernum 99129d7786
Fix exit code for Security CLI tools (#37956)
The certgen, certutil and saml-metadata tools did not correctly return
their exit code to the calling shell.

These commands now explicitly exit with the code that was returned
from the main(args, terminal) method.
2019-01-30 17:51:11 +11:00
Armin Braun 57823c484f
Streamline S3 Repository- and Client-Settings (#37393)
* Make repository settings override static settings
* Cache clients according to settings
   * Introduce custom implementations for the AWS credentials here to be able to use them as part of a hash key
2019-01-30 06:22:31 +01:00
Marios Trivyzas f5b9b4d89c Add version 6.6.1 (#37975) 2019-01-30 15:33:01 +11:00
Tim Brooks 55b916afc0
Ensure task metadata not null in follow test (#37993)
This commit fixes a potential race in the IndexFollowingIT. Currently it
is possible that we fetch the task metadata, it is null, and that throws
a null pointer exception. Assertbusy does not catch null pointer
exceptions. This commit assertions that the metadata is not null.
2019-01-29 15:58:31 -07:00
markharwood 0470ee1fcc Docs fix - missing callout 2019-01-29 21:24:08 +00:00
markharwood b889221f75
Types removal - deprecate include_type_name with index templates (#37484)
Added deprecation warnings for use of include_type_name in put/get index templates.
HLRC changes:
GetIndexTemplateRequest has a new client-side class which is a copy of server's GetIndexTemplateResponse but modified to be typeless.
PutIndexTemplateRequest has a new client-side counterpart which doesn't use types in the mappings
Relates to #35190
2019-01-29 20:52:41 +00:00
jimczi 193017672a Handle completion suggestion without contexts
This change fixes the handling of completion suggestion without contexts.

Relates #36996
2019-01-29 20:31:46 +01:00
Tim Brooks f3f9cabd67
Add timeout for ccr recovery action (#37840)
This is related to #35975. It adds a action timeout setting that allows
timeouts to be applied to the individual transport actions that are
used during a ccr recovery.
2019-01-29 12:29:06 -07:00
Marios Trivyzas e9332331a3
SQL: Make error msg for validation of 2nd arg of PERCENTILE[_RANK] consistent (#37937)
Use `first` and `second` instead of `1st` and `2nd`.
2019-01-29 21:20:09 +02:00
Ryan Ernst 8e5f9c4b14
Add OS/architecture classifier to distributions (#37881)
This commit adds classifiers to the distributions indicating the
OS (for archives) and platform. The current OSes are for windows, darwin (ie
macos) and linux. This change will allow future OS/architecture specific
changes to the distributions. Note the docs using distribution links
have been updated, but will be reworked in a followup to make OS
specific instructions for the archives.
2019-01-29 11:18:30 -08:00
Albert Zaharovits 697b2fbe52
Remove implicit index monitor privilege (#37774)
Restricted indices (currently only .security-6 and .security) are special
internal indices that require setting the `allow_restricted_indices` flag
on every index permission that covers them. If this flag is `false`
(default) the permission will not cover these and actions against them
will not be authorized.
However, the monitoring APIs were the only exception to this rule.

This exception is herein forfeited and index monitoring privileges have to be
granted explicitly, using the `allow_restricted_indices` flag on the permission,
as is the case for any other index privilege.
2019-01-29 21:10:03 +02:00
Benjamin Trent 34d61d3231
ML: ignore unknown fields for JobTaskState (#37982) 2019-01-29 12:51:34 -06:00
Julie Tibshirani 9ca26b7e63
Remove more references to type in docs. (#37946)
* Update the top-level 'getting started' guide.
* Remove custom types from the painless getting started documentation.
* Fix an incorrect references to '_doc' in the cardinality query docs.
* Update the _update docs to use the typeless API format.
2019-01-29 10:51:07 -08:00
Tim Brooks 00ace369af
Use `CcrRepository` to init follower index (#35719)
This commit modifies the put follow index action to use a
CcrRepository when creating a follower index. It routes 
the logic through the snapshot/restore process. A 
wait_for_active_shards parameter can be used to configure
how long to wait before returning the response.
2019-01-29 11:47:29 -07:00