Commit Graph

44497 Commits

Author SHA1 Message Date
Mayya Sharipova e4fa32470b
Types removal fix FullClusterRestartIT warnings (#38445)
Backport PR #38389 for 6.7 produces warnings for rollover test.
This fixes FullClusterRestartIT warning expectations
for rollover request

Relates to #38389
2019-02-05 14:15:43 -05:00
Julie Tibshirani 3ce7d2c9b6
Make sure to reject mappings with type _doc when include_type_name is false. (#38270)
`CreateIndexRequest#source(Map<String, Object>, ... )`, which is used when
deserializing index creation requests, accidentally accepts mappings that are
nested twice under the type key (as described in the bug report #38266).

This in turn causes us to be too lenient in parsing typeless mappings. In
particular, we accept the following index creation request, even though it
should not contain the type key `_doc`:

```
PUT index?include_type_name=false
{
  "mappings": {
    "_doc": {
      "properties": { ... }
    }
  }
}
```

There is a similar issue for both 'put templates' and 'put mappings' requests
as well.

This PR makes the minimal changes to detect and reject these typed mappings in
requests. It does not address #38266 generally, or attempt a larger refactor
around types in these server-side requests, as I think this should be done at a
later time.
2019-02-05 10:52:32 -08:00
austintp 8ebff0512b Updates the grok patterns to be consistent with logstash (#27181) 2019-02-05 12:37:02 -06:00
Christoph Büscher ca47f68091
Ignore type-removal warnings in XPackRestTestHelper (#38431)
The backport of #38022 introduced types-deprecation warning for get/put template requests
that cause problems on tests master in mixed cluster scenarios. While these warnings are
caught and ignored in regular Rest tests, the get template requests in XPackRestTestHelper
were missed.

Closes #38412
2019-02-05 19:07:53 +01:00
Zachary Tong 54e684bedd
testHlrcFromXContent() should respect assertToXContentEquivalence() (#38232)
Tests can override assertToXContentEquivalence() in case their xcontent
cannot be directly compared (e.g. due to insertion order in maps
affecting the xcontent ordering).  But the `testHlrcFromXContent` test
hardcoded the equivalence test to `true` instead of consulting
`assertToXContentEquivalence()`

Fixes #36034
2019-02-05 12:59:05 -05:00
Tal Levy ae47c025e2
add basic REST test for geohash_grid (#37996) 2019-02-05 09:44:47 -08:00
David Turner f2dd5dd6eb
Remove DiscoveryPlugin#getDiscoveryTypes (#38414)
With this change we no longer support pluggable discovery implementations. No
known implementations of `DiscoveryPlugin` actually override this method, so in
practice this should have no effect on the wider world. However, we were using
this rather extensively in tests to provide the `test-zen` discovery type. We
no longer need a separate discovery type for tests as we no longer need to
customise its behaviour.

Relates #38410
2019-02-05 17:42:24 +00:00
Przemyslaw Gomulka 963b474f2f
Fix the clock resolution to millis in GetWatchResponseTests (#38405)
the clock resolution changed from jdk8->jdk10, hence the test is passing
in jdk8 but failing in jdk10. The Watcher's objects are serialised and
deserialised with milliseconds precision, making test to fail in jdk 10
and higher

closes #38400
2019-02-05 18:27:24 +01:00
David Turner b7ab521eb1
Throw AssertionError when no master (#38432)
Today we throw a fatal `RuntimeException` if an exception occurs in
`getMasterName()`, and this includes the case where there is currently no
master. However, sometimes we call this method inside an `assertBusy()` in
order to allow for a cluster that is in the process of stabilising and electing
a master. The trouble is that `assertBusy()` only retries on an
`AssertionError` and not on a general `RuntimeException`, so the lack of a
master is immediately fatal.

This commit fixes the issue by asserting there is a master, triggering a retry
if there is not.

Fixes #38331
2019-02-05 17:11:20 +00:00
Boaz Leskes 12657fda44
`if_seq_no` and `if_primary_term` parameters aren't wired correctly in REST Client's CRUD API (#38411) 2019-02-05 18:05:56 +01:00
Przemyslaw Gomulka df4eb0485d
Enable CronEvalToolTest.testEnsureDateIsShownInRootLocale (#38394)
The test is now expected to be always passing no matter what the random
locale is. This is fixed with using jdk ZoneId.systemDefault() in both
the test and CronEvalTool

closes #35687
2019-02-05 17:48:47 +01:00
Julie Tibshirani 440d1eda8a
Fix failures in BulkProcessorIT#testGlobalParametersAndBulkProcessor. (#38129)
This PR fixes a couple test issues:
* It narrows an assertWarnings call that was too broad, and wasn't always
  applicable with certain random sequences.
* Previously, we could send a typeless bulk request containing '_type: 'null'.
  Now we omit the _type key altogether for typeless requests.
2019-02-05 08:42:37 -08:00
Marios Trivyzas c9701be1e8
SQL: Implement CURRENT_DATE (#38175)
Since DATE data type is now available, this implements the
`CURRENT_DATE/CURRENT_DATE()/TODAY()` similar to `CURRENT_TIMESTAMP`.

Closes: #38160
2019-02-05 18:15:26 +02:00
Armin Braun 887fa2c97a
Mute testReadRequestsReturnLatestMappingVersion (#38438)
* Relates #37807
2019-02-05 17:10:12 +01:00
David Roberts 92bc681705
[ML] Report index unavailable instead of waiting for lazy node (#38423)
If a job cannot be assigned to a node because an index it
requires is unavailable and there are lazy ML nodes then
index unavailable should be reported as the assignment
explanation rather than waiting for a lazy ML node.
2019-02-05 16:10:00 +00:00
Michael Basnight 8742db3afe
Update Rollup Caps to allow unknown fields (#38339)
This commit ensures that the parts of rollup caps that can allow unknown
fields will allow them. It also modifies the test such that we can use
the features we need for disallowing fields in spots where they would
not be allowed.

Relates #36938
2019-02-05 10:08:08 -06:00
Martijn van Groningen 0beb3c93d1
Clean up duplicate follow config parameter code (#37688)
Introduced FollowParameters class that put follow, resume follow,
put auto follow pattern requests and follow info response classes reuse.

The FollowParameters class had the fields, getters etc. for the common parameters
that all these APIs have.  Also binary and xcontent serialization /
parsing is handled by this class.

The follow, resume follow, put auto follow pattern request classes originally
used optional non primitive fields, so FollowParameters has that too and the follow info api can handle that now too.

Also the followerIndex field can in production only be specified via
the url path. If it is also specified via the request body then
it must have the same value as is specified in the url path. This
option only existed to xcontent testing. However the AbstractSerializingTestCase
base class now also supports createXContextTestInstance() to provide
a different test instance when testing xcontent, so allowing followerIndex
to be specified via the request body is no longer needed.

By moving the followerIndex field from Body to ResumeFollowAction.Request
class and not allowing the followerIndex field to be specified via
the request body the Body class is redundant and can be removed. The
ResumeFollowAction.Request class can then directly use the
FollowParameters class.

For consistency I also removed the ability to specified followerIndex
in the put follow api and the name in put auto follow pattern api via
the request body.
2019-02-05 17:05:19 +01:00
Armin Braun 2f6afd290e
Fix Concurrent Snapshot Ending And Stabilize Snapshot Finalization (#38368)
* The problem in #38226 is that in some corner cases multiple calls to `endSnapshot` were made concurrently, leading to non-deterministic behavior (`beginSnapshot` was triggering a repository finalization while one that was triggered by a `deleteSnapshot` was already in progress)
   * Fixed by:
      * Making all `endSnapshot` calls originate from the cluster state being in a "completed" state (apart from on short-circuit on initializing an empty snapshot). This forced putting the failure string into `SnapshotsInProgress.Entry`.
      * Adding deduplication logic to `endSnapshot`
* Also:
  * Streamlined the init behavior to work the same way (keep state on the `SnapshotsService` to decide which snapshot entries are stale)
* closes #38226
2019-02-05 16:44:18 +01:00
Lee Hinman d862453d68
Support unknown fields in ingest pipeline map configuration (#38352)
We already support unknown objects in the list of pipelines, this changes the
`PipelineConfiguration` to support fields other than just `id` and `config`.

Relates to #36938
2019-02-05 07:52:17 -07:00
Jason Tedor 638ba4a59a
Mute failing API key integration test (#38409)
This commit mutes the test
testGetAndInvalidateApiKeysWithExpiredAndInvalidatedApiKey as it failed
during a PR build.
2019-02-05 06:08:03 -05:00
Andrei Stefan cea81b199d
Change the milliseconds precision to 3 digits for intervals. (#38297) 2019-02-05 12:00:49 +02:00
Albert Zaharovits 8e2eb39cef
SecuritySettingsSource license.self_generated: trial (#38233)
Authn is enabled only if `license_type` is non `basic`, but `basic` is
what the `LicenseService` generates implicitly. This commit explicitly sets
license type to `trial`, which allows for authn, in the `SecuritySettingsSource`
which is the settings configuration parameter for `InternalTestCluster`s.

The real problem, that had created tests failures like #31028 and #32685, is
that the check `licenseState.isAuthAllowed()` can change sporadically. If it were
to return `true` or `false` during the whole test there would be no problem.
The problem manifests when it turns from `true` to `false` right before `Realms.asList()`.
There are other license checks before this one (request filter, token service, etc)
that would not cause a problem if they would suddenly see the check as `false`.
But switching to `false` before `Realms.asList()` makes it appear that no installed
realms could have handled the authn token which is an authentication error, as can
be seen in the failing tests.

Closes #31028 #32685
2019-02-05 10:49:08 +02:00
David Turner 3b2a0d7959
Rename no-master-block setting (#38350)
Replaces `discovery.zen.no_master_block` with `cluster.no_master_block`. Any
value set for the old setting is now ignored.
2019-02-05 08:47:56 +00:00
David Turner 2d114a02ff
Rename static Zen1 settings (#38333)
Renames the following settings to remove the mention of `zen` in their names:

- `discovery.zen.hosts_provider` -> `discovery.seed_providers`
- `discovery.zen.ping.unicast.concurrent_connects` -> `discovery.seed_resolver.max_concurrent_resolvers`
- `discovery.zen.ping.unicast.hosts.resolve_timeout` -> `discovery.seed_resolver.timeout`
- `discovery.zen.ping.unicast.hosts` -> `discovery.seed_addresses`
2019-02-05 08:46:52 +00:00
Przemyslaw Gomulka 89feaa0e23
Migration doc for audit json log file (#38165)
The migration documentation for an audit logging changes. Removal of plaintext logs and rename of json log file

relates #32850
2019-02-05 09:09:15 +01:00
Brandon Kobel 64ff75f04e
Add apm_user reserved role (#38206)
* Adding apm_user

* Fixing SecurityDocumentationIT testGetRoles test

* Adding access to .ml-anomalies-*

* Fixing APM test, we don't have access to the ML state index
2019-02-04 21:45:28 -08:00
Yogesh Gaikwad fe36861ada
Add support for API keys to access Elasticsearch (#38291)
X-Pack security supports built-in authentication service
`token-service` that allows access tokens to be used to 
access Elasticsearch without using Basic authentication.
The tokens are generated by `token-service` based on
OAuth2 spec. The access token is a short-lived token
(defaults to 20m) and refresh token with a lifetime of 24 hours,
making them unsuitable for long-lived or recurring tasks where
the system might go offline thereby failing refresh of tokens.

This commit introduces a built-in authentication service
`api-key-service` that adds support for long-lived tokens aka API
keys to access Elasticsearch. The `api-key-service` is consulted
after `token-service` in the authentication chain. By default,
if TLS is enabled then `api-key-service` is also enabled.
The service can be disabled using the configuration setting.

The API keys:-
- by default do not have an expiration but expiration can be
  configured where the API keys need to be expired after a
  certain amount of time.
- when generated will keep authentication information of the user that
   generated them.
- can be defined with a role describing the privileges for accessing
   Elasticsearch and will be limited by the role of the user that
   generated them
- can be invalidated via invalidation API
- information can be retrieved via a get API
- that have been expired or invalidated will be retained for 1 week
  before being deleted. The expired API keys remover task handles this.

Following are the API key management APIs:-
1. Create API Key - `PUT/POST /_security/api_key`
2. Get API key(s) - `GET /_security/api_key`
3. Invalidate API Key(s) `DELETE /_security/api_key`

The API keys can be used to access Elasticsearch using `Authorization`
header, where the auth scheme is `ApiKey` and the credentials, is the 
base64 encoding of API key Id and API key separated by a colon.
Example:-
```
curl -H "Authorization: ApiKey YXBpLWtleS1pZDphcGkta2V5" http://localhost:9200/_cluster/health
```

Closes #34383
2019-02-05 14:21:57 +11:00
Christoph Büscher d255303584
Add typless client side GetIndexRequest calls and response class (#37778)
The HLRC client currently uses `org.elasticsearch.action.admin.indices.get.GetIndexRequest`
and `org.elasticsearch.action.admin.indices.get.GetIndexResponse` in its get index calls. Both request and
response are designed for the typed APIs, including some return types e.g. for `getMappings()` which in
the maps it returns still use a level including the type name.
In order to change this without breaking existing users of the HLRC API, this PR introduces two new request
and response objects in the `org.elasticsearch.client.indices` client package. These are used by the
IndicesClient#get and IndicesClient#exists calls now by default and support the type-less API. The old request
and response objects are still kept for use in similarly named, but deprecated methods.

The newly introduced client side classes are simplified versions of the server side request/response classes since
they don't need to support wire serialization, and only the response needs fromXContent parsing (but no
xContent-serialization, since this is the responsibility of the server-side class).
Also changing the return type of `GetIndexResponse#getMapping` to
`Map<String, MappingMetaData> getMappings()`, while it previously was returning another map
keyed by the type-name. Similar getters return simple Maps instead of the ImmutableOpenMaps that the 
server side response objects return.
2019-02-05 03:41:05 +01:00
Yogesh Gaikwad 9d3f057894
Limit token expiry to 1 hour maximum (#38244)
We mention in our documentation for the token
expiration configuration maximum value is 1 hour
but do not enforce it. This commit adds max limit
to the TOKEN_EXPIRATION setting.
2019-02-05 12:02:36 +11:00
Tal Levy 48f09471f8
add docs saying mixed-cluster ILM is not supported (#37954)
Closes #37085.
2019-02-04 17:00:40 -08:00
Yogesh Gaikwad b5b319ec9a
Skip unsupported languages for tests (#38328)
Skip the languages in tests for which SimpleKdcServer
does not handle generalized time correctly.

Closes#38320
2019-02-05 11:01:13 +11:00
Gordon Brown 292e0f6fb7
Deprecate `_type` in simulate pipeline requests (#37949)
As mapping types are being removed throughout Elasticsearch, the use of
`_type` in pipeline simulation requests is deprecated. Additionally, the
default `_type` used if one is not supplied has been changed to `_doc` for
consistency with the rest of Elasticsearch.
2019-02-04 16:11:44 -07:00
Gordon Brown b866417650
Mute testCannotShrinkLeaderIndex (#38374)
This test should not pass until CCR finishes integrating shard history
retention leases. It currently sometimes passes (which is a bug in the
test), but cannot pass reliably until the linked issue is resolved.
2019-02-04 16:06:19 -07:00
Nhat Nguyen cecfa5bd6d
Tighten mapping syncing in ccr remote restore (#38071)
There are two issues regarding the way that we sync mapping from leader
to follower when a ccr restore is completed:

1.  The returned mapping from a cluster service might not be up to date
as the mapping of the restored index commit.

2. We should not compare the mapping version of the follower and the
leader. They are not related to one another.

Moreover, I think we should only ensure that once the restore is done,
the mapping on the follower should be at least the mapping of the copied
index commit. We don't have to sync the mapping which is updated after
we have opened a session.

Relates #36879
Closes #37887
2019-02-04 17:53:41 -05:00
Tim Brooks 5a33816c86
Add test for `PutFollowAction` on a closed index (#38236)
This is related to #35975. Currently when an index falls behind a leader
it encounters a fatal exception. This commit adds a test for that
scenario. Additionally, it tests that the user can stop following, close
the follower index, and put follow again. After the indexing is
re-bootstrapped, it will recover the documents it lost in normal
following operations.
2019-02-04 16:37:42 -06:00
Jay Modi c3cdf84c04
Fix SSLContext pinning to TLSV1.2 in reload tests (#38341)
This commit fixes the pinning of SSLContexts to TLSv1.2 in the
SSLConfigurationReloaderTests. The pinning was added for the initial
creation of clients and webservers but the updated contexts would
default to TLSv1.3, which is known to cause hangs with the
MockWebServer that we use.

Relates #38103
Closes #38247
2019-02-04 14:34:37 -07:00
Christoph Büscher 0ced775389
Mute RareClusterStateIT.testDelayedMappingPropagationOnReplica (#38357) 2019-02-04 22:30:34 +01:00
Mayya Sharipova 641704464d
Deprecate types in rollover index API (#38039)
Relates to #35190
2019-02-04 16:07:45 -05:00
markharwood 578fd14257
Types removal - fix FullClusterRestartIT warning expectations (#38310)
Relax test warning message checking to pre-empt PR 38022 landing in 6.7 with new warning messages.
The relaxed test now just assumes any warning message starting with “[types removal]” is tolerated rather than the precise phrasing used in the 6.7 branch.
2019-02-04 20:09:07 +00:00
Michael Basnight 24db053465
Fix ILM explain response to allow unknown fields (#38054)
IndexLifecycleExplainResponse did not allow unknown fields. This commit
fixes the test and ConstructingObjectParser such that it allows unknown
fields.
2019-02-04 14:05:00 -06:00
Nhat Nguyen fb1e350c81
Mute testFollowIndexAndCloseNode (#38360)
Tracked at #33337
2019-02-04 15:04:46 -05:00
Nik Everett 5d949dddfb
Docs: Drop inline callout from scroll example (#38340)
Coalesces two calls into one in a scroll example so all callouts are at
the end of the line. This is the only sort of callouts that are
supported by asciidoctor and we'd like to start building our docs with
asciidoctor.

At present we don't have any mechanism to stop folks adding more inline
callouts but we ought to be able to have one in a few weeks. For now,
though, removing these inline callouts is a step in the right direction.

Relates to #38335
2019-02-04 14:57:38 -05:00
Michael Basnight 66530dbde0
Deprecate HLRC security methods (#37883)
This commit deprecates the few methods that had their parameters
reordered to facilitate the move from EmptyResponse to boolean. This
commit also readds the boolean based methods with the proper
signatures.

Relates #37540
Relates #36938
2019-02-04 13:23:00 -06:00
Shaunak Kashyap be1bb0ec7d
Remove types from Monitoring plugin "backend" code (#37745)
This PR removes the use of document types from the monitoring exporters and template + watches setup code.

It does not remove the notion of types from the monitoring bulk API endpoint "front end" code as that code will eventually just go away in 8.0 and be replaced with Beats as collectors/shippers directly to the monitoring cluster.
2019-02-04 10:58:03 -08:00
Zachary Tong ab1150378b
Add Composite to AggregationBuilders (#38207) 2019-02-04 13:47:04 -05:00
David Turner 2c1eab2b8a
Clarify slow cluster-state log messages (#38302)
The message `... took [31s] above the warn threshold of 30s` suggests
incorrectly that the task took 61 seconds. This commit adds the clarifying
words `which is`.
2019-02-04 17:44:00 +00:00
Adrien Grand ac07386eab
Ignore the `>test-mute` label when generating release notes. (#38307)
Muted tests are irrelevant for release notes.
2019-02-04 17:48:08 +01:00
Gordon Brown f872c721ac
Run Node deprecation checks locally (#38065) (#38250)
At times, we need to check for usage of deprecated settings in settings
which should not be returned by the NodeInfo API.  This commit changes
the deprecation info API to run all node checks locally so that these
settings can be checked without exposing them via any externally
accessible API.
2019-02-04 09:43:28 -07:00
Christoph Büscher 1899658a38
Mute ClusterClientIT#testClusterHealthYellowSpecificIndex (#38343) 2019-02-04 17:13:54 +01:00
Andrey Ershov 7bc8bc9605
ensureGreen (#38324) 2019-02-04 16:36:04 +01:00