Commit Graph

458 Commits

Author SHA1 Message Date
Gordon Brown 454ce99b01 Merge branch 'master' into index-lifecycle 2018-08-29 08:28:23 -06:00
Jay Modi 1e11b05b58
Remove unused User class from protocol (#33137)
This commit removes the unused User class from the protocol project.
This class was originally moved into protocol in preparation for moving
more request and response classes, but given the change in direction
for the HLRC this is no longer needed. Additionally, this change also
changes the package name for the User object in x-pack/plugin/core to
its original name.
2018-08-28 08:55:29 -06:00
Gordon Brown 50368656ee Merge branch 'master' into index-lifecycle 2018-08-27 15:35:19 -06:00
Jay Modi 5d9c270608
Token API supports the client_credentials grant (#33106)
This change adds support for the client credentials grant type to the
token api. The client credentials grant allows for a client to
authenticate with the authorization server and obtain a token to access
as itself. Per RFC 6749, a refresh token should not be included with
the access token and as such a refresh token is not issued when the
client credentials grant is used.

The addition of the client credentials grant will allow users
authenticated with mechanisms such as kerberos or PKI to obtain a token
that can be used for subsequent access.
2018-08-27 10:56:21 -06:00
Tal Levy 5783545222 Merge branch 'master' into index-lifecycle 2018-08-27 08:19:05 -07:00
Andrei Stefan 3d9ca4baee
SQL: Enable aggregations to create a separate bucket for missing values (#32832)
Enable aggregations to create a separate bucket for missing values.
2018-08-27 17:56:28 +03:00
Shaunak Kashyap 1779d3376a
APM server monitoring (#32515)
* Adding new MonitoredSystem for APM server

* Teaching Monitoring template utils about APM server monitoring indices

* Documenting new monitoring index for APM server

* Adding monitoring index template for APM server

* Copy pasta typo

* Removing metrics.libbeat.config section from mapping

* Adding built-in user and role for APM server user

* Actually define the role :)

* Adding missing import

* Removing index template and system ID for apm server

* Shortening line lengths

* Updating expected number of built-in users in integration test

* Removing "system" from role and user names

* Rearranging users to make tests pass
2018-08-27 08:42:40 -04:00
Nik Everett 1e9144d8e6
Switch remaining x-pack tests to new style Requests (#33108)
In #29623 we added `Request` object flavored requests to the low level
REST client and in #30315 we deprecated the old `performRequest`s. This
changes all calls in the `x-pack/qa/saml-idp-tests` and
`x-pack/qa/security-setup-password-tests` projects to use the new
versions.
2018-08-24 16:39:08 -04:00
Gordon Brown 1f13c77b49 Merge branch 'master' into index-lifecycle 2018-08-23 11:52:59 -06:00
Andrei Stefan de95dead2d
SQL: skip uppercasing/lowercasing function tests for AZ locales as well (#32910)
* Added the rest of the Locales that have different behavior for uppercasing/lowercasing scenarios to the skip list
2018-08-23 00:21:38 +03:00
Nik Everett c3438bc8d8
Switch some watcher tests to new style Requests (#33044)
In #29623 we added `Request` object flavored requests to the low level
REST client and in #30315 we deprecated the old `performRequest`s. This
changes all calls in the `x-pack/qa/smoke-test-monitoring-with-watcher`,
`x-pack/qa/smoke-test-watcher`, and
`x-pack/qa/smoke-test-watcher-with-security` projects to use the new
versions.
2018-08-22 14:02:39 -04:00
Alpar Torok 82d10b484a
Run forbidden api checks with runtimeJavaVersion (#32947)
Run forbidden APIs checks with runtime hava version
2018-08-22 09:05:22 +03:00
Nik Everett 2c81d7f77e
Build: Rework shadow plugin configuration (#32409)
This reworks how we configure the `shadow` plugin in the build. The major
change is that we no longer bundle dependencies in the `compile` configuration,
instead we bundle dependencies in the new `bundle` configuration. This feels
more right because it is a little more "opt in" rather than "opt out" and the
name of the `bundle` configuration is a little more obvious.

As an neat side effect of this, the `runtimeElements` configuration used when
one project depends on another now contains exactly the dependencies needed
to run the project so you no longer need to reference projects that use the
shadow plugin like this:

```
testCompile project(path: ':client:rest-high-level', configuration: 'shadow')
```

You can instead use the much more normal:

```
testCompile "org.elasticsearch.client:elasticsearch-rest-high-level-client:${version}"
```
2018-08-21 20:03:28 -04:00
Nik Everett fcf8cadd9a
Switch some x-pack tests to new style Requests (#32500)
In #29623 we added `Request` object flavored requests to the low level
REST client and in #30315 we deprecated the old `performRequest`s. This
changes all calls in the `x-pack/qa/audit-tests`,
`x-pack/qa/ml-disabled`, and `x-pack/qa/multi-node` projects to use the
new versions.
2018-08-21 14:48:53 -04:00
Jason Tedor 28d12b05b7
Move ML tests to be sub-projects of ML (#33026)
This commit moves the ML QA tests to be a sub-project of ML. The purpose
of this refactoring is to enable ML developers to run
:x-pack:plugin:ml:check and run the vast majority of a ML tests with a
single command (this still does not contain the ML REST tests, nor the
upgrade tests). This simplifies local development for faster iteration.
2018-08-21 12:23:21 -04:00
Benjamin Trent 3f91bbfa6b
[ML] Allowing _close to accept body payloads for options (#32989) (#33000) 2018-08-21 08:08:26 -05:00
Colin Goodheart-Smithe 10c60fae93
Merge branch 'master' into index-lifecycle 2018-08-21 11:54:06 +01:00
Jason Tedor ad0a965db9
Protect scheduler engine against throwing listeners (#32998)
There are two problems with the scheduler engine today. Both relate to
listeners that throw.

The first problem is that any triggered listener that throws a plain old
exception will cause no additional listeners to be triggered for the
event, and will also cause the scheduler to never be invoked again. This
leads to lost events and is bad.

The second problem is that any triggered listener that throws an error
of the fatal kind will not lead to that error because caught by the
uncaught exception handler. This is because the triggered listener is
executed as a future task under a scheduled thread pool executor. A
throwable there goes caught by the JDK framework and set as the outcome
on the future task. Since we never inspect these tasks for their
outcomes, nor is there a good place to do this, we have to handle these
errors ourselves. To do this, we catch them and dispatch them to the
uncaught exception handler via a forked thread. This is similar to our
handling in Netty.
2018-08-20 22:07:16 -04:00
Tal Levy a26e108590 Merge branch 'master' into index-lifecycle 2018-08-17 13:57:28 -07:00
Benjamin Trent 647705e00a Bypassing failing test PainlessDomainSplitIT#testHRDSplit (#32966) 2018-08-17 15:30:31 -05:00
Dimitris Athanasiou 46c35db1df [ML][TEST] Fix BasicRenormalizationIT after adding multibucket feature
As the multibucket feature was merged in, this test hit a side effect
which means buckets trailing an anomaly could become anomalous.
This commit fixes the problem by filtering low score records when
we request them.
2018-08-17 17:52:29 +01:00
Ioannis Kakavas e3aa68b0a9
[TEST] Run pre 6.4 nodes in non-FIPS JVMs (#32901)
Elasticsearch versions earlier than 6.4.0 cannot properly run in a
FIPS 140 JVM. This commit ensures that we use a non-FIPS JVM for
nodes that we spin up in BWC tests even when we're testing FIPS.
2018-08-17 18:23:13 +03:00
Benjamin Trent 9cec4aa14b
[ML] fix updating opened jobs scheduled events (#31651) (#32881)
* ML: fix updating opened jobs scheduled events (#31651)

* Adding UpdateParamsTests license header

* Adding integration test and addressing PR comments

* addressing test and job names
2018-08-17 07:21:17 -05:00
Tal Levy 711ac402b6 Merge branch 'master' into index-lifecycle 2018-08-16 15:08:22 -07:00
Ed Savage 62559d2b3c
Re enable ml bwc tests (#32916)
[ML] Re-enabling BWC tests

Re-enable BWC tests for ML now that #32816 has been backported to 6.x
2018-08-16 18:23:26 +01:00
Ed Savage d604b3e3a1 Temporarily disabled ML BWC tests for backporting
https://github.com/elastic/elasticsearch/pull/32816
2018-08-16 17:18:51 +01:00
Tal Levy c9de707f58 Merge branch 'master' into index-lifecycle 2018-08-16 08:41:57 -07:00
Adrien Grand eaaf37a1f9 AwaitFix FullClusterRestartIT#testRollupIDSchemeAfterRestart. 2018-08-16 16:35:10 +02:00
Jay Modi 1a45b27d8b
Move CharArrays to core lib (#32851)
This change cleans up some methods in the CharArrays class from x-pack, which
includes the unification of char[] to utf8 and utf8 to char[] conversions that
intentionally do not use strings. There was previously an implementation in
x-pack and in the reloading of secure settings. The method from the reloading
of secure settings was adopted as it handled more scenarios related to the
backing byte and char buffers that were used to perform the conversions. The
cleaned up class is moved into libs/core to allow it to be used by requests
that will be migrated to the high level rest client.

Relates #32332
2018-08-15 15:26:00 -06:00
Tal Levy ec93756600 Merge branch 'master' into index-lifecycle 2018-08-15 12:56:01 -07:00
Ed Savage 51cece1900 [ML] Temporarily disabling rolling-upgrade tests
BWC tests disabled while backporting https://github.com/elastic/elasticsearch/pull/32816
2018-08-15 15:48:50 +01:00
Lee Hinman 48281ac5bc
Use generic AcknowledgedResponse instead of extended classes (#32859)
This removes custom Response classes that extend `AcknowledgedResponse` and do nothing, these classes are not needed and we can directly use the non-abstract super-class instead.

While this appears to be a large PR, no code has actually changed, only class names have been changed and entire classes removed.
2018-08-15 08:06:14 -06:00
Tal Levy e78f537e58 Merge branch 'master' into index-lifecycle 2018-08-13 16:31:40 -07:00
Nhat Nguyen cba7fceb53 AwaitsFix case-functions.sql-spec
Relates #32589
2018-08-13 15:06:33 -04:00
Zachary Tong 4da0b52a19
[TEST] Force a stop to save rollup state before continuing (#32787)
We only upgrade the ID when the state is saved in one of four scenarios:

- when we reach a checkpoint (every 50 pages)
- when we run out of data
- when explicitly stopped
- on failure

The test was relying on the pre-upgrade to finish, save state and then
the post-upgrade to start, hit the end of data and upgrade ID.  THEN
get the new doc and apply the new ID.

But I think this is vulnerable to timing issues. If the pre-upgrade
portion shutdown before it saved the state, when restarting we would run
through all the data from the beginning with the old ID, meaning both
docs would still have the old scheme.

This change makes the pre-upgrade wait for the job to go back to STARTED
so that we know it persisted the end point.  Post-upgrade, it stops and
restarts the job to ensure the state was persisted and the ID upgraded.

That _should_ rule out the above timing issue.

Closes #32773
2018-08-13 12:53:22 -04:00
Tal Levy a771478940 Merge branch 'master' into index-lifecycle 2018-08-13 09:14:00 -07:00
Benjamin Trent b08416b899
Clear Job#finished_time when it is opened (#32605) (#32755)
* Clear Job#finished_time when it is opened (#32605)

* not returning failure when Job#finished_time is not reset

* Changing error log string and source string
2018-08-10 13:52:00 -05:00
Tal Levy 93637e2135 Merge branch 'master' into index-lifecycle 2018-08-10 10:23:14 -07:00
Alexander Reelsen e49717393a
Tests: Disable rolling upgrade tests with system key on fips JVM (#32775)
This disables the x-pack rolling upgrade tests using a fips JVM, as
there are problems creating the keystore.

Relates #32737
2018-08-10 16:50:11 +02:00
Tal Levy c7a2c357a3 Merge branch 'master' into index-lifecycle 2018-08-09 18:00:40 -07:00
Tal Levy 41e6d98af8
move qa yaml tests to inside the ILM plugin (#32693)
The qa tests with security haven't actually gone as far as testing security roles yet, so this is a start in the hopes of both bringing the tests into the ilm plugin
2018-08-09 16:09:20 -07:00
Alpar Torok 5bbed5ed9a Merge branch 'master' of github.com:elastic/elasticsearch 2018-08-09 21:58:03 +03:00
Alpar Torok c5e66cb345 mute test #32737 2018-08-09 21:54:58 +03:00
Andrei Stefan 6750e15773
SQL: Ignore H2 comparative tests for uppercasing/lowercasing string functions (#32604)
Skip the comparative tests using lowercasing/uppercasing against H2 (which considers the Locale).
ES-SQL is, so far, ignoring the Locale.
Still, the same queries are executed against ES-SQL alone and results asserted to be correct.
2018-08-09 19:28:48 +03:00
Colin Goodheart-Smithe b9c04adb29
Merge branch 'master' into index-lifecycle 2018-08-07 12:35:32 +01:00
Yogesh Gaikwad 615aa85f4e
[Kerberos] Use canonical host name (#32588)
The Apache Http components support for Spnego scheme
uses canonical name by default.
Also when resolving host name, on centos by default
there are other aliases so adding them to the
DelegationPermission.

Closes#32498
2018-08-06 23:51:43 +10:00
Tal Levy 457661e0da fix smoke test with security reference to renamed ilm 2018-08-03 14:00:27 -07:00
Lee Hinman 47a3e980b2 Merge remote-tracking branch 'origin/master' into index-lifecycle 2018-08-03 13:18:34 -06:00
Zachary Tong 992ec4be5d [Rollup] Update wire version check after backport
Bumping down the version to 6.4 since the backport is complete.  Also
adds some missing version checks to the bwc tests to make sure it
only runs on the correct versions
2018-08-03 14:09:01 -04:00
Tal Levy 78e94e1b00 Merge branch 'master' into index-lifecycle 2018-08-03 11:06:29 -07:00
Zachary Tong fc9fb64ad5
[Rollup] Improve ID scheme for rollup documents (#32558)
Previously, we were using a simple CRC32 for the IDs of rollup documents.
This is a very poor choice however, since 32bit IDs leads to collisions
between documents very quickly.

This commit moves Rollups over to a 128bit ID.  The ID is a concatenation
of all the keys in the document (similar to the rolling CRC before),
hashed with 128bit Murmur3, then base64 encoded.  Finally, the job
ID and a delimiter (`$`) are prepended to the ID.

This gurantees that there are 128bits per-job.  128bits should
essentially remove all chances of collisions, and the prepended
job ID means that _if_ there is a collision, it stays "within"
the job.

BWC notes:

We can only upgrade the ID scheme after we know there has been a good
checkpoint during indexing.  We don't rely on a STARTED/STOPPED
status since we can't guarantee that resulted from a real checkpoint,
or other state.  So we only upgrade the ID after we have reached
a checkpoint state during an active index run, and only after the
checkpoint has been confirmed.

Once a job has been upgraded and checkpointed, the version increments
and the new ID is used in the future.  All new jobs use the
new ID from the start
2018-08-03 11:13:25 -04:00
Colin Goodheart-Smithe 20915a9baf
Merge branch 'master' into index-lifecycle
client/rest-high-level/src/main/java/org/elasticsearch/client/XPackClien
t.java
/Users/colings86/dev/work/git/elasticsearch/.git/worktrees/elasticsearch
-ilm/MERGE_HEAD

client/rest-high-level/src/main/java/org/elasticsearch/client/RestHighLe
velClient.java
client/rest-high-level/src/main/java/org/elasticsearch/client/XPackClien
t.java
client/rest-high-level/src/test/java/org/elasticsearch/client/IndexLifec
ycleIT.java
client/rest-high-level/src/test/java/org/elasticsearch/client/RestHighLe
velClientTests.java
client/rest-high-level/src/test/java/org/elasticsearch/client/WatcherIT.
java
client/rest-high-level/src/test/java/org/elasticsearch/client/documentat
ion/LicensingDocumentationIT.java
client/rest-high-level/src/test/java/org/elasticsearch/client/documentat
ion/WatcherDocumentationIT.java
modules/lang-painless/src/main/java/org/elasticsearch/painless/Compiler.
java
modules/lang-painless/src/main/java/org/elasticsearch/painless/Def.java
modules/lang-painless/src/main/java/org/elasticsearch/painless/FunctionR
ef.java
modules/lang-painless/src/main/java/org/elasticsearch/painless/PainlessE
xplainError.java
modules/lang-painless/src/main/java/org/elasticsearch/painless/ScriptCla
ssInfo.java
modules/lang-painless/src/main/java/org/elasticsearch/painless/antlr/Enh
ancedPainlessLexer.java
modules/lang-painless/src/main/java/org/elasticsearch/painless/lookup/Pa
inlessLookup.java
modules/lang-painless/src/main/java/org/elasticsearch/painless/node/EExp
licit.java
modules/lang-painless/src/main/java/org/elasticsearch/painless/node/EFun
ctionRef.java
modules/lang-painless/src/main/java/org/elasticsearch/painless/node/EIns
tanceof.java
modules/lang-painless/src/main/java/org/elasticsearch/painless/node/ELam
bda.java
modules/lang-painless/src/main/java/org/elasticsearch/painless/node/ELis
tInit.java
modules/lang-painless/src/main/java/org/elasticsearch/painless/node/EMap
Init.java
modules/lang-painless/src/main/java/org/elasticsearch/painless/node/ENew
Array.java
modules/lang-painless/src/main/java/org/elasticsearch/painless/node/ENew
Obj.java
modules/lang-painless/src/main/java/org/elasticsearch/painless/node/ESta
tic.java
modules/lang-painless/src/main/java/org/elasticsearch/painless/node/PCal
lInvoke.java
modules/lang-painless/src/main/java/org/elasticsearch/painless/node/PFie
ld.java
modules/lang-painless/src/main/java/org/elasticsearch/painless/node/PSub
ListShortcut.java
modules/lang-painless/src/main/java/org/elasticsearch/painless/node/PSub
MapShortcut.java
modules/lang-painless/src/main/java/org/elasticsearch/painless/node/SCat
ch.java
modules/lang-painless/src/main/java/org/elasticsearch/painless/node/SDec
laration.java
modules/lang-painless/src/main/java/org/elasticsearch/painless/node/SEac
h.java
modules/lang-painless/src/main/java/org/elasticsearch/painless/node/SFun
ction.java
modules/lang-painless/src/main/java/org/elasticsearch/painless/node/SSub
EachIterable.java
modules/lang-painless/src/test/java/org/elasticsearch/painless/OverloadT
ests.java
modules/lang-painless/src/test/java/org/elasticsearch/painless/PainlessD
ocGenerator.java
modules/lang-painless/src/test/java/org/elasticsearch/painless/RegexTest
s.java
modules/lang-painless/src/test/java/org/elasticsearch/painless/node/Node
ToStringTests.java
rest-api-spec/src/main/resources/rest-api-spec/test/cluster.stats/10_bas
ic.yml
server/src/main/java/org/elasticsearch/action/admin/cluster/stats/Cluste
rStatsResponse.java
server/src/main/java/org/elasticsearch/action/admin/cluster/stats/Transp
ortClusterStatsAction.java
server/src/main/java/org/elasticsearch/action/bulk/TransportShardBulkAct
ion.java
server/src/main/java/org/elasticsearch/action/support/replication/Transp
ortReplicationAction.java
server/src/main/java/org/elasticsearch/index/engine/Engine.java
server/src/main/java/org/elasticsearch/index/engine/InternalEngine.java
server/src/main/java/org/elasticsearch/index/seqno/ReplicationTracker.ja
va
server/src/main/java/org/elasticsearch/index/shard/IndexShard.java
server/src/main/java/org/elasticsearch/index/shard/IndexShardOperationPe
rmits.java
server/src/main/java/org/elasticsearch/index/shard/PrimaryReplicaSyncer.
java
server/src/main/java/org/elasticsearch/index/shard/StoreRecovery.java
server/src/main/java/org/elasticsearch/index/translog/Translog.java
server/src/main/java/org/elasticsearch/indices/recovery/RecoverySourceHa
ndler.java
server/src/main/java/org/elasticsearch/indices/recovery/RecoveryTarget.j
ava
server/src/test/java/org/elasticsearch/action/bulk/TransportShardBulkAct
ionTests.java
server/src/test/java/org/elasticsearch/action/search/SearchAsyncActionTe
sts.java
server/src/test/java/org/elasticsearch/action/support/replication/Transp
ortReplicationActionTests.java
server/src/test/java/org/elasticsearch/action/support/replication/Transp
ortWriteActionTests.java
server/src/test/java/org/elasticsearch/cluster/routing/allocation/ShardS
tateIT.java
server/src/test/java/org/elasticsearch/index/replication/IndexLevelRepli
cationTests.java
server/src/test/java/org/elasticsearch/index/replication/RecoveryDuringR
eplicationTests.java
server/src/test/java/org/elasticsearch/index/seqno/ReplicationTrackerTes
ts.java
server/src/test/java/org/elasticsearch/index/shard/IndexShardOperationPe
rmitsTests.java
server/src/test/java/org/elasticsearch/index/shard/IndexShardTests.java
server/src/test/java/org/elasticsearch/index/shard/IndexingOperationList
enerTests.java
server/src/test/java/org/elasticsearch/index/shard/PrimaryReplicaSyncerT
ests.java
server/src/test/java/org/elasticsearch/index/translog/TranslogTests.java
server/src/test/java/org/elasticsearch/indices/recovery/RecoverySourceHa
ndlerTests.java
server/src/test/java/org/elasticsearch/indices/recovery/RecoveryTests.ja
va
server/src/test/java/org/elasticsearch/search/profile/query/QueryProfile
rIT.java
test/framework/src/main/java/org/elasticsearch/index/replication/ESIndex
LevelReplicationTestCase.java
test/framework/src/main/java/org/elasticsearch/index/shard/IndexShardTes
tCase.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/rollup/job
/MetricConfig.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/rollup/job
/RollupJobConfig.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/S
ecurityContext.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/U
serSettings.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/a
ction/user/AuthenticateResponse.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/a
ction/user/ChangePasswordRequestBuilder.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/a
ction/user/GetUsersResponse.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/a
ction/user/PutUserRequestBuilder.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/a
uthc/Authentication.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/a
uthc/AuthenticationResult.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/a
uthc/Realm.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/a
uthz/RoleDescriptor.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/a
uthz/accesscontrol/SecurityIndexSearcherWrapper.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/s
upport/MetadataUtils.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/u
ser/AnonymousUser.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/u
ser/BeatsSystemUser.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/u
ser/ElasticUser.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/u
ser/InternalUserSerializationHelper.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/u
ser/KibanaUser.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/u
ser/LogstashSystemUser.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/u
ser/SystemUser.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/u
ser/XPackSecurityUser.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/u
ser/XPackUser.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/rollup/Con
figTestHelpers.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/rollup/job
/MetricsConfigSerializingTests.java ->
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/rollup/job
/MetricConfigSerializingTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/security/a
uthz/accesscontrol/SecurityIndexSearcherWrapperUnitTests.java
x-pack/plugin/monitoring/src/test/java/org/elasticsearch/xpack/monitorin
g/collector/cluster/ClusterStatsMonitoringDocTests.java
x-pack/plugin/rollup/src/test/java/org/elasticsearch/xpack/rollup/Rollup
JobIdentifierUtilTests.java
x-pack/plugin/rollup/src/test/java/org/elasticsearch/xpack/rollup/Rollup
RequestTranslationTests.java
x-pack/plugin/rollup/src/test/java/org/elasticsearch/xpack/rollup/action
/SearchActionTests.java
x-pack/plugin/rollup/src/test/java/org/elasticsearch/xpack/rollup/config
/ConfigTests.java
x-pack/plugin/rollup/src/test/java/org/elasticsearch/xpack/rollup/job/In
dexerUtilsTests.java
x-pack/plugin/rollup/src/test/java/org/elasticsearch/xpack/rollup/job/Ro
llupIndexerIndexingTests.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/ac
tion/saml/TransportSamlLogoutAction.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/ac
tion/user/TransportAuthenticateAction.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/ac
tion/user/TransportGetUsersAction.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/ac
tion/user/TransportHasPrivilegesAction.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/au
dit/AuditTrail.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/au
dit/AuditTrailService.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/au
dit/index/IndexAuditTrail.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/au
dit/logfile/LoggingAuditTrail.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/au
thc/AuthenticationService.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/au
thc/esnative/NativeRealm.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/au
thc/esnative/NativeUsersStore.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/au
thc/esnative/ReservedRealm.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/au
thc/esnative/UserAndPassword.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/au
thc/file/FileRealm.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/au
thc/file/FileUserPasswdStore.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/au
thc/kerberos/KerberosRealm.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/au
thc/ldap/LdapRealm.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/au
thc/pki/PkiRealm.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/au
thc/saml/SamlRealm.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/au
thc/support/CachingUsernamePasswordRealm.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/au
thz/AuthorizationService.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/au
thz/AuthorizedIndices.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/in
gest/SetSecurityUserProcessor.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/re
st/action/RestAuthenticateAction.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/re
st/action/user/RestChangePasswordAction.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/re
st/action/user/RestGetUsersAction.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/tr
ansport/ServerTransportFilter.java
x-pack/plugin/security/src/test/java/org/elasticsearch/integration/Clear
RealmsCacheTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/Se
curityContextTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/ac
tion/filter/SecurityActionFilterTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/ac
tion/interceptor/IndicesAliasesRequestInterceptorTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/ac
tion/interceptor/ResizeRequestInterceptorTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/ac
tion/saml/TransportSamlInvalidateSessionActionTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/ac
tion/saml/TransportSamlLogoutActionTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/ac
tion/user/TransportAuthenticateActionTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/ac
tion/user/TransportChangePasswordActionTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/ac
tion/user/TransportDeleteUserActionTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/ac
tion/user/TransportGetUsersActionTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/ac
tion/user/TransportHasPrivilegesActionTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/ac
tion/user/TransportPutUserActionTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/ac
tion/user/TransportSetEnabledActionTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
dit/AuditTrailServiceTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
dit/index/IndexAuditTrailMutedTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
dit/index/IndexAuditTrailTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
dit/logfile/LoggingAuditTrailFilterTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
dit/logfile/LoggingAuditTrailTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
thc/AuthenticationServiceTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
thc/RealmsTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
thc/TokenServiceTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
thc/UserTokenTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
thc/esnative/NativeRealmIntegTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
thc/esnative/NativeUsersStoreTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
thc/esnative/ReservedRealmTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
thc/file/FileRealmTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
thc/file/FileUserPasswdStoreTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
thc/kerberos/KerberosRealmAuthenticateFailedTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
thc/kerberos/KerberosRealmCacheTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
thc/kerberos/KerberosRealmTestCase.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
thc/kerberos/KerberosRealmTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
thc/ldap/ActiveDirectoryRealmTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
thc/ldap/CancellableLdapRunnableTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
thc/ldap/LdapRealmTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
thc/pki/PkiRealmTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
thc/support/CachingUsernamePasswordRealmTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
thc/support/mapper/NativeRoleMappingStoreTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
thz/AuthorizationServiceTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
thz/AuthorizationUtilsTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
thz/AuthorizedIndicesTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
thz/IndicesAndAliasesResolverTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
thz/SecuritySearchOperationListenerTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/in
gest/SetSecurityUserProcessorTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/tr
ansport/SecurityServerTransportInterceptorTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/tr
ansport/ServerTransportFilterTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/us
er/AnonymousUserTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/us
er/UserTests.java ->
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/us
er/UserSerializationTests.java
x-pack/plugin/sql/src/main/java/org/elasticsearch/xpack/sql/expression/f
unction/scalar/string/StringFunctionUtils.java
x-pack/plugin/watcher/src/test/java/org/elasticsearch/xpack/watcher/exec
ution/ExecutionServiceTests.java
x-pack/protocol/src/main/java/org/elasticsearch/protocol/xpack/ml/job/co
nfig/DefaultDetectorDescription.java
x-pack/protocol/src/main/java/org/elasticsearch/protocol/xpack/ml/job/co
nfig/DetectionRule.java
x-pack/protocol/src/main/java/org/elasticsearch/protocol/xpack/ml/job/co
nfig/Detector.java
x-pack/protocol/src/main/java/org/elasticsearch/protocol/xpack/ml/job/co
nfig/DetectorFunction.java
x-pack/protocol/src/main/java/org/elasticsearch/protocol/xpack/ml/job/co
nfig/FilterRef.java
x-pack/protocol/src/main/java/org/elasticsearch/protocol/xpack/ml/job/co
nfig/MlFilter.java
x-pack/protocol/src/main/java/org/elasticsearch/protocol/xpack/ml/job/co
nfig/Operator.java
x-pack/protocol/src/main/java/org/elasticsearch/protocol/xpack/ml/job/co
nfig/RuleAction.java
x-pack/protocol/src/main/java/org/elasticsearch/protocol/xpack/ml/job/co
nfig/RuleCondition.java
x-pack/protocol/src/main/java/org/elasticsearch/protocol/xpack/ml/job/co
nfig/RuleScope.java
x-pack/protocol/src/main/java/org/elasticsearch/protocol/xpack/ml/packag
e-info.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/u
ser/User.java ->
x-pack/protocol/src/main/java/org/elasticsearch/protocol/xpack/security/
User.java
x-pack/protocol/src/test/java/org/elasticsearch/protocol/xpack/ml/job/co
nfig/DetectionRuleTests.java
x-pack/protocol/src/test/java/org/elasticsearch/protocol/xpack/ml/job/co
nfig/DetectorTests.java
x-pack/protocol/src/test/java/org/elasticsearch/protocol/xpack/ml/job/co
nfig/FilterRefTests.java
x-pack/protocol/src/test/java/org/elasticsearch/protocol/xpack/ml/job/co
nfig/MlFilterTests.java
x-pack/protocol/src/test/java/org/elasticsearch/protocol/xpack/ml/job/co
nfig/RuleConditionTests.java
x-pack/protocol/src/test/java/org/elasticsearch/protocol/xpack/ml/job/co
nfig/RuleScopeTests.java
x-pack/protocol/src/test/java/org/elasticsearch/protocol/xpack/security/
UserTests.java
x-pack/qa/rolling-upgrade/src/test/java/org/elasticsearch/upgrades/Watch
erRestartIT.java
x-pack/qa/rolling-upgrade/src/test/resources/rest-api-spec/test/mixed_cl
uster/60_watcher.yml
x-pack/qa/rolling-upgrade/src/test/resources/rest-api-spec/test/old_clus
ter/60_watcher.yml
x-pack/qa/rolling-upgrade/src/test/resources/rest-api-spec/test/upgraded
_cluster/60_watcher.yml
x-pack/qa/security-example-spi-extension/src/main/java/org/elasticsearch
/example/realm/CustomRealm.java
x-pack/qa/security-example-spi-extension/src/test/java/org/elasticsearch
/example/realm/CustomRealmTests.java
x-pack/qa/security-migrate-tests/src/test/java/org/elasticsearch/xpack/s
ecurity/MigrateToolIT.java
2018-08-03 12:25:38 +01:00
Alexander Reelsen f809d6fff4
Tests: Add rolling upgrade tests for watcher (#32428)
These tests ensure, that the basic watch APIs are tested in the rolling
upgrade tests. After initially adding a watch, the tests try to get,
execute, deactivate and activate a watch. Watcher stats are tested as
well, and an own java based test has been added for restarting, as that
requires waiting for a state change. Watcher history is also checked.

Closes #31216
2018-08-03 09:41:29 +02:00
Jay Modi eb3accb721 Security: move User to protocol project (#32367)
The User class has been moved to the protocol project for upcoming work
to add more security APIs to the high level rest client. As part of
this change, the toString method no longer uses a custom output method
from MetadataUtils and instead just relies on Java's toString
implementation.
2018-08-03 00:21:34 +03:00
Tal Levy f7162588df Merge branch 'master' into index-lifecycle 2018-08-02 10:22:46 -07:00
Jason Tedor 5de236e1e7
Rename ILM, ILM endpoints and drop _xpack (#32564)
This commit does the following:
 - renames index-lifecycle plugin to ilm
 - modifies the endpoints to ilm instead of index_lifecycle
 - drops _xpack from the endpoints
 - drops a few duplicate endpoints
2018-08-02 13:05:11 -04:00
Colin Goodheart-Smithe 4cdbb42542
Mutes failing SQL string function tests due to #32589 2018-08-02 16:15:59 +01:00
Tal Levy ea1e26ca27 Merge branch 'master' into index-lifecycle 2018-08-01 10:15:18 -07:00
Andrei Stefan 4c388539a0
SQL: Added support for string manipulating functions with more than one parameter (#32356)
Added support for string manipulating functions with more than one parameter:
CONCAT, LEFT, RIGHT, REPEAT, POSITION, LOCATE, REPLACE, SUBSTRING, INSERT
2018-08-01 12:29:06 +03:00
Jay Modi f2f33f3149 Use hostname instead of IP with SPNEGO test (#32514)
This change updates KerberosAuthenticationIT to resolve the host used
to connect to the test cluster. This is needed because the host could
be an IP address but SPNEGO requires a hostname to work properly. This
is done by adding a hook in ESRestTestCase for building the HttpHost
from the host and port.

Additionally, the project now specifies the IPv4 loopback address as
the http host. This is done because we need to be able to resolve the
address used for the HTTP transport before the node starts up, but the
http.ports file is not written until the node is started.

Closes #32498
2018-08-01 12:57:33 +10:00
Nik Everett 99d9a0a40c
Switch x-pack rolling restart to new style Requests (#32339)
In #29623 we added `Request` object flavored requests to the low level
REST client and in #30315 we deprecated the old `performRequest`s. This
changes all calls in the `x-pack:qa:rolling-upgrade*` projects to use
the new versions.
2018-07-31 21:04:33 -04:00
Tal Levy 73cdfaf07b Merge branch 'master' into index-lifecycle 2018-07-31 11:50:33 -07:00
Igor Motov 5fd7202808
SQL: Add test for handling of partial results (#32474)
Verifies that partial results are rejected by SQL requests.

Closes #32284
2018-07-31 11:06:00 -07:00
Daniel Mitterdorfer adb93da974 Mute KerberosAuthenticationIT
Relates #32498
2018-07-31 15:55:52 +02:00
Nik Everett 670630948b
Switch security spi example to new style Requests (#32341)
In #29623 we added `Request` object flavored requests to the low level
REST client and in #30315 we deprecated the old `performRequest`s. This
changes all calls in the `x-pack/qa/security-example-spi-extension`
project to use the new versions.
2018-07-30 18:07:49 -04:00
Colin Goodheart-Smithe dfb277cfb7
Merge branch 'master' into index-lifecycle 2018-07-27 10:51:09 +01:00
Yogesh Gaikwad 9198bcf78e
[Kerberos] Avoid vagrant update on precommit (#32416)
This commit avoids dependency during compile on copy keytab to
be present in the generated sources so pre-commit does not
stall for updating vagrant box.

Closes#32387
2018-07-27 12:01:52 +10:00
Nik Everett 11a2189c6f
Switch x-pack full restart to new style Requests (#32294)
In #29623 we added `Request` object flavored requests to the low level
REST client and in #30315 we deprecated the old `performRequest`s. This
changes all calls in the `x-pack:qa:full-cluster-restart` project to use
the new versions.
2018-07-26 12:11:31 -04:00
Tal Levy 86928e7d24 Merge branch 'master' into index-lifecycle 2018-07-25 09:01:35 -07:00
Igor Motov e12e2e0cdd
Add opaque_id to index audit logging (#32260)
Logs opaque_id if it is available with all audit log messages using
index-based audit log.

Closes #31521
2018-07-25 08:07:09 -07:00
Colin Goodheart-Smithe 81faa2fe6b
Merge branch 'master' into index-lifecycle 2018-07-25 10:39:14 +01:00
Tim Vernum 33e3822c5d Fix JarHell on X-Pack protocol
Ports 466bc8c from 6.x
2018-07-25 14:14:19 +10:00
Tal Levy e523030670 Merge branch 'master' into index-lifecycle 2018-07-24 11:23:44 -07:00
Nik Everett e6b9f59e4e
Build: Shadow x-pack:protocol into x-pack:plugin:core (#32240)
This bundles the x-pack:protocol project into the x-pack:plugin:core
project because we'd like folks to consider it an implementation detail
of our build rather than a separate artifact to be managed and depended
on. It is now bundled into both x-pack:plugin:core and
client:rest-high-level. To make this work I had to fix a few things.

Firstly, I had to make PluginBuildPlugin work with the shadow plugin.
In that case we have to bundle only the `shadow` dependencies and the
shadow jar.

Secondly, every reference to x-pack:plugin:core has to use the `shadow`
configuration. Without that the reference is missing all of the
un-shadowed dependencies. I tried to make it so that applying the shadow
plugin automatically redefines the `default` configuration to mirror the
`shadow` configuration which would allow us to use bare project references
to the x-pack:plugin:core project but I couldn't make it work. It'd *look*
like it works but then fail for transitive dependencies anyway. I think
it is still a good thing to do but I don't have the willpower to do it
now.

Finally, I had to fix an issue where Eclipse and IntelliJ didn't properly
reference shadowed transitive dependencies. Neither IDE supports shadowing
natively so they have to reference the shadowed projects. We fix this by
detecting `shadow` dependencies when in "Intellij mode" or "Eclipse mode"
and adding `runtime` dependencies to the same target. This convinces
IntelliJ and Eclipse to play nice.
2018-07-24 11:53:04 -04:00
Yogesh Gaikwad a525c36c60 [Kerberos] Add Kerberos authentication support (#32263)
This commit adds support for Kerberos authentication with a platinum
license. Kerberos authentication support relies on SPNEGO, which is
triggered by challenging clients with a 401 response with the
`WWW-Authenticate: Negotiate` header. A SPNEGO client will then provide
a Kerberos ticket in the `Authorization` header. The tickets are
validated using Java's built-in GSS support. The JVM uses a vm wide
configuration for Kerberos, so there can be only one Kerberos realm.
This is enforced by a bootstrap check that also enforces the existence
of the keytab file.

In many cases a fallback authentication mechanism is needed when SPNEGO
authentication is not available. In order to support this, the
DefaultAuthenticationFailureHandler now takes a list of failure response
headers. For example, one realm can provide a
`WWW-Authenticate: Negotiate` header as its default and another could
provide `WWW-Authenticate: Basic` to indicate to the client that basic
authentication can be used in place of SPNEGO.

In order to test Kerberos, unit tests are run against an in-memory KDC
that is backed by an in-memory ldap server. A QA project has also been
added to test against an actual KDC, which is provided by the krb5kdc
fixture.

Closes #30243
2018-07-24 08:44:26 -06:00
Colin Goodheart-Smithe a2abc86595
Merge branch 'master' into index-lifecycle 2018-07-24 13:13:52 +01:00
Ioannis Kakavas a2dbd83db1
Allow Integ Tests to run in a FIPS-140 JVM (#31989)
* Complete changes for running IT in a fips JVM

- Mute :x-pack:qa:sql:security:ssl:integTest as it
  cannot run in FIPS 140 JVM until the SQL CLI supports key/cert.
- Set default JVM keystore/truststore password in top level build
  script for all integTest tasks in a FIPS 140 JVM
- Changed top level x-pack build script to use keys and certificates
  for trust/key material when spinning up clusters for IT
2018-07-24 12:48:14 +03:00
Christoph Büscher fe6bb75eb4
Rename ranking evaluation `quality_level` to `metric_score` (#32168)
The notion of "quality" is an overloaded term in the search ranking evaluation 
context. Its usually used to decribe certain levels of "good" vs. "bad" of a 
seach result with respect to the users information need. We currently report the 
result of the ranking evaluation as `quality_level` which is a bit missleading.
This changes the response parameter name to `metric_score` which fits better.
2018-07-23 22:25:02 +02:00
Tal Levy 1a1ae5de1e
add qa project for running ILM tests against security (#32218)
This is a bare-bones skeleton for running existing yaml tests with security
enabled. Additional tests which test users and roles should follow
2018-07-23 11:14:41 -07:00
Colin Goodheart-Smithe 5f696e15ea
Merge branch 'master' into index-lifecycle 2018-07-23 14:49:40 +01:00
Christoph Büscher ff87b7aba4
Remove unnecessary warning supressions (#32250) 2018-07-23 11:31:04 +02:00
Ioannis Kakavas aaa8f842d6
Remove BouncyCastle dependency from runtime (#32193)
* Remove BouncyCastle dependency from runtime

This commit introduces a new gradle  project that contains
 the classes that have a dependency on BouncyCastle. For 
the default distribution, It builds  a jar from those and
 in puts it in a subdirectory of lib
 (/tools/security-cli) along with the BouncyCastle jars. 
This directory is then passed in the
ES_ADDITIONAL_CLASSPATH_DIRECTORIES of the CLI tools 
that use these classes.

BouncyCastle is removed as a runtime dependency (remains
as a compileOnly one) from x-pack core and x-pack security.
2018-07-21 00:03:58 +03:00
Luca Cavanna 00a6ad0e9e
Remove aliases resolution limitations when security is enabled (#31952)
Resolving wildcards in aliases expression is challenging as we may end
up with no aliases to replace the original expression with, but if we
replace with an empty array that means _all which is quite the opposite.
Now that we support and serialize the original requested aliases,
whenever aliases are replaced we will be able to know what was
initially requested. `MetaData#findAliases` can then be updated to not
return anything in case it gets empty aliases, but the original aliases
were not empty. That means that empty aliases are interpreted as _all
only if they were originally requested that way.

Relates to #31516
2018-07-20 09:23:32 +02:00
Colin Goodheart-Smithe 573a9e6443
Merge branch 'master' into index-lifecycle 2018-07-20 07:56:57 +01:00
Tim Vernum 6f2b7dc9fe Fix AD / vagrant based tests for #30953
These tests were creating a SSL service that was not aware of the
realm that they were trying to test.
This no longer works.
2018-07-20 12:51:52 +10:00
Tal Levy 1acdd37968 Merge branch 'master' into index-lifecycle 2018-07-19 12:33:10 -07:00
Jim Ferenczi 644a92f158
Fix rollup on date fields that don't support epoch_millis (#31890)
The rollup indexer uses a range query to select the next page
of results based on the last time bucket of the previous round
and the `delay` configured on the rollup job. This query uses
the `epoch_millis` format implicitly but doesn't set the `format`.
This result in errors during the rollup job if the field
definition doesn't allow this format. It can also miss documents
if the format is not accepted but another format in the field
definition is able to parse the query (e.g.: `epoch_second`).
This change ensures that we use `epoch_millis` as the only format
to parse the rollup range query.
2018-07-19 09:34:23 +02:00
Alpar Torok 38e2e1d553
Detect and prevent configuration that triggers a Gradle bug (#31912)
* Detect and prevent configuration that triggers a Gradle bug

As we found in #31862, this can lead to a lot of wasted time as it's not
immediatly obvius what's going on.
Givent how many projects we have it's getting increasingly easier to run
into gradle/gradle#847.
2018-07-19 06:46:58 +00:00
Michael Basnight 63749498ea Remove empty @return from JavaDoc 2018-07-18 13:28:25 -05:00
Colin Goodheart-Smithe 6c77449a87
Merge branch 'master' into index-lifecycle 2018-07-18 11:41:23 +01:00
David Kyle 2bd40a7927
[ML] Wait for aliases in multi-node tests (#32086) 2018-07-17 16:21:53 +01:00
Ioannis Kakavas 9e529d9d58
Enable testing in FIPS140 JVM (#31666)
Ensure our tests can run in a FIPS JVM

JKS keystores cannot be used in a FIPS JVM as attempting to use one
in order to init a KeyManagerFactory or a TrustManagerFactory is not
allowed.( JKS keystore algorithms for private key encryption are not
FIPS 140 approved)
This commit replaces JKS keystores in our tests with the
corresponding PEM encoded key and certificates both for key and trust
configurations.
Whenever it's not possible to refactor the test, i.e. when we are
testing that we can load a JKS keystore, etc. we attempt to
mute the test when we are running in FIPS 140 JVM. Testing for the
JVM is naive and is based on the name of the security provider as
we would control the testing infrastrtucture and so this would be
reliable enough.
Other cases of tests being muted are the ones that involve custom
TrustStoreManagers or KeyStoreManagers, null TLS Ciphers and the
SAMLAuthneticator class as we cannot sign XML documents in the
way we were doing. SAMLAuthenticator tests in a FIPS JVM can be
reenabled with precomputed and signed SAML messages at a later stage.

IT will be covered in a subsequent PR
2018-07-17 10:54:10 +03:00
Yogesh Gaikwad 6717df3c2d
[Test] Modify assert statement for ssl handshake (#32072)
There have been changes in error messages for `SSLHandshakeException`.
This has caused a couple of failures in our tests.
This commit modifies test verification to assert on exception type of
class `SSLHandshakeException`.
There was another issue in Java11 which caused NPE. The bug has now
been fixed on Java11 - early access build 22.
Bug Ref: https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8206355
Enable the skipped tests due to this bug.

Closes #31940
2018-07-17 15:43:03 +10:00
Michael Basnight 637cac9061
Watcher: Store username on watch execution (#31873)
There is currently no way to see what user executed a watch. This commit
adds the decrypted username to each execution in the watch history, in a
new field "user".

Closes #31772
2018-07-16 14:20:16 -05:00
Zachary Tong 59191b4998
[Rollup] Replace RollupIT with a ESRestTestCase version (#31977)
The old RollupIT was a node IT, an flaky for a number of reasons.
This new version is an ESRestTestCase and should be a little more robust.

This was added to the multi-node QA tests as that seemed like the most
appropriate location.  It didn't seem necessary to create a whole new
QA module.

Note: The only test that was ported was the "Big" test for validating
a larger dataset.  The rest of the tests are represented in existing
yaml tests.

Closes #31258
Closes #30232
Related to #30290
2018-07-16 10:47:46 -04:00
Jim Ferenczi a14db2f9d6 [Test] Mute MlJobIT#testDeleteJobAfterMissingAliases
Relates #32034
2018-07-16 10:53:51 +02:00
Colin Goodheart-Smithe 5d3a53843a
Merge branch 'master' into index-lifecycle 2018-07-16 09:30:07 +01:00
Tim Vernum edbea73f24 Fix broken OpenLDAP Vagrant QA test
This was broken due to c662565 but the problem didn't get detected as
CI builds typically don't run vagrant tests
2018-07-16 15:43:41 +10:00
Christoph Büscher 82cdb574cf
Mute ML AutodetectMemoryLimitIT#testTooManyPartitions on Windows (#32044)
Adding assumption to not run this test on Windows temporarily.

Relates to #32033
2018-07-13 17:41:28 +02:00
Zachary Tong b7f07f03ed
[Rollup] Use composite's missing_bucket (#31402)
We can leverage the composite agg's new `missing_bucket` feature on
terms groupings.  This means the aggregation criteria used in the indexer
will now return null buckets for missing keys.  

Because all buckets are now returned (even if a key is null),
we can guarantee correct doc counts with
"combined" jobs (where a job rolls up multiple schemas).  This was
previously impossible since composite would ignore documents that
didn't have _all_ the keys, meaning non-overlapping schemas would
cause composite to return no buckets.

Note: date_histo does not use `missing_bucket`, since a timestamp is
always required.

The docs have been adjusted to recommend a single, combined job.  It
also makes reference to the previous issue to help users that are upgrading
(rather than just deleting the sections).
2018-07-13 10:07:42 -04:00
Tim Vernum c662565f29
Access SSL contexts using names instead of Settings (#30953)
Historically we have loaded SSL objects (such as SSLContext,
SSLIOSessionStrategy) by passing in the SSL settings, constructing a
new SSL configuration from those settings and then looking for a
cached object that matches those settings.

The primary issue with this approach is that it requires a fully
configured Settings object to be available any time the SSL context
needs to be loaded. If the Settings include SecureSettings (such as
passwords for keys or keystores) then this is not true, and the cached
SSL object cannot be loaded at runtime.

This commit introduces an alternative approach of naming every cached
ssl configuration, so that it is possible to load the SSL context for
a named configuration (such as "xpack.http.ssl"). This means that the
calling code does not need to have ongoing access to the secure
settings that were used to load the configuration.

This change also allows monitoring exporters to use SSL passwords
from secure settings, however an exporter that uses a secure SSL setting
(e.g. truststore.secure_password) may not have its SSL settings updated
dynamically (this is prevented by a settings validator).
Exporters without secure settings can continue to be defined and updated
dynamically.
2018-07-13 16:40:09 +10:00
Tom Veasey de2eb85aaf
[ML] Re-enable memory limit integration tests (#31328) 2018-07-12 20:56:05 +01:00
David Kyle 6fcd606536
[ML] Get ForecastRequestStats doc in RestoreModelSnapshotIT (#31973) 2018-07-12 13:59:34 +01:00
Andrei Stefan edf83c1d87
SQL: Add support for single parameter text manipulating functions (#31874)
Added support for ASCII, BIT_LENGTH, CHAR, CHAR_LENGTH, LCASE, LENGTH, LTRIM, RTRIM, SPACE, UCASE functions.
Wherever Painless scripting is necessary (WHERE conditions, ORDER BY etc), those scripts are being used.
2018-07-12 15:05:42 +03:00
Dimitris Athanasiou 2cfe703299
[ML] Ensure immutability of MlMetadata (#31957)
The test failure in #31916 revealed that updating
rules on a job was modifying the detectors list
in-place. That meant the old cluster state and the
updated cluster state had no difference and thus the
change was not propagated to non-master nodes.

This commit fixes that and also reviews all of ML
metadata in order to ensure immutability.

Closes #31916
2018-07-12 11:49:06 +01:00
Martijn van Groningen ae5c70e603
muted tests due to #31940 2018-07-12 11:51:33 +02:00
Colin Goodheart-Smithe cd89ca2685
Merge branch 'master' into index-lifecycle
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/XPackSetti
ngs.java
/Users/colings86/dev/work/git/elasticsearch/.git/worktrees/elasticsearch
-ilm/MERGE_HEAD

buildSrc/src/main/groovy/org/elasticsearch/gradle/vagrant/VagrantTestPlu
gin.groovy
client/rest-high-level/src/main/java/org/elasticsearch/client/RestHighLe
velClient.java
client/rest-high-level/src/main/java/org/elasticsearch/client/XPackClien
t.java
client/rest-high-level/src/test/java/org/elasticsearch/client/PingAndInf
oIT.java
client/rest-high-level/src/test/java/org/elasticsearch/client/documentat
ion/MiscellaneousDocumentationIT.java
docs/reference/analysis/tokenfilters/synonym-graph-tokenfilter.asciidoc
docs/reference/analysis/tokenfilters/synonym-tokenfilter.asciidoc
modules/ingest-common/src/test/java/org/elasticsearch/ingest/common/Grok
ProcessorTests.java
modules/ingest-common/src/test/java/org/elasticsearch/ingest/common/Json
ProcessorTests.java
qa/vagrant/src/main/java/org/elasticsearch/packaging/test/ArchiveTestCas
e.java
qa/vagrant/src/main/java/org/elasticsearch/packaging/util/Archives.java
qa/vagrant/src/main/java/org/elasticsearch/packaging/util/Cleanup.java
qa/vagrant/src/main/java/org/elasticsearch/packaging/util/FileUtils.java
qa/vagrant/src/main/java/org/elasticsearch/packaging/util/Installation.j
ava
qa/vagrant/src/main/java/org/elasticsearch/packaging/util/Platforms.java
qa/vagrant/src/main/java/org/elasticsearch/packaging/util/ServerUtils.ja
va
qa/vagrant/src/main/java/org/elasticsearch/packaging/util/Shell.java
qa/vagrant/src/test/resources/packaging/tests/20_tar_package.bats
server/src/main/java/org/elasticsearch/index/analysis/ESSolrSynonymParse
r.java
server/src/main/java/org/elasticsearch/index/analysis/ESWordnetSynonymPa
rser.java
server/src/main/java/org/elasticsearch/index/analysis/SynonymGraphTokenF
ilterFactory.java
server/src/main/java/org/elasticsearch/index/analysis/SynonymTokenFilter
Factory.java
server/src/test/java/org/elasticsearch/index/analysis/ESSolrSynonymParse
rTests.java
server/src/test/java/org/elasticsearch/index/analysis/ESWordnetSynonymPa
rserTests.java
server/src/test/java/org/elasticsearch/index/shard/IndexShardIT.java
test/framework/src/main/java/org/elasticsearch/ingest/IngestDocumentMatc
her.java
test/framework/src/test/java/org/elasticsearch/ingest/IngestDocumentMatc
herTests.java
x-pack/plugin/core/src/main/java/org/elasticsearch/license/License.java
x-pack/plugin/core/src/main/java/org/elasticsearch/license/XPackLicenseS
tate.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/XPackClien
tPlugin.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/XPackField
.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/XPackSetti
ngs.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/beats/Beat
sFeatureSetUsage.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/a
uthz/permission/Role.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/a
uthz/store/ReservedRolesStore.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/action/Tra
nsportXPackInfoActionTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/security/a
uthz/store/ReservedRolesStoreTests.java
x-pack/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/datafeed/MlRem
oteLicenseChecker.java
x-pack/plugin/ml/src/test/java/org/elasticsearch/xpack/ml/datafeed/MlRem
oteLicenseCheckerTests.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/au
thz/store/CompositeRolesStore.java
x-pack/protocol/src/main/java/org/elasticsearch/protocol/xpack/XPackInfo
Response.java
x-pack/protocol/src/main/java/org/elasticsearch/protocol/license/License
Status.java ->
x-pack/protocol/src/main/java/org/elasticsearch/protocol/xpack/license/L
icenseStatus.java
x-pack/protocol/src/main/java/org/elasticsearch/protocol/license/package
-info.java ->
x-pack/protocol/src/main/java/org/elasticsearch/protocol/xpack/license/p
ackage-info.java
x-pack/protocol/src/main/java/org/elasticsearch/protocol/security/packag
e-info.java ->
x-pack/protocol/src/main/java/org/elasticsearch/protocol/xpack/security/
package-info.java
x-pack/protocol/src/main/java/org/elasticsearch/protocol/watcher/package
-info.java ->
x-pack/protocol/src/main/java/org/elasticsearch/protocol/xpack/watcher/p
ackage-info.java
x-pack/protocol/src/test/java/org/elasticsearch/protocol/xpack/XPackInfo
ResponseTests.java
x-pack/protocol/src/test/java/org/elasticsearch/protocol/license/License
StatusTests.java ->
x-pack/protocol/src/test/java/org/elasticsearch/protocol/xpack/license/L
icenseStatusTests.java
x-pack/qa/rolling-upgrade/src/test/resources/rest-api-spec/test/mixed_cl
uster/10_basic.yml
2018-07-11 10:26:44 +01:00
Boaz Leskes c6666fc6cb rolling upgrade should use a replica to prevent relocations while running a scroll
A scroll holds a reference to the shard store. If the cluster is moving shards
around that reference can prevent a shard from relocating back to node it used
to be on, causing test failures.

Closes #31827
2018-07-11 08:13:38 +02:00
Tal Levy 952859d87d Merge branch 'master' into index-lifecycle 2018-07-10 10:55:50 -04:00
David Kyle 1f0421aa69 [ML] Mute failing DetectionRulesIT.testCondition() test 2018-07-10 11:47:47 +01:00
David Kyle 991d17c8c4
[ML] Switch native QA tests to a 3 node cluster (#31757) 2018-07-10 08:26:48 +01:00
Tal Levy e9cd0556c1 Merge branch 'master' into index-lifecycle 2018-07-09 19:44:12 -04:00
Costin Leau 9ffb26ab02
SQL: Remove restriction for single column grouping (#31818)
For historical reasons SQL restricts GROUP BY to only one field.
This commit removes the restriction and improves the test suite with
multi group by tests.

Close #31793
2018-07-06 20:55:27 +03:00
Costin Leau 6f5d8caeb7
QA: build improvements related to SQL projects (#31862)
Register SQL as an xpackModule
Specify group for SQL QA to disambiguate projects (otherwise due to an
old Gradle bug (https://github.com/gradle/gradle/issues/847) any
subprojects under SQL QA will not be able to refer to SQL xpackModule

Co-authored-by: Alpar Torok <torokalpar@gmail.com>
2018-07-06 18:24:12 +03:00
Costin Leau 89cb0872cf
SQL: Fix incorrect HAVING equality (#31820)
Fix bug that causes `HAVING a = b` to be translated ad-litteram in
Painless which uses `==` for equality checks not `=`.

Close #31796
2018-07-06 12:46:57 +03:00
Armin Braun ed41d4f566
Fix not waiting for Netty ThreadDeathWatcher in IT (#31758)
Same problem and solution as in #30763
Fixes #30547
2018-07-03 18:55:23 +02:00
Costin Leau de9e56aa01
DOC: Add examples to the SQL docs (#31633)
Significantly improve the example snippets in the documentation.
The examples are part of the test suite and checked nightly.
To help readability, the existing dataset was extended (test_emp renamed
to emp plus library).
Improve output of JDBC tests to be consistent with the CLI
Add lenient flag to JDBC asserts to allow type widening (a long is
equivalent to a integer as long as the value is the same).
2018-07-03 16:56:31 +03:00
Andrei Stefan ce78925732
JDBC: Fix stackoverflow on getObject and timestamp conversion (#31735)
StackOverflowError fix in JdbcResultSet getObject method.
Fix Timestamp conversion bug when getting the value of a time column.
2018-07-03 13:55:33 +03:00
Dimitris Athanasiou 8e838ea12e
[ML] Validate ML filter_id (#31535)
Like job and datafeed ids, the filter id
should be validated with the same rules
to avoid document ids that can be problematic.
2018-07-02 12:48:04 +01:00
Konrad Beiske 2971dd56ca Enable setting client path prefix to / (#30119)
Some proxies require all requests to have paths starting with / since
there are no relative paths at the HTTP connection level. Elasticsearch
assumes paths are absolute. In order to run rest tests against a cluster
behind such a proxy, set the system property
tests.rest.client_path_prefix to /.
2018-07-01 13:42:03 -04:00
Ioannis Kakavas db6b33978e
Configurable password hashing algorithm/cost (#31234)
Make password hashing algorithm/cost configurable for the 
stored passwords of users for the realms that this applies
(native, reserved). Replaces predefined choice of bcrypt with
cost factor 10.
This also introduces PBKDF2 with configurable cost
(number of iterations) as an algorithm option for password hashing
both for storing passwords and for the user cache.
Password hash validation algorithm selection takes into
consideration the stored hash prefix and only a specific number
of algorithnm and cost factor options for brypt and pbkdf2 are 
whitelisted and can be selected in the relevant setting.
2018-06-28 19:39:39 +03:00
Igor Motov 0b1a0641ef
SQL: Refactor package names of sql-proto and sql-shared-proto projects (#31622)
- renames project `sql-proto` to `sql-action`,
- renames package `sql.plugin` to `sql.action`
- renames project `sql-shared-client` to `sql-client`
- renames package `sql.shard.client` to `sql.client`
- renames project `sql-shared-proto` to `sql-proto`

Closes #31593
2018-06-28 05:56:16 -07:00
Alpar Torok 8557bbab28
Upgrade gradle wrapper to 4.8 (#31525)
* Move to Gradle 4.8 RC1

* Use latest version of plugin

The current does not work with Gradle 4.8 RC1

* Switch to Gradle GA

* Add and configure build compare plugin

* add work-around for https://github.com/gradle/gradle/issues/5692

* work around https://github.com/gradle/gradle/issues/5696

* Make use of Gradle build compare with reference project

* Make the manifest more compare friendly

* Clear the manifest in compare friendly mode

* Remove animalsniffer from buildscript classpath

* Fix javadoc errors

* Fix doc issues

* reference Gradle issues in comments

* Conditionally configure build compare

* Fix some more doclint issues

* fix typo in build script

* Add sanity check to make sure the test task was replaced

Relates to #31324. It seems like Gradle has an inconsistent behavior and
the taks is not always replaced.

* Include number of non conforming tasks in the exception.

* No longer replace test task, create implicit instead

Closes #31324. The issue has full context in comments.

With this change the `test` task becomes nothing more than an alias for `utest`.
Some of the stand alone tests that had a `test` task now have `integTest`, and a
few of them that used to have `integTest` to run multiple tests now only
have `check`.
This will also help separarate unit/micro tests from integration tests.

* Revert "No longer replace test task, create implicit instead"

This reverts commit f1ebaf7d93e4a0a19e751109bf620477dc35023c.

* Fix replacement of the test task

Based on information from gradle/gradle#5730 replace the task taking
into account the task providres.
Closes #31324.

* Only apply build comapare plugin if needed

* Make sure test runs before integTest

* Fix doclint aftter merge

* PR review comments

* Switch to Gradle 4.8.1 and remove workaround

* PR review comments

* Consolidate task ordering
2018-06-28 08:13:21 +03:00
Nik Everett d0c276c456
QA: Merge query-builder-bwc to restart test (#30979)
Merges the `query-builder-bwc` qa project into the
`full-cluster-restart` qa project, saving a cluster starts on every
build and *many* cluster starts on `./gradlew bwcTests`.
2018-06-27 16:37:04 -04:00
Nik Everett 232c71b6bf
QA: Create xpack yaml features (#31403)
This creates a YAML test "features" that indices if the cluster being
tested has xpack installed (`xpack`) or if it does *not* have xpack
installed (`no_xpack`). It uses those features to centralize skipping
a few tests that fail if xpack is installed.

The plan is to use this in a followup to skip docs tests that require
xpack when xpack is not installed. We *plan* to use the declaration
of required license level on the docs page to generate the required
`skip`.

Closes #30933.
2018-06-26 09:26:48 -04:00
Tanguy Leroux be9292cac6
[Test] Add full cluster restart test for Rollup (#31533)
This pull request adds a full cluster restart test for a Rollup job. 
The test creates and starts a Rollup job on the cluster and checks 
that the job already exists and is correctly started on the upgraded 
cluster.

This test allows to test that the persistent task state is correctly 
parsed from the cluster state after the upgrade, as the status field 
has been renamed to state in #31031.

The test undercovers a ClassCastException that can be thrown in 
the RollupIndexer when the timestamp as a very low value that fits 
into an integer. When it's the case, the value is parsed back as an 
Integer instead of Long object and (long) position.get(rollupFieldName) 
fails.
2018-06-26 10:07:25 +02:00
Tal Levy 1d11fdaad8 Merge branch 'master' into index-lifecycle 2018-06-22 10:11:15 -07:00
Dimitris Athanasiou c6cbc99f9c
[ML] Add ML filter update API (#31437)
This adds an api to allow updating a filter:

POST _xpack/ml/filters/{filter_id}/_update

The request body may have:

- description: setting a new description
- add_items: a list of the items to add
- remove_items: a list of the items to remove

This commit also changes the PUT filter api to
error when the filter_id is already used. As
now there is an api for updating filters, the
put api should only be used to create new ones.

Also, updating a filter results into a notification
message auditing the change for every job that is
using that filter.
2018-06-22 15:13:31 +01:00
Tal Levy 5d52f297cc Merge branch 'master' into index-lifecycle 2018-06-18 10:06:47 -07:00
Ryan Ernst f3297ed23a
Packaging: Remove windows bin files from the tar distribution (#30596)
This commit removes windows specific files from the tar distribution.
Windows users use the zip, linux users use the tar.
2018-06-18 19:02:51 +02:00
Dimitris Athanasiou c6a5a6d924
[ML] Put ML filter API response should contain the filter (#31362) 2018-06-15 21:15:35 +01:00
Tal Levy 85d0613ea6 Merge branch 'master' into index-lifecycle 2018-06-15 08:28:23 -07:00
David Kyle ca00deb8ad [ML] Re-enable tests muted in #30982 2018-06-15 10:54:13 +01:00
Tanguy Leroux 992c7889ee
Uncouple persistent task state and status (#31031)
This pull request removes the relationship between the state 
of persistent task (as stored in the cluster state) and the status 
of the task (as reported by the Task APIs and used in various 
places) that have been confusing for some time (#29608).

In order to do that, a new PersistentTaskState interface is added. 
This interface represents the persisted state of a persistent task. 
The methods used to update the state of persistent tasks are 
renamed: updatePersistentStatus() becomes updatePersistentTaskState() 
and now takes a PersistentTaskState as a parameter. The 
Task.Status type as been changed to PersistentTaskState in all 
places were it make sense (in persistent task customs in cluster 
state and all other methods that deal with the state of an allocated 
persistent task).
2018-06-15 09:26:47 +02:00
Tal Levy 2af05e5480 Merge branch 'master' into index-lifecycle 2018-06-14 15:32:41 -07:00
Dimitris Athanasiou 9b293275af
[ML] Add description to ML filters (#31330)
This adds a `description` to ML filters in order
to allow users to describe their filters in a human
readable form which is also editable (filter updates
to be added shortly).
2018-06-14 16:52:32 +01:00
Colin Goodheart-Smithe ce38127a16
Merge branch 'master' into index-lifecycle 2018-06-14 13:02:24 +01:00
Luca Cavanna ce245a7320
Remove RestGetAllAliasesAction (#31308)
We currently have a specific REST action to retrieve all aliaes, which
uses internally the get index API. This doesn't seem to be required
anymore though as the existing RestGetAliaesAction could as well take
the requests with no indices and aliases specified.

This commit removes the RestGetAllAliasesAction in favour of using
RestGetAliasesAction also for requests that don't specify indices nor
aliases. Similar to #31129.
2018-06-14 11:21:16 +02:00
Tanguy Leroux 4d7447cb5e
Reenable Checkstyle's unused import rule (#31270) 2018-06-14 09:52:46 +02:00
Colin Goodheart-Smithe 9fd4f3f854
Merge branch 'master' into index-lifecycle 2018-06-13 13:41:17 +01:00
Tom Veasey 66f7dd2c4d
[ML] Update test thresholds to account for changes to memory control (#31289)
To avoid temporary failures, this also disables these tests until elastic/ml-cpp#122 is committed.
2018-06-13 13:12:53 +01:00
Dimitris Athanasiou 5c77ebe89d
[ML] Implement new rules design (#31110)
Rules allow users to supply a detector with domain
knowledge that can improve the quality of the results.
The model detects statistically anomalous results but it
has no knowledge of the meaning of the values being modelled.

For example, a detector that performs a population analysis
over IP addresses could benefit from a list of IP addresses
that the user knows to be safe. Then anomalous results for
those IP addresses will not be created and will not affect
the quantiles either.

Another example would be a detector looking for anomalies
in the median value of CPU utilization. A user might want
to inform the detector that any results where the actual
value is less than 5 is not interesting.

This commit introduces a `custom_rules` field to the `Detector`.
A detector may have multiple rules which are combined with `or`.

A rule has 3 fields: `actions`, `scope` and `conditions`.

Actions is a list of what should happen when the rule applies.
The current options include `skip_result` and `skip_model_update`.
The default value for `actions` is the `skip_result` action.

Scope is optional and allows for applying filters on any of the
partition/over/by field. When not defined the rule applies to
all series. The `filter_id` needs to be specified to match the id
of the filter to be used. Optionally, the `filter_type` can be specified
as either `include` (default) or `exclude`. When set to `include`
the rule applies to entities that are in the filter. When set to
`exclude` the rule only applies to entities not in the filter.

There may be zero or more conditions. A condition requires `applies_to`,
`operator` and `value` to be specified. The `applies_to` value can be
either `actual`, `typical` or `diff_from_typical` and it specifies
the numerical value to which the condition applies. The `operator`
(`lt`, `lte`, `gt`, `gte`) and `value` complete the definition.
Conditions are combined with `and` and allow to specify numerical
conditions for when a rule applies.

A rule must either have a scope or one or more conditions. Finally,
a rule with scope and conditions applies when all of them apply.
2018-06-13 11:20:38 +01:00
Jason Tedor 0bfd18cc8b
Revert upgrade to Netty 4.1.25.Final (#31282)
This reverts upgrading to Netty 4.1.25.Final until we have a cleaner
solution to dealing with the object cleaner thread.
2018-06-12 19:26:18 -04:00
Colin Goodheart-Smithe ff80202353
Merge branch 'master' into index-lifecycle 2018-06-12 12:52:04 +01:00
Dimitris Athanasiou 5f84e18c72
[ML][TEST] Mute tests using rules (#31204)
This is in preparation of pushing the new
rules design in the `ml-cpp` side. These
tests will be switched on again after merging
in the new rules implementation.
2018-06-12 11:36:26 +01:00
Tal Levy 0541684df3 Merge branch 'master' into index-lifecycle 2018-06-11 17:20:05 -07:00
Jason Tedor 563141c6c9
Upgrade to Netty 4.1.25.Final (#31232)
This commit upgrades us to Netty 4.1.25. This upgrade is more
challenging than past upgrades, all because of a new object cleaner
thread that they have added. This thread requires an additional security
permission (set context class loader, needed to avoid leaks in certain
scenarios). Additionally, there is not a clean way to shutdown this
thread which means that the thread can fail thread leak control during
tests. As such, we have to filter this thread from thread leak control.
2018-06-11 16:55:07 -04:00
Jason Tedor cb952bd9ec
Enable custom credentials for core REST tests (#31235)
The core REST tests with security currently use a hardcoded username and
password. This is not amenable to running these tests in scenarios where
the user controls the creation of the cluster and owns the credentials
for this cluster. This commit enables running the core REST tests with
security with a custom username and password.
2018-06-11 16:53:40 -04:00
Tanguy Leroux bf58660482
Remove all unused imports and fix CRLF (#31207)
The X-Pack opening and the recent other refactorings left a lot of 
unused imports in the codebase. This commit removes them all.
2018-06-11 15:12:12 +02:00
Tanguy Leroux a1916658a9 [Tests] Fix self-referencing tests
This commit adapts some test after #31044 has been merged.
2018-06-11 12:45:27 +02:00
Colin Goodheart-Smithe 9ee492a3f0
Merge branch 'master' into index-lifecycle 2018-06-11 09:47:26 +01:00
Jason Tedor 65c107b47d
Fix unknown licenses (#31223)
The goal of this commit is to address unknown licenses when producing
the dependencies info report. We have two different checks that we run
on licenses. The first check is whether or not we have stashed a copy of
the license text for a dependency in the repository. The second is to
map every dependency to a license type (e.g., BSD 3-clause). The problem
here is that the way we were handling licenses in the second check
differs from how we handle licenses in the first check. The first check
works by finding a license file with the name of the artifact followed
by the text -LICENSE.txt. Yet in some cases we allow mapping an artifact
name to another name used to check for the license (e.g., we map
lucene-.* to lucene, and opensaml-.* to shibboleth. The second check
understood the first way of looking for a license file but not the
second way. So in this commit we teach the second check about the
mappings from artifact names to license names. We do this by copying the
configuration from the dependencyLicenses task to the dependenciesInfo
task and then reusing the code from the first check in the second
check. There were some other challenges here though. For example,
dependenciesInfo was checking too many dependencies. For now, we should
only be checking direct dependencies and leaving transitive dependencies
from another org.elasticsearch artifact to that artifact (we want to do
this differently in a follow-up). We also want to disable
dependenciesInfo for projects that we do not publish, users only care
about licenses they might be exposed to if they use our assembled
products. With all of the changes in this commit we have eliminated all
unknown licenses. A follow-up will enforce that when we add a new
dependency it does not get mapped to unknown, these will be forbidden in
the future. Therefore, with this change and earlier changes are left
having no unknown licenses and two custom licenses; custom here means it
does not map to an SPDX license type. Those two licenses are xz and
ldapsdk. A future change will not allow additional custom licenses
unless they are explicitly whitelisted. This ensures that if a new
dependency is added it is mapped to an SPDX license or mapped to custom
because it does not have an SPDX license.
2018-06-09 07:28:41 -04:00
Igor Motov 01140a3ad8
SQL: Make a single JDBC driver jar (#31012)
Replaces zip archive containing multiple jars with a single JDBC driver jar 
that shades all external dependencies.

Closes #29856
2018-06-08 10:15:28 -04:00
Hendrik Muhs 253b998681
flush job to ensure all results have been written (#31187)
flush ml job to ensure all results have been written

fixes #31173
2018-06-08 07:51:45 +02:00
Nik Everett dfcc939ef8 QA: Better seed nodes for rolling restart
Use all running nodes as unicast seeds in the rolling restart tests to
avoid a race between pinging and the tests. Without this if the tests
are too fast then when a new node comes up and pings its single
configured seed node that node *might* not have a ping from the other
running node.
2018-06-07 13:30:37 -04:00
Nik Everett 56207ea43d QA: Set better node names on rolling restart tests
These should help with debugging failures.
2018-06-07 11:25:41 -04:00
Nik Everett dc4bb62a78 QA: Remove mistaken timeout
I pushed a test that `assertBusy`s for a whole hour accidentally. I was
testing something and forgot to revert my local hack but caught it on
backport. This removes it.
2018-06-06 13:51:54 -04:00
Tal Levy 6b6882900d Merge branch 'master' into index-lifecycle 2018-06-06 10:08:03 -07:00
Nik Everett 7c59e7690e
QA: Switch xpack rolling upgrades to three nodes (#31112)
This is much more realistic and can find more issues. This causes the
"mixed cluster" tests to be run twice so I had to fix the tests to work
in that case. In most cases I did as little as possible to get them
working but in a few cases I went a little beyond that to make them
easier for me to debug while getting them to work. My test changes:

1. Remove the "basic indexing" tests and replace them with a copy of the
tests used in the OSS. We have no way of sharing code between these two
projects so for now I copy.
2. Skip the a few tests in the "one third" upgraded scenario:
  * creating a scroll to be reused when the cluster is fully upgraded
  * creating some ml data to be used when the cluster is fully ugpraded
3. Drop many "assert yellow and that the cluster has two nodes"
assertions. These assertions duplicate those made by the wait condition
and they fail now that we have three nodes.
4. Switch many "assert green and that the cluster has two nodes" to 3
nodes. These assertions are unique from the wait condition and, while
I imagine they aren't required in all cases, now is not the time to
find that out. Thus, I made them work.
5. Rework the index audit trail test so it is more obvious that it is
the same test expecting different numbers based on the shape of the
cluster. The conditions for which number are expected are fairly
complex because the index audit trail is shut down until the template
for it is upgraded and the template is upgraded when a master node is
elected that has the new version of the software.
6. Add some more information to debug the index audit trail test because
it helped me figure out what was going on.

I also dropped the `waitCondition` from the `rolling-upgrade-basic`
tests because it wasn't needed.

Closes #25336
2018-06-06 11:59:16 -04:00
Tal Levy 6b0008c668 Merge branch 'master' into index-lifecycle 2018-06-05 08:09:02 -07:00
Hendrik Muhs 5e48ba7cbd
run overflow forecast a 2nd time as regression test for elastic/ml-cpp#110 (#30969)
Improve test to run overflow forecast a 2nd time as regression test for elastic/ml-cpp#110
2018-06-05 08:52:06 +02:00
Tal Levy fbfb4ff8c7 Merge branch 'master' into index-lifecycle 2018-06-01 11:44:38 -07:00
Dimitris Athanasiou 9141108334
[ML][TEST] Fix bucket count assertion in all tests in ModelPlotsIT (#31026)
This fixes the last remaining test that was missed in #30717.

Closes #30715
2018-06-01 10:51:12 +01:00
Colin Goodheart-Smithe 830198fa00
Merge branch 'master' into index-lifecycle 2018-06-01 09:04:22 +01:00
Luca Cavanna 70749e01c4
Cross Cluster Search: preserve remote status code (#30976)
In case an error is returned when calling search_shards on a remote
cluster, which will lead to throwing an exception in the coordinating
 node, we should make sure that the status code returned by the
 coordinating node is the same as the one returned by the remote
 cluster. Up until now a 500 - Internal Server Error was always
 returned. This commit changes this behaviour so that for instance if an
 index is not found, which causes an 404, a 404 is also returned by the
 coordinating node to the client.

 Closes #27461
2018-06-01 08:53:53 +02:00
Nik Everett b225f5e5c6
HLRest: Allow caller to set per request options (#30490)
This modifies the high level rest client to allow calling code to
customize per request options for the bulk API. You do the actual
customization by passing a `RequestOptions` object to the API call
which is set on the `Request` that is generated by the high level
client. It also makes the `RequestOptions` a thing in the low level
rest client. For now that just means you use it to customize the
headers and the `httpAsyncResponseConsumerFactory` and we'll add
node selectors and per request timeouts in a follow up.

I only implemented this on the bulk API because it is the first one
in the list alphabetically and I wanted to keep the change small
enough to review. I'll convert the remaining APIs in a followup.
2018-05-31 13:59:52 -04:00
Ryan Ernst 46e8d97813
Core: Remove RequestBuilder from Action (#30966)
This commit removes the RequestBuilder generic type from Action. It was
needed to be used by the newRequest method, which in turn was used by
client.prepareExecute. Both of these methods are now removed, along with
the existing users of prepareExecute constructing the appropriate
builder directly.
2018-05-31 16:15:00 +02:00
Colin Goodheart-Smithe 976935967b
Merge branch 'master' into index-lifecycle 2018-05-31 14:29:00 +01:00
David Roberts 0ff2c605b6 [CI] Mute Ml rolling upgrade test for mixed cluster too
It can fail in either the mixed cluster or the upgraded cluster,
so it needs to be muted in both.

Tracked by #30982
2018-05-31 11:17:18 +01:00
Igor Motov 8e4ab82e3d [CI] Mute Ml rolling upgrade tests
Tracked by #30982
2018-05-30 21:30:18 -04:00
Christoph Büscher 1ea9f11b03
Change ScriptException status to 400 (bad request) (#30861)
Currently failures to compile a script usually lead to a ScriptException, which
inherits the 500 INTERNAL_SERVER_ERROR from ElasticsearchException if it does
not contain another root cause. Instead, this should be a 400 Bad Request error.
This PR changes this more generally for script compilation errors by changing 
ScriptException to return 400 (bad request) as status code.

Closes #12315
2018-05-30 14:00:07 +02:00
Tal Levy bdf70e4f2f Merge branch 'master' into index-lifecycle 2018-05-29 12:29:53 -07:00
Andy Bristol ba8bb1d4a1 [test] packaging test logging for suse distros 2018-05-29 11:06:57 -07:00
Ioannis Kakavas a8faf9768a
Limit the scope of BouncyCastle dependency (#30358)
Limits the scope of the runtime dependency on
BouncyCastle so that it can be eventually removed.

* Splits functionality related to reading and generating certificates
and keys in two utility classes so that reading certificates and
keys doesn't require BouncyCastle.
* Implements a class for parsing PEM Encoded key material (which also
adds support for reading PKCS8 encoded encrypted private keys).
* Removes BouncyCastle dependency for all of our test suites(except
for the tests that explicitly test certificate generation) by using
pre-generated keys/certificates/keystores.
2018-05-29 19:11:09 +03:00
Colin Goodheart-Smithe 2ccb4df005
Merge branch 'master' into index-lifecycle 2018-05-29 09:47:43 +01:00
Igor Motov dbb2e8143c
SQL: Remove the last remaining server dependencies from jdbc (#30771)
Removes the last remaining server dependencies from jdbc client. In order to do that it introduces the new project sql-shared-proto that contains only XContent-serializable classes. HTTP Client and JDBC now depend only on sql-shared-proto. I had to keep the original sql-proto project since it is used as a dependency by sql-cli and security integration tests.

Relates #29856
2018-05-25 15:41:41 -04:00
Colin Goodheart-Smithe 84c4598e47
Merge branch 'master' into index-lifecycle
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/XPackClien
tPlugin.java
/Users/colings86/dev/work/git/elasticsearch/.git/worktrees/elasticsearch
-ilm/MERGE_HEAD

buildSrc/src/main/groovy/org/elasticsearch/gradle/vagrant/VagrantTestPlu
gin.groovy
docs/reference/migration/migrate_7_0/settings.asciidoc
modules/transport-netty4/src/main/java/org/elasticsearch/http/netty4/Net
ty4HttpChannel.java
modules/transport-netty4/src/main/java/org/elasticsearch/http/netty4/Net
ty4HttpPipeliningHandler.java
modules/transport-netty4/src/main/java/org/elasticsearch/http/netty4/Net
ty4HttpRequestHandler.java
modules/transport-netty4/src/main/java/org/elasticsearch/http/netty4/Net
ty4HttpResponse.java
modules/transport-netty4/src/main/java/org/elasticsearch/http/netty4/Net
ty4HttpServerTransport.java
modules/transport-netty4/src/main/java/org/elasticsearch/http/netty4/pip
elining/HttpPipelinedRequest.java
modules/transport-netty4/src/main/java/org/elasticsearch/http/netty4/pip
elining/HttpPipelinedResponse.java
modules/transport-netty4/src/main/java/org/elasticsearch/http/netty4/pip
elining/HttpPipeliningHandler.java
modules/transport-netty4/src/test/java/org/elasticsearch/http/netty4/Net
ty4HttpChannelTests.java
modules/transport-netty4/src/test/java/org/elasticsearch/http/netty4/pip
elining/Netty4HttpPipeliningHandlerTests.java ->
modules/transport-netty4/src/test/java/org/elasticsearch/http/netty4/Net
ty4HttpPipeliningHandlerTests.java
modules/transport-netty4/src/test/java/org/elasticsearch/http/netty4/Net
ty4HttpServerPipeliningTests.java
modules/transport-netty4/src/test/java/org/elasticsearch/http/netty4/Net
ty4PipeliningEnabledIT.java ->
modules/transport-netty4/src/test/java/org/elasticsearch/http/netty4/Net
ty4PipeliningIT.java
plugins/transport-nio/src/main/java/org/elasticsearch/http/nio/HttpReadW
riteHandler.java
plugins/transport-nio/src/main/java/org/elasticsearch/http/nio/HttpWrite
Operation.java
plugins/transport-nio/src/main/java/org/elasticsearch/http/nio/NettyAdap
tor.java
plugins/transport-nio/src/main/java/org/elasticsearch/http/nio/NettyList
ener.java
plugins/transport-nio/src/main/java/org/elasticsearch/http/nio/NioHttpCh
annel.java
plugins/transport-nio/src/main/java/org/elasticsearch/http/nio/NioHttpPi
peliningHandler.java
plugins/transport-nio/src/main/java/org/elasticsearch/http/nio/NioHttpRe
sponse.java
plugins/transport-nio/src/main/java/org/elasticsearch/http/nio/NioHttpSe
rverTransport.java
plugins/transport-nio/src/test/java/org/elasticsearch/NioIntegTestCase.j
ava
plugins/transport-nio/src/test/java/org/elasticsearch/http/nio/HttpReadW
riteHandlerTests.java
plugins/transport-nio/src/test/java/org/elasticsearch/http/nio/NioHttpPi
peliningHandlerTests.java
modules/transport-netty4/src/test/java/org/elasticsearch/http/netty4/Net
ty4PipeliningDisabledIT.java ->
plugins/transport-nio/src/test/java/org/elasticsearch/http/nio/NioPipeli
ningIT.java
qa/vagrant/src/main/java/org/elasticsearch/packaging/PackagingTests.java
qa/vagrant/src/main/java/org/elasticsearch/packaging/VMTestRunner.java
qa/vagrant/src/main/java/org/elasticsearch/packaging/test/ArchiveTestCas
e.java
qa/vagrant/src/main/java/org/elasticsearch/packaging/test/DefaultTarTest
s.java
qa/vagrant/src/main/java/org/elasticsearch/packaging/test/DefaultZipTest
s.java
qa/vagrant/src/main/java/org/elasticsearch/packaging/test/OssTarTests.ja
va
qa/vagrant/src/main/java/org/elasticsearch/packaging/test/OssZipTests.ja
va
qa/vagrant/src/main/java/org/elasticsearch/packaging/util/Archives.java
qa/vagrant/src/main/java/org/elasticsearch/packaging/util/Cleanup.java
qa/vagrant/src/main/java/org/elasticsearch/packaging/util/Distribution.j
ava
qa/vagrant/src/main/java/org/elasticsearch/packaging/util/FileMatcher.ja
va
qa/vagrant/src/main/java/org/elasticsearch/packaging/util/FileUtils.java
qa/vagrant/src/main/java/org/elasticsearch/packaging/util/Installation.j
ava
qa/vagrant/src/main/java/org/elasticsearch/packaging/util/Platforms.java
qa/vagrant/src/main/java/org/elasticsearch/packaging/util/Shell.java
server/src/main/java/org/elasticsearch/client/transport/TransportClient.
java
server/src/main/java/org/elasticsearch/cluster/ClusterModule.java
server/src/main/java/org/elasticsearch/cluster/service/ClusterApplier.ja
va
server/src/main/java/org/elasticsearch/cluster/service/ClusterApplierSer
vice.java
server/src/main/java/org/elasticsearch/common/settings/ClusterSettings.j
ava
server/src/main/java/org/elasticsearch/discovery/single/SingleNodeDiscov
ery.java
server/src/main/java/org/elasticsearch/discovery/zen/ZenDiscovery.java
server/src/main/java/org/elasticsearch/http/HttpHandlingSettings.java
server/src/main/java/org/elasticsearch/http/HttpPipelinedMessage.java
server/src/main/java/org/elasticsearch/http/HttpPipelinedRequest.java
server/src/main/java/org/elasticsearch/http/HttpPipeliningAggregator.jav
a
server/src/main/java/org/elasticsearch/http/HttpTransportSettings.java
server/src/main/java/org/elasticsearch/index/query/GeoShapeQueryBuilder.
java
server/src/main/java/org/elasticsearch/persistent/PersistentTasksCustomM
etaData.java
server/src/main/java/org/elasticsearch/transport/TcpTransport.java
server/src/test/java/org/elasticsearch/client/AbstractClientHeadersTestC
ase.java
server/src/test/java/org/elasticsearch/client/transport/TransportClientT
ests.java
server/src/test/java/org/elasticsearch/cluster/service/ClusterApplierSer
viceTests.java
server/src/test/java/org/elasticsearch/discovery/single/SingleNodeDiscov
eryIT.java
server/src/test/java/org/elasticsearch/discovery/single/SingleNodeDiscov
eryTests.java
server/src/test/java/org/elasticsearch/discovery/zen/ZenDiscoveryUnitTes
ts.java
server/src/test/java/org/elasticsearch/index/query/GeoShapeQueryBuilderT
ests.java
server/src/test/java/org/elasticsearch/indices/store/IndicesStoreIntegra
tionIT.java
server/src/test/java/org/elasticsearch/persistent/TestPersistentTasksPlu
gin.java
server/src/test/java/org/elasticsearch/search/geo/GeoShapeIntegrationIT.
java
test/framework/src/main/java/org/elasticsearch/test/ClusterServiceUtils.
java
test/framework/src/main/java/org/elasticsearch/test/ESIntegTestCase.java
test/framework/src/main/java/org/elasticsearch/test/InternalTestCluster.
java
test/framework/src/test/java/org/elasticsearch/test/test/InternalTestClu
sterTests.java
x-pack/docs/en/security/auditing/event-types.asciidoc
x-pack/docs/en/security/auditing/forwarding-logs.asciidoc
x-pack/docs/en/security/auditing/output-index.asciidoc
x-pack/docs/en/security/auditing/output-logfile.asciidoc
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/XPackClien
tPlugin.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
dit/index/IndexAuditTrailTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
dit/index/RemoteIndexAuditTrailStartingTests.java
x-pack/qa/ml-native-tests/src/test/java/org/elasticsearch/xpack/ml/integ
ration/MlNativeAutodetectIntegTestCase.java
2018-05-24 10:07:44 +01:00
David Roberts aafcd85f50
Move persistent task registrations to core (#30755)
Persistent tasks was moved from X-Pack to core in #28455.
However, registration of the named writables and named
X-content was left in X-Pack.

This change moves the registration of the named writables
and named X-content into core.  Additionally, the persistent
task actions are no longer registered in the X-Pack client
plugin, as they are already registered in ActionModule.
2018-05-24 09:17:17 +01:00
Tal Levy d1479dbaed Merge branch 'master' into index-lifecycle 2018-05-23 10:29:52 -07:00
Adrien Grand a19df4ab3b
Add a `format` option to `docvalue_fields`. (#29639)
This commit adds the ability to configure how a docvalue field should be
formatted, so that it would be possible eg. to return a date field
formatted as the number of milliseconds since Epoch.

Closes #27740
2018-05-23 14:39:04 +02:00
Luca Cavanna a17d6cab98
Replace Request#setHeaders with addHeader (#30588)
Adding headers rather than setting them all at once seems more
user-friendly and we already do it in a similar way for parameters
(see Request#addParameter).
2018-05-22 20:32:30 +02:00
Colin Goodheart-Smithe bfd6e60a75
Merge branch 'master' into index-lifecycle 2018-05-22 10:32:49 +01:00
Costin Leau dcf0f9f8dd
SQL: Preserve scoring in bool queries (#30730)
Make all bool constructs use match/should (that is a query context) as
that is controlled and changed to a filter context by ES automatically
based on the sort order (_doc, field vs _sort) and trackScores.

Fix #29685
2018-05-21 21:50:06 +03:00
Colin Goodheart-Smithe 46a3aba798
Merge branch 'master' into index-lifecycle 2018-05-21 12:05:24 +01:00
David Roberts 2b72adc8ac
[TEST] Reduce forecast overflow to disk test memory limit (#30727)
By default ML native processes are only allowed to use
30% of RAM, so the previous 2GB setting prevented the
test passing on VMs with only 4GB RAM.  This change
reduces the limit to 1200MB, which means it can now
pass on VMs with 4GB RAM.
2018-05-18 19:01:43 +01:00
Ryan Ernst b3f3a4312b
Plugins: Remove meta plugins (#30670)
Meta plugins existed only for a short time, in order to enable breaking
up x-pack into multiple plugins. However, now that x-pack is no longer
installed as a plugin, the need for them has disappeared. This commit
removes the meta plugins infrastructure.
2018-05-18 10:56:08 -07:00
Dimitris Athanasiou 6bb2a1da22
[ML][TEST] Fix bucket count assertion in ModelPlotsIT (#30717)
As the first record is random, there's a chance it will
be aligned on a bucket start. Thus we need to check the
bucket count is in [23, 24].

Closes #30715
2018-05-18 17:59:01 +03:00
Dimitris Athanasiou 1484a31be5
[ML][TEST] Make AutodetectMemoryLimitIT less fragile (#30716)
These tests aim to check the set model memory limit is
respected. Additionally, it was asserting counts of
partition, by, over fields in an attempt to check that
the used memory is spent meaningfully. However, this
made the tests fragile, as changes in the ml-cpp could
lead to CI failures.

This commit removes those assertions. We are working on
adding tests in ml-cpp that will compensate.
2018-05-18 17:57:20 +03:00
Hendrik Muhs 6c313a9871 This implementation lazily (on 1st forecast request) checks for available
diskspace and creates a subfolder for storing data outside of Lucene
indexes, but as part of the ES data paths.

Details:
 - tmp storage is managed and does not allow allocation if disk space is
   below a threshold (5GB at the moment)
 - tmp storage is supposed to be managed by the native component but in
   case this fails cleanup is provided:
    - on job close
    - on process crash
    - after node crash, on restart
 - available space is re-checked for every forecast call (the native
   component has to check again before writing)

Note: The 1st path that has enough space is chosen on job open (job
close/reopen triggers a new search)
2018-05-18 14:04:09 +02:00
Tal Levy 190a7efd2a Merge branch 'master' into index-lifecycle 2018-05-17 09:21:17 -07:00
Dimitris Athanasiou 75665a2d3e
[ML] Clean left behind model state docs (#30659)
It is possible for state documents to be
left behind in the state index. This may be
because of bugs or uncontrollable scenarios.
In any case, those documents may take up quite
some disk space when they add up. This commit
adds a step in the expired data deletion that
is part of the daily maintenance service. The
new step searches for state documents that
do not belong to any of the current jobs and
deletes them.

Closes #30551
2018-05-17 17:51:26 +03:00
David Roberts ef0daee850
[TEST] Account for increase in ML C++ memory usage (#30675)
Recent changes to the ML C++ have resulted in higher memory usage,
so fewer "by" fields can be analyzed in a given amount of model
memory.
2018-05-17 12:59:20 +01:00
Tim Vernum 9f824c4aa8
Add detailed assert message to IndexAuditUpgradeIT (#30669)
Print out the returned buckets if the size does not match the
expectation.
2018-05-17 21:36:13 +10:00
Alexander Reelsen 11d776ecf0
Watcher: Fix watch history template for dynamic slack attachments (#30172)
The part of the history template responsible for slack attachments had a
dynamic mapping configured which could lead to problems, when a string
value looking like a date was configured in the value field of an
attachment.

This commit fixes the template by setting this field always to text.
This also requires a change in the template numbering to be sure this
will be applied properly when starting watcher.
2018-05-17 11:57:54 +02:00
Tal Levy 4e757fff21 Merge branch 'master' into index-lifecycle 2018-05-16 15:46:44 -07:00
Ryan Ernst a4c9c2fa2a
Make xpack modules instead of a meta plugin (#30589)
This commit removes xpack from being a meta-plugin-as-a-module.
It also fixes a couple tests which were missing task dependencies, which
failed once the gradle execution order changed.
2018-05-16 15:35:57 -07:00
Colin Goodheart-Smithe f4fad07113
Merge branch 'master' into index-lifecycle 2018-05-16 11:43:26 +01:00
David Kyle 16f5a515f3
[ML] Wait for ML indices in rolling upgrade tests (#30615) 2018-05-16 09:52:25 +01:00
Tal Levy 8c12a59c5e Merge branch 'master' into index-lifecycle 2018-05-15 22:41:23 -07:00
Ryan Ernst c7d82b378b
Build: Add task interdependencies for ssl configuration (#30633)
This commit fixes the tasks creating ssl certs for tests to have correct
dependsOn to ensure the right tasks are run before tests run.
2018-05-15 16:09:15 -07:00
Yannick Welsch af2b9dd779 Revert "Mute ML upgrade test (#30458)"
This reverts commit 4b36ea7433.
2018-05-15 11:20:57 +02:00
Colin Goodheart-Smithe 8059aa7d55
Merge branch 'master' into index-lifecycle 2018-05-15 09:59:55 +01:00
Jason Tedor 4a4e3d70d5
Default to one shard (#30539)
This commit changes the default out-of-the-box configuration for the
number of shards from five to one. We think this will help address a
common problem of oversharding. For users with time-based indices that
need a different default, this can be managed with index templates. For
users with non-time-based indices that find they need to re-shard with
the split API in place they no longer need to resort only to
reindexing.

Since this has the impact of changing the default number of shards used
in REST tests, we want to ensure that we still have coverage for issues
that could arise from multiple shards. As such, we randomize (rarely)
the default number of shards in REST tests to two. This is managed via a
global index template. However, some tests check the templates that are
in the cluster state during the test. Since this template is randomly
there, we need a way for tests to skip adding the template used to set
the number of shards to two. For this we add the default_shards feature
skip. To avoid having to write our docs in a complicated way because
sometimes they might be behind one shard, and sometimes they might be
behind two shards we apply the default_shards feature skip to all docs
tests. That is, these tests will always run with the default number of
shards (one).
2018-05-14 12:22:35 -04:00
David Roberts 6a8aa99e3f [TEST] Mute ML test that needs updating to following ml-cpp changes
Relates #30399
2018-05-14 12:49:37 +01:00
Colin Goodheart-Smithe b08d7c872b
Merge branch 'master' into index-lifecycle 2018-05-14 10:10:33 +01:00
Ryan Ernst be8c094e8c
Build: Add mavenPlugin cluster configuration method (#30541)
This commit adds the ability to specify a plugin from maven for a
test cluster to use. Currently, only local projects may be used as
plugins, except when testing bwc, where the coordinates of the project
are used. However, that assumes all projects always keep the same
coordinates, or are even still plugins, which is no longer the case for
x-pack. The full cluster and rolling restart tests are changed to use
this new method when pulling x-pack versions before 6.3.0.
2018-05-11 21:58:18 -07:00
Nik Everett 069fec83a8
SQL: Use request flavored methods in tests (#30345)
Modifies the SQL tests to use the new `Request` object flavored methods
introduced onto the `RestClient` in #29623. We'd like to remove the old
methods eventually so we should stop using them.
2018-05-11 13:10:01 -04:00
Colin Goodheart-Smithe 0e417e3112
Merge branch 'master' into index-lifecycle 2018-05-11 10:00:49 +01:00
Costin Leau 2594c1fb38
SQL: Improve correctness of SYS COLUMNS & TYPES (#30418)
Tweak the return data, in particular with regards for ODBC columns to
better align with the spec
Fix order for SYS TYPES and TABLES according to the JDBC/ODBC spec

Fix #30386
Fix #30521
2018-05-11 10:17:01 +03:00
Ryan Ernst a82acac3e7 Fix incorrect template name in test case 2018-05-10 22:34:38 -07:00
Ryan Ernst 8d1756cd12
Security: Simplify security index listeners (#30466)
This commit adds a general state listener to the SecurityIndexManager,
and replaces the existing health and up-to-date listeners with that. It
also moves helper methods relating to health to SecurityIndexManager
from SecurityLifecycleService.
2018-05-10 08:23:29 -07:00
Jay Modi 37bb8f8075
Build: move generated-resources to build (#30366)
This commit moves the generated-resources directory to be within
the build directory for the openldap-tests and saml-idp-tests
projects. Both projects create a generated-resources directory that
should have been in the build directory but were instead at the same
level as the build directory.
2018-05-10 07:35:23 -06:00
Colin Goodheart-Smithe a03241d590
Merge branch 'master' into index-lifecycle 2018-05-09 10:50:33 +01:00
Alpar Torok 4b36ea7433 Mute ML upgrade test (#30458)
It fails fairly frequently. Some ML expert will look soon.
2018-05-08 18:11:14 -04:00
Ryan Ernst ce008c446b
Security: Rename IndexLifecycleManager to SecurityIndexManager (#30442)
This commit renames IndexLifecycleManager to SecurityIndexManager as it
is not actually a general purpose class, but specific to security. It
also removes indirection in code calling the lifecycle service, instead
calling the security index manager directly.
2018-05-08 10:03:11 -07:00
Jason Tedor ec71144040 Merge branch 'master' into index-lifecycle
* master: (35 commits)
  DOCS: Correct mapping tags in put-template api
  DOCS: Fix broken link in the put index template api
  Add put index template api to high level rest client (#30400)
  Relax testAckedIndexing to allow document updating
  [Docs] Add snippets for POS stop tags default value
  Move respect accept header on no handler to 6.3.1
  Respect accept header on no handler (#30383)
  [Test] Add analysis-nori plugin to the vagrant tests
  [Docs] Fix bad link
  [Docs] Fix end of section in the korean plugin docs
  Expose the Lucene Korean analyzer module in a plugin (#30397)
  Docs: remove transport_client from CCS role example (#30263)
  [Rollup] Validate timezone in range queries (#30338)
  Use readFully() to read bytes from CipherInputStream (#28515)
  Fix  docs Recently merged #29229 had a doc bug that broke the doc build. This commit fixes.
  Test: remove cluster permission from CCS user (#30262)
  Add Get Settings API support to java high-level rest client (#29229)
  Watcher: Remove unneeded index deletion in tests
  Set the new lucene version for 6.4.0
  [ML][TEST] Clean up jobs in ModelPlotIT
  ...
2018-05-06 18:17:36 -04:00
Jason Tedor beee5fe004
Respect accept header on no handler (#30383)
Today when processing a request for a URL path for which we can not find
a handler we send back a plain-text response. Yet, we have the accept
header in our hand and can respect the accepted media type of the
request. This commit addresses this.
2018-05-04 18:13:50 -04:00
Jay Modi 2426d65dda
Test: remove cluster permission from CCS user (#30262)
This commit updates the multi cluster search test with security so that
the user that is simply performing a multi cluster search does not have
any cluster permissions. This is done as none are needed by this user
and excess privileges could mask a behavior change.
2018-05-04 10:14:09 -06:00
Alexander Reelsen bf51a21b6c Watcher: Remove unneeded index deletion in tests
A leftover invalid TODO comment and `.watches` index deletion command
can be removed again.
2018-05-04 17:10:54 +02:00
Dimitris Athanasiou 2751790aea [ML][TEST] Clean up jobs in ModelPlotIT
Closes #30377
2018-05-04 10:46:19 +01:00
jaymode 9573492a9e
Test: use trial license in qa tests with security
QA tests that use security need to use a trial license instead of a
basic license. Basic licenses do not enable security so these tests are
not running in the expected configuration. This can also lead to issues
that seem completely unrelated such as authentication failures right
after cluster formation.

The authentication failure right after cluster formation happens since
a request makes it past the authentication license checks and the
request starts the authentication process. During authentication the
cluster forms and the license is updated to a basic license, which
disables all realms. The request that is being authenticated then tries
to iterate over the realms, but the realms are empty and the request
cannot be authenticated. This results in a authentication failure even
though the credentials provided are correct.

Closes #30306
2018-05-03 10:33:08 -06:00
Dimitris Athanasiou a1e23feba2
[ML] Add integration test for model plots (#30359)
Relates #30004
2018-05-03 17:02:45 +01:00
Costin Leau 65dbc17510
SQL: Fix bug caused by empty composites (#30343)
When dealing with filtering, a composite aggregation might return empty
buckets (which have been filtered) which gets sent as is to the client.
Unfortunately this interprets the response as no more data instead of
retrying.

This now has changed and the listener keeps retrying until either the
query has ended or data passes the filter.

Fix #30292
2018-05-03 17:13:32 +03:00
Dimitris Athanasiou 3b260dcfc1
[ML] Account for gaps in data counts after job is reopened (#30294)
This commit fixes an issue with the data diagnostics were
empty buckets are not reported even though they should. Once
a job is reopened, the diagnostics do not get initialized from
the current data counts (especially the latest record timestamp).
The result is that if the data that is sent have a time gap compared
to the previous ones, that gap is not accounted for in the empty bucket
count.

This commit fixes that by initializing the diagnostics with the current
data counts.

Closes #30080
2018-05-03 15:08:24 +01:00
Alexander Reelsen 18394540ab
Tests: Use different watch ids per test in smoke test (#30331)
Each test now has its own watch id that is being used.
This ensures there are no old history entries, which can potentially
lead to broken test assertions.
2018-05-03 11:41:06 +02:00
Colin Goodheart-Smithe 6546535a8b
Merge branch 'master' into index-lifecycle 2018-05-03 08:54:15 +01:00
Alexander Reelsen 2c38d12e23
Watcher: Make start/stop cycle more predictable and synchronous (#30118)
The current implementation starts/stops watcher using an executor. This
can result in our of order operations.

This commit reduces those executor calls to an absolute minimum in order
to be able to do state changes within the cluster state listener method,
which runs in sequence.

When a state change occurs that forces the watcher service to pause
(like no watcher index, no master node, no local shards), the service is
now in a paused state.

Pausing is a super lightweight operation, which marks the
ExecutionService as paused and waits for the currently executing watches
to finish in the background via an executor. The same applies for
stopping, the potentially long running operation is outsourced in to an
executor, as waiting for executed watches is decoupled from the current
state.

The only other long running operation is starting, where watches need to
be loaded. This is also done via an executor, but has an additional
protection by checking the cluster state version it was started with. If
another cluster state version was trying to load the watches, then this
loading will not take effect.

This PR also cleans up some unused states, like the a simple boolean in
the HistoryStore/TriggeredWatchStore marking it as started or stopped,
as this can now be caught in the execution service.

Another advantage of this approach is the fact, that now only triggered
watches are not getting executed, while watches that are run via the
Execute Watch API will still be executed regardless if watcher is
stopped or not.

Lastly the TickerScheduleTriggerEngine thread now only starts on data nodes.
2018-05-03 09:47:12 +02:00
Andy Bristol 824e648662 [test] add debug logging for packaging test 2018-05-02 18:09:44 -07:00
Adrien Grand 231a63fdf8
Remove useless version checks in REST tests. (#30165)
Many tests are added with a version check so that they do not run against a
version that doesn't have the feature yet. Master is 7.0, so all tests that
do not run against 6.0+ can be removed and the version check can be removed
on all tests that always run on 6.0+.
2018-05-02 11:34:15 +02:00
Colin Goodheart-Smithe 4ad44a76b0
Merge branch 'master' into index-lifecycle 2018-05-02 09:01:08 +01:00
Tim Vernum 66daaaa1cc
Fix message content in users tool (#30293)
The elasticsearch-users utility had various messages that were
outdated or incorrect. This commit updates the output from this
command to reflect current terminology and configuration.
2018-05-02 11:04:28 +10:00
jaymode cdae8ff5a0
Test: increase authentication logging for debugging
This commit increases the logging for authentication in the x-pack
multi-node qa test project. This is needed to assist in debugging HTTP
authorization failures in waiting for the second node in these tests.

See #30306
2018-05-01 13:15:59 -06:00
David Turner b9e1860f36 Mark JdbcSqlSpecIT as @AwaitsFix
Relates #30292
2018-05-01 10:03:18 +01:00
Colin Goodheart-Smithe d7d709a5dc
Merge branch 'master' into index-lifecycle 2018-04-30 09:29:48 +01:00
Nik Everett f4ed902698
CCS: Drop http address from remote cluster info (#29568)
They are expensive to fetch and no longer needed by Kibana so they
*shouldn't* be needed by anyone else either.

Closes #29207
2018-04-27 14:19:00 -04:00
Zachary Tong fee000a37f
[TEST] Redirect links to new locations (#30179)
We had a number of awaitsFix links that weren't updated after the xpack
merge.

Where possible I changed the links to the new locations, but in some
circumstances the original ticket was closed (suggesting the awaitsfix
should be removed) or was otherwise unclear the status.
2018-04-27 09:24:46 -07:00
Tanguy Leroux 7ae3b3b155
Move repository-s3 fixture tests to QA test project (#29372)
This commit moves the repository-s3 fixture test added in #29296 in a
new `repository-s3/qa/amazon-s3` project. This new project allows the
REST integration tests to be executed using the real S3 service when
all the required environment variables are provided. When no env var
is provided, then the tests are executed using the fixture added
in #29296.

The REST tests located at the `repository-s3`plugin  project now only 
verify that the plugin is correctly loaded.

The REST tests have been adapted to allow a bucket name and a base 
path to be specified as env vars. This way it is possible to run the tests
with different base paths (could be anything, like a CI job name or a
branch name) without multiplicating buckets.

Related to #29349
2018-04-27 16:49:06 +02:00
Colin Goodheart-Smithe ec9f1a9f3b
Merge branch 'master' into index-lifecycle 2018-04-27 15:32:02 +01:00
Ryan Ernst 55e1b1e8b5
Tests: Use buildDir as base for generated-resources (#30191)
This commit moves the generated-resources directory created by many qa
projects into the build directory, so it is not seen as unknown files to
git.
2018-04-26 15:36:14 -07:00
Tal Levy 053e9fb234 Merge branch 'master' into index-lifecycle 2018-04-26 15:15:45 -07:00
Chris Earle 1d81ec1562
[Monitoring] Remove unhelpful Monitoring tests (#30144)
This removes some monitoring tests that have been silenced for a long
time. These tests don't really provide any value for the upgrade suite and
they just create noise due to their occasional timing-related failures.
2018-04-26 11:32:46 -04:00
Andy Bristol 67c0cf1dbf
[test] include oss tar in packaging tests (#30155)
Add the oss tar distribution to the packaging test plugin. Test the oss
tar distribution in the core packaging tests, and the non-oss tar
distribution in the x-pack packaging tests.
2018-04-26 06:58:41 -07:00
Alexander Reelsen fadcce8367
Watcher: Fold two smoke test projects into smoke-test-watcher (#30137)
In order to have less qa/ projects, this commit removes the
watcher-smoke-test-mustache and the watcher-smoke-test-painless projects
and moves the tests of both into the watcher-smoke-test projects.

This results in less projects to parse when calling gradle and a shorter test
time due to being able to run everything within one elasticsearch
instance.
2018-04-26 09:23:54 +02:00
Tal Levy 1df6b6c65d Applying changes required for ILM after merge 2018-04-25 13:04:06 -07:00
Michael Basnight 72f57c8e72
Add comments inadvertently removed during migrate
A few files had their first comment removed even though it did not
contain a license. This re-adds those comments.
2018-04-24 13:41:09 -05:00
David Kyle cfc66a1fd5 [ML] Wait for updates to established memory usage
Tests need to wait for changes to the job's established memory usage to
propagate and an over enthusiastic optimisation meant jobs were updated
from stale state causing recent change to be lost.
2018-04-24 13:46:58 -04:00
Ryan Ernst d8636d0fe3 Rename syskeygen
This commit renames syskeygen to elasticsearch-syskeygen
2018-04-20 15:34:01 -07:00
Jason Tedor c7c0e330b8 Rename users
This commit renames users to elasticsearch-users.
2018-04-20 15:34:01 -07:00
Jason Tedor b05c8bf781 Rename migrate
This commit renames migrate to elasticsearch-migrate.
2018-04-20 15:34:00 -07:00
Jason Tedor 2d3fac307a Rename saml-metadata
This commit renames saml-metadata to elasticsearch-saml-metadata.
2018-04-20 15:34:00 -07:00