Commit Graph

4331 Commits

Author SHA1 Message Date
Jason Tedor 3f5d24ae55 Add shutdown hook for closing CLI commands
This commit adapts some CLI command tests for some changes in core.

Relates elastic/elasticsearch#4342

Original commit: elastic/x-pack-elasticsearch@1e48c1404f
2016-12-13 19:10:35 -05:00
Alexander Reelsen 1e7985563a Tests: Make MockWebServer thread-safe (elastic/elasticsearch#4351)
The new MockWebServer assumed that responses are inserted at the beginning
and removed later on. This was not thread safe.

Also this fixes a bug in the HttpExporterIT where there was no wait time
for a bulk request, even though the request execution is asynchronous.

Closes elastic/elasticsearch#4335

Original commit: elastic/x-pack-elasticsearch@11f31f68bd
2016-12-13 19:34:33 +01:00
Daniel Mitterdorfer 8be3250601 Merge remote-tracking branch 'origin/master' into json_strict_duplicate_checks
Original commit: elastic/x-pack-elasticsearch@a010b1b4e9
2016-12-13 16:22:47 +01:00
Daniel Mitterdorfer 7cb0771c46 Use assume* instead of custom skipping logic
Original commit: elastic/x-pack-elasticsearch@0f3a01735f
2016-12-13 16:21:46 +01:00
Luca Cavanna 1c846dd893 Stop execution once destructive operations check has failed (elastic/elasticsearch#4337)
Otherwise we do return an error, but we also go ahead and open/close/delete the indices anyways.

Original commit: elastic/x-pack-elasticsearch@176eca4cff
2016-12-13 11:22:48 +01:00
Alexander Reelsen b57c4f6ebe Watcher: Remove in memory watch store (elastic/elasticsearch#4201)
In order to prepare to the distributed watch execution, this commit
removes the in memory watch store.

Whenever a watch is needed now, a get request is executed and the parsing
is done. This happens when

* Put
* Get
* Ack
* Activate/Deactivate
* Execute

Note: This also means there are no usage stats currently regarding
the watch count, because we would need to execute a query. This would
require the usage stats to be async, see elastic/elasticsearch#3569

Another advantage is, that there is no dirty flag in the watch itself
needed anymore, because the watch is always the latest. Also write
operations store immediately and dont leave anything in memory.

Also ActionListener.wrap() was used a lot instead of more verbose anonmyous
inner classes.

Original commit: elastic/x-pack-elasticsearch@c47465b47c
2016-12-13 08:54:03 +01:00
Tim Vernum 7192c46307 Define explicit settings for security realms/ssl (elastic/elasticsearch#4311)
Modified the definition and loading of settings in Security to provide early detection and failure of invalid (unrecognised or syntactically invalid) settings.
Also consolidates the number of places where settings were defined.

Each realm now defines its supported settings.
This is facilitated for custom realms via a new "getRealmSettings" method on XPackExtension.
The realm group setting performs validation of the child settings with reference to the "type".
For backwards compatibility reasons, realm types that have no configuration defined, will be accepted during validation, but may fail at realm creation time.

All SSL settings have been centralised into SSLConfigurationSettings, which supports a variable "prefix" to accommodate the multiple places we define SSL config.

HTTP Proxy settings are explicitly defined rather than being a generic group.

Where possible all security settings now reference a Setting object, and there are less magic strings scattered in the code.

Closes: elastic/elasticsearch#3965

Original commit: elastic/x-pack-elasticsearch@2c76a137a9
2016-12-13 16:14:02 +11:00
javanna 85679fcf19 Adapt to removal of support for empty queries
Original commit: elastic/x-pack-elasticsearch@87915a9cde
2016-12-12 13:34:30 +01:00
Daniel Mitterdorfer 050a36bb28 Address review comments in ES core
Original commit: elastic/x-pack-elasticsearch@925c78d4fe
2016-12-12 13:30:39 +01:00
Daniel Mitterdorfer b7d2ef0160 Merge remote-tracking branch 'origin/master' into json_strict_duplicate_checks
Original commit: elastic/x-pack-elasticsearch@601a663982
2016-12-12 12:36:38 +01:00
Alexander Reelsen 779eb44b66 Watcher: Cleanup - Replace list of integers with bitset (elastic/elasticsearch#4324)
In the triggered watch store a list of integers was returned to keep track
which watches had been successfully stored and thus should be executed.

This means, that an arraylist, plus autoboxing/unboxing needs to be done for
all the triggered watches. This data structure can easily be replaced with a BitSet,
resulting in much less objects being created or parsed - also it's a bit faster.

Original commit: elastic/x-pack-elasticsearch@e9fba67e34
2016-12-12 09:48:58 +01:00
Simon Willnauer 5716a20fd3 Fix compile issues after elastic/elasticsearchelastic/elasticsearch#22037
Original commit: elastic/x-pack-elasticsearch@c286839ba5
2016-12-10 11:05:58 +01:00
Nik Everett f6cb60204a Handle core moving some methods around
We're trying to move all the XContentParser creation into
`RestRequest`.

Original commit: elastic/x-pack-elasticsearch@87c05c4a9a
2016-12-09 20:01:53 -05:00
Stacey Gammon e321337d4c Merge pull request elastic/elasticsearch#4296 from Stacey-Gammon/save-rename-graph-update
Incorporate new save/rename flow for graph

Original commit: elastic/x-pack-elasticsearch@27d2031381
2016-12-09 08:49:10 -05:00
Jay Modi ac34c3c37f Remove deprecated methods from the Realm class
This change removes the deprecated methods from the realm class. These methods include
blocking authentication and lookup and the lookup supported method.

Original commit: elastic/x-pack-elasticsearch@cff21e21ee
2016-12-09 07:28:11 -05:00
Daniel Mitterdorfer 8153d2ca69 Enable strict duplicate checks for JSON content
With this commit we enable the Jackson feature 'STRICT_DUPLICATE_DETECTION'
by default. This ensures that JSON keys are always unique. While this has
a performance impact, benchmarking has indicated that the typical drop in
indexing throughput is around 1 - 2%.

As a last resort, we allow users to still disable strict duplicate checks
by setting `-Des.json.strict_duplicate_detection=false` which is
intentionally undocumented.

Relates elastic/elasticsearchelastic/elasticsearch#19614

Original commit: elastic/x-pack-elasticsearch@cced57b884
2016-12-09 12:48:28 +01:00
Alexander Reelsen 37b0d52882 Watcher: Remove mock web server from square (elastic/elasticsearch#4221)
The latest release of the mock web server requires more security permissions, and
we dont need all the functionality anyway.

This introduces a small MockWebServer using the JDK internal HttpServer, yet fullfilling
all our needs and supporting SSL as well for testing.

The MockWebServer allows to enqueue responses and also requires you to enqueue as many responses
as you requests will be executed - there is no fallback at the moment.

SSL is also supported by passing an SSL context - for which the TestsSSLService is needed, which
makes the required methods public.

Original commit: elastic/x-pack-elasticsearch@55f4a172a2
2016-12-09 09:07:09 +01:00
Ryan Ernst b1846190af Remove uses of 2.0 prerelease constants
This is the xplugins side of elastic/elasticsearchelastic/elasticsearch#22004

Original commit: elastic/x-pack-elasticsearch@55433d760c
2016-12-08 23:11:20 -08:00
Jason Tedor e6d758b92b Bump version to 5.1.2
This commit bumps the version to 5.1.2.

Relates elastic/elasticsearch#4318

Original commit: elastic/x-pack-elasticsearch@a922dc7288
2016-12-08 18:43:16 -05:00
Tanguy Leroux 1f9871c31e [TEST] Mute OldMonitoringIndicesBackwardsCompatibilityTests
This tests fails because of multiple issues in monitoring service and exporters lifecycle. These issues are tracked in https://github.com/elastic/x-plugins/issues/4314.

Original commit: elastic/x-pack-elasticsearch@f461d98a4c
2016-12-08 10:48:39 +01:00
Areek Zillur b274ddc0da Log license type on license update (elastic/elasticsearch#4308)
Now when a cluster gets updated with a new license,
the license uid and type will be logged as info. Making
it easier to verify the current license mode of the cluster.

Original commit: elastic/x-pack-elasticsearch@c205cb42ae
2016-12-07 18:01:58 -05:00
Yannick Welsch 754fada43b Use autoMinMasterNodes for most of the license tests
There were some test failures as LicenseServiceClusterTests was not properly setting minimum_master_nodes while having autoMinMasterNodes disabled.

Original commit: elastic/x-pack-elasticsearch@2030db2424
2016-12-07 17:59:25 +01:00
jaymode 714bf929af test: LocalExporterTests should wait for exporters to be started
Original commit: elastic/x-pack-elasticsearch@978933f9d6
2016-12-07 09:22:05 -05:00
Jim Ferenczi fba54d02d7 fix compilation error due to a missing parameter in SortedSetDVOrdinalsIndexFieldData.
Original commit: elastic/x-pack-elasticsearch@f9803968d9
2016-12-07 11:33:29 +01:00
Boaz Leskes affdf10274 Remove `InternalTestCluster.startNode(s)Async` (elastic/elasticsearch#4198)
A companion PR to https://github.com/elastic/elasticsearch/pull/21846 where the above methods were removed. See ES PR for details.

With the concurrent starting the issues with licenses and time freeze became more apparent and I had to apply my suggestion to only freeze time once the license has been applied (as opposed to freeze on node start up). Since this also means that a node that starts up after the cluster time has been frozen need to also immediately freeze, it felt natural to use a `ServiceDisruptionScheme`. Although the name doesn't really make sense here, it all has all the logic we need. 

Original commit: elastic/x-pack-elasticsearch@5641742f60
2016-12-06 12:07:28 +01:00
Boaz Leskes 7a996da86d ci: don't use ssh to check elasticsearch repo for a sibling branch
ssh keys are not always available

Original commit: elastic/x-pack-elasticsearch@5c974724ea
2016-12-06 11:37:50 +01:00
Boaz Leskes 7b044a6e16 ci: remove unneeded AWAITSFIX comment
Original commit: elastic/x-pack-elasticsearch@726d26fcdb
2016-12-06 10:00:22 +01:00
Boaz Leskes 11ae30cf90 ci: properly accommodate for repo name to be part of `GIT_BRANCH`
previous commit missed a `*`

Original commit: elastic/x-pack-elasticsearch@33b9f32c62
2016-12-06 09:52:03 +01:00
Boaz Leskes 373c350920 ci: accommodate for repo name to be part of `GIT_BRANCH`
It seems that that variable is set by the git jenkins plugin and it's format is `origin/master`. See https://wiki.jenkins-ci.org/display/JENKINS/Git+Plugin#GitPlugin-Environmentvariables

Original commit: elastic/x-pack-elasticsearch@17f8ec1a9e
2016-12-06 09:43:28 +01:00
Ryan Ernst 7483abb226 Add fallback branch until CI configuration is fixed to pass GIT_BRANCH
Original commit: elastic/x-pack-elasticsearch@68d12b33e9
2016-12-05 16:34:08 -08:00
Ryan Ernst 13e427d83f Remove action filter response override (elastic/elasticsearch#4268)
This is the xplugins side of elastic/elasticsearchelastic/elasticsearch#21950

Original commit: elastic/x-pack-elasticsearch@0034a3384c
2016-12-05 16:14:10 -08:00
Ryan Ernst 75a72b34cb Build: Checkout relevant branch of elasticsearch for ci (elastic/elasticsearch#4270)
The ci script checks out elasticsearch for x-plugins to use. However, it
always checks out the master branch. This change makes the script look
for the source branch of a PR (falling back to GIT_BRANCH which is provided by jenkins for non PR CI jobs) and checks if that branch exists in elasticsearch. If it does not, it falls back to the target branch for the PR.

Original commit: elastic/x-pack-elasticsearch@73146bb3b3
2016-12-05 16:10:33 -08:00
Jay Modi ffeaea0663 Migrate authentication calls to be asynchronous
This commit migrates the actual authentication calls that are performed by internal realms
to be asynchronous so that we do not execute blocking calls on network threads. The realms
that used LDAP were especially bad as they can issue many different requests and/or open
multiple connections per authentication attempt.

The LdapRealm now uses the ThreadPool to fork a thread for authentication. This is necessary
because a LDAP bind operation is not allowed to be performed asynchronously and must be done
in a blocking manner. After the bind request has completed, all other requests can be done
asynchrnously. The LdapUtils class now provides asynchronous methods for searching and also
includes the handling of referral following as automated following is not supported by
UnboundID when using the asynchronous calls.

Original commit: elastic/x-pack-elasticsearch@dfb259e498
2016-12-05 10:07:31 -05:00
Tanguy Leroux 549629b1aa [Watcher] HistoryStore should hide secrets (elastic/elasticsearch#4256)
* [Watcher] HistoryStore should hide secrets

Some watcher actions like Jira stores the whole HTTP request in case of a failure. This is very helpful when something goes wrong, but it has to hide the password used for Basic authentication otherwise it will appear in the watch record in the .watcher-history index. In general, I would consider better to always hide secrets when storing something in the history index.

 This commit removes the XContentParams "hide secrets" set at the JiraIssue so that it is now set at the HistoryStore level.

 It also fixes test failures that can be reproduced with:
 gradle :x-plugins:elasticsearch:test -Dtests.seed=BA730F93E726AECC -Dtests.class=org.elasticsearch.xpack.notification.jira.JiraIssueTests -Dtests.method="testToXContent" -Dtests.security.manager=true -Dtests.locale=es -Dtests.timezone=Asia/Choibalsan

Original commit: elastic/x-pack-elasticsearch@d686b3b3ad
2016-12-05 15:30:48 +01:00
Jay Modi 9afb6dd4f2 monitoring: local exporter waits for in flight requests before retrying
The local exporter previously fired off asynchronous requests every time a cluster state was
observed that did not contain all of the required items for monitoring. This change adds a
flag so that monitoring can wait for the pending requests to complete before retrying. This
will reduce the number of duplicated log messages as well.

Additionally, the log message for adding modern aliases now contains the name of the indices.

Closes elastic/elasticsearch#3756

Original commit: elastic/x-pack-elasticsearch@727a0adfbe
2016-12-05 09:01:24 -05:00
jaymode 2b7c03848f test: disable xpack plugins on tribe client nodes
The license test for the tribe node disables xpack features but these settings
are not passed to the tribe client nodes, so they need to manually be added.

Original commit: elastic/x-pack-elasticsearch@7273817c47
2016-12-03 14:02:55 -05:00
Ryan Ernst 97336ea946 Fix rest filter tests to expect passthrough when security is disabled
Original commit: elastic/x-pack-elasticsearch@ea6adabb4d
2016-12-02 16:28:18 -08:00
Ryan Ernst dbbf470734 Fix line length to appease checkstyle
Original commit: elastic/x-pack-elasticsearch@35df54d342
2016-12-02 15:06:33 -08:00
Ryan Ernst 923926ef28 Convert security rest filter to rest handler wrapper (elastic/elasticsearch#4234)
* Convert security rest filter to rest handler wrapper

This is the xpack side of elastic/elasticsearchelastic/elasticsearch#21905

Original commit: elastic/x-pack-elasticsearch@38bfa771b6
2016-12-02 14:55:10 -08:00
Tanguy Leroux f1a4a2fb73 [Monitoring] Remove unused code in Collectors (elastic/elasticsearch#4240)
This commit removes some unnecessary code: collectors do not need to extend AbstractLifecycleComponent and do not need to be started/stopped. The extra
interface Collector is removed and AbstractCollector renamed to Collector.

Original commit: elastic/x-pack-elasticsearch@75893d66e3
2016-12-02 13:29:09 +01:00
Simon Willnauer 7f17896927 Revert "Followup for elastic/elasticsearchelastic/elasticsearch#21915 - removal of legacy BWC test infrastructure (elastic/elasticsearch#4247)"
This reverts commit elastic/x-pack@c6c0ffd5d9.

Original commit: elastic/x-pack-elasticsearch@8b7386fb99
2016-12-02 10:55:46 +01:00
Alexander Reelsen 7c04897392 Watcher: Compile scripts on each invocation (elastic/elasticsearch#4239)
Transform and condition scripts were only compiled on its initial creation, so
when a new watch is created or when the master node loads all the watches. However
changing a script (like a stored one) did not lead to any changes in the in memory
watch store and thus the old script was executed again.

We do however have a mechanism in Elasticsearch's ScriptService that already does some
caching, and should reuse that one.

Closes elastic/elasticsearch#4237

Original commit: elastic/x-pack-elasticsearch@477548e237
2016-12-02 10:36:05 +01:00
Alexander Reelsen 946d943868 Watcher: Removing unused upgradeSource boolean in watch parsing (elastic/elasticsearch#4202)
This seems to be a leftover from elastic/elasticsearch#4162

However the boolean parameters is completely unused already and
can be removed.

Original commit: elastic/x-pack-elasticsearch@3371b089d6
2016-12-02 10:34:11 +01:00
Simon Willnauer ace1a7e6af Followup for elastic/elasticsearchelastic/elasticsearch#21915 - removal of legacy BWC test infrastructure (elastic/elasticsearch#4247)
Original commit: elastic/x-pack-elasticsearch@07cecdbf00
2016-12-02 08:06:46 +01:00
Aaron Bull Schaefer 417ccdd7cc Merge pull request elastic/elasticsearch#4137 from dliappis/master
Allow defining gradle test params via cli argument

Original commit: elastic/x-pack-elasticsearch@3d312de1cc
2016-12-01 10:24:10 -08:00
Jay Modi e8836f99b0 test: add tests that exercise the security index during rolling upgrades
This commit adds basic tests that store a user and a role using the native API. The test checks
that the user and role can be used prior to starting the upgrade. The realm and roles caches are
also cleared to ensure the next authentication will require a read from the security index; this
ensures we are actually testing reads from the index.

Original commit: elastic/x-pack-elasticsearch@396862da94
2016-12-01 11:15:15 -05:00
Jay Modi 139073e8f7 security: improve migrate tool output and remove trappy config option
This commit improves the output of the migrate tool in cases when there are errors parsing entries
in the roles or users files. This is done through the use of a logger that delegates its output to
the terminal.

Additionally, the `-c` option has been removed. This option was used to set the configuration directory
but this should be handled one way only and that is through the use of the `-Epath.conf` setting.

Closes elastic/elasticsearch#3757
Closes elastic/elasticsearch#3758

Original commit: elastic/x-pack-elasticsearch@811e367766
2016-12-01 10:17:28 -05:00
Boaz Leskes d881e4d9ad properly deal with overriding parent's tearDown method
Original commit: elastic/x-pack-elasticsearch@0edd4b1e20
2016-12-01 13:18:36 +01:00
Boaz Leskes 9e3ae67423 HttpClientTests don't override parent tearDown
Original commit: elastic/x-pack-elasticsearch@3cf9f6f352
2016-12-01 13:15:25 +01:00
Boaz Leskes a4cec2316b Add before and after logging for unit tests
Currently we have these logs for integration tests only.

This adds the following log at the start:
```
logger.info("[{}]: before test", getTestName());
```

and this is logged at the end, but before any clean up done in sub classes

```
 logger.info("[{}]: after test", getTestName());
```

Original commit: elastic/x-pack-elasticsearch@2ca7296665
2016-12-01 12:56:38 +01:00