Commit Graph

1645 Commits

Author SHA1 Message Date
Shaunak Kashyap 78f7c0e27a Fix license messaging for Logstash functionality (elastic/x-pack-elasticsearch#3268)
* Fix license messaging for Logstash functionality

With a Basic license, users are still able to perform CRUD operations on the `.logstash` index, therefore manage their Logstash pipelines. However, Logstash itself will not pick up any changes from this index and act on them. With an expired license Logstash functionality continues to operate as normal.

* Fixing messages after feedback

* Removing extraneous tabs at end of line

* Fixing typo

Original commit: elastic/x-pack-elasticsearch@bc069cf00f
2017-12-15 05:56:01 -08:00
Costin Leau f2792b8d93 Update cli to work on windows
Original commit: elastic/x-pack-elasticsearch@84f6ba3c1f
2017-12-14 18:53:21 +02:00
Nik Everett 2268e592c6 Merge branch 'master' into feature/sql_2
Original commit: elastic/x-pack-elasticsearch@ded26a3834
2017-12-14 10:31:51 -05:00
Nik Everett 59e6d34c29 SQL: Bundle the CLI into x-pack (elastic/x-pack-elasticsearch#3316)
This adds:

* The CLI jar itself into the `bin`. It is an executable jar.
* A shell and bat script to start the CLI. This isn't strictly required but folks will appreciate the consistency.
* Basic packaging tests for the CLI.

Relates to elastic/x-pack-elasticsearch#2979

Original commit: elastic/x-pack-elasticsearch@158f70a530
2017-12-14 09:57:03 -05:00
Colin Goodheart-Smithe 51b80aa998 Fixes test to support BytesSizeValue changes (elastic/x-pack-elasticsearch#3321)
Original commit: elastic/x-pack-elasticsearch@0bc8b0e847
2017-12-14 12:17:25 +00:00
Ioannis Kakavas d24921ea60 Check for existing x-pack directory when running the `users` CLI tool (elastic/x-pack-elasticsearch#3271)
Verify that the configuration directory `$ES_PATH_CONF/x-pack`
exists before attempting to run any of the `users` commands, and 
return a helpful error message to the user if it doesn't.

Original commit: elastic/x-pack-elasticsearch@6d663b6654
2017-12-14 13:45:53 +02:00
Chris Earle 876fc5612a [Monitoring] Use index.auto_expand_replicas: 0-1 (elastic/x-pack-elasticsearch#3282)
This changes the default behavior of .monitoring indices to be green on one-node clusters, instead of constantly yellow.

Note: This only impacts .monitoring* indices. Watcher indices currently still require a replica.

Original commit: elastic/x-pack-elasticsearch@6eb8a48a9f
2017-12-13 13:46:31 -05:00
Costin Leau cdfe0f1f5a Revert "Revert "Merge branch 'feature/sql'""
This reverts commit elastic/x-pack-elasticsearch@cc79e19911.

We'll merge this when we're good and ready.

Original commit: elastic/x-pack-elasticsearch@b3ef4f2836
2017-12-13 10:19:31 -05:00
Costin Leau 2e60e831c0 Revert "Merge branch 'feature/sql'"
This reverts commit elastic/x-pack-elasticsearch@2b3f7697a5, reversing
changes made to elastic/x-pack-elasticsearch@b79f16673c.

We're backing out all of SQL.

Original commit: elastic/x-pack-elasticsearch@cc79e19911
2017-12-13 09:33:13 -05:00
Lee Hinman ffeebc366d Add sql folder for upcoming migration (elastic/x-pack-elasticsearch#3307)
* Add sql folder for upcoming migration

This is the skeleton folder for SQL now that it's been merged.

Relates to elastic/x-pack-elasticsearch#3233

Original commit: elastic/x-pack-elasticsearch@c46548bd6c
2017-12-12 16:48:43 -07:00
Lee Hinman 86a04acb01 Rename folder x-pack-core -> core (elastic/x-pack-elasticsearch#3305)
* Rename folder x-pack-core -> core

The jar remains 'x-pack-core-*.jar'

* Put group in top-level build.gradle instead of plugin/core/build.gradle

Original commit: elastic/x-pack-elasticsearch@b23452fa55
2017-12-12 13:23:29 -07:00
Nik Everett aef5ea6527 Merge branch 'feature/sql'
It is time.

Original commit: elastic/x-pack-elasticsearch@2b3f7697a5
2017-12-12 14:49:27 -05:00
Jay Modi d48ad6dbd6 Enhance security for alias and resize operations (elastic/x-pack-elasticsearch#3302)
This commit adds additional checks around resize operations and alias creation operations to
add an extra layer of security around these APIs.

Original commit: elastic/x-pack-elasticsearch@b79f16673c
2017-12-12 12:01:27 -07:00
Nik Everett eaa75cfdbb Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@b5fc2b484d
2017-12-12 11:29:23 -05:00
Dimitris Athanasiou 92c40061c5 [ML] Skip index health checks for remote indices on datafeed node sel (elastic/x-pack-elasticsearch#3301)
Upon selecting a node to run a datafeed we normally check that the
data indices exists and their primaries are active. However, these
checks cannot be applied for CCS to a remote cluster. This commit
skips these checks for remote indices.

This removes the last obstacle for running CCS datafeeds.

Relates elastic/x-pack-elasticsearch#1071

Original commit: elastic/x-pack-elasticsearch@092f44feee
2017-12-12 15:12:33 +00:00
Tanguy Leroux 711254fd24 [Monitoring] Use the same Cluster State for all Collectors (elastic/x-pack-elasticsearch#3216)
This commit changes the Collectors so that they all use the 
same instance of ClusterState.

relates elastic/x-pack-elasticsearch#3156

Original commit: elastic/x-pack-elasticsearch@4f537b026c
2017-12-12 12:29:40 +01:00
David Kyle 0d46e9035c [ML] Get Filters should use executeAsyncWithOrigin (elastic/x-pack-elasticsearch#3295)
* Get Filters should use executeAsyncWithOrigin

Original commit: elastic/x-pack-elasticsearch@786c7bfd06
2017-12-12 11:15:54 +00:00
David Kyle 6113b86bdb [ML] Special events calendar CRUD endpoints (elastic/x-pack-elasticsearch#3267)
* Calendar CRUD endpoints

* Get calendars requires monitor permission

* Address review comments

* Add page params to get calendars

Original commit: elastic/x-pack-elasticsearch@badd1e6add
2017-12-12 09:21:44 +00:00
Nik Everett 15f84840f2 SQL: Revert some security changes so NOOP merge
SQL used to have some changes to security. We've since reverted them but
we have some leftover stuff like import reordering and spacing changes.
We may as well remove them so merging SQL to master is smaller.

Original commit: elastic/x-pack-elasticsearch@c632256ddd
2017-12-11 16:06:48 -05:00
David Roberts 249d06b256 [ML] Fix permissions for field caps in scroll data extractor
Follow up to elastic/x-pack-elasticsearch#3254

Original commit: elastic/x-pack-elasticsearch@d4df9446c0
2017-12-11 17:45:32 +00:00
Igor Motov 4bebc307c3 SQL: Add ability to close cursors (elastic/x-pack-elasticsearch#3249)
This commits adds a new end point for closing in-flight cursors, it also ensures that all cursors are properly closed by adding after test checks that ensures that we don't leave any search context open.

relates elastic/x-pack-elasticsearch#2878

Original commit: elastic/x-pack-elasticsearch@1052ea28dc
2017-12-11 11:36:02 -05:00
David Kyle 04c07688a2 [ML] Align special events with buckets (elastic/x-pack-elasticsearch#3258)
Original commit: elastic/x-pack-elasticsearch@71f9d0fb13
2017-12-11 15:42:06 +00:00
David Roberts 5fd68959a0 [ML] Make datafeeds run-as the user who created/updated them (elastic/x-pack-elasticsearch#3254)
This is the ML equivalent of what was done for Watcher in elastic/x-pack-elasticsearch#2808.

For security reasons, ML datafeeds should not run as the _xpack
user.  Instead, they record the security headers from the request
to create/update them, and reuse these when performing the search
to retrieve data for analysis.

Relates elastic/x-pack-elasticsearch#1071

Original commit: elastic/x-pack-elasticsearch@29f85de404
2017-12-11 13:01:16 +00:00
Alexander Reelsen 6bae4681e2 Tests: Replace http input integration test with REST test (elastic/x-pack-elasticsearch#3215)
Remove HttpInputIntegrationTests, which only tested existing
functionality, the remaining part was moved over into a REST tests.

relates elastic/x-pack-elasticsearch#3210

Original commit: elastic/x-pack-elasticsearch@474e5337b6
2017-12-11 13:08:58 +01:00
Igor Motov 018d4d7722 Merge remote-tracking branch 'elastic/master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@a113c54b94
2017-12-09 12:21:07 -05:00
Tim Brooks 9dbb887a58 Update x-pack for change to channel tracking (elastic/x-pack-elasticsearch#3261)
This commit updates x-pack to be compatible with
elastic/elasticsearch#27711. That commit removed the need for channels
to be internally tracked inside transport implementations. This commit
removes a test mocking class that is not necessary after that change.

Original commit: elastic/x-pack-elasticsearch@75d99ba1d1
2017-12-08 16:57:08 -07:00
Lee Hinman 91d1812cb8 Rename x-pack-common -> x-pack-core (elastic/x-pack-elasticsearch#3272)
The folder has to be named 'x-pack-core' (not 'core'), otherwise Gradle flips
out and has circular dependencies.

Relates to elastic/x-pack-elasticsearch#3233

Original commit: elastic/x-pack-elasticsearch@f756e5d356
2017-12-08 14:42:50 -07:00
Igor Motov d3cd462700 Merge remote-tracking branch 'elastic/master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@496f9cc46b
2017-12-08 13:22:20 -05:00
Tim Brooks a1d72fc004 Modify security transport to take PageCacheRecycler (elastic/x-pack-elasticsearch#3251)
This is related to elastic/elasticsearhc#27696. That commit introduced
signature changes to `NetworkPlugin.getTransports`. This commit updates
x-pack to be compatible with that change.

Original commit: elastic/x-pack-elasticsearch@2e331ef6cd
2017-12-08 10:39:41 -07:00
Igor Motov 4330195cdf Merge remote-tracking branch 'elastic/master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@f4ef2d9cdf
2017-12-08 12:35:59 -05:00
Jason Tedor 27a3a80c07 Adapt to core Strings method rename
This commit adapts a single usage of a method that was renamed in core.

Relates elastic/x-pack-elasticsearch#3265

Original commit: elastic/x-pack-elasticsearch@107c495d63
2017-12-08 12:17:29 -05:00
Igor Motov b72a9b5674 SQL: switch from _sql/ endpoint to _xpack/sql endpoint (elastic/x-pack-elasticsearch#3270)
As a part of xpack, SQL should use _xpack/sql endpoint instead of _sql endpoint.

relates elastic/x-pack-elasticsearch#3114

Original commit: elastic/x-pack-elasticsearch@f561b57f16
2017-12-08 12:17:26 -05:00
Lee Hinman 24d91298db Create skeleton for plugin split (elastic/x-pack-elasticsearch#3233)
This creates a basic skeleton for the plugin split by adding folders and example
`build.gradle` files. It also includes a non-implemented `migrate-plugins.sh`
script that we can fill in at a later time.

Relates to elastic/x-pack-elasticsearch#2925

Original commit: elastic/x-pack-elasticsearch@2ab035d6b6
2017-12-08 09:41:32 -07:00
Ioannis Kakavas aff5cc840f Make generated passwords shell safe (elastic/x-pack-elasticsearch#3253)
Generate passwords from  [A-Za-z0-9] so that they are safe to be
used in shell scripts.
Entropy deterioration is not significant (124.9 -> 119), generated
passwords still meet guidelines and best practices regarding length
and complexity.

Resolves elastic/x-pack-elasticsearch#3087

Original commit: elastic/x-pack-elasticsearch@078639e7c2
2017-12-08 18:03:13 +02:00
Dimitrios Athanasiou 4082d6fa48 [Watcher] Remove line that accidentally sneaked in
Original commit: elastic/x-pack-elasticsearch@e63b1942ca
2017-12-08 13:47:53 +00:00
Dimitris Athanasiou 434dc94eb2 [ML] Avoid all overhead when renormalization window is zero (elastic/x-pack-elasticsearch#3255)
relates elastic/x-pack-elasticsearch#3244

Original commit: elastic/x-pack-elasticsearch@fe41c23ad7
2017-12-08 12:22:52 +00:00
Yannick Welsch 9ddbef0641 Fix long path issue of bwc checkouts on Windows (elastic/x-pack-elasticsearch#3259)
Hopefully fixes the Windows CI failures that break on cloning the repository into a target directory with a lengthy path name.

Original commit: elastic/x-pack-elasticsearch@fe18e95d3f
2017-12-07 19:04:40 +01:00
Jason Tedor 0c6ec82317 Grant Netty necessary permissions
When using the security networking implementations, the Netty jars that
are in play are those that are loaded in the X-Pack classloader. This
means that permissions granted to the Netty jars loaded in the
transport-netty4 module classloader do nothing. Instead, we have to
grant the same permissions to the Netty jars in the X-Pack
classloader. This commit does this.

Relates elastic/x-pack-elasticsearch#3247

Original commit: elastic/x-pack-elasticsearch@91780597b9
2017-12-07 12:57:48 -05:00
Jay Modi c6799de2a4 Do not enforce TLS if discovery type is single-node (elastic/x-pack-elasticsearch#3245)
This commit adds a check for the discovery type so that the TLS join validator does not fail join
requests when the discovery type is single-node.

relates elastic/x-pack-elasticsearch#2828

Original commit: elastic/x-pack-elasticsearch@fdfdb76b0b
2017-12-07 09:50:25 -07:00
David Kyle e9d9199205 [ML] Special Events (elastic/x-pack-elasticsearch#2930)
* Add Special Event

* Add special events to update process

* Add time condition and skip rule actions.

* Update special events

* Address review comments

Original commit: elastic/x-pack-elasticsearch@80500ded76
2017-12-07 11:44:12 +00:00
javanna 8c61cabe87 remove audit logging changes added for delayed actions that are now removed
This commit reverts part of elastic/x-pack-elasticsearch#2210

Original commit: elastic/x-pack-elasticsearch@75fda79851
2017-12-06 13:58:17 +01:00
javanna 08950ff491 Remove security filter, replaced by get index api call which returns filtered mappings
Given that we get now filtered mappings directly from the get index API (in case security is configured with FLS), we don't need the security filter nor the filtered catalog. That means we can remove the delayed action support also from AuthorizationService and rather make SQLAction a composite action like others. It will be authorized as an action, but its indices won't be checked while that will happen with its inner actions (get index and search) which need to be properly authorized.

Also, SQLGetIndicesAction is not needed anymore, as its purpose was to retrieve the indices access resolver put in the context by the security plugin for delayed actions, which are not supported anymore.

This commit kind of reverts elastic/x-pack-elasticsearch#2162, as it is now possible to integrate with security out-of-the-box

relates elastic/x-pack-elasticsearch#2934

Original commit: elastic/x-pack-elasticsearch@64d5044426
2017-12-06 13:58:17 +01:00
Tim Vernum 628dfaa843 Add API for SSL certificate information (elastic/x-pack-elasticsearch#3088)
Exposes the certificate location (configured path), serial number, and expiry date

Closes: elastic/x-pack-elasticsearch#2795

Original commit: elastic/x-pack-elasticsearch@a0773f6840
2017-12-06 19:57:25 +10:00
javanna 66c4962be9 Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@00ac1d607b
2017-12-06 09:31:20 +01:00
Nhat Nguyen 7f553f391f TEST: fix index template without index patterns
Relates https://github.com/elastic/elasticsearch/pull/27662

Original commit: elastic/x-pack-elasticsearch@7f2766695f
2017-12-05 21:25:53 -05:00
Nhat Nguyen 3cbb69eb3d Fix index template tests
Relates https://github.com/elastic/elasticsearch/pull/27662

Original commit: elastic/x-pack-elasticsearch@a2175c1689
2017-12-05 18:27:45 -05:00
javanna 915d95bca1 fix line length
Original commit: elastic/x-pack-elasticsearch@c2e67e68bd
2017-12-05 22:22:17 +01:00
javanna 626c74a437 Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@67f8321368
2017-12-05 21:50:35 +01:00
Luca Cavanna 81dcd8c5f1 Filter mappings fields when field level security is configured (elastic/x-pack-elasticsearch#3173)
This PR uses a new extension point that's being added to Elasticsearch (see https://github.com/elastic/elasticsearch/pull/27603) so that the security plugin can filter the mappings fields returned by get index, get mappings, get field mappings and field capabilities API.

This effort aims at filtering information returned by API in the `indices/admin` category and field capabilities. It doesn't filter what the cluster state api returns as that is a cluster level operation.

One question is about backwards compatibility given that we would like to have this in 6.2. Shall we treat this as a bug as mappings should have been filtered before? Not sure if it's going to break existing integrations.

relates elastic/x-pack-elasticsearch#340

Original commit: elastic/x-pack-elasticsearch@d7e3fd3fa1
2017-12-05 20:32:17 +01:00
Chris Earle 48c8aed373 [Monitoring] Support Beats Monitoring (elastic/x-pack-elasticsearch#3208)
This reintroduces support for Beats monitoring.

Original commit: elastic/x-pack-elasticsearch@539da3afa1
2017-12-05 13:44:22 -05:00
David Kyle 4cd8f075b9 [ML] Add missing job Ids to log messages
Original commit: elastic/x-pack-elasticsearch@56855a3808
2017-12-05 17:43:34 +00:00
David Roberts 751caaae76 [ML] Set established model memory on job open for pre-6.1 jobs (elastic/x-pack-elasticsearch#3222)
Before this was done it was easy to get into the situation where a
job created in 5.x with a default model memory limit of 4GB could not
be opened on any node in the cluster.  Following this change this
problem will no longer occur for jobs that ran for a decent amount of
time on the old cluster.

relates elastic/x-pack-elasticsearch#3181

Original commit: elastic/x-pack-elasticsearch@cb029debba
2017-12-05 17:05:58 +00:00
Alexander Reelsen 9eb2e14981 Watcher: Ensure watcher thread pool size is reasonably bound (elastic/x-pack-elasticsearch#3056)
The watcher threadpool size was always five times the CPU core
count, resulting in a huge threadpool when with even 24 cores.

This changes the behaviour to be five times the number of cores
by default - as watcher is usually waiting on I/O you should have more
threads than cores, but it maxes out with 50 threads, unless the number
of available cores is higher than that.

relates elastic/x-pack-elasticsearch#3052

Original commit: elastic/x-pack-elasticsearch@eab5deb113
2017-12-05 16:09:46 +01:00
Alexander Reelsen c3e5a20242 Monitoring: Disable security for integration tests (elastic/x-pack-elasticsearch#3174)
In order to support the source directory repo split, this commit
disables security for the regular integration tests.

The MonitoringSettingsFilterTests already existed as REST test, so
this test has been removed.

Relates elastic/x-pack-elasticsearch#2925

Original commit: elastic/x-pack-elasticsearch@519154dd5f
2017-12-05 12:07:04 +01:00
Alexander Reelsen fdb02f4f99 Watcher: Fix pagerduty action to send context data (elastic/x-pack-elasticsearch#3185)
The pagerduty action allows to send contexts, which contains an array
of texts or images, each with a link.

The field of this data was named 'context' instead of 'contexts' and
thus those contects were never correctly parsed on the pagerduty side.

Unfortunately pagerduty accepts any JSON, thus this was not caught so
far.

This commit allows parsing of the old field name to retain BWC, but when
written out via toXContent, it will always use the 'contexts' field name.

relates elastic/x-pack-elasticsearch#3184

Original commit: elastic/x-pack-elasticsearch@50f0b65d56
2017-12-05 11:31:48 +01:00
jaymode 6487557e61 Test: fix min value being greater than max value in IndexLifecycleManagerIntegTests
This commit fixes the minimum value being smaller than the maximum value in a call to the
scaledRandomIntBetween, which causes the test to fail.

Original commit: elastic/x-pack-elasticsearch@da7d0ce0ce
2017-12-04 10:45:18 -07:00
Dimitris Athanasiou 30b745f846 [ML] Frequency in datafeeds with aggs must be multiple of hist interval (elastic/x-pack-elasticsearch#3205)
relates elastic/x-pack-elasticsearch#3204

Original commit: elastic/x-pack-elasticsearch@0bbd9addd4
2017-12-04 15:22:56 +00:00
David Kyle d39c8b76db [ML] Fix test after elastic/x-pack-elasticsearch#3202
Original commit: elastic/x-pack-elasticsearch@c83c3ebcc1
2017-12-04 13:48:24 +00:00
David Kyle cb9314ba78 [ML] Remove confusing datafeed log message (elastic/x-pack-elasticsearch#3202)
Original commit: elastic/x-pack-elasticsearch@b8ec3d06c9
2017-12-04 13:08:49 +00:00
David Roberts 2c978842da [ML] Avoid spurious logging when deleting lookback job from the UI (elastic/x-pack-elasticsearch#3193)
When you click "delete" in the UI it force-deletes the datafeed then
force-deletes the job.  For a datafeed doing lookback, this results
in a close followed very quickly by a kill on the autodetect process.
Depending on thread scheduling this could cause a lot of spurious
errors and exception traces to be logged.

This change prevents the log spam in this scenario.

relates elastic/x-pack-elasticsearch#3149

Original commit: elastic/x-pack-elasticsearch@091240f32a
2017-12-04 10:29:05 +00:00
jaymode 409492808f Test: bound number of requests in IndexLifecycleManagerIntegTests
The IndexLifecycleManagerIntegTests has a method that tests concurrency and executes a large number
of requests in parallel. On some machines, this can actually overwhelm a thread pool and cause the
test to fail for the wrong reasons. This commit bounds the total number of requests to 100.

Original commit: elastic/x-pack-elasticsearch@07613ada51
2017-12-01 09:40:53 -07:00
Alexander Reelsen f816b2e850 Monitoring: Move watcher tests for repository split preparations (elastic/x-pack-elasticsearch#3183)
Created a smoke-test-monitoring-with-watcher project that runs REST
tests with watcher enabled to ensure that the proper watcher are
installed either when the local or the HTTP exporter are set up.

Also removed two more watcher imports in the tests.

Relates elastic/x-pack-elasticsearch#2925

Original commit: elastic/x-pack-elasticsearch@0a9abc3185
2017-12-01 13:20:05 +01:00
David Roberts d08944b1ff [TEST] Fix seed-specific failure in ML node allocation test
This was due to a hardcoded job ID that wouldn't exist when two random
numbers generated by the test framework were both small.

Original commit: elastic/x-pack-elasticsearch@9529e6d280
2017-12-01 11:12:55 +00:00
Tim Sullivan 6f4484c287 [Monitoring] Update Beats Template with Metrics (elastic/x-pack-elasticsearch#2902)
* [Monitoring] Update Beats Template with Metrics

- remove source_node.timestamp
- includes mapping for beat type = Metricbeat

* remove metricbeat and xpack objects which are not used in the UI

* use long instead of integer type

Original commit: elastic/x-pack-elasticsearch@083b9cc575
2017-11-30 13:33:16 -07:00
Costin Leau 7cab29760d SQL: Introduce PreAnalyze phase to resolve catalogs async (elastic/x-pack-elasticsearch#2962)
SQL: Introduce PreAnalyze phase to resolve catalogs async

The new preanalyze phase collects all unresolved relations and tries
to resolve them as indices through typical async calls _before_ starting the analysis process.
The result is loaded into a catalog which is then passed to the analyzer.
While at it, the analyzer was made singleton and state across the engine
is done through SqlSession#currentContext().


Commit missing fix
Fix typo
Fix license
Fix line length
remove redundant static modifier
Remove redundant generics type
Rename catalogResolver instance member to indexResolver
Fix translate action to return a response through the listener, it hangs otherwise
IndexResolver improvements

Make sure that get index requests calls are locally executed by providing local flag.
Don't replace index/alias name with concrete index name in asCatalog response conversion. We need to preserve the original alias name for security, so it is reused in the subsequent search.
Update roles and actions names for security tests
Get index is now executed instead of sql get indices, and sql get indices has been removed.
Also made cluster privileges more restrictive to make sure that cluster state calls are no longer executed.
Fix most of the security IT tests
indices options are now unified, always lenient. The only situation where we get authorization exception back is when the user is not authorized for the sql action (besides for which indices).
Improve SessionContext handling
Fix context being invalid in non-executable phases
Make Explain & Debug command fully async
Resolve checkstyle error about redundant modifiers
Temporarily restore SqlGetIndicesAction

SqlGetIndicesAction action is still needed in RestSqlJdbcAction (metaTable and metaColumn methods), where we can't at the moment call IndexResolver directly, as security (FLS) needs index resolver to be called as part of the execution of an indices action. Once mappings are returned filtered, delayed action and the security filter will go away, as well as SqlGetIndicesAction.
SqlGetIndicesAction doesn't need to be a delayed action, my bad

[TEST] remove unused expectSqlWithAsyncLookup and rename expectSqlWithSyncLookup to expectSqlCompositeAction
Polish and feedback
Add unit test for PreAnalyzer

Original commit: elastic/x-pack-elasticsearch@57846ed613
2017-11-30 18:18:08 +02:00
Albert Zaharovits 3ea5a6df91 Augment audit authz event with role names data (elastic/x-pack-elasticsearch#3100)
Audit authz events (accessGranted, accessDenied, runAsGranted
and runAsDenied) include role names.

Original commit: elastic/x-pack-elasticsearch@6a94f65962
2017-11-30 15:56:00 +02:00
Tim Vernum 4262b29188 [Security] Add DEBUG logging on role resolution (elastic/x-pack-elasticsearch#3138)
This change adds some debug and trace logging when we look up role names, to explain how each role was resolved.

At the moment we have very little insight into how roles are being resolved which can make it difficult to diagnose some issues.

Original commit: elastic/x-pack-elasticsearch@1b3c246186
2017-11-30 21:34:07 +10:00
Albert Zaharovits a5fe074b5c LoggingAuditTrail emit events before local address available (elastic/x-pack-elasticsearch#3061)
Register LoggingAuditTrail as a ClusterStateListener.
Avoid querying for the localNode while on the ClusterStateApplier thread,
which tripps assertion. This can happen when logging audit events that
originate from the system.

relates elastic/x-pack-elasticsearch#3057

Original commit: elastic/x-pack-elasticsearch@66bc59682d
2017-11-30 12:42:34 +02:00
Hendrik Muhs 756d878983 [ML-FC] do not allow durations below the bucket span (elastic/x-pack-elasticsearch#3166)
do not allow durations below the bucket span

Original commit: elastic/x-pack-elasticsearch@0e895c1ddd
2017-11-30 07:49:21 +01:00
Tim Vernum c2ff796fea Support realm validation when Keystore is closed (elastic/x-pack-elasticsearch#3096)
If the KeyStoreWrapper is closed, then we cannot validate secure settings (because we no longer have access to them)
The Realm group setting uses the "validate" method to ensure that child settings are correct, but it must ignore secure settings as it might get called
after startup (e.g. during a settings diff)

Original commit: elastic/x-pack-elasticsearch@b30db6bc62
2017-11-30 11:22:17 +10:00
Michael Basnight bf27cd1457 Use SPI to glean reserved roles (elastic/x-pack-elasticsearch#3012)
This commit uses SPI to get the list of system wide reserved roles in
security. It does not yet split out the RoleDescriptor to a common
location so the implementing services still depend on security. Each
role, however, only depends on its own feature as well as security.

ref elastic/x-pack-elasticsearch#2925

Original commit: elastic/x-pack-elasticsearch@efebc3e5c8
2017-11-29 09:00:23 -06:00
David Kyle 171c48fd2f [TESTS] Refactor yml test suite classes (elastic/x-pack-elasticsearch#3145)
* Use XPackRestIT as base class for XDocsClientYamlTestSuiteIT

* Remove the XPackRestTestCase class

* Address review comments

* Fix checkstyle checks

Original commit: elastic/x-pack-elasticsearch@c2a5e60c12
2017-11-29 12:43:53 +00:00
David Roberts ef96831515 [ML] Don't mention unrelated indices when deleting job index aliases (elastic/x-pack-elasticsearch#3160)
This change fixes a problem that would cause job deletion to fail if ANY
index had a block on it, e.g. read-only.

The problem was that we were requesting the job aliases be deleted from
ALL indices in the system due to a misunderstanding with the format of the
get_aliases response.  This didn't usually cause any noticable effects, as
only the ML indices would have the aliases.  But in the case of a read-only
index it would cause an error, leading to unnecessary failure of the job
deletion.

Fixes elastic/machine-learning-cpp#428

Original commit: elastic/x-pack-elasticsearch@a573f85a00
2017-11-29 11:39:30 +00:00
Dimitris Athanasiou 9ef9edc1ca [ML] Rework ScrollDataExtractorTests to make scroll clearing more robust (elastic/x-pack-elasticsearch#3150)
Original commit: elastic/x-pack-elasticsearch@7784497a3d
2017-11-29 11:19:13 +00:00
David Roberts 8cb6e63a0e [ML] Increase default limit on ML jobs per node from 10 to 20 (elastic/x-pack-elasticsearch#3141)
Following the changes of elastic/x-pack-elasticsearch#2975 the hard limit on the number of ML jobs
per node is no longer the only limiting factor.  Additionally there is
now a limit based on the estimated memory usage of the jobs, and this is
expected to provide a more sensible limit that accounts for differing
resource requirements per job.

As a result, it makes sense to raise the default for the hard limit on
the number of jobs, on the assumption that the memory limit will prevent
the node becoming overloaded if an attempt is made to run many large jobs.
Increasing the hard limit will allow more small jobs to be run than was
previously the case by default.

Of course, this change to the default will have no effect for customers
who have already overridden the default in their config files.

Original commit: elastic/x-pack-elasticsearch@9fed1d1237
2017-11-28 20:40:55 +00:00
Nik Everett 0cc153f6d3 Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@ccc2fc708e
2017-11-28 13:24:36 -05:00
Dimitris Athanasiou d96be6c51f [ML] Only clear scroll whan a scroll id exists (elastic/x-pack-elasticsearch#3148)
The issue here is that if the first search request fails
(initScroll), then we do not have a scroll_id. However,
in order to retry the search, we reset the scroll. That
involves clearing the current search, but since we do
not have a scroll_id, the clear scroll request fails.
We end up reporting the failure for the scroll clearing,
rather than the actual problem.

This commit fixes that by avoiding clearing the scroll
when the scroll_id is null.

relates elastic/x-pack-elasticsearch#3146

Original commit: elastic/x-pack-elasticsearch@b5086028f6
2017-11-28 16:48:24 +00:00
Jay Modi fa33f45cfb Test: remove KnownActionsTests (elastic/x-pack-elasticsearch#3133)
This commit removes the KnownActionsTests as it no longer serves the intended purpose of catching
actions/handlers added to elasticsearch that security needs to be aware of. Today, it is common
for this test to break the build and as a mechanical response developers just add to the actions
or handlers file to get the build green again.

Relates elastic/x-pack-elasticsearch#1489

Original commit: elastic/x-pack-elasticsearch@0bdb5341f5
2017-11-28 08:06:36 -07:00
Michael Basnight c7fea95476 Revert " Add "client-api-objects" dependency for xpack plugin and transport-client (elastic/x-pack-elasticsearch#2995)" (elastic/x-pack-elasticsearch#3083)
This reverts commit elastic/x-pack-elasticsearch@a6d83299d0.

Original commit: elastic/x-pack-elasticsearch@ca55ee747c
2017-11-28 09:05:00 -06:00
Dimitris Athanasiou 83e28bea3e [ML] Increase autodetect thread pool queue size (elastic/x-pack-elasticsearch#3142)
relates elastic/x-pack-elasticsearch#2981

Original commit: elastic/x-pack-elasticsearch@86719ada89
2017-11-28 14:46:04 +00:00
Adrien Grand 4e25ffad39 Fix compilation with Lucene 6.2.
Original commit: elastic/x-pack-elasticsearch@a34002a11c
2017-11-28 15:00:53 +01:00
Alexander Reelsen cdb85d8317 Watcher: Run tests without security enabled (elastic/x-pack-elasticsearch#3060)
In order to support the repository split, this changes the
`AbstractWatcherIntegrationTestCase` to not run with security enabled.

We have a dedicated QA project called `smoke-test-watcher-with-security`,
where tests that explicitely need security should be running.

This commit removes the possibility to enable security as part of the
test case. In addition some tests have been moved over to the dedicated
project.

In addition the `timewarp` functionality cannot be configured with a
system property anymore. This would not have worked anyway, because
tests were already dependent on that  functionality and did not have any
other means of running. A bit of redundant code was removed due to this.

Relates elastic/x-pack-elasticsearch#2925

Original commit: elastic/x-pack-elasticsearch@b24b365ad1
2017-11-28 13:11:49 +01:00
Dimitris Athanasiou 3e52e0ba48 [ML] Validate duration and expires_in params in forecast API (elastic/x-pack-elasticsearch#3139)
Relates elastic/machine-learning-cpp#443

Original commit: elastic/x-pack-elasticsearch@f42e4490d1
2017-11-28 10:57:48 +00:00
Dimitris Athanasiou e396c61afc [ML] Remove forecast end param (elastic/x-pack-elasticsearch#3121)
The forecast API provides a `duration` parameters
which is the most convenient way of specifying
the span of the forecast. End time is now unnecessary
and possibly confusing.

Relates elastic/machine-learning-cpp#443

Original commit: elastic/x-pack-elasticsearch@04eb0408e7
2017-11-28 10:49:15 +00:00
David Roberts 220d0647b8 [ML] Specify ML_ORIGIN when batch scrolling results (elastic/x-pack-elasticsearch#3125)
This change applies the same pattern that was applied in elastic/x-pack-elasticsearch#3054 to the
ML batched results iterators, which are used to scroll through ML results
during some internal ML implementation details, such as renormalization
and nightly maintenance.

Additionally the thread context is reset before submitting the results
processor to a thread pool, to avoid masking the problem in situations
where the user opening the job coincidentally had workable permissions.

Fixes elastic/machine-learning-cpp#438

Original commit: elastic/x-pack-elasticsearch@bd1e2dc7d4
2017-11-28 09:48:49 +00:00
Igor Motov 5c88fa0b3b SQL: Add support for plain text output to /_sql endpoint (elastic/x-pack-elasticsearch#3124)
The /_sql endpoint now returns the results in the text format by default. Structured formats are also supported using the format parameter or accept header similar to _cat endpoints.

Original commit: elastic/x-pack-elasticsearch@4353793b83
2017-11-27 18:10:13 -05:00
Nik Everett 378abf1d8f SQL: Basic REST spec and tests (elastic/x-pack-elasticsearch#3128)
Adds a basic REST spec and tests for the SQL and translate endpoints so
that clients can can execute these endpoints. We'll keep our complex
REST testing in Java REST tests though.

relates elastic/x-pack-elasticsearch#3115

Original commit: elastic/x-pack-elasticsearch@c5de301f3d
2017-11-27 14:57:12 -05:00
Igor Motov 626e9b87a1 Merge remote-tracking branch 'elastic/master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@6c5a683209
2017-11-27 09:26:14 -05:00
Alexander Reelsen 6406c9816a Watcher: Add transform input for chained input (elastic/x-pack-elasticsearch#2861)
The chained input in watcher is a useful feature to
call several endpoints before execution a condition.
However it was pretty hard to modify data from a previous
input in order to be able to execute it in another input.

This commit adds a another input, called a `transform` input,
which allows you to do a transform as another input in a chained
input.

See this example

```
"input" : {
  "chain" : {
    "inputs" : [ <1>
      {
        "first" : {
          "simple" : { "path" : "/_search" }
        }
      },
      {
        "second" : {
          "transform" : {
            "script" : "return [ 'path' : 'ctx.payload.first.path' + '/' ]"
          }
        }
      },
      {
        "third" : {
          "http" : {
            "request" : {
              "host" : "localhost",
              "port" : 9200,
              "path" : "{{ctx.payload.second.path}}" <2>
            }
          }
        }
      }
    ]
  }
}
```

This allows for far more flexibility before executing the next input in a chained
one.

Original commit: elastic/x-pack-elasticsearch@3af9ba6e9b
2017-11-27 13:27:56 +01:00
David Roberts 304330e1bc [ML] Specify ML_ORIGIN when calling the job update endpoint internally (elastic/x-pack-elasticsearch#3110)
This change applies the same pattern that was applied in elastic/x-pack-elasticsearch#3054 to a new
method that was introduced in elastic/x-pack-elasticsearch#2975 which was in-flight at the same time
so missed the original batch of changes.

relates elastic/x-pack-elasticsearch#3109

Original commit: elastic/x-pack-elasticsearch@8076c6cf6a
2017-11-27 09:18:24 +00:00
Jason Tedor c777c1d36f Refactor CLI commands as logging-aware commands
We have to ensure logging is configured for any CLI command that depends
on core Elasticsearch (since it might directly or indirectly touch
logging). This commit does this for all commands in X-Pack.

Relates elastic/x-pack-elasticsearch#3112

Original commit: elastic/x-pack-elasticsearch@f77f9b5052
2017-11-25 11:40:29 -05:00
Dimitris Athanasiou eb4186dd5c [ML] Stop datafeed when job fails (elastic/x-pack-elasticsearch#3107)
The problem here was that when the autodetect process crashes
we set the job state to FAILED but we did not remove the
communicator from the map in AutodetectProcessManager.

relates elastic/x-pack-elasticsearch#2773

Original commit: elastic/x-pack-elasticsearch@9b8eafb4d0
2017-11-24 15:04:29 +00:00
Alexander Reelsen d89d8abec9 Watcher: Fix equals/hashcode for WatchStatus (elastic/x-pack-elasticsearch#3105)
This was missed in elastic/x-pack-elasticsearch#3103 and fixes to add the headers variable to both
methods to ensure comparisons work as expected.

Original commit: elastic/x-pack-elasticsearch@df5e422698
2017-11-24 15:32:25 +01:00
Alexander Reelsen 4fe9ac734b Watcher: Store thread context headers in watch (elastic/x-pack-elasticsearch#2808)
In order to be able to execute a watch as the user, who stored the
watch, this commit stores certain headers of the thread context, that
was used when the watch was stored.

Upon loading the watch the headers are loaded and applied for the
following watcher execution features

* search transform
* search input
* index action

A special case is the execute watch API, which overrides the headers loaded
from the watch with the one of the current request, so that a user
cannot execute this watch with other privileges of the user who stored it.

Only the headers "es-security-runas-user", "_xpack_security_authentication" are
copied for now, as those are needed for our security features.

The headers are stored in watch status in the watch and are not returned by default,
when the GET Watch API is used. A search reveals those of course.

relates elastic/x-pack-elasticsearch#2201

Original commit: elastic/x-pack-elasticsearch@9803bd51c2
2017-11-24 09:15:54 +01:00
David Turner 3e8b3491d5 Consolidate version numbering semantics (elastic/x-pack-elasticsearch#3078)
Fixes to the build system, particularly around BWC testing, and to make future
version bumps less painful.

Original commit: elastic/x-pack-elasticsearch@a1d456f30a
2017-11-23 20:23:05 +00:00
Alexander Reelsen 00ea3e8fc7 Watcher: Transform also needs to use stashing client (elastic/x-pack-elasticsearch#3098)
... in order to ensure that the xpack user is used.

Relates elastic/x-pack-elasticsearch#3054

Original commit: elastic/x-pack-elasticsearch@6edfcc3d87
2017-11-23 16:13:31 +01:00
Dimitris Athanasiou e0affd455d [ML] Change forecast_id to UUid, add create_time and start_time (elastic/x-pack-elasticsearch#3095)
relates elastic/x-pack-elasticsearch#3093

Original commit: elastic/x-pack-elasticsearch@f586189851
2017-11-23 14:46:52 +00:00
jaymode a8b5b138a7 Test: print cluster state xcontent on security index check failures
Original commit: elastic/x-pack-elasticsearch@0ff85f851c
2017-11-22 12:57:10 -07:00
Nik Everett f97f56ba54 SQL: Throw exceptions on errors (elastic/x-pack-elasticsearch#3066)
Instead of returning "error response" objects and then translating them
into SQL exceptions this just throws the SQL exceptions directly. This
means the CLI catches exceptions and prints out the messages which isn't
ideal if this were hot code but it isn't and this is a much simpler way
of doing things.

Original commit: elastic/x-pack-elasticsearch@08431d3941
2017-11-22 11:22:31 -05:00
Jay Modi 0a683a0e18 Remove InternalClient and InternalSecurityClient (elastic/x-pack-elasticsearch#3054)
This change removes the InternalClient and the InternalSecurityClient. These are replaced with
usage of the ThreadContext and a transient value, `action.origin`, to indicate which component the
request came from. The security code has been updated to look for this value and ensure the
request is executed as the proper user. This work comes from elastic/x-pack-elasticsearch#2808 where @s1monw suggested
that we do this.

While working on this, I came across index template registries and rather than updating them to use
the new method, I replaced the ML one with the template upgrade framework so that we could
remove this template registry. The watcher template registry is still needed as the template must be
updated for rolling upgrades to work (see elastic/x-pack-elasticsearch#2950).

Original commit: elastic/x-pack-elasticsearch@7dbf2f263e
2017-11-22 08:35:18 -07:00
Alexander Reelsen c7a64667d4 Watcher: Properly url encode room names (elastic/x-pack-elasticsearch#2896)
Room names in hipchat were not properly URL encoded, thus room names
with spaces would not work as expected. This fixes all the hipchat
accounts by properly using spaces.

Also the hipchat tests are reenabled, as the IT team gave me new access to hipchat, 
allowing to create a fresh set of oauth tokens for the integration account type.

The HipchatServiceTests have also been converted to XPackSingleNodeTestCase

relates elastic/x-pack-elasticsearch#2371
relates elastic/x-pack-elasticsearch#2429

Original commit: elastic/x-pack-elasticsearch@9f8872f686
2017-11-22 15:50:18 +01:00
Dimitrios Athanasiou 02c83a3b6a [ML][TEST] Add delays between forecasts to ensure id uniqueness
relates elastic/x-pack-elasticsearch#3090

Original commit: elastic/x-pack-elasticsearch@5d6b091607
2017-11-22 12:04:44 +00:00
David Kyle 0dea758022 [ML] Log deprecation warning for jobs with delimited formats (elastic/x-pack-elasticsearch#3092)
Original commit: elastic/x-pack-elasticsearch@bd75fae990
2017-11-22 11:53:08 +00:00
Dimitris Athanasiou 74beb9ca64 [ML] Remove expired forecasts (elastic/x-pack-elasticsearch#3077)
Closes elastic/machine-learning-cpp#322


Original commit: elastic/x-pack-elasticsearch@5249452a86
2017-11-21 17:18:04 +00:00
Jay Modi 4ae1ca5fa5 Security: IndexLifecycleManager provides a consistent view of index state (elastic/x-pack-elasticsearch#3008)
This commit changes the IndexLifecycleManager's handling of variables about an index to only update
all of the values at a single time. Previously, all of the index state variables were volatile
members of the IndexLifecycleManager, which meant we could get an inconsistent view of the index
state. Although rare, this is still incorrect so this change adds a single volatile variable that
holds the state as of the last processed cluster state update.

Additionally, the IndexLifecycleManagerIntegTests were updated to have more concurrency and further
stress this portion of the code and its checks.

relates elastic/x-pack-elasticsearch#2973

Original commit: elastic/x-pack-elasticsearch@5f1552b298
2017-11-21 10:17:08 -07:00
Jay Modi d86e7870da Security: add manage_index_templates to the kibana_system role (elastic/x-pack-elasticsearch#3009)
This commit adds the manage_index_templates permission to the kibana_system role that is used by
the kibana system user. This is needed due to an upcoming feature in kibana where a index template
will be used to create the saved objects index.

relates elastic/x-pack-elasticsearch#2937

Original commit: elastic/x-pack-elasticsearch@85a67c73aa
2017-11-21 08:45:07 -07:00
Nik Everett 0d4a91af50 Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@989e27840f
2017-11-21 10:34:23 -05:00
David Roberts 35551859c5 [TEST] Fix side effects of elastic/x-pack-elasticsearch#2975 on build servers with very little RAM
Some of our REST tests open many jobs, and assuming each will use 1GB of
RAM on a single node cluster could fail the test.  The solution is to
explicitly say the test jobs will use very little RAM.

Original commit: elastic/x-pack-elasticsearch@a3fcfc4589
2017-11-21 15:11:28 +00:00
Hendrik Muhs cc66020cf3 [ML-FC] add expires_in parameter and change forecast_start_timestamp to timestamp (elastic/x-pack-elasticsearch#3073)
add expires_in parameter and change forecast_start_timestamp to timestamp

depends on elastic/machine-learning-cpp#421

Original commit: elastic/x-pack-elasticsearch@3a3eebd49c
2017-11-21 15:32:06 +01:00
Simon Willnauer 601222903d X-Pack side of elastic/elasticsearch#27469 (elastic/x-pack-elasticsearch#3071)
Original commit: elastic/x-pack-elasticsearch@99499b6bd6
2017-11-21 15:15:24 +01:00
Dimitris Athanasiou e71b5639de [ML] Rename id to forecast_id in forecast API response (elastic/x-pack-elasticsearch#3074)
Original commit: elastic/x-pack-elasticsearch@c05d9fc602
2017-11-21 13:57:41 +00:00
javanna 61f13b9642 Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@d11ddc7a2c
2017-11-21 14:13:24 +01:00
Dimitris Athanasiou 83ca6e8064 [ML] Report errors back to listener in DeleteExpiredDataAction (elastic/x-pack-elasticsearch#3072)
Currently, any errors that occur during the DeleteExpiredDataAction are logged and the deletion proceeds to the next job. The user will get no indication in the response that something went wrong although nothing should really go wrong unless the cluster is messed up.

This commit changes this so that errors are reported back to the action.

Original commit: elastic/x-pack-elasticsearch@489cf03c3e
2017-11-21 12:03:04 +00:00
Dimitris Athanasiou 754623753a [ML] Make it easier to add various ml data removal (elastic/x-pack-elasticsearch#3048)
Original commit: elastic/x-pack-elasticsearch@3e4ac6033b
2017-11-21 11:43:01 +00:00
David Roberts 402852a4ee Update BWC version after backporting to 6.1
Relates elastic/x-pack-elasticsearch#2975

Original commit: elastic/x-pack-elasticsearch@a63c56a019
2017-11-21 10:57:44 +00:00
Hendrik Muhs 41b254cdf4 change forecast message into an array of messages (elastic/x-pack-elasticsearch#3070)
depends on elastic/machine-learning-cpp#419

Turns the forecast message into an array of messages.

Original commit: elastic/x-pack-elasticsearch@7598342712
2017-11-21 11:47:34 +01:00
Luca Cavanna 941c0a5701 Cross Cluster Search: optionally skip disconnected clusters (elastic/x-pack-elasticsearch#2823)
Original commit: elastic/x-pack-elasticsearch@3b0017df1f
2017-11-21 11:42:39 +01:00
David Roberts f06acdc219 [ML] Improve the way ML jobs are allocated to nodes (elastic/x-pack-elasticsearch#2975)
This change modifies the way ML jobs are assigned to nodes to primarily
base the decision on the estimated memory footprint of the jobs. The
memory footprint comes from the model size stats if the job has been
running long enough, otherwise from the model memory limit. In addition,
an allowance for the program code and stack is added.

If insufficient information is available to base the allocation decision on
memory requirements then the decision falls back to using simple job
counts per node.

relates elastic/x-pack-elasticsearch#546

Original commit: elastic/x-pack-elasticsearch@b276aedf2f
2017-11-21 09:51:52 +00:00
Igor Motov 0d398b19ce Merge remote-tracking branch 'elastic/master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@6b82e2c12e
2017-11-20 17:41:43 -05:00
Michael Basnight f8cb1e603b Removed unused Terminal param (elastic/x-pack-elasticsearch#2854)
Relates elastic/elasticsearch#27216

Original commit: elastic/x-pack-elasticsearch@70bdda51ef
2017-11-19 22:33:27 -06:00
Michael Basnight 316da9a970 Move the CLI into its own subproject (elastic/x-pack-elasticsearch#3032)
relates elastic/elasticsearch#27114

Original commit: elastic/x-pack-elasticsearch@70e8488223
2017-11-18 21:43:25 -06:00
Nhat Nguyen 7c9af72d6e Removes the old handler for shard snapshot status
Relates https://github.com/elastic/elasticsearch/pull/27443

Original commit: elastic/x-pack-elasticsearch@7f0021e3f4
2017-11-17 20:14:44 -05:00
jaymode 34ecd18e76 Remove use of forbidden API URL#getPath
tool. However, this is a forbidden API so this commit replaces it with URI#getPath. Additionally,
the tests fail with a security manager permission error due to the use of Mockito for exception
throwing. This commit still uses Mockito for throwing exceptions but does it differently in a way
that is acceptable by our test security policy.

Original commit: elastic/x-pack-elasticsearch@5e1d45acf8
2017-11-17 14:03:26 -07:00
Igor Motov d011247970 SQL: Fix testJdbcActionLicense test
The testJdbcActionLicense test broke because we changed the error serialization mechanism in elastic/x-pack-elasticsearch#3034. This commit updates that check to capture an exception instead of checking for ErrorResponse.

Original commit: elastic/x-pack-elasticsearch@17c41426d0
2017-11-17 15:37:02 -05:00
Igor Motov ffcd54de77 Merge remote-tracking branch 'elastic/master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@e67ceb1d1e
2017-11-17 13:33:28 -05:00
Albert Zaharovits 5b73c77011 SetupPasswordTool handle url cmd option correctly (elastic/x-pack-elasticsearch#2899)
Fixes bug when the url option had trailing slashes. The URL built
was invalid (consecutive fwd slashes) but the failure errors of
the subsequent requests were ignored.

URL is built correctly from the option spec.
True HTTP errors and Exceptions are logged and the cmd fails.

relates elastic/x-pack-elasticsearch#2778

Original commit: elastic/x-pack-elasticsearch@62b2d94ca0
2017-11-17 20:30:22 +02:00
Nhat Nguyen 418a6632e5 Registers a new handler for shard snapshot status
Relates elastic/x-pack-elasticsearch#27165

Original commit: elastic/x-pack-elasticsearch@11199642bb
2017-11-17 13:13:50 -05:00
Nik Everett b8e082107f Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@bbc72c0368
2017-11-17 12:05:47 -05:00
David Roberts 6741a62e80 [ML] Adjust memory limit test to account for greater accuracy (elastic/x-pack-elasticsearch#3047)
Due to elastic/machine-learning-cpp#409 the ML C++ code now instruments
memory more accurately.  This change modifies the expectations of the Java
integration test to account for the change.

Original commit: elastic/x-pack-elasticsearch@2ed7a75af4
2017-11-17 17:02:19 +00:00
Alexander Reelsen 0f97e28074 Watcher: Further preparations for source repo split (elastic/x-pack-elasticsearch#3006)
Changes to further prepare for feature split with regards to watcher:

- CryptoService has been moved into watcher
- CryptoService.generateKey() has been moved into SystemKeyTools, only
  used there
- The creation of the http client/notification classes have been moved
  into watcher, no further dependencies on watcher in XPackPlugin
- Each subproject now registers it's own named writeables

Relates elastic/x-pack-elasticsearch#2925

Original commit: elastic/x-pack-elasticsearch@a60c98ba7e
2017-11-17 17:05:07 +01:00
Alexander Reelsen 3d5fb54522 Watcher: Create templates on nodes newer than the master (elastic/x-pack-elasticsearch#2950)
This problem was introduced due to distributed watch execution.

When a node newer than the master node joins the cluster and gets a
.watches shard assigned it is supposed to start watcher. However
when a new version of the watch history template is part of that new
node (and we might increase that version anytime), this template does
not get installed, because only the master node is updating watcher
templates.

This commit checks if the local node version is higher than the master
node version and then also puts missing templates.

Currently this is done for all watcher templates, not only the watcher history.

relates elastic/x-pack-elasticsearch#2944

Original commit: elastic/x-pack-elasticsearch@4960231ea7
2017-11-17 16:56:29 +01:00
Hendrik Muhs a583cf270b [ML-FC] implement endpoint parameter that takes a duration (elastic/x-pack-elasticsearch#3027)
Adds a duration parameter to the forecast API. Also fixes issue if no parameter is given (forecast 1 day from time of last bucket), in which case it lets autodetect decide

depends on elastic/machine-learning-cpp#407

Original commit: elastic/x-pack-elasticsearch@3387478872
2017-11-17 06:37:03 +01:00
Tim Brooks f94e6d427b Modify x-pack to for `TcpTransport` changes
This is related to elatic/elasticsearch#27407. That commit removed
parameterization from TcpTranport. This commit modifies the security
transport to be compatible with those changes.

Original commit: elastic/x-pack-elasticsearch@9878c26e14
2017-11-16 14:32:40 -07:00
Hendrik Muhs 5efdb54eb8 add memoryUsage stats (elastic/x-pack-elasticsearch#3025)
adds memory usage statistics to forecast request stats

depends on: elastic/machine-learning-cpp#406

Original commit: elastic/x-pack-elasticsearch@abf90030fc
2017-11-16 18:38:13 +01:00
Dimitrios Athanasiou 80f4797f7a [ML][TEST] Add integ test for memory redution in population analysis
Original commit: elastic/x-pack-elasticsearch@ccd1f4835d
2017-11-16 17:10:59 +00:00
Igor Motov 8a1dd59178 Merge remote-tracking branch 'elastic/master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@4805de1874
2017-11-15 17:27:45 -05:00
Tim Brooks d45c2d784c Update x-pack to support TcpChannel (elastic/x-pack-elasticsearch#2983)
This is related to elastic/elasticsearch#27132. This commit updates
x-pack to support signature changes introduced in that PR.

Original commit: elastic/x-pack-elasticsearch@461c96ffe6
2017-11-15 12:38:56 -07:00
Lee Hinman 889b008298 Add "client-api-objects" dependency for xpack plugin and transport-client (elastic/x-pack-elasticsearch#2995)
* Add "client-api-objects" dependency for xpack plugin and transport-client

This adds another gradle project, "client-api-objects" which is intended to be a
common dependency so that the xpack plugin and transport-client can share the
same Request and Response objects.

Relates to elastic/x-pack-elasticsearch#2925

Original commit: elastic/x-pack-elasticsearch@a6d83299d0
2017-11-15 09:49:00 -07:00
Tim Brooks 5d47b96905 Modify x-pack to for `TransportChannel` changes (elastic/x-pack-elasticsearch#3013)
This is related to elastic/elasticsearch#elastic/x-pack-elasticsearch#27388. It modifies x-pack to
be compatible with the removal of the delegating transport channel.

Original commit: elastic/x-pack-elasticsearch@3bd7bf6773
2017-11-15 09:48:17 -07:00
Alexander Reelsen 36105231c3 Watcher: Return useful error message when no accounts are found (elastic/x-pack-elasticsearch#2897)
When there were no accounts configured, watcher returned a cryptic
error message containing 'null' in the description. This fix returns
a more clear error message.

On top a dedicated NotificationServiceTests class was added to remove
redundant test cases in the hipchat/jira/slack unit tests, that all
basically tested NotificationService capabilties.

relates elastic/x-pack-elasticsearch#2666

Original commit: elastic/x-pack-elasticsearch@45d0d1df31
2017-11-15 11:51:23 +01:00
Alexander Reelsen f286f9b0f2 Watcher: Remove unused test helper class
Original commit: elastic/x-pack-elasticsearch@09f66a5f52
2017-11-15 11:05:59 +01:00
Clinton Gormley d833af2046 Rest spec fixes (elastic/x-pack-elasticsearch#2965)
* Rename REST spec xpack.deprecation.info to xpack.migration.deprecations

* Fixed parameter-type naming in xpack.ml.get_model_snapshots

* Fixed QS multi-cluster search test to use cluster.remote_info

Original commit: elastic/x-pack-elasticsearch@ccd35b4a6c
2017-11-15 09:33:19 +01:00
Martijn van Groningen 56e7512500 Make the persistent task status available to PersistentTasksExecutor.nodeOperation(...) method
Original commit: elastic/x-pack-elasticsearch@ce6d788e77
2017-11-15 07:54:05 +01:00
Martijn van Groningen 439830890f Move the initialization of persistent tasks from the machine learning level to xpack level.
Today persistent tasks is only usable from machine learning, but others like ccr will need to use it too.
With this change ccr and other will be able to make use of it too.

Original commit: elastic/x-pack-elasticsearch@c90f01d5f6
2017-11-15 07:44:53 +01:00
Nik Everett 1a434636fe Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@4c504025ce
2017-11-14 14:30:12 -05:00
jaymode 2f8cd77349 Test: TribeWithSecurityIT should wait for security index to be writeable
This commit adds checks to the TribeWithSecurityIT tests to ensure that the security index is
writeable before making modification operations. Otherwise, we hit errors in tests that are not
always reproducible.

relates elastic/x-pack-elasticsearch#2977

Original commit: elastic/x-pack-elasticsearch@c29bdff7ae
2017-11-14 08:18:55 -07:00
Jay Modi 5888174f24 Cleanup leftover tribe references in the plugin build.gradle file (elastic/x-pack-elasticsearch#2987)
In elastic/x-pack-elasticsearch#2901, the dependency on the tribe module was removed but a few leftover references were missed
in the build.gradle file of the x-pack-elasticsearch plugin. This commit removes these leftover
references.

Original commit: elastic/x-pack-elasticsearch@03f1cae1f5
2017-11-14 08:11:21 -07:00
Alexander Reelsen dc42887396 Watcher: Move watcher-only packages into watcher hierarchy (elastic/x-pack-elasticsearch#2933)
In order to prepare for separate source directories, this commit moves
a few packages back into the watcher namespaces. A few of them have been
moved out previously as we thought that it might make sense to have a
dedicated notification API. This wont be the case for watcher on ES
anymore, so we can safely move those back into the watcher space.

Packages affected by this move:

* org.elasticsearch.xpack.common.http
* org.elasticsearch.xpack.common.text
* org.elasticsearch.xpack.common.secret
* org.elasticsearch.xpack.common.stats
* org.elasticsearch.xpack.support
* org.elasticsearch.xpack.notification

Tests have been moved accordingly.

The class `XContentUtils` has been split into one implementation for
watcher and one for security as different methods were used.

Relates elastic/x-pack-elasticsearch#2925

Original commit: elastic/x-pack-elasticsearch@0aec64a7e2
2017-11-14 11:35:10 +01:00
Dimitris Athanasiou e00b4bd197 [ML][TEST] Improve RevertModelSnapshotIT (elastic/x-pack-elasticsearch#2954)
X-Pack side testing of elastic/machine-learning-cpp#390



Original commit: elastic/x-pack-elasticsearch@64e8ca85eb
2017-11-14 10:32:42 +00:00
Igor Motov ea0e58f971 SQL: introduce setting to disable SQL (elastic/x-pack-elasticsearch#2966)
Adds xpack.sql.enabled setting to provide ability to disable SQL on elasticsearch nodes.

relates elastic/x-pack-elasticsearch#2872

Original commit: elastic/x-pack-elasticsearch@d13b72e9ea
2017-11-13 15:10:47 -05:00