Commit Graph

7282 Commits

Author SHA1 Message Date
Igor Motov d3cd462700 Merge remote-tracking branch 'elastic/master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@496f9cc46b
2017-12-08 13:22:20 -05:00
Tim Brooks a1d72fc004 Modify security transport to take PageCacheRecycler (elastic/x-pack-elasticsearch#3251)
This is related to elastic/elasticsearhc#27696. That commit introduced
signature changes to `NetworkPlugin.getTransports`. This commit updates
x-pack to be compatible with that change.

Original commit: elastic/x-pack-elasticsearch@2e331ef6cd
2017-12-08 10:39:41 -07:00
Igor Motov 4330195cdf Merge remote-tracking branch 'elastic/master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@f4ef2d9cdf
2017-12-08 12:35:59 -05:00
Jason Tedor 27a3a80c07 Adapt to core Strings method rename
This commit adapts a single usage of a method that was renamed in core.

Relates elastic/x-pack-elasticsearch#3265

Original commit: elastic/x-pack-elasticsearch@107c495d63
2017-12-08 12:17:29 -05:00
Igor Motov b72a9b5674 SQL: switch from _sql/ endpoint to _xpack/sql endpoint (elastic/x-pack-elasticsearch#3270)
As a part of xpack, SQL should use _xpack/sql endpoint instead of _sql endpoint.

relates elastic/x-pack-elasticsearch#3114

Original commit: elastic/x-pack-elasticsearch@f561b57f16
2017-12-08 12:17:26 -05:00
Lisa Cawley 8a0e7f58b8 [DOCS] Split ML overview topics (elastic/x-pack-elasticsearch#3262)
Original commit: elastic/x-pack-elasticsearch@793e25ff62
2017-12-08 09:07:14 -08:00
Lee Hinman 24d91298db Create skeleton for plugin split (elastic/x-pack-elasticsearch#3233)
This creates a basic skeleton for the plugin split by adding folders and example
`build.gradle` files. It also includes a non-implemented `migrate-plugins.sh`
script that we can fill in at a later time.

Relates to elastic/x-pack-elasticsearch#2925

Original commit: elastic/x-pack-elasticsearch@2ab035d6b6
2017-12-08 09:41:32 -07:00
Dimitris Athanasiou a9535c0b5a [ML][DOCS] Correct get-overall-buckets API example (elastic/x-pack-elasticsearch#3269)
Also fixes the score filters explanation for the results APIs.

Original commit: elastic/x-pack-elasticsearch@18cb31ab56
2017-12-08 16:03:51 +00:00
Ioannis Kakavas aff5cc840f Make generated passwords shell safe (elastic/x-pack-elasticsearch#3253)
Generate passwords from  [A-Za-z0-9] so that they are safe to be
used in shell scripts.
Entropy deterioration is not significant (124.9 -> 119), generated
passwords still meet guidelines and best practices regarding length
and complexity.

Resolves elastic/x-pack-elasticsearch#3087

Original commit: elastic/x-pack-elasticsearch@078639e7c2
2017-12-08 18:03:13 +02:00
Dimitrios Athanasiou 4082d6fa48 [Watcher] Remove line that accidentally sneaked in
Original commit: elastic/x-pack-elasticsearch@e63b1942ca
2017-12-08 13:47:53 +00:00
Dimitris Athanasiou 434dc94eb2 [ML] Avoid all overhead when renormalization window is zero (elastic/x-pack-elasticsearch#3255)
relates elastic/x-pack-elasticsearch#3244

Original commit: elastic/x-pack-elasticsearch@fe41c23ad7
2017-12-08 12:22:52 +00:00
Alexander Reelsen 3f17c28f9b Docs: Remove note about running watches with elevated privileges
The documentation still mentioned that watches are running with elevated
privileges, which is not true anymore since elastic/x-pack-elasticsearch#2808.

Original commit: elastic/x-pack-elasticsearch@6b510d999a
2017-12-08 11:01:28 +01:00
Yannick Welsch 9ddbef0641 Fix long path issue of bwc checkouts on Windows (elastic/x-pack-elasticsearch#3259)
Hopefully fixes the Windows CI failures that break on cloning the repository into a target directory with a lengthy path name.

Original commit: elastic/x-pack-elasticsearch@fe18e95d3f
2017-12-07 19:04:40 +01:00
Jason Tedor 0c6ec82317 Grant Netty necessary permissions
When using the security networking implementations, the Netty jars that
are in play are those that are loaded in the X-Pack classloader. This
means that permissions granted to the Netty jars loaded in the
transport-netty4 module classloader do nothing. Instead, we have to
grant the same permissions to the Netty jars in the X-Pack
classloader. This commit does this.

Relates elastic/x-pack-elasticsearch#3247

Original commit: elastic/x-pack-elasticsearch@91780597b9
2017-12-07 12:57:48 -05:00
Jay Modi c6799de2a4 Do not enforce TLS if discovery type is single-node (elastic/x-pack-elasticsearch#3245)
This commit adds a check for the discovery type so that the TLS join validator does not fail join
requests when the discovery type is single-node.

relates elastic/x-pack-elasticsearch#2828

Original commit: elastic/x-pack-elasticsearch@fdfdb76b0b
2017-12-07 09:50:25 -07:00
Lisa Cawley c7b100a8c3 [DOCS] Added X-Pack bootstrap checks (elastic/x-pack-elasticsearch#2747)
* [DOCS] Added X-Pack bootstrap checks

* [DOCS] Added more bootstrap checks

* [DOCS] Added link to TLS configuration

* [DOCS] Added Encrypt Sensitive Data bootstrap check

* [DOCS] Added link to watcher encryption task

* [DOCS] Added role mapping bootstrap check

* [DOCS] Added PKI realm boostrap check

* [DOCS] Added token SSL bootstrap check

* [DOCS] Added role mapping bootstrap check

* [DOCS] Added bootstrap check file names

Original commit: elastic/x-pack-elasticsearch@4f3eb72d94
2017-12-07 08:42:02 -08:00
Alexander Reelsen eaf67f8bc2 Tests: Replace YAML tests with ESTestRestCase to be able to wait for … (elastic/x-pack-elasticsearch#3252)
* Tests: Replace YAML tests with ESTestRestCase to be able to wait for events

The YAML tests did not have any possibility to wait for the watches to
be created. A hard ten second timeout was used, that could not be
aborted, by simulating a sleep when waiting for a number of nodes that
never occured in the cluster.

This commit replaces those waiting YAML tests with ESRestTestCases, that
use `assertBusy()` to exit early once the watches have been added. Also
this increases the wait time if needed, as these tests tend to fail on
CI.

relates elastic/x-pack-elasticsearch#3217

Original commit: elastic/x-pack-elasticsearch@74b9945d88
2017-12-07 14:40:08 +01:00
David Kyle e9d9199205 [ML] Special Events (elastic/x-pack-elasticsearch#2930)
* Add Special Event

* Add special events to update process

* Add time condition and skip rule actions.

* Update special events

* Address review comments

Original commit: elastic/x-pack-elasticsearch@80500ded76
2017-12-07 11:44:12 +00:00
Costin Leau 432f10e736 SQL: Introduce LiteralAttribute for values that are not extracted from ES (elastic/x-pack-elasticsearch#3209)
This is mainly a promotion of Literal to Attribute to better handle folding expressions from extracted queries

Original commit: elastic/x-pack-elasticsearch@c3bb48bb61
2017-12-06 19:35:34 +02:00
Costin Leau aa9199f5d1 SQL: Push limit down to GroupByColumn aggs (elastic/x-pack-elasticsearch#3200)
Limit operators if available are pushed into `GROUP BY`

Original commit: elastic/x-pack-elasticsearch@e5f82df810
2017-12-06 19:29:48 +02:00
Igor Motov 6839f99ed0 SQL: Switch CLI to core-cli's Command and make it testable (elastic/x-pack-elasticsearch#3232)
Switches CLI to use the standard Elasticsearch Command and refactors it to be more testable. It doesn't change any cli functionality except using the bright color while displaying query results.

relates elastic/x-pack-elasticsearch#2881, elastic/x-pack-elasticsearch#3203, elastic/x-pack-elasticsearch#2990

Original commit: elastic/x-pack-elasticsearch@841f306d50
2017-12-06 11:27:51 -05:00
javanna 9664363575 remove Catalog abstraction and rename its package from catalog to index
Given that the Catalog was only ever used to hold a single index, the corresponding abstraction can be removed in favour of the abstraction that it holds, namely `GetIndexResult`.

Original commit: elastic/x-pack-elasticsearch@6932db642c
2017-12-06 13:58:17 +01:00
javanna 90aee54251 Trim down the Catalog implementations to a single one
Catalog is now a final class rather than an interface with different implementations.

relates elastic/x-pack-elasticsearch#3179

Original commit: elastic/x-pack-elasticsearch@4cc927e113
2017-12-06 13:58:17 +01:00
javanna 8c61cabe87 remove audit logging changes added for delayed actions that are now removed
This commit reverts part of elastic/x-pack-elasticsearch#2210

Original commit: elastic/x-pack-elasticsearch@75fda79851
2017-12-06 13:58:17 +01:00
javanna 08950ff491 Remove security filter, replaced by get index api call which returns filtered mappings
Given that we get now filtered mappings directly from the get index API (in case security is configured with FLS), we don't need the security filter nor the filtered catalog. That means we can remove the delayed action support also from AuthorizationService and rather make SQLAction a composite action like others. It will be authorized as an action, but its indices won't be checked while that will happen with its inner actions (get index and search) which need to be properly authorized.

Also, SQLGetIndicesAction is not needed anymore, as its purpose was to retrieve the indices access resolver put in the context by the security plugin for delayed actions, which are not supported anymore.

This commit kind of reverts elastic/x-pack-elasticsearch#2162, as it is now possible to integrate with security out-of-the-box

relates elastic/x-pack-elasticsearch#2934

Original commit: elastic/x-pack-elasticsearch@64d5044426
2017-12-06 13:58:17 +01:00
Tim Vernum 628dfaa843 Add API for SSL certificate information (elastic/x-pack-elasticsearch#3088)
Exposes the certificate location (configured path), serial number, and expiry date

Closes: elastic/x-pack-elasticsearch#2795

Original commit: elastic/x-pack-elasticsearch@a0773f6840
2017-12-06 19:57:25 +10:00
javanna 66c4962be9 Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@00ac1d607b
2017-12-06 09:31:20 +01:00
Nhat Nguyen 7f553f391f TEST: fix index template without index patterns
Relates https://github.com/elastic/elasticsearch/pull/27662

Original commit: elastic/x-pack-elasticsearch@7f2766695f
2017-12-05 21:25:53 -05:00
Igor Motov 3b2667048a SQL: Cleanup of Cursor wrappers
The follow up for elastic/x-pack-elasticsearch#3197, addresses nik9000's review comments.

Original commit: elastic/x-pack-elasticsearch@2307029bd8
2017-12-05 20:54:12 -05:00
Nhat Nguyen 3cbb69eb3d Fix index template tests
Relates https://github.com/elastic/elasticsearch/pull/27662

Original commit: elastic/x-pack-elasticsearch@a2175c1689
2017-12-05 18:27:45 -05:00
javanna 4e84a1d658 [TEST] update audit logging parsing
Original commit: elastic/x-pack-elasticsearch@041cb09e2b
2017-12-05 23:00:28 +01:00
javanna 915d95bca1 fix line length
Original commit: elastic/x-pack-elasticsearch@c2e67e68bd
2017-12-05 22:22:17 +01:00
javanna 626c74a437 Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@67f8321368
2017-12-05 21:50:35 +01:00
Luca Cavanna 81dcd8c5f1 Filter mappings fields when field level security is configured (elastic/x-pack-elasticsearch#3173)
This PR uses a new extension point that's being added to Elasticsearch (see https://github.com/elastic/elasticsearch/pull/27603) so that the security plugin can filter the mappings fields returned by get index, get mappings, get field mappings and field capabilities API.

This effort aims at filtering information returned by API in the `indices/admin` category and field capabilities. It doesn't filter what the cluster state api returns as that is a cluster level operation.

One question is about backwards compatibility given that we would like to have this in 6.2. Shall we treat this as a bug as mappings should have been filtered before? Not sure if it's going to break existing integrations.

relates elastic/x-pack-elasticsearch#340

Original commit: elastic/x-pack-elasticsearch@d7e3fd3fa1
2017-12-05 20:32:17 +01:00
Chris Earle 48c8aed373 [Monitoring] Support Beats Monitoring (elastic/x-pack-elasticsearch#3208)
This reintroduces support for Beats monitoring.

Original commit: elastic/x-pack-elasticsearch@539da3afa1
2017-12-05 13:44:22 -05:00
David Kyle 4cd8f075b9 [ML] Add missing job Ids to log messages
Original commit: elastic/x-pack-elasticsearch@56855a3808
2017-12-05 17:43:34 +00:00
David Roberts 751caaae76 [ML] Set established model memory on job open for pre-6.1 jobs (elastic/x-pack-elasticsearch#3222)
Before this was done it was easy to get into the situation where a
job created in 5.x with a default model memory limit of 4GB could not
be opened on any node in the cluster.  Following this change this
problem will no longer occur for jobs that ran for a decent amount of
time on the old cluster.

relates elastic/x-pack-elasticsearch#3181

Original commit: elastic/x-pack-elasticsearch@cb029debba
2017-12-05 17:05:58 +00:00
Costin Leau 88b8794801 SQL: Forbid multi field groups (elastic/x-pack-elasticsearch#3199)
* SQL: GROUP BY with multiple fields are forbidden

The check is performed in the folder Verifier as the optimizer can eliminate some fields (like those with constants)

Original commit: elastic/x-pack-elasticsearch@8d49f4ab02
2017-12-05 18:41:19 +02:00
Costin Leau 4e49769efb SQL: Extend HAVING support (elastic/x-pack-elasticsearch#3155)
SQL: Extend HAVING support

Enhance Analyzer to support HAVING scalar functions over aggregates
Enhance Analyzer to push down undeclared aggs into the Aggregate
Fix bug in Analyzer$MissingRef that caused invalid groupings to still be resolved when pushed into an Aggregate
Preserve location information across the plan
Add AttributeMap as a backing for AttributeSet
Add Optimizer rule for combining projects
Add tz to DT functions toString
Change formatTemplate to not use String.format and thus to avoid
interfering with special % chars
Extend dataset with random salary and languages
Add unit tests for AttributeMap
Fix MathFunction scripting
Improve MissingRefs to enrich UnresolvedAttribute with metadata
During the Analysis unpushed attributes are automatically enriched to
provide more accurate error information
Enhance Verifier to deal with invalid (but resolved) ORDER/HAVING
Add OrderBy arithmetic tests
Improve Verifier to prevent GROUP BY on aggregations
Add tests on grouping by scalar functions

Original commit: elastic/x-pack-elasticsearch@5030d7a755
2017-12-05 18:14:15 +02:00
Dimitrios Athanasiou 6c6b72db25 [TEST] Increase timeout for CoreWithSecurityClientYamlTestSuiteIT
This suite seems to be timing out on the CI slaves. Hopefully,
some extra time will stop the failures.

Original commit: elastic/x-pack-elasticsearch@95ba86be79
2017-12-05 15:57:27 +00:00
Alexander Reelsen 9eb2e14981 Watcher: Ensure watcher thread pool size is reasonably bound (elastic/x-pack-elasticsearch#3056)
The watcher threadpool size was always five times the CPU core
count, resulting in a huge threadpool when with even 24 cores.

This changes the behaviour to be five times the number of cores
by default - as watcher is usually waiting on I/O you should have more
threads than cores, but it maxes out with 50 threads, unless the number
of available cores is higher than that.

relates elastic/x-pack-elasticsearch#3052

Original commit: elastic/x-pack-elasticsearch@eab5deb113
2017-12-05 16:09:46 +01:00
Alexander Reelsen e3a7e4bee4 Tests: Increase timeout to wait to monitoring watches being created
Relates elastic/x-pack-elasticsearch#3217

Original commit: elastic/x-pack-elasticsearch@a6256f9e42
2017-12-05 14:20:46 +01:00
Alexander Reelsen c3e5a20242 Monitoring: Disable security for integration tests (elastic/x-pack-elasticsearch#3174)
In order to support the source directory repo split, this commit
disables security for the regular integration tests.

The MonitoringSettingsFilterTests already existed as REST test, so
this test has been removed.

Relates elastic/x-pack-elasticsearch#2925

Original commit: elastic/x-pack-elasticsearch@519154dd5f
2017-12-05 12:07:04 +01:00
Alexander Reelsen fdb02f4f99 Watcher: Fix pagerduty action to send context data (elastic/x-pack-elasticsearch#3185)
The pagerduty action allows to send contexts, which contains an array
of texts or images, each with a link.

The field of this data was named 'context' instead of 'contexts' and
thus those contects were never correctly parsed on the pagerduty side.

Unfortunately pagerduty accepts any JSON, thus this was not caught so
far.

This commit allows parsing of the old field name to retain BWC, but when
written out via toXContent, it will always use the 'contexts' field name.

relates elastic/x-pack-elasticsearch#3184

Original commit: elastic/x-pack-elasticsearch@50f0b65d56
2017-12-05 11:31:48 +01:00
Lisa Cawley 0def4dfbf8 [DOCS] Add Data Visualizer to the ML Getting Started tutorial (elastic/x-pack-elasticsearch#3171)
* [DOCS] Refreshed ML screenshots

* [DOCS] Added screenshots for ML Data Visualizer

* [DOCS] Addressed feedback about data visualizer

* [DOCS] Fixed typo in ML tutorial

Original commit: elastic/x-pack-elasticsearch@2603536a93
2017-12-04 13:22:42 -08:00
jaymode 6487557e61 Test: fix min value being greater than max value in IndexLifecycleManagerIntegTests
This commit fixes the minimum value being smaller than the maximum value in a call to the
scaledRandomIntBetween, which causes the test to fail.

Original commit: elastic/x-pack-elasticsearch@da7d0ce0ce
2017-12-04 10:45:18 -07:00
Alexander Reelsen b4a317ee94 Documentation: Fix checksum link to point to sha512 link
Original commit: elastic/x-pack-elasticsearch@cac9cb1a84
2017-12-04 17:54:51 +01:00
Dimitris Athanasiou 30b745f846 [ML] Frequency in datafeeds with aggs must be multiple of hist interval (elastic/x-pack-elasticsearch#3205)
relates elastic/x-pack-elasticsearch#3204

Original commit: elastic/x-pack-elasticsearch@0bbd9addd4
2017-12-04 15:22:56 +00:00
David Kyle d39c8b76db [ML] Fix test after elastic/x-pack-elasticsearch#3202
Original commit: elastic/x-pack-elasticsearch@c83c3ebcc1
2017-12-04 13:48:24 +00:00
David Kyle cb9314ba78 [ML] Remove confusing datafeed log message (elastic/x-pack-elasticsearch#3202)
Original commit: elastic/x-pack-elasticsearch@b8ec3d06c9
2017-12-04 13:08:49 +00:00