Commit Graph

7745 Commits

Author SHA1 Message Date
Costin Leau 88b8794801 SQL: Forbid multi field groups (elastic/x-pack-elasticsearch#3199)
* SQL: GROUP BY with multiple fields are forbidden

The check is performed in the folder Verifier as the optimizer can eliminate some fields (like those with constants)

Original commit: elastic/x-pack-elasticsearch@8d49f4ab02
2017-12-05 18:41:19 +02:00
Costin Leau 4e49769efb SQL: Extend HAVING support (elastic/x-pack-elasticsearch#3155)
SQL: Extend HAVING support

Enhance Analyzer to support HAVING scalar functions over aggregates
Enhance Analyzer to push down undeclared aggs into the Aggregate
Fix bug in Analyzer$MissingRef that caused invalid groupings to still be resolved when pushed into an Aggregate
Preserve location information across the plan
Add AttributeMap as a backing for AttributeSet
Add Optimizer rule for combining projects
Add tz to DT functions toString
Change formatTemplate to not use String.format and thus to avoid
interfering with special % chars
Extend dataset with random salary and languages
Add unit tests for AttributeMap
Fix MathFunction scripting
Improve MissingRefs to enrich UnresolvedAttribute with metadata
During the Analysis unpushed attributes are automatically enriched to
provide more accurate error information
Enhance Verifier to deal with invalid (but resolved) ORDER/HAVING
Add OrderBy arithmetic tests
Improve Verifier to prevent GROUP BY on aggregations
Add tests on grouping by scalar functions

Original commit: elastic/x-pack-elasticsearch@5030d7a755
2017-12-05 18:14:15 +02:00
Dimitrios Athanasiou 6c6b72db25 [TEST] Increase timeout for CoreWithSecurityClientYamlTestSuiteIT
This suite seems to be timing out on the CI slaves. Hopefully,
some extra time will stop the failures.

Original commit: elastic/x-pack-elasticsearch@95ba86be79
2017-12-05 15:57:27 +00:00
Alexander Reelsen 9eb2e14981 Watcher: Ensure watcher thread pool size is reasonably bound (elastic/x-pack-elasticsearch#3056)
The watcher threadpool size was always five times the CPU core
count, resulting in a huge threadpool when with even 24 cores.

This changes the behaviour to be five times the number of cores
by default - as watcher is usually waiting on I/O you should have more
threads than cores, but it maxes out with 50 threads, unless the number
of available cores is higher than that.

relates elastic/x-pack-elasticsearch#3052

Original commit: elastic/x-pack-elasticsearch@eab5deb113
2017-12-05 16:09:46 +01:00
Alexander Reelsen e3a7e4bee4 Tests: Increase timeout to wait to monitoring watches being created
Relates elastic/x-pack-elasticsearch#3217

Original commit: elastic/x-pack-elasticsearch@a6256f9e42
2017-12-05 14:20:46 +01:00
Alexander Reelsen c3e5a20242 Monitoring: Disable security for integration tests (elastic/x-pack-elasticsearch#3174)
In order to support the source directory repo split, this commit
disables security for the regular integration tests.

The MonitoringSettingsFilterTests already existed as REST test, so
this test has been removed.

Relates elastic/x-pack-elasticsearch#2925

Original commit: elastic/x-pack-elasticsearch@519154dd5f
2017-12-05 12:07:04 +01:00
Alexander Reelsen fdb02f4f99 Watcher: Fix pagerduty action to send context data (elastic/x-pack-elasticsearch#3185)
The pagerduty action allows to send contexts, which contains an array
of texts or images, each with a link.

The field of this data was named 'context' instead of 'contexts' and
thus those contects were never correctly parsed on the pagerduty side.

Unfortunately pagerduty accepts any JSON, thus this was not caught so
far.

This commit allows parsing of the old field name to retain BWC, but when
written out via toXContent, it will always use the 'contexts' field name.

relates elastic/x-pack-elasticsearch#3184

Original commit: elastic/x-pack-elasticsearch@50f0b65d56
2017-12-05 11:31:48 +01:00
Lisa Cawley 0def4dfbf8 [DOCS] Add Data Visualizer to the ML Getting Started tutorial (elastic/x-pack-elasticsearch#3171)
* [DOCS] Refreshed ML screenshots

* [DOCS] Added screenshots for ML Data Visualizer

* [DOCS] Addressed feedback about data visualizer

* [DOCS] Fixed typo in ML tutorial

Original commit: elastic/x-pack-elasticsearch@2603536a93
2017-12-04 13:22:42 -08:00
jaymode 6487557e61 Test: fix min value being greater than max value in IndexLifecycleManagerIntegTests
This commit fixes the minimum value being smaller than the maximum value in a call to the
scaledRandomIntBetween, which causes the test to fail.

Original commit: elastic/x-pack-elasticsearch@da7d0ce0ce
2017-12-04 10:45:18 -07:00
Alexander Reelsen b4a317ee94 Documentation: Fix checksum link to point to sha512 link
Original commit: elastic/x-pack-elasticsearch@cac9cb1a84
2017-12-04 17:54:51 +01:00
Dimitris Athanasiou 30b745f846 [ML] Frequency in datafeeds with aggs must be multiple of hist interval (elastic/x-pack-elasticsearch#3205)
relates elastic/x-pack-elasticsearch#3204

Original commit: elastic/x-pack-elasticsearch@0bbd9addd4
2017-12-04 15:22:56 +00:00
David Kyle d39c8b76db [ML] Fix test after elastic/x-pack-elasticsearch#3202
Original commit: elastic/x-pack-elasticsearch@c83c3ebcc1
2017-12-04 13:48:24 +00:00
David Kyle cb9314ba78 [ML] Remove confusing datafeed log message (elastic/x-pack-elasticsearch#3202)
Original commit: elastic/x-pack-elasticsearch@b8ec3d06c9
2017-12-04 13:08:49 +00:00
David Roberts 2c978842da [ML] Avoid spurious logging when deleting lookback job from the UI (elastic/x-pack-elasticsearch#3193)
When you click "delete" in the UI it force-deletes the datafeed then
force-deletes the job.  For a datafeed doing lookback, this results
in a close followed very quickly by a kill on the autodetect process.
Depending on thread scheduling this could cause a lot of spurious
errors and exception traces to be logged.

This change prevents the log spam in this scenario.

relates elastic/x-pack-elasticsearch#3149

Original commit: elastic/x-pack-elasticsearch@091240f32a
2017-12-04 10:29:05 +00:00
David Roberts df9dd77656 [TEST] Fix more side effects of elastic/x-pack-elasticsearch#2975 on machines with < 16GB RAM
Our rolling upgrade tests were failing on many machines due to using
the 5.x default of 4GB model_memory_limit, which then propagated forward
to 6.1+ even though the default is now lower.

Original commit: elastic/x-pack-elasticsearch@3b23d8fe9d
2017-12-04 09:48:51 +00:00
Igor Motov 7b701cbf88 SQL: Don't override calendar for Thai locale
The Thai Buddhist calendar is close enough to Gregorian calendar that it doesn't trip H2 if timestamps with timezones are used. Related to elastic/x-pack-elasticsearch#3169.

Original commit: elastic/x-pack-elasticsearch@4f8f1b603d
2017-12-03 14:51:23 -05:00
Igor Motov a732d751c4 SQL: Replace calendar in H2 tests if a non-Gregorian calendar is detected
Instead of replacing entire locale with the ROOT locale, just strip the calendar settings if a non-Gregorian calendar is detected. Related to elastic/x-pack-elasticsearch#3169.

Original commit: elastic/x-pack-elasticsearch@33b1c63768
2017-12-03 13:33:24 -05:00
Igor Motov 1971adda68 SQL: Improve SqlResponseTests
This commit fixes a bug in the testVersionHandling test and adds more randomization to serialized cursors.

Original commit: elastic/x-pack-elasticsearch@fab8d50518
2017-12-02 18:40:52 -05:00
Igor Motov 773cdf0f9f SQL: Switch to the standard way of cursor serialization (elastic/x-pack-elasticsearch#3197)
While working on cursor cleanup, I realized that we still have two ways to serialize the cursor and the second way doesn't contain the cursor version (only client version, that can be potentially different from the cursor version). This commit switches to the unified way of serializing the cursor.

This is a follow up for elastic/x-pack-elasticsearch#3064.

Original commit: elastic/x-pack-elasticsearch@ef1a6427dd
2017-12-02 16:03:22 -05:00
Costin Leau 5c6e5c1cb5 Fix MetaTableRequest constructor
The pattern argument can be null which triggers a NPE when saved

Original commit: elastic/x-pack-elasticsearch@348d731b06
2017-12-01 20:48:48 +02:00
jaymode 409492808f Test: bound number of requests in IndexLifecycleManagerIntegTests
The IndexLifecycleManagerIntegTests has a method that tests concurrency and executes a large number
of requests in parallel. On some machines, this can actually overwhelm a thread pool and cause the
test to fail for the wrong reasons. This commit bounds the total number of requests to 100.

Original commit: elastic/x-pack-elasticsearch@07613ada51
2017-12-01 09:40:53 -07:00
Alexander Reelsen f816b2e850 Monitoring: Move watcher tests for repository split preparations (elastic/x-pack-elasticsearch#3183)
Created a smoke-test-monitoring-with-watcher project that runs REST
tests with watcher enabled to ensure that the proper watcher are
installed either when the local or the HTTP exporter are set up.

Also removed two more watcher imports in the tests.

Relates elastic/x-pack-elasticsearch#2925

Original commit: elastic/x-pack-elasticsearch@0a9abc3185
2017-12-01 13:20:05 +01:00
Luca Cavanna 55a19ed394 Deprecate the transport client in favour of the high-level REST client (elastic/x-pack-elasticsearch#2779)
Original commit: elastic/x-pack-elasticsearch@2aeef5df3f
2017-12-01 12:24:26 +01:00
David Roberts d08944b1ff [TEST] Fix seed-specific failure in ML node allocation test
This was due to a hardcoded job ID that wouldn't exist when two random
numbers generated by the test framework were both small.

Original commit: elastic/x-pack-elasticsearch@9529e6d280
2017-12-01 11:12:55 +00:00
Tim Sullivan 6f4484c287 [Monitoring] Update Beats Template with Metrics (elastic/x-pack-elasticsearch#2902)
* [Monitoring] Update Beats Template with Metrics

- remove source_node.timestamp
- includes mapping for beat type = Metricbeat

* remove metricbeat and xpack objects which are not used in the UI

* use long instead of integer type

Original commit: elastic/x-pack-elasticsearch@083b9cc575
2017-11-30 13:33:16 -07:00
Lisa Cawley d2172be562 [DOCS] Refresh screenshots in ML single metric job tutorial (elastic/x-pack-elasticsearch#3172)
* [DOCS] Refresh screenshots in ML tutorial

* [DOCS] Refreshed screenshots for single metric job

* [DOCS] Removed outdated index screenshot

Original commit: elastic/x-pack-elasticsearch@14f39c3091
2017-11-30 10:55:46 -08:00
Igor Motov d8ef52dc0d SQL: Allow H2-based tests to run only with Gregorian calendar (elastic/x-pack-elasticsearch#3170)
This PR detects non-gregorian calendars in H2 tests and overrides them. This is a temporary workaround to reduce amount of noise in CI. The permanent solution is tracked in elastic/x-pack-elasticsearch#3169.

Original commit: elastic/x-pack-elasticsearch@60991d1a11
2017-11-30 11:42:59 -05:00
Costin Leau 7cab29760d SQL: Introduce PreAnalyze phase to resolve catalogs async (elastic/x-pack-elasticsearch#2962)
SQL: Introduce PreAnalyze phase to resolve catalogs async

The new preanalyze phase collects all unresolved relations and tries
to resolve them as indices through typical async calls _before_ starting the analysis process.
The result is loaded into a catalog which is then passed to the analyzer.
While at it, the analyzer was made singleton and state across the engine
is done through SqlSession#currentContext().


Commit missing fix
Fix typo
Fix license
Fix line length
remove redundant static modifier
Remove redundant generics type
Rename catalogResolver instance member to indexResolver
Fix translate action to return a response through the listener, it hangs otherwise
IndexResolver improvements

Make sure that get index requests calls are locally executed by providing local flag.
Don't replace index/alias name with concrete index name in asCatalog response conversion. We need to preserve the original alias name for security, so it is reused in the subsequent search.
Update roles and actions names for security tests
Get index is now executed instead of sql get indices, and sql get indices has been removed.
Also made cluster privileges more restrictive to make sure that cluster state calls are no longer executed.
Fix most of the security IT tests
indices options are now unified, always lenient. The only situation where we get authorization exception back is when the user is not authorized for the sql action (besides for which indices).
Improve SessionContext handling
Fix context being invalid in non-executable phases
Make Explain & Debug command fully async
Resolve checkstyle error about redundant modifiers
Temporarily restore SqlGetIndicesAction

SqlGetIndicesAction action is still needed in RestSqlJdbcAction (metaTable and metaColumn methods), where we can't at the moment call IndexResolver directly, as security (FLS) needs index resolver to be called as part of the execution of an indices action. Once mappings are returned filtered, delayed action and the security filter will go away, as well as SqlGetIndicesAction.
SqlGetIndicesAction doesn't need to be a delayed action, my bad

[TEST] remove unused expectSqlWithAsyncLookup and rename expectSqlWithSyncLookup to expectSqlCompositeAction
Polish and feedback
Add unit test for PreAnalyzer

Original commit: elastic/x-pack-elasticsearch@57846ed613
2017-11-30 18:18:08 +02:00
Albert Zaharovits 3ea5a6df91 Augment audit authz event with role names data (elastic/x-pack-elasticsearch#3100)
Audit authz events (accessGranted, accessDenied, runAsGranted
and runAsDenied) include role names.

Original commit: elastic/x-pack-elasticsearch@6a94f65962
2017-11-30 15:56:00 +02:00
Tim Vernum 4262b29188 [Security] Add DEBUG logging on role resolution (elastic/x-pack-elasticsearch#3138)
This change adds some debug and trace logging when we look up role names, to explain how each role was resolved.

At the moment we have very little insight into how roles are being resolved which can make it difficult to diagnose some issues.

Original commit: elastic/x-pack-elasticsearch@1b3c246186
2017-11-30 21:34:07 +10:00
Albert Zaharovits a5fe074b5c LoggingAuditTrail emit events before local address available (elastic/x-pack-elasticsearch#3061)
Register LoggingAuditTrail as a ClusterStateListener.
Avoid querying for the localNode while on the ClusterStateApplier thread,
which tripps assertion. This can happen when logging audit events that
originate from the system.

relates elastic/x-pack-elasticsearch#3057

Original commit: elastic/x-pack-elasticsearch@66bc59682d
2017-11-30 12:42:34 +02:00
Hendrik Muhs 756d878983 [ML-FC] do not allow durations below the bucket span (elastic/x-pack-elasticsearch#3166)
do not allow durations below the bucket span

Original commit: elastic/x-pack-elasticsearch@0e895c1ddd
2017-11-30 07:49:21 +01:00
Tim Vernum c2ff796fea Support realm validation when Keystore is closed (elastic/x-pack-elasticsearch#3096)
If the KeyStoreWrapper is closed, then we cannot validate secure settings (because we no longer have access to them)
The Realm group setting uses the "validate" method to ensure that child settings are correct, but it must ignore secure settings as it might get called
after startup (e.g. during a settings diff)

Original commit: elastic/x-pack-elasticsearch@b30db6bc62
2017-11-30 11:22:17 +10:00
Igor Motov 6fceb2fdde SQL: Fix more JDK vs Joda TZ issues (elastic/x-pack-elasticsearch#3137)
Adds a random timezone known to both JDK and Joda as a connection parameter to all JdbcIntegrationTestCase-based tests.

Original commit: elastic/x-pack-elasticsearch@6be6a3b69b
2017-11-29 19:15:29 -05:00
Lisa Cawley 90a1da82ee [DOCS] Subdivided getting started with ML pages (elastic/x-pack-elasticsearch#3167)
* [DOCS] Subdivided getting started with ML pages

* [DOCS] Added new getting started page to build.gradle

Original commit: elastic/x-pack-elasticsearch@968187b048
2017-11-29 12:03:51 -08:00
Lisa Cawley 11ab50d9dc [DOCS] Clarified watcher cleaner service behaviour (elastic/x-pack-elasticsearch#3135)
* [DOCS] Clarified watcher cleaner service behaviour

* [DOCS] Fixed typo in watcher settings

* [DOC] Clarified watcher and monitor setting dependencies

Original commit: elastic/x-pack-elasticsearch@1386139460
2017-11-29 08:25:59 -08:00
Michael Basnight bf27cd1457 Use SPI to glean reserved roles (elastic/x-pack-elasticsearch#3012)
This commit uses SPI to get the list of system wide reserved roles in
security. It does not yet split out the RoleDescriptor to a common
location so the implementing services still depend on security. Each
role, however, only depends on its own feature as well as security.

ref elastic/x-pack-elasticsearch#2925

Original commit: elastic/x-pack-elasticsearch@efebc3e5c8
2017-11-29 09:00:23 -06:00
Alexander Reelsen c641a30bc5 Docs: Explain watcher security integration/limitations (elastic/x-pack-elasticsearch#3106)
Original commit: elastic/x-pack-elasticsearch@991e1de267
2017-11-29 14:48:06 +01:00
Costin Leau c08eb56238 SQL: Improve grammar to better handle quotes (elastic/x-pack-elasticsearch#3051)
SQL: Improve grammar to better handle quotes

Fix typo in handling (back)quoted identifiers
Clarify use of unquote (dedicated for literals) and text (generic)
Address feedback
clarify that ` are picked up but not supported/recommended
Fix merge and adjust json errors to work on windows

Original commit: elastic/x-pack-elasticsearch@67e0f3f38e
2017-11-29 15:42:04 +02:00
Igor Motov d5525f38f3 SQL: Return proper NUM_PREC_RADIX for non-numeric types (elastic/x-pack-elasticsearch#3086)
NUM_PREC_RADIX should be 10 for integer types, 2 for rational types and null for everything else.

relates elastic/x-pack-elasticsearch#3085

Original commit: elastic/x-pack-elasticsearch@81d5ee04b3
2017-11-29 08:21:42 -05:00
David Kyle 171c48fd2f [TESTS] Refactor yml test suite classes (elastic/x-pack-elasticsearch#3145)
* Use XPackRestIT as base class for XDocsClientYamlTestSuiteIT

* Remove the XPackRestTestCase class

* Address review comments

* Fix checkstyle checks

Original commit: elastic/x-pack-elasticsearch@c2a5e60c12
2017-11-29 12:43:53 +00:00
David Roberts ef96831515 [ML] Don't mention unrelated indices when deleting job index aliases (elastic/x-pack-elasticsearch#3160)
This change fixes a problem that would cause job deletion to fail if ANY
index had a block on it, e.g. read-only.

The problem was that we were requesting the job aliases be deleted from
ALL indices in the system due to a misunderstanding with the format of the
get_aliases response.  This didn't usually cause any noticable effects, as
only the ML indices would have the aliases.  But in the case of a read-only
index it would cause an error, leading to unnecessary failure of the job
deletion.

Fixes elastic/machine-learning-cpp#428

Original commit: elastic/x-pack-elasticsearch@a573f85a00
2017-11-29 11:39:30 +00:00
Dimitris Athanasiou 9ef9edc1ca [ML] Rework ScrollDataExtractorTests to make scroll clearing more robust (elastic/x-pack-elasticsearch#3150)
Original commit: elastic/x-pack-elasticsearch@7784497a3d
2017-11-29 11:19:13 +00:00
David Roberts 8cb6e63a0e [ML] Increase default limit on ML jobs per node from 10 to 20 (elastic/x-pack-elasticsearch#3141)
Following the changes of elastic/x-pack-elasticsearch#2975 the hard limit on the number of ML jobs
per node is no longer the only limiting factor.  Additionally there is
now a limit based on the estimated memory usage of the jobs, and this is
expected to provide a more sensible limit that accounts for differing
resource requirements per job.

As a result, it makes sense to raise the default for the hard limit on
the number of jobs, on the assumption that the memory limit will prevent
the node becoming overloaded if an attempt is made to run many large jobs.
Increasing the hard limit will allow more small jobs to be run than was
previously the case by default.

Of course, this change to the default will have no effect for customers
who have already overridden the default in their config files.

Original commit: elastic/x-pack-elasticsearch@9fed1d1237
2017-11-28 20:40:55 +00:00
Nik Everett 0cc153f6d3 Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@ccc2fc708e
2017-11-28 13:24:36 -05:00
Igor Motov da180bd9f9 Tests: Muted FullClusterRestartIT.testMonitoring
Tracked by elastic/x-pack-elasticsearch#3068

Original commit: elastic/x-pack-elasticsearch@199d4bb6b9
2017-11-28 12:06:10 -05:00
Nik Everett 18e88122eb SQL: Add more error integration tests (elastic/x-pack-elasticsearch#3134)
We didn't have many integration tests for errors other than the security
errors. This adds some and sets up a way we can make sure we are
consistent across the REST, JDBC, and CLI.

relates elastic/x-pack-elasticsearch#3033

Original commit: elastic/x-pack-elasticsearch@debbb2ec46
2017-11-28 11:54:51 -05:00
Dimitris Athanasiou d96be6c51f [ML] Only clear scroll whan a scroll id exists (elastic/x-pack-elasticsearch#3148)
The issue here is that if the first search request fails
(initScroll), then we do not have a scroll_id. However,
in order to retry the search, we reset the scroll. That
involves clearing the current search, but since we do
not have a scroll_id, the clear scroll request fails.
We end up reporting the failure for the scroll clearing,
rather than the actual problem.

This commit fixes that by avoiding clearing the scroll
when the scroll_id is null.

relates elastic/x-pack-elasticsearch#3146

Original commit: elastic/x-pack-elasticsearch@b5086028f6
2017-11-28 16:48:24 +00:00
Lisa Cawley b6f322e72e [DOCS] Enable code snippet testing in open job API (elastic/x-pack-elasticsearch#3053)
* [DOCS] Enable code snippet testing in open job API

* [DOCS] Fixed open job API example

Original commit: elastic/x-pack-elasticsearch@f789041c2a
2017-11-28 08:26:58 -08:00
David Kyle ac01962a62 [DOCS] Disable monitoring for the docs tests (elastic/x-pack-elasticsearch#3143)
Original commit: elastic/x-pack-elasticsearch@114fdbf2d9
2017-11-28 15:57:17 +00:00