Commit Graph

7495 Commits

Author SHA1 Message Date
Nik Everett 2b2bf89fb1 Fix broken SQL security test
Used the wrong user.

Original commit: elastic/x-pack-elasticsearch@334955fbfa
2017-11-01 21:48:46 -04:00
lcawley 61864c3a67 [DOCS] Added troubleshooting for setup-passwords command
Original commit: elastic/x-pack-elasticsearch@6196c1e2bf
2017-11-01 09:35:53 -07:00
Lisa Cawley da3d9dcf69 [DOCS] Added hide_settings to security settings (elastic/x-pack-elasticsearch#2801)
* [DOCS] Added hide_settings to security settings

* [DOCS] Addressed feedback about hide_settings

Original commit: elastic/x-pack-elasticsearch@6a6d394c71
2017-11-01 09:21:11 -07:00
lcawley b8aefcc1e7 [DOCS] Added testresponse substitution to avoid gradle issues
Original commit: elastic/x-pack-elasticsearch@531579a626
2017-11-01 08:25:46 -07:00
Albert Zaharovits fb13299714 Log when encountered cert but expecting key and vice-versa.. (elastic/x-pack-elasticsearch#2670)
Log when encountered cert but expecting key and vice-versa.

relates elastic/x-pack-elasticsearch#2657

Original commit: elastic/x-pack-elasticsearch@4e26d8044f
2017-11-01 15:20:28 +02:00
David Kyle fe21003341 [DOCS] Mute failing test snippet
Original commit: elastic/x-pack-elasticsearch@0a2a90bbed
2017-11-01 11:05:17 +00:00
Chris Earle 31741a85d9 [Monitoring] Add Shard-level Details to Index Stats Collection (elastic/x-pack-elasticsearch#2817)
This adds details about the shards and the health of the index. By adding these stats directly to the document, the UI can avoid many aggregations and enable better searching and sorting against indices.

Original commit: elastic/x-pack-elasticsearch@f38ae5ce69
2017-10-31 16:41:40 +00:00
Tanguy Leroux f416c5b3c9 Replace empty index block checks with global block checks in get/put license actions (elastic/x-pack-elasticsearch#2766)
Related to elastic/x-pack-elasticsearch#10530

Original commit: elastic/x-pack-elasticsearch@f4c9924d62
2017-10-31 16:15:14 +01:00
Nik Everett 33f4a8317c JDBC metadata integration with security (elastic/x-pack-elasticsearch#2806)
I realized that we weren't running our DatabaseMetaData tests. One thing led to another and I made these changes:
1. Got the DatabaseMetaData tests running in all three of our QA projects.
2. Fixed the SecurityCatalogFilter to work with `SqlGetIndicesAction`. It worked before, but only for requests that were a `SqlAction` as well as `SqlGetIndicesAction`.
3. Added security test for the JDBC DatabaseMetaData requests. These mirror exactly the security tests that we use for `SHOW TABLES` and `DESCRIBE` but cover the JDBC actions.

Original commit: elastic/x-pack-elasticsearch@7026d83c06
2017-10-30 23:22:12 +00:00
Nik Everett 43b3a7d7aa Fix import
I missed it in the last PR.

Original commit: elastic/x-pack-elasticsearch@58b2047165
2017-10-30 19:01:15 -04:00
Nik Everett c13f0a8272 Rename SQL's net-client project to shared-client (elastic/x-pack-elasticsearch#2756)
The `net-client` project contained more then just the `net-client`.
It contains stuff like `SuppressForbidden` and `Strings` and `IOUtil`
and other things shared between the CLI and JDBC. It also does contain
the http client. Anyway, it makes more sense to call it `shared-client`,
I think.

Alos updated the copywrite dates on the files that I touched because
they are all 2017 files.

Removed some uses of `String.EMPTY` because they don't buy us anything
and require an extra import. `""` is just one less step.

Original commit: elastic/x-pack-elasticsearch@465c6445c4
2017-10-30 22:59:01 +00:00
Nik Everett b0dc14f639 Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@7af47176fc
2017-10-30 13:50:33 -04:00
Nik Everett d933b1b48b Initial docs for SQL (elastic/x-pack-elasticsearch#2810)
Adds docs for the REST API, translate API, the CLI, and JDBC.

Next we need to add more example queries and documentation for our
extensions.

Original commit: elastic/x-pack-elasticsearch@ed6d1360d2
2017-10-30 17:23:27 +00:00
Jay Modi 4f65d9b527 Retry startup for IndexAuditTrail and version templates (elastic/x-pack-elasticsearch#2755)
This commit removes the FAILED state for the IndexAuditTrail so that we always try to keep starting
the service. Previously, on any exception during startup we moved to a failed state and never tried
to start again. The users only option was to restart the node. This was problematic in the case of
large clusters as there could be common timeouts of cluster state listeners that would cause the
startup of this service to fail.

Additionally, the logic in the IndexAuditTrail to update the template on the current cluster has
been removed and replaced with the use of the TemplateUpgradeService. However, we still need to
maintain the ability to determine if a template on a remote cluster should be PUT. To avoid always
PUTing the template, the version field has been added so it only needs to be PUT once on upgrade.

Finally, the default queue size has been increased as this is another common issue that users hit
with high traffic clusters.

relates elastic/x-pack-elasticsearch#2658

Original commit: elastic/x-pack-elasticsearch@27e2ce7223
2017-10-30 09:11:18 -06:00
Lisa Cawley f69f6cd341 [DOCS] Enabled code snippet testing for datafeed APIs (elastic/x-pack-elasticsearch#2811)
* [DOCS] Enabled code snippet testing for put datafeed API

* [DOCS] Addressed gradle errors in put datafeed API

* [DOCS] Added job creation test to build.gradle

Original commit: elastic/x-pack-elasticsearch@3548d920c7
2017-10-30 07:54:33 -07:00
Chris Earle c335b00c9b [Monitoring] Add "cluster_state.nodes_hash" to document (elastic/x-pack-elasticsearch#2798)
Adding this field enables a very simple mechanism for detecting node changes in the cluster state via Watcher (and other mechanisms). The next step is to add the cluster alert that uses it.

Original commit: elastic/x-pack-elasticsearch@1eacc25cff
2017-10-30 13:13:57 +00:00
Martijn van Groningen 9706d07209 disabled test assertion
Original commit: elastic/x-pack-elasticsearch@5a05c607e5
2017-10-30 10:15:45 +01:00
Martijn van Groningen f96153143b test: added logging and don't use replicas
Original commit: elastic/x-pack-elasticsearch@917ef8177d
2017-10-30 09:39:10 +01:00
Martijn van Groningen 2d89394896 test: removed refresh workaround
Original commit: elastic/x-pack-elasticsearch@75b571c23f
2017-10-30 09:29:43 +01:00
Martijn van Groningen 9a1c103bb2 security: Fail search request if profile is used and DLS is active.
Original commit: elastic/x-pack-elasticsearch@b83536460d
2017-10-30 09:12:27 +01:00
Tim Vernum 0c7caabea1 Usability enhancements for certificate generation (elastic/x-pack-elasticsearch#2561)
This commit adds a new `certutil` command and deprecates the `certgen` command.
 
The new certuil consists of sub commands that are (by default) are simpler to use than the old monolithic command, but still support all the previous behaviours.

Original commit: elastic/x-pack-elasticsearch@3f57687da9
2017-10-30 13:08:31 +11:00
Nhat ba29971323 test: updates DocsStats with totalSizeInBytes
Relates https://github.com/elastic/elasticsearch/pull/27117

Original commit: elastic/x-pack-elasticsearch@9bf177d90b
2017-10-28 13:04:21 -04:00
Costin Leau 66719f7a92 Improve DateTime script template
When dealing with TZ, the script is reused across extractors by
externalizing the tz & chronofield

Original commit: elastic/x-pack-elasticsearch@9e63ea8cd8
2017-10-28 13:35:16 +03:00
Costin Leau a7b559c825 temporarily set UTC as timezone to comply with H2 behavior
Original commit: elastic/x-pack-elasticsearch@00be17cebb
2017-10-28 11:59:13 +03:00
Nik Everett f92684c436 Fix SQL test
Original commit: elastic/x-pack-elasticsearch@14d07c2607
2017-10-27 17:38:34 -04:00
Nik Everett b91e9d85e9 Fix SHAs
Original commit: elastic/x-pack-elasticsearch@15c2669416
2017-10-27 17:12:15 -04:00
Costin Leau f26a9acdc4 Update ANTLR in qa project as well
Original commit: elastic/x-pack-elasticsearch@58c10398b4
2017-10-27 23:43:37 +03:00
Costin Leau 660cb3b814 Upgrade ANTLR to 4.5.3
Relates to elastic/x-pack-elasticsearch#26834

Original commit: elastic/x-pack-elasticsearch@57ba8f1d5f
2017-10-27 23:29:00 +03:00
Costin Leau 8d5572a77b SQL: improve validation of properties (both names and values) (elastic/x-pack-elasticsearch#2759)
* refactor Configuration class

move away from Properties
perform validation of settings names and values at startup
better exception handling

Original commit: elastic/x-pack-elasticsearch@d8a9edeccf
2017-10-27 22:54:03 +03:00
Costin Leau af591b9edd SQL: Remove usage of Settings inside SqlSettings (elastic/x-pack-elasticsearch#2757)
* Remove usage of Settings inside SqlSettings

Also hook client timeouts to the backend
Set UTC as default timezone when using CSV
As the JVM timezone changes, make sure to pin it to UTC since this is what the results are computed against

Original commit: elastic/x-pack-elasticsearch@3e7aad8c1f
2017-10-27 18:55:59 +03:00
Costin Leau 6b1d0d1c8e SQL: Remove time threshold for rule executor (elastic/x-pack-elasticsearch#2750)
Refactor RuleExecutor to eliminate time-based thresholds

Original commit: elastic/x-pack-elasticsearch@ba131f8058
2017-10-27 18:43:13 +03:00
Alexander Reelsen 940eabd5f3 Watcher: Add thread pool rejection to execution state (elastic/x-pack-elasticsearch#2805)
The execution state of a watch did not differentiate between failures of
the execution like a broken painless script and a thread pool rejection.

This adds an own state, which allows to aggregate on such data in the
watch history, which should ease debugging issues a bit.

Original commit: elastic/x-pack-elasticsearch@351e64e14d
2017-10-27 16:37:14 +02:00
Martijn van Groningen 96b0b4e96d test: refresh only once to workaround phrase suggester edge case
Relates to elastic/x-pack-elasticsearch#2804

Original commit: elastic/x-pack-elasticsearch@3f2b6b7eea
2017-10-27 15:05:18 +02:00
Hendrik Muhs f74e680142 [ML] add detectorIndex to modelplot and forecast (elastic/x-pack-elasticsearch#2796)
add detector_index to model plots and forecast

relates elastic/x-pack-elasticsearch#2547

corresponding ml-cpp change: elastic/machine-learning-cpp#361

Original commit: elastic/x-pack-elasticsearch@5927d8578e
2017-10-27 12:54:42 +02:00
Dimitris Athanasiou c7e94b3b4c [ML] Enable overall buckets aggregation at a custom bucket span (elastic/x-pack-elasticsearch#2782)
For the purpose of getting this API consumed by our UI, returning
overall buckets that match the job's largest `bucket_span` can
result in too much data. The UI only ever displays a few buckets
in the swimlane. Their span depends on the time range selected and
the screen resolution, but it will only ever be a relatively
low number.

This PR adds the ability to aggregate overall buckets in a user
specified `bucket_span`. That `bucket_span` may be equal or
greater to the largest job's `bucket_span`. The `overall_score`
of the result overall buckets is the max score of the
corresponding overall buckets with a span equal to the job's
largest `bucket_span`.

The implementation is now chunking the bucket requests
as otherwise the aggregation would fail when too many buckets
are matching.

Original commit: elastic/x-pack-elasticsearch@981f7a40e5
2017-10-27 11:14:13 +01:00
Martijn van Groningen e028716bec test: use a single primary shard to workaround an edge case with the phrase suggester
Relates to elastic/x-pack-elasticsearch#2804

Original commit: elastic/x-pack-elasticsearch@afd028faf7
2017-10-27 10:41:19 +02:00
Lisa Cawley 215f289a8c [DOCS] Reformatted security troubleshooting pages (elastic/x-pack-elasticsearch#2799)
Original commit: elastic/x-pack-elasticsearch@ec9969ec7a
2017-10-26 13:56:57 -07:00
Nik Everett 3d0f57d976 Add remaining security tests (elastic/x-pack-elasticsearch#2797)
This adds all of the security tests I think SQL is going to need for the initial release. SQL is still missing an entire scenario though: SSL enabled. Either way, this removes some `NOCOMMIT`s in `qa/sql/security`. Adding the SSL testing can come later.

Original commit: elastic/x-pack-elasticsearch@851620b606
2017-10-26 17:23:35 +00:00
Lisa Cawley 61bfa39331 [DOCS] Added setup-passwords command parameters (elastic/x-pack-elasticsearch#2735)
* [DOCS] Added setup-passwords command parameters

* [DOCS] Addressed feedback in setup-passwords command

Original commit: elastic/x-pack-elasticsearch@5401994c56
2017-10-26 08:28:32 -07:00
Nik Everett 52d9de1de7 Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@dd027d9ca5
2017-10-26 09:51:04 -04:00
Martijn van Groningen 62215f1fae security: Fail request if suggesters are used and DLS is active.
Original commit: elastic/x-pack-elasticsearch@056c735e77
2017-10-26 08:02:31 +02:00
Tim Vernum 8985625ea5 [Security] BulkShardRequest may have multiple indices (elastic/x-pack-elasticsearch#2742)
If a bulk update references aliases rather than concrete indices,
it is possible that a single shard level request could have multiple distinct "index names", potentially including "date math".
Those names will resolve to the same concrete index, but they might have different privileges.

Original commit: elastic/x-pack-elasticsearch@34cfd11df8
2017-10-26 15:34:58 +11:00
Jason Tedor 70a38ec545 Enable BWC testing against other remotes
This commit enables BWC testing against remotes on GitHub other than
elastic/elasticsearch.git and elastic/x-pack-elasticsearch.git.

Relates elastic/x-pack-elasticsearch#2707

Original commit: elastic/x-pack-elasticsearch@9028f2e089
2017-10-25 22:39:58 -04:00
lcawley 615de68711 [DOCS] Added xpack.watcher.history.cleaner_service_enabled
Original commit: elastic/x-pack-elasticsearch@ceead2a1c2
2017-10-25 16:35:20 -07:00
Lisa Cawley c67ec73cb5 [DOCS] Add xpack.ml.max_model_memory_limit (elastic/x-pack-elasticsearch#2787)
* [DOCS] Add xpack.ml.max.model_memory_limit

* [DOCS] Addressed feedback on model limit setting

Original commit: elastic/x-pack-elasticsearch@77a10bfe0e
2017-10-25 09:15:19 -07:00
Lee Hinman cdaa047d56 [TEST] Fix compilation for ARS stats exposing
Original commit: elastic/x-pack-elasticsearch@eb0ad99434
2017-10-24 11:09:19 -06:00
Lisa Cawley 73e819b0bd [DOCS] Add secure versions of SSL passphrases (elastic/x-pack-elasticsearch#2478)
* [DOCS] Add secure versions of SSL passphrases

* [DOCS] Add secure xpack settings

* [DOCS] Clarify type of keystore

* [DOCS] Added secure settings to security page

* [DOCS] Clarify X-Pack secure settings

* [DOCS] Reformat secure X-Pack settings

Original commit: elastic/x-pack-elasticsearch@efe043fb67
2017-10-24 16:38:37 +01:00
Lisa Cawley 3a6870f0dc [DOCS] Added SSL settings to X-Pack installation (elastic/x-pack-elasticsearch#2733)
Original commit: elastic/x-pack-elasticsearch@c88cce763a
2017-10-24 08:34:07 -07:00
Albert Zaharovits 403912b8a2 SecureSettings ignored by customAuditIndexSettings (elastic/x-pack-elasticsearch#2748)
customAuditIndexSettings does not submit SecureSettings with putIndexMapping.

relates elastic/x-pack-elasticsearch#2705

* Randomize SecureSetting in testcase

* Apply feedback

Original commit: elastic/x-pack-elasticsearch@1a5414b057
2017-10-24 13:50:35 +03:00
Alexander Reelsen 6f437c973a Watcher: Ensure all templates exist before starting watcher (elastic/x-pack-elasticsearch#2765)
This is to ensure that the required templates exist (which are added by
the WatcherIndexTemplateRegistry) before actually starting watcher.

Relates elastic/x-pack-elasticsearch#2761 

Original commit: elastic/x-pack-elasticsearch@568088061f
2017-10-23 11:57:40 +02:00