Commit Graph

1361 Commits

Author SHA1 Message Date
Nik Everett 562117a7b7 Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@c8368be422
2017-11-03 16:16:50 -04:00
David Roberts ba5dbc4daf Remove uses of single argument Environment constructor from production code (elastic/x-pack-elasticsearch#2852)
Following elastic/elasticsearch#27235 the single argument Environment constructor
is forbidden in production code.  This change removes the last such uses from
X-Pack.

Original commit: elastic/x-pack-elasticsearch@87e72d0d07
2017-11-03 09:12:35 +00:00
David Roberts 14211b47f2 Use the correct Environment object in NativeControllerHolder (elastic/x-pack-elasticsearch#2847)
We should not be constructing a temporary Environment object in production
code.  This currently isn't causing any problems, but it might in the future
if elastic/elasticsearch#27144 or something similar is ever merged.  Instead
the master Environment of the node should always be used.

Original commit: elastic/x-pack-elasticsearch@6276a54a45
2017-11-02 16:36:09 +00:00
Chris Earle 2acd0afdd5 [Monitoring] Add Data to Warn on lack of Transport TLS (elastic/x-pack-elasticsearch#2820)
This adds the data necessary to add a warning to the alerts UI representing each cluster when xpack.security.transport.tls.enabled is not set to true for a trial licensed cluster running with
xpack.security.enabled.

Original commit: elastic/x-pack-elasticsearch@28fe8bad76
2017-11-02 15:26:04 +00:00
Nik Everett 28dc53ac5e Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@ad5707a44f
2017-11-02 00:14:51 -04:00
Nik Everett d66d88c5cd Fix tset compile
Test wasn't compiling after a change to core.

Original commit: elastic/x-pack-elasticsearch@8017b29f0b
2017-11-02 00:14:27 -04:00
Albert Zaharovits fb13299714 Log when encountered cert but expecting key and vice-versa.. (elastic/x-pack-elasticsearch#2670)
Log when encountered cert but expecting key and vice-versa.

relates elastic/x-pack-elasticsearch#2657

Original commit: elastic/x-pack-elasticsearch@4e26d8044f
2017-11-01 15:20:28 +02:00
Chris Earle 31741a85d9 [Monitoring] Add Shard-level Details to Index Stats Collection (elastic/x-pack-elasticsearch#2817)
This adds details about the shards and the health of the index. By adding these stats directly to the document, the UI can avoid many aggregations and enable better searching and sorting against indices.

Original commit: elastic/x-pack-elasticsearch@f38ae5ce69
2017-10-31 16:41:40 +00:00
Tanguy Leroux f416c5b3c9 Replace empty index block checks with global block checks in get/put license actions (elastic/x-pack-elasticsearch#2766)
Related to elastic/x-pack-elasticsearch#10530

Original commit: elastic/x-pack-elasticsearch@f4c9924d62
2017-10-31 16:15:14 +01:00
Nik Everett 33f4a8317c JDBC metadata integration with security (elastic/x-pack-elasticsearch#2806)
I realized that we weren't running our DatabaseMetaData tests. One thing led to another and I made these changes:
1. Got the DatabaseMetaData tests running in all three of our QA projects.
2. Fixed the SecurityCatalogFilter to work with `SqlGetIndicesAction`. It worked before, but only for requests that were a `SqlAction` as well as `SqlGetIndicesAction`.
3. Added security test for the JDBC DatabaseMetaData requests. These mirror exactly the security tests that we use for `SHOW TABLES` and `DESCRIBE` but cover the JDBC actions.

Original commit: elastic/x-pack-elasticsearch@7026d83c06
2017-10-30 23:22:12 +00:00
Nik Everett b0dc14f639 Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@7af47176fc
2017-10-30 13:50:33 -04:00
Jay Modi 4f65d9b527 Retry startup for IndexAuditTrail and version templates (elastic/x-pack-elasticsearch#2755)
This commit removes the FAILED state for the IndexAuditTrail so that we always try to keep starting
the service. Previously, on any exception during startup we moved to a failed state and never tried
to start again. The users only option was to restart the node. This was problematic in the case of
large clusters as there could be common timeouts of cluster state listeners that would cause the
startup of this service to fail.

Additionally, the logic in the IndexAuditTrail to update the template on the current cluster has
been removed and replaced with the use of the TemplateUpgradeService. However, we still need to
maintain the ability to determine if a template on a remote cluster should be PUT. To avoid always
PUTing the template, the version field has been added so it only needs to be PUT once on upgrade.

Finally, the default queue size has been increased as this is another common issue that users hit
with high traffic clusters.

relates elastic/x-pack-elasticsearch#2658

Original commit: elastic/x-pack-elasticsearch@27e2ce7223
2017-10-30 09:11:18 -06:00
Chris Earle c335b00c9b [Monitoring] Add "cluster_state.nodes_hash" to document (elastic/x-pack-elasticsearch#2798)
Adding this field enables a very simple mechanism for detecting node changes in the cluster state via Watcher (and other mechanisms). The next step is to add the cluster alert that uses it.

Original commit: elastic/x-pack-elasticsearch@1eacc25cff
2017-10-30 13:13:57 +00:00
Martijn van Groningen 9706d07209 disabled test assertion
Original commit: elastic/x-pack-elasticsearch@5a05c607e5
2017-10-30 10:15:45 +01:00
Martijn van Groningen f96153143b test: added logging and don't use replicas
Original commit: elastic/x-pack-elasticsearch@917ef8177d
2017-10-30 09:39:10 +01:00
Martijn van Groningen 2d89394896 test: removed refresh workaround
Original commit: elastic/x-pack-elasticsearch@75b571c23f
2017-10-30 09:29:43 +01:00
Martijn van Groningen 9a1c103bb2 security: Fail search request if profile is used and DLS is active.
Original commit: elastic/x-pack-elasticsearch@b83536460d
2017-10-30 09:12:27 +01:00
Tim Vernum 0c7caabea1 Usability enhancements for certificate generation (elastic/x-pack-elasticsearch#2561)
This commit adds a new `certutil` command and deprecates the `certgen` command.
 
The new certuil consists of sub commands that are (by default) are simpler to use than the old monolithic command, but still support all the previous behaviours.

Original commit: elastic/x-pack-elasticsearch@3f57687da9
2017-10-30 13:08:31 +11:00
Nhat ba29971323 test: updates DocsStats with totalSizeInBytes
Relates https://github.com/elastic/elasticsearch/pull/27117

Original commit: elastic/x-pack-elasticsearch@9bf177d90b
2017-10-28 13:04:21 -04:00
Nik Everett b91e9d85e9 Fix SHAs
Original commit: elastic/x-pack-elasticsearch@15c2669416
2017-10-27 17:12:15 -04:00
Alexander Reelsen 940eabd5f3 Watcher: Add thread pool rejection to execution state (elastic/x-pack-elasticsearch#2805)
The execution state of a watch did not differentiate between failures of
the execution like a broken painless script and a thread pool rejection.

This adds an own state, which allows to aggregate on such data in the
watch history, which should ease debugging issues a bit.

Original commit: elastic/x-pack-elasticsearch@351e64e14d
2017-10-27 16:37:14 +02:00
Martijn van Groningen 96b0b4e96d test: refresh only once to workaround phrase suggester edge case
Relates to elastic/x-pack-elasticsearch#2804

Original commit: elastic/x-pack-elasticsearch@3f2b6b7eea
2017-10-27 15:05:18 +02:00
Hendrik Muhs f74e680142 [ML] add detectorIndex to modelplot and forecast (elastic/x-pack-elasticsearch#2796)
add detector_index to model plots and forecast

relates elastic/x-pack-elasticsearch#2547

corresponding ml-cpp change: elastic/machine-learning-cpp#361

Original commit: elastic/x-pack-elasticsearch@5927d8578e
2017-10-27 12:54:42 +02:00
Dimitris Athanasiou c7e94b3b4c [ML] Enable overall buckets aggregation at a custom bucket span (elastic/x-pack-elasticsearch#2782)
For the purpose of getting this API consumed by our UI, returning
overall buckets that match the job's largest `bucket_span` can
result in too much data. The UI only ever displays a few buckets
in the swimlane. Their span depends on the time range selected and
the screen resolution, but it will only ever be a relatively
low number.

This PR adds the ability to aggregate overall buckets in a user
specified `bucket_span`. That `bucket_span` may be equal or
greater to the largest job's `bucket_span`. The `overall_score`
of the result overall buckets is the max score of the
corresponding overall buckets with a span equal to the job's
largest `bucket_span`.

The implementation is now chunking the bucket requests
as otherwise the aggregation would fail when too many buckets
are matching.

Original commit: elastic/x-pack-elasticsearch@981f7a40e5
2017-10-27 11:14:13 +01:00
Martijn van Groningen e028716bec test: use a single primary shard to workaround an edge case with the phrase suggester
Relates to elastic/x-pack-elasticsearch#2804

Original commit: elastic/x-pack-elasticsearch@afd028faf7
2017-10-27 10:41:19 +02:00
Nik Everett 52d9de1de7 Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@dd027d9ca5
2017-10-26 09:51:04 -04:00
Martijn van Groningen 62215f1fae security: Fail request if suggesters are used and DLS is active.
Original commit: elastic/x-pack-elasticsearch@056c735e77
2017-10-26 08:02:31 +02:00
Tim Vernum 8985625ea5 [Security] BulkShardRequest may have multiple indices (elastic/x-pack-elasticsearch#2742)
If a bulk update references aliases rather than concrete indices,
it is possible that a single shard level request could have multiple distinct "index names", potentially including "date math".
Those names will resolve to the same concrete index, but they might have different privileges.

Original commit: elastic/x-pack-elasticsearch@34cfd11df8
2017-10-26 15:34:58 +11:00
Jason Tedor 70a38ec545 Enable BWC testing against other remotes
This commit enables BWC testing against remotes on GitHub other than
elastic/elasticsearch.git and elastic/x-pack-elasticsearch.git.

Relates elastic/x-pack-elasticsearch#2707

Original commit: elastic/x-pack-elasticsearch@9028f2e089
2017-10-25 22:39:58 -04:00
Lee Hinman cdaa047d56 [TEST] Fix compilation for ARS stats exposing
Original commit: elastic/x-pack-elasticsearch@eb0ad99434
2017-10-24 11:09:19 -06:00
Albert Zaharovits 403912b8a2 SecureSettings ignored by customAuditIndexSettings (elastic/x-pack-elasticsearch#2748)
customAuditIndexSettings does not submit SecureSettings with putIndexMapping.

relates elastic/x-pack-elasticsearch#2705

* Randomize SecureSetting in testcase

* Apply feedback

Original commit: elastic/x-pack-elasticsearch@1a5414b057
2017-10-24 13:50:35 +03:00
Alexander Reelsen 6f437c973a Watcher: Ensure all templates exist before starting watcher (elastic/x-pack-elasticsearch#2765)
This is to ensure that the required templates exist (which are added by
the WatcherIndexTemplateRegistry) before actually starting watcher.

Relates elastic/x-pack-elasticsearch#2761 

Original commit: elastic/x-pack-elasticsearch@568088061f
2017-10-23 11:57:40 +02:00
Martijn van Groningen c9682d02d4 fix test
Original commit: elastic/x-pack-elasticsearch@7ca5e0fb54
2017-10-23 09:58:33 +02:00
Martijn van Groningen 652f6560b8 security: Always allow access to a rootdoc's nested documents if access to rootdoc is allowed
relates elastic/x-pack-elasticsearch#2665

Original commit: elastic/x-pack-elasticsearch@2bbddd1dd2
2017-10-23 09:28:53 +02:00
Nik Everett 65f2b9fe01 Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@9fc67cbfee
2017-10-18 10:27:19 -04:00
Simon Willnauer 2d1ce76194 Adopt core that `_flush` and `_force_merge` doesn't refresh anymore (elastic/x-pack-elasticsearch#2752)
Relates to elastic/elasticsearch#27000

Original commit: elastic/x-pack-elasticsearch@52e9951094
2017-10-16 10:16:50 +02:00
Igor Motov e1d4fdc6d3 SQL: Unknown filter type [pattern_capture] warning in SQL IT Tests (elastic/x-pack-elasticsearch#2746)
Fixes java.lang.IllegalArgumentException: Unknown filter type [pattern_capture] for [email] warnings that pollute log files during execution of the AbstractSqlIntegTestCase-based tests.

Related to elastic/x-pack-elasticsearch#2501

Original commit: elastic/x-pack-elasticsearch@b9ec3c20d0
2017-10-13 11:55:48 -04:00
Nik Everett 99fea5f448 Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@d521ecda35
2017-10-13 10:39:28 -04:00
Tanguy Leroux a6776cef97 [Monitoring] Add interval_ms to Monitoring documents (elastic/x-pack-elasticsearch#2650)
This commit adds a new interval_ms field to the monitoring documents. 
This field indicates the current collection interval for Elasticsearch or 
external monitored systems. The value is indexed as a long.

Related to elastic/x-pack-elasticsearch#212

Original commit: elastic/x-pack-elasticsearch@2ceb20455c
2017-10-13 11:18:47 +02:00
Jay Modi 9028c0a642 Allow PkiRealm to use truststore.password setting (elastic/x-pack-elasticsearch#2727)
This change fixes an incorrect check for a missing password setting for the PKI realm. The check
only allowed the secure setting to be used for the PkiRealm password even though the legacy setting
is still valid. This change fixes the check.

Relates elastic/x-pack-elasticsearch#2487

Original commit: elastic/x-pack-elasticsearch@a4524c2c05
2017-10-12 10:07:08 -06:00
Tanguy Leroux 0299886388 [Tests] Use XPack Usage API to verify Monitoring exporters are disabled (elastic/x-pack-elasticsearch#2648)
This commit changes the MonitoringIt and XPackRestIT tests so that the
disableMonitoring() method now use the XPack Usage API in order to check
that the monitoring exporters are correctly disabled. It checks at the
beginning of the tests (all exporters must be disabled before running
the test) and also at the end of the test.

This commit also fixes a bug in MonitoringIT where the Bulk thread pool
active queue was wrongly extracted from the response's map, forcing the
test to always wait for 30sec.

relates elastic/x-pack-elasticsearch#2459

Original commit: elastic/x-pack-elasticsearch@2d349e840f
2017-10-12 09:36:44 +02:00
Costin Leau 2026198dd4 Add size for column tests (elastic/x-pack-elasticsearch#2685)
Add displaySize to columnInfo

Original commit: elastic/x-pack-elasticsearch@ed1d265e98
2017-10-12 00:03:41 +03:00
Tanguy Leroux ea4bff1d43 [Monitoring] Align MonitoringBulkDoc serialization with 6.0 (elastic/x-pack-elasticsearch#2736)
The version used in serialization must be aligned with 6.0/6.x.

Original commit: elastic/x-pack-elasticsearch@db63b91bc6
2017-10-11 17:56:24 +02:00
Tim Vernum a4f7db4f66 [Security] Improve error messages in setup-passwords (elastic/x-pack-elasticsearch#2724)
Provides more verbose messaging around errors and possible causes when the tool aborts.

This change is primarily focused on errors connecting to the Elasticsearch node when TLS is enabled on the HTTP connection.

Original commit: elastic/x-pack-elasticsearch@aa8f7c6143
2017-10-11 12:32:35 +10:00
Tim Vernum bc038b323d [Security] Apply validation when parsing certgen input (elastic/x-pack-elasticsearch#2711)
When certgen configuration was read from an input file (`-in` option) validation errors were collected but never reported. Depending on the type of error this may have caused the tool to exit with an internal error (e.g. NPE).

Validation is now applied after parsing the file and if errors are found the tool exits.

Original commit: elastic/x-pack-elasticsearch@b2262ed1d7
2017-10-11 12:30:19 +10:00
David Roberts 5d0388ccb3 [TEST] Fix ML node attribute test
When ML is disabled the attribute checking is stricter, but the test
did not reflect this

Original commit: elastic/x-pack-elasticsearch@613e97c595
2017-10-10 16:22:03 +01:00
David Roberts ab9cc25a8e [ML] Prevent ML node attributes being set directly (elastic/x-pack-elasticsearch#2725)
ML uses node attributes to ensure that the master node knows how many
ML jobs may be allocated to each node.  This change prevents a user
messing up the way these attributes are used by setting them differently
using node.attr.* entries in their elasticsearch.yml.

This covers the "very short term" change outlined in elastic/x-pack-elasticsearch#2649

Original commit: elastic/x-pack-elasticsearch@9c381801d9
2017-10-10 15:12:59 +01:00
Dimitris Athanasiou 5eea355b33 [ML] Add overall buckets api (elastic/x-pack-elasticsearch#2713)
Adds the GET overall_buckets API.

The REST end point is: GET
/_xpack/ml/anomaly_detectors/job_id/results/overall_buckets

The API returns overall bucket results. An overall bucket
is a summarized bucket result over multiple jobs.
It has the `bucket_span` of the longest job's `bucket_span`.
It also has an `overall_score` that is the `top_n` average of the
max anomaly scores per job.

relates elastic/x-pack-elasticsearch#2693

Original commit: elastic/x-pack-elasticsearch@ba6061482d
2017-10-10 14:41:24 +01:00
Alexander Reelsen 80593fb23c Watcher: Add execution state to watch status (elastic/x-pack-elasticsearch#2699)
The execution state is kind of a global indicator if a watch has been
running successfully and is used by the watcher UI.

However this field is only stored in the watch history but not part of
the watch status, thus it is not available everywhere. In order to
simplify the watcher UI this commit also adds the field to the
watch status which is stored together with the watch.

It is stored under the `status.execution_state` field as `status.state`
is already taken. This is also reflects with the name of the java class.

The WatchStatus class does not contain serialization checks, as this is
intended to be backported to 6.x, where those checks will be added.

Once the backport is done, the old execution state field can be fully
deleted from the master branch in another commit (syncing with Kibana
folks required).

relates elastic/x-pack-elasticsearch#2385

* fix doc tests

Original commit: elastic/x-pack-elasticsearch@26e8f99571
2017-10-10 09:07:33 +02:00
Alexander Reelsen cadfd03529 Watcher: Allow JIRA path to be custom chosen (elastic/x-pack-elasticsearch#2682)
The path of a JIRA endpoint used to be fixed. This commit allows the
path to be dynamic, so that users can deploy their JIRA instance under
an arbitrary prefix.

Original commit: elastic/x-pack-elasticsearch@7702505114
2017-10-10 08:55:28 +02:00