Commit Graph

6720 Commits

Author SHA1 Message Date
Martijn van Groningen 9a1c103bb2 security: Fail search request if profile is used and DLS is active.
Original commit: elastic/x-pack-elasticsearch@b83536460d
2017-10-30 09:12:27 +01:00
Tim Vernum 0c7caabea1 Usability enhancements for certificate generation (elastic/x-pack-elasticsearch#2561)
This commit adds a new `certutil` command and deprecates the `certgen` command.
 
The new certuil consists of sub commands that are (by default) are simpler to use than the old monolithic command, but still support all the previous behaviours.

Original commit: elastic/x-pack-elasticsearch@3f57687da9
2017-10-30 13:08:31 +11:00
Nhat ba29971323 test: updates DocsStats with totalSizeInBytes
Relates https://github.com/elastic/elasticsearch/pull/27117

Original commit: elastic/x-pack-elasticsearch@9bf177d90b
2017-10-28 13:04:21 -04:00
Alexander Reelsen 940eabd5f3 Watcher: Add thread pool rejection to execution state (elastic/x-pack-elasticsearch#2805)
The execution state of a watch did not differentiate between failures of
the execution like a broken painless script and a thread pool rejection.

This adds an own state, which allows to aggregate on such data in the
watch history, which should ease debugging issues a bit.

Original commit: elastic/x-pack-elasticsearch@351e64e14d
2017-10-27 16:37:14 +02:00
Martijn van Groningen 96b0b4e96d test: refresh only once to workaround phrase suggester edge case
Relates to elastic/x-pack-elasticsearch#2804

Original commit: elastic/x-pack-elasticsearch@3f2b6b7eea
2017-10-27 15:05:18 +02:00
Hendrik Muhs f74e680142 [ML] add detectorIndex to modelplot and forecast (elastic/x-pack-elasticsearch#2796)
add detector_index to model plots and forecast

relates elastic/x-pack-elasticsearch#2547

corresponding ml-cpp change: elastic/machine-learning-cpp#361

Original commit: elastic/x-pack-elasticsearch@5927d8578e
2017-10-27 12:54:42 +02:00
Dimitris Athanasiou c7e94b3b4c [ML] Enable overall buckets aggregation at a custom bucket span (elastic/x-pack-elasticsearch#2782)
For the purpose of getting this API consumed by our UI, returning
overall buckets that match the job's largest `bucket_span` can
result in too much data. The UI only ever displays a few buckets
in the swimlane. Their span depends on the time range selected and
the screen resolution, but it will only ever be a relatively
low number.

This PR adds the ability to aggregate overall buckets in a user
specified `bucket_span`. That `bucket_span` may be equal or
greater to the largest job's `bucket_span`. The `overall_score`
of the result overall buckets is the max score of the
corresponding overall buckets with a span equal to the job's
largest `bucket_span`.

The implementation is now chunking the bucket requests
as otherwise the aggregation would fail when too many buckets
are matching.

Original commit: elastic/x-pack-elasticsearch@981f7a40e5
2017-10-27 11:14:13 +01:00
Martijn van Groningen e028716bec test: use a single primary shard to workaround an edge case with the phrase suggester
Relates to elastic/x-pack-elasticsearch#2804

Original commit: elastic/x-pack-elasticsearch@afd028faf7
2017-10-27 10:41:19 +02:00
Lisa Cawley 215f289a8c [DOCS] Reformatted security troubleshooting pages (elastic/x-pack-elasticsearch#2799)
Original commit: elastic/x-pack-elasticsearch@ec9969ec7a
2017-10-26 13:56:57 -07:00
Lisa Cawley 61bfa39331 [DOCS] Added setup-passwords command parameters (elastic/x-pack-elasticsearch#2735)
* [DOCS] Added setup-passwords command parameters

* [DOCS] Addressed feedback in setup-passwords command

Original commit: elastic/x-pack-elasticsearch@5401994c56
2017-10-26 08:28:32 -07:00
Martijn van Groningen 62215f1fae security: Fail request if suggesters are used and DLS is active.
Original commit: elastic/x-pack-elasticsearch@056c735e77
2017-10-26 08:02:31 +02:00
Tim Vernum 8985625ea5 [Security] BulkShardRequest may have multiple indices (elastic/x-pack-elasticsearch#2742)
If a bulk update references aliases rather than concrete indices,
it is possible that a single shard level request could have multiple distinct "index names", potentially including "date math".
Those names will resolve to the same concrete index, but they might have different privileges.

Original commit: elastic/x-pack-elasticsearch@34cfd11df8
2017-10-26 15:34:58 +11:00
Jason Tedor 70a38ec545 Enable BWC testing against other remotes
This commit enables BWC testing against remotes on GitHub other than
elastic/elasticsearch.git and elastic/x-pack-elasticsearch.git.

Relates elastic/x-pack-elasticsearch#2707

Original commit: elastic/x-pack-elasticsearch@9028f2e089
2017-10-25 22:39:58 -04:00
lcawley 615de68711 [DOCS] Added xpack.watcher.history.cleaner_service_enabled
Original commit: elastic/x-pack-elasticsearch@ceead2a1c2
2017-10-25 16:35:20 -07:00
Lisa Cawley c67ec73cb5 [DOCS] Add xpack.ml.max_model_memory_limit (elastic/x-pack-elasticsearch#2787)
* [DOCS] Add xpack.ml.max.model_memory_limit

* [DOCS] Addressed feedback on model limit setting

Original commit: elastic/x-pack-elasticsearch@77a10bfe0e
2017-10-25 09:15:19 -07:00
Lee Hinman cdaa047d56 [TEST] Fix compilation for ARS stats exposing
Original commit: elastic/x-pack-elasticsearch@eb0ad99434
2017-10-24 11:09:19 -06:00
Lisa Cawley 73e819b0bd [DOCS] Add secure versions of SSL passphrases (elastic/x-pack-elasticsearch#2478)
* [DOCS] Add secure versions of SSL passphrases

* [DOCS] Add secure xpack settings

* [DOCS] Clarify type of keystore

* [DOCS] Added secure settings to security page

* [DOCS] Clarify X-Pack secure settings

* [DOCS] Reformat secure X-Pack settings

Original commit: elastic/x-pack-elasticsearch@efe043fb67
2017-10-24 16:38:37 +01:00
Lisa Cawley 3a6870f0dc [DOCS] Added SSL settings to X-Pack installation (elastic/x-pack-elasticsearch#2733)
Original commit: elastic/x-pack-elasticsearch@c88cce763a
2017-10-24 08:34:07 -07:00
Albert Zaharovits 403912b8a2 SecureSettings ignored by customAuditIndexSettings (elastic/x-pack-elasticsearch#2748)
customAuditIndexSettings does not submit SecureSettings with putIndexMapping.

relates elastic/x-pack-elasticsearch#2705

* Randomize SecureSetting in testcase

* Apply feedback

Original commit: elastic/x-pack-elasticsearch@1a5414b057
2017-10-24 13:50:35 +03:00
Alexander Reelsen 6f437c973a Watcher: Ensure all templates exist before starting watcher (elastic/x-pack-elasticsearch#2765)
This is to ensure that the required templates exist (which are added by
the WatcherIndexTemplateRegistry) before actually starting watcher.

Relates elastic/x-pack-elasticsearch#2761 

Original commit: elastic/x-pack-elasticsearch@568088061f
2017-10-23 11:57:40 +02:00
Martijn van Groningen c9682d02d4 fix test
Original commit: elastic/x-pack-elasticsearch@7ca5e0fb54
2017-10-23 09:58:33 +02:00
Martijn van Groningen 652f6560b8 security: Always allow access to a rootdoc's nested documents if access to rootdoc is allowed
relates elastic/x-pack-elasticsearch#2665

Original commit: elastic/x-pack-elasticsearch@2bbddd1dd2
2017-10-23 09:28:53 +02:00
Alexander Reelsen b76c85e7fd Docs: Change port to be an integer in htttp input docs
Original commit: elastic/x-pack-elasticsearch@768ec54e03
2017-10-19 13:53:24 +02:00
Simon Willnauer 2d1ce76194 Adopt core that `_flush` and `_force_merge` doesn't refresh anymore (elastic/x-pack-elasticsearch#2752)
Relates to elastic/elasticsearch#27000

Original commit: elastic/x-pack-elasticsearch@52e9951094
2017-10-16 10:16:50 +02:00
Lisa Cawley 84b00995e3 [DOCS] Removed ML GA limitation (elastic/x-pack-elasticsearch#2751)
Original commit: elastic/x-pack-elasticsearch@30aeb0d269
2017-10-13 16:52:40 +01:00
Jason Tedor c35efb7adf Replace global checkpoint sync test
This commit replaces the REST test that the global checkpoint sync
action runs successfully as a privileged user. The test needs to be
replaced because it has a small race condition. Namely, the check that
the post-operation global checkpoint sync was successful could run
before the sync finishes running. To address this, we replace the REST
test with a test where we have a little more control and can assert busy
to avoid this race from failing the test.

Relates elastic/x-pack-elasticsearch#2749

Original commit: elastic/x-pack-elasticsearch@ea585b843c
2017-10-13 10:05:59 -04:00
Tanguy Leroux a6776cef97 [Monitoring] Add interval_ms to Monitoring documents (elastic/x-pack-elasticsearch#2650)
This commit adds a new interval_ms field to the monitoring documents. 
This field indicates the current collection interval for Elasticsearch or 
external monitored systems. The value is indexed as a long.

Related to elastic/x-pack-elasticsearch#212

Original commit: elastic/x-pack-elasticsearch@2ceb20455c
2017-10-13 11:18:47 +02:00
David Roberts 10cc0088e4 [DOCS] Make clearer that xpack.xyz.enabled settings are node settings (elastic/x-pack-elasticsearch#2731)
The discussion in elastic/x-pack-elasticsearch#2697 shows that this was not clear before.

relates elastic/x-pack-elasticsearch#2697

Original commit: elastic/x-pack-elasticsearch@87553faa2c
2017-10-13 09:22:21 +01:00
Jay Modi 9028c0a642 Allow PkiRealm to use truststore.password setting (elastic/x-pack-elasticsearch#2727)
This change fixes an incorrect check for a missing password setting for the PKI realm. The check
only allowed the secure setting to be used for the PkiRealm password even though the legacy setting
is still valid. This change fixes the check.

Relates elastic/x-pack-elasticsearch#2487

Original commit: elastic/x-pack-elasticsearch@a4524c2c05
2017-10-12 10:07:08 -06:00
lcawley 7f37c2c431 [DOCS] Fixed broken link to java transport client
Original commit: elastic/x-pack-elasticsearch@f3036bb2a0
2017-10-12 14:29:20 +01:00
Jason Tedor aece28c286 Add test for global checkpoint sync with security
After a write operation on an index, a post-operation global checkpoint
sync fires. Previously, this action fired on the same user as executed
the write action. If the user did not have priviledges for this action,
the global checkpoint sync would fail. With an upstream change in core,
this action now fires as the system user. This commit adds a test that
create a user that has minimal write permissions on an index, but none
that would imply it could execute the global checkpoint sync. This then
serves as a test that the upstream change to fire the global checkpoint
sync as the system user is correct. This test must run as a mulit-node
test so that a replica is a assigned so that the global checkpoint sync
fires in the first place. This test does indeed fail without the
upstream change, and passes with it.

Relates elastic/x-pack-elasticsearch#2744

Original commit: elastic/x-pack-elasticsearch@bf7e771756
2017-10-12 09:19:17 -04:00
lcawley b628815dbe [DOCS] Fixed link to X-Pack transport client
Original commit: elastic/x-pack-elasticsearch@0870334e4b
2017-10-12 13:41:14 +01:00
Lisa Cawley 604229cd4d [DOCS] Added transport client info for X-Pack (elastic/x-pack-elasticsearch#2737)
* [DOCS] Added transport client info for X-Pack

* [DOCS] Relocated X-Pack java client info

* [DOCS] Added transport client deprecation info

Original commit: elastic/x-pack-elasticsearch@416aab1d76
2017-10-12 13:18:44 +01:00
Tanguy Leroux 0299886388 [Tests] Use XPack Usage API to verify Monitoring exporters are disabled (elastic/x-pack-elasticsearch#2648)
This commit changes the MonitoringIt and XPackRestIT tests so that the
disableMonitoring() method now use the XPack Usage API in order to check
that the monitoring exporters are correctly disabled. It checks at the
beginning of the tests (all exporters must be disabled before running
the test) and also at the end of the test.

This commit also fixes a bug in MonitoringIT where the Bulk thread pool
active queue was wrongly extracted from the response's map, forcing the
test to always wait for 30sec.

relates elastic/x-pack-elasticsearch#2459

Original commit: elastic/x-pack-elasticsearch@2d349e840f
2017-10-12 09:36:44 +02:00
Tanguy Leroux ea4bff1d43 [Monitoring] Align MonitoringBulkDoc serialization with 6.0 (elastic/x-pack-elasticsearch#2736)
The version used in serialization must be aligned with 6.0/6.x.

Original commit: elastic/x-pack-elasticsearch@db63b91bc6
2017-10-11 17:56:24 +02:00
Lisa Cawley 2455415a04 [DOCS] Small fixes in the overall buckets API (elastic/x-pack-elasticsearch#2732)
* [DOCS] Small fixes in the overall buckets API

* [DOCS] Addressed feedback in overall buckets API

Original commit: elastic/x-pack-elasticsearch@4f79bc9a50
2017-10-11 16:25:05 +01:00
lcawley 723dd49905 [DOCS] Fixed typo in count function
Original commit: elastic/x-pack-elasticsearch@34c821796b
2017-10-11 16:16:28 +01:00
Tanguy Leroux 8484680007 Few fixes in packaging tests
This commit fixes indentation in certgen.bash, adds a check on cluster
health in bootstrap_password.bash and fixes a bug in xpack.bash

Original commit: elastic/x-pack-elasticsearch@d6847f6640
2017-10-11 11:53:10 +02:00
David Roberts c84d69fde3 [DOCS] Fix ML post_data docs (elastic/x-pack-elasticsearch#2689)
It was pointed out in
https://github.com/elastic/elasticsearch-net/pull/2856#discussion_r142830656
that our post_data docs incorrectly say that reset_start and reset_end are
body parameters.  In fact they are query parameters.

There were also a number of other errors and ommissions on this page that I
have attempted to correct.

Original commit: elastic/x-pack-elasticsearch@c83decacc7
2017-10-11 10:47:07 +01:00
Tim Vernum a4f7db4f66 [Security] Improve error messages in setup-passwords (elastic/x-pack-elasticsearch#2724)
Provides more verbose messaging around errors and possible causes when the tool aborts.

This change is primarily focused on errors connecting to the Elasticsearch node when TLS is enabled on the HTTP connection.

Original commit: elastic/x-pack-elasticsearch@aa8f7c6143
2017-10-11 12:32:35 +10:00
Tim Vernum bc038b323d [Security] Apply validation when parsing certgen input (elastic/x-pack-elasticsearch#2711)
When certgen configuration was read from an input file (`-in` option) validation errors were collected but never reported. Depending on the type of error this may have caused the tool to exit with an internal error (e.g. NPE).

Validation is now applied after parsing the file and if errors are found the tool exits.

Original commit: elastic/x-pack-elasticsearch@b2262ed1d7
2017-10-11 12:30:19 +10:00
David Roberts 5d0388ccb3 [TEST] Fix ML node attribute test
When ML is disabled the attribute checking is stricter, but the test
did not reflect this

Original commit: elastic/x-pack-elasticsearch@613e97c595
2017-10-10 16:22:03 +01:00
David Roberts ab9cc25a8e [ML] Prevent ML node attributes being set directly (elastic/x-pack-elasticsearch#2725)
ML uses node attributes to ensure that the master node knows how many
ML jobs may be allocated to each node.  This change prevents a user
messing up the way these attributes are used by setting them differently
using node.attr.* entries in their elasticsearch.yml.

This covers the "very short term" change outlined in elastic/x-pack-elasticsearch#2649

Original commit: elastic/x-pack-elasticsearch@9c381801d9
2017-10-10 15:12:59 +01:00
Dimitris Athanasiou 5eea355b33 [ML] Add overall buckets api (elastic/x-pack-elasticsearch#2713)
Adds the GET overall_buckets API.

The REST end point is: GET
/_xpack/ml/anomaly_detectors/job_id/results/overall_buckets

The API returns overall bucket results. An overall bucket
is a summarized bucket result over multiple jobs.
It has the `bucket_span` of the longest job's `bucket_span`.
It also has an `overall_score` that is the `top_n` average of the
max anomaly scores per job.

relates elastic/x-pack-elasticsearch#2693

Original commit: elastic/x-pack-elasticsearch@ba6061482d
2017-10-10 14:41:24 +01:00
Dimitris Athanasiou 90e327032e [ML][DOCS] Fix bucket_span type in results resources (elastic/x-pack-elasticsearch#2714)
Original commit: elastic/x-pack-elasticsearch@af24bde71f
2017-10-10 12:51:12 +01:00
Tanguy Leroux 1ed4be1471 Show exit code in Bootstrap Password packaging tests
Also cleans up some files before the test is executed,
and explicitly binds to 127.0.0.1/9200.

Original commit: elastic/x-pack-elasticsearch@778584ea78
2017-10-10 09:51:22 +02:00
Alexander Reelsen 80593fb23c Watcher: Add execution state to watch status (elastic/x-pack-elasticsearch#2699)
The execution state is kind of a global indicator if a watch has been
running successfully and is used by the watcher UI.

However this field is only stored in the watch history but not part of
the watch status, thus it is not available everywhere. In order to
simplify the watcher UI this commit also adds the field to the
watch status which is stored together with the watch.

It is stored under the `status.execution_state` field as `status.state`
is already taken. This is also reflects with the name of the java class.

The WatchStatus class does not contain serialization checks, as this is
intended to be backported to 6.x, where those checks will be added.

Once the backport is done, the old execution state field can be fully
deleted from the master branch in another commit (syncing with Kibana
folks required).

relates elastic/x-pack-elasticsearch#2385

* fix doc tests

Original commit: elastic/x-pack-elasticsearch@26e8f99571
2017-10-10 09:07:33 +02:00
Alexander Reelsen cadfd03529 Watcher: Allow JIRA path to be custom chosen (elastic/x-pack-elasticsearch#2682)
The path of a JIRA endpoint used to be fixed. This commit allows the
path to be dynamic, so that users can deploy their JIRA instance under
an arbitrary prefix.

Original commit: elastic/x-pack-elasticsearch@7702505114
2017-10-10 08:55:28 +02:00
Chris Earle 69ab7797be [Monitoring] Cleaner Service should be able to cleanup .watcher-history* (elastic/x-pack-elasticsearch#2696)
This adds a dynamic setting, which defaults to `false` currently, that can be used to delete all `.watcher-history*` indices that match the same age requirements as Monitoring indices.

Original commit: elastic/x-pack-elasticsearch@8ca3bdbca3
2017-10-09 15:46:07 -06:00
David Roberts 9ad961088d [TEST] Wait a little longer for named pipes to open in unit tests (elastic/x-pack-elasticsearch#2712)
Same fix as elastic/x-pack-elasticsearch#987, but for the unit tests.  The slowness affecting EBS
volumes created from snapshots can affect CI as it runs on AWS instances.

Original commit: elastic/x-pack-elasticsearch@306b8110b7
2017-10-09 13:09:17 +01:00