Commit Graph

7831 Commits

Author SHA1 Message Date
Lee Hinman 418eefbcf1 Remove BytesArray and BytesReference usage from XContentFactory (elastic/x-pack-elasticsearch#4158)
This is the x-pack side of https://github.com/elastic/elasticsearch/pull/29151

Original commit: elastic/x-pack-elasticsearch@be71338057
2018-03-20 11:52:31 -06:00
Tim Brooks 0a1e09c644 Remove date from rest resp for non-exp licenses (elastic/x-pack-elasticsearch#4149)
This is related to elastic/x-pack-elasticsearch#3877. This commit removes the expiration from the
json rest response for licenses that do not expire.

Original commit: elastic/x-pack-elasticsearch@f767e9d756
2018-03-20 12:19:07 -04:00
Alan Woodward c16e5f1f92 Watcher comparisons don't deal with NaN correctly (elastic/x-pack-elasticsearch#4133)
Aggregations may return `NaN`, and the comparison code would return `true` if this result was passed to a `gte` or `lte` condition.

Original commit: elastic/x-pack-elasticsearch@3b16ae6675
2018-03-20 15:55:59 +00:00
Tim Vernum 063ed78c42 Provide clearer errors if SAML is not licensed (elastic/x-pack-elasticsearch#4096)
SAML is only available on a Platinum license. If you try and use SAML
on a Gold license, then the error message is misleading - it gives the
equivalent of "cannot find saml realm".

This change adds a standard license error of
    "current license is non-compliant for [saml]"
if SAML rest actions are used when SAML is not licensed.

Original commit: elastic/x-pack-elasticsearch@7c0e26d58e
2018-03-20 18:54:51 +10:00
Yogesh Gaikwad 0de6376452 [DOCS] `xpack.ssl.client_authentication` setting does not apply to HTTP (elastic/x-pack-elasticsearch#4113)
Fix the documentation to mention the global setting does not apply to HTTP.

relates elastic/x-pack-elasticsearch#3413

Original commit: elastic/x-pack-elasticsearch@f9dc545b4a
2018-03-20 18:45:36 +11:00
Tim Vernum 7cd5e1d516 Check cluster heath before setup-passwords (elastic/x-pack-elasticsearch#4104)
Trying to setup passwords on a red cluster (or a cluster that cannot
reach a quorum) is generally not a good idea.

This commit:
- Adds a check for RED cluster status
- Prompts to confirm execution if the cluster is red
- Prints out the reason/type is an error response is received
- Increases the HTTP read timeout so that master election failures are
  reported correctly.

Original commit: elastic/x-pack-elasticsearch@4ffbda23db
2018-03-20 17:03:17 +10:00
Tim Vernum bc95ad80ce Add beats_system user to security (elastic/x-pack-elasticsearch#4103)
This creates a new "beats_system" user and role with the same
privileges as the existing "logstash_system" user/role.

The "beat_system" user is also added as a managed user within
the "setup-passwords" command.

Users who upgrade from an earlier version of Elasticsearch/X-Pack
will need to manually set a password for the beats_system user via
the change password API (or Kibana UI)

Original commit: elastic/x-pack-elasticsearch@6087d3a18e
2018-03-20 17:01:53 +10:00
Yogesh Gaikwad 10bb78c3d6 X-Pack-Security: Correct attribute_patterns regex in saml guide (elastic/x-pack-elasticsearch#4159)
X-Pack-Security: Correct attribute_patterns regex in saml guide

relates elastic/x-pack-elasticsearch#4157

Original commit: elastic/x-pack-elasticsearch@546d408e5a
2018-03-20 17:37:05 +11:00
Lisa Cawley 9d697f3c8b [DOCS] Add monitoring overview (elastic/x-pack-elasticsearch#4148)
Original commit: elastic/x-pack-elasticsearch@750ca4abc1
2018-03-19 16:48:07 -07:00
jaymode 0493f64e3f [Test] mute MonitoringIT#testMonitoringService
The MonitoringIT#testMonitoringService test fails occasionally in CI
due to a failed wait for documents condition, so this commit adds an
awaits fix.

See elastic/x-pack-elasticsearch#4150

Original commit: elastic/x-pack-elasticsearch@c9415cba63
2018-03-19 12:47:51 -06:00
Tim Brooks 762bacbc03 Update start_trial version compatibility (elastic/x-pack-elasticsearch#4122)
This is related to elastic/x-pack-elasticsearch#3877. With the backport of elastic/x-pack-elasticsearch#4102 the type
parameter for start_trial is now on 6.3. This commit updates the
version check during stream serialization / deserialization.

Original commit: elastic/x-pack-elasticsearch@75e8537ab6
2018-03-19 12:08:13 -04:00
Lisa Cawley b6f16a6b0d [DOCS] Add aggregation limitation details (elastic/x-pack-elasticsearch#4143)
Original commit: elastic/x-pack-elasticsearch@953103bcd7
2018-03-19 07:55:14 -07:00
Alexander Reelsen 955b0dddad Watcher: Ignore deactivated state with execute watch API (elastic/x-pack-elasticsearch#4054)
If a watch is not active, it should still be executed, if it is called
via the execute watch API.

This commit adds an additional method to the execution context to check
for this, which returns true for a manual execution context but checks
the watch status for the triggered one.

Original commit: elastic/x-pack-elasticsearch@18f3f9e84b
2018-03-19 07:31:06 -07:00
Jason Tedor a68051405a Fix Javadocs for MonitoringTempalteUtils
This commit fixes the Javadoc build for MonitoringTemplateUtils after
changes to core removed the string and bytes methods from
XContentBuilder.

Original commit: elastic/x-pack-elasticsearch@49f3b95b22
2018-03-19 10:19:50 -04:00
Alexander Reelsen d46bdd887f Watcher: Fix SmokeTestWatcherWithSecurityIT test (elastic/x-pack-elasticsearch#4140)
These tests were configured to run every second, some of the watch
history entries could have an execution state throttled, when two
executions happened while waiting. The tests who were affected by 
this are now explicitely searching for the watch history entry with 
the correct state.

relates elastic/x-pack-elasticsearch#4137

Original commit: elastic/x-pack-elasticsearch@e463cc4770
2018-03-19 06:51:28 -07:00
Alexander Reelsen e53121206c Watcher: Clear usage stats when execution is paused (elastic/x-pack-elasticsearch#4132)
This ensures that the per watch stats in the trigger service are
properly cleared, when execution is paused.

Original commit: elastic/x-pack-elasticsearch@f5119e4072
2018-03-19 06:15:10 -07:00
Tanguy Leroux c0be4d7055 Fix compilation errors in ML integration tests
After elastic/elasticsearch#29109, the `needsReassignment` method has
been moved to the PersistentTasksClusterService. This commit fixes
some compilation in tests I introduced.

Original commit: elastic/x-pack-elasticsearch@b5bca2a191
2018-03-19 09:46:52 +01:00
Jason Tedor 83aae92353 Use bash in packaging heredocs (elastic/x-pack-elasticsearch#4145)
In some places in the packaging tests we use heredocs to run some
scripts, for example, under the root user. However, we were running
these heredocs under sh instead of bash. This is confusing since we use
bash everywhere and we end up with syntax in the herdocs that expects
bash but will not necessarily be run under bash. For example, one
circumstance where this arises is the bash syntax [[. This leads to
packaging test failures on some systems (e.g., the debian-8 box) that
are solely due to this sh/bash distinction. We should really be using
bash everywhere, it is the shell that we settled on for all of our
scripts everywhere awhile ago and then we can avoid this confusion. This
commit replaces the usage of sh in the X-Pack packaging tests with bash.

Original commit: elastic/x-pack-elasticsearch@7555bb32c8
2018-03-18 13:06:11 -04:00
Albert Zaharovits 7ea79c88ab Realm along the principal in audit authz events (elastic/x-pack-elasticsearch#3260)
Add realm name to all authz audit events: accessDenied, accessGranted,
runAsDenied and runAsGranted.
These event types receive the following attributes: realm,
run_by_realm and run_as_realm to go along with with the existing
attributes: principal, run_by_principal and run_as_principal. The
'effective realm name' (run_as_realm or run_by_realm) is certainly
filterable by ignore policies.

Original commit: elastic/x-pack-elasticsearch@cb3801e197
2018-03-18 11:27:28 +02:00
Albert Zaharovits 14acdcb4f7 Fix test after elastic/x-pack-elasticsearch#28919
After elastic/x-pack-elasticsearch#28919 some rest client exceptions are wrapped by the
IOException.

Original commit: elastic/x-pack-elasticsearch@6c658ae5b7
2018-03-17 16:58:12 +02:00
Albert Zaharovits 891013a17b Update audit trail filter policy settings (elastic/x-pack-elasticsearch#3984)
Audit trail filter policies can now be updated by the cluster update
settings API. Previously, policies were static inside the conf file.
This is helpful because, in practice, coming up with a reasonable
policy set is an iterative process, adding and changing policies to
shave off yet another pesky audit event.

Original commit: elastic/x-pack-elasticsearch@e8a670c427
2018-03-17 11:30:11 +02:00
Nik Everett f1c83820f7 Watcher: Mark test AwaitsFix
Looks like sometimes it catches a throttled watch.

```
18:23:19 FAILURE 7.93s | SmokeTestWatcherWithSecurityIT.testSearchTransformHasPermissions <<< FAILURES!
18:23:19    > Throwable elastic/x-pack-elasticsearch#1: java.lang.AssertionError:
18:23:19    > Expected: is "executed"
18:23:19    >      but: was "throttled"
18:23:19    > 	at __randomizedtesting.SeedInfo.seed([C40A591DC5A7785E:ED7ABE031CA86AB3]:0)
18:23:19    > 	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
18:23:19    > 	at org.elasticsearch.smoketest.SmokeTestWatcherWithSecurityIT.testSearchTransformHasPermissions(SmokeTestWatcherWithSecurityIT.java:189)
18:23:19    > 	at java.lang.Thread.run(Thread.java:748)
```

Original commit: elastic/x-pack-elasticsearch@b96b6438d3
2018-03-16 15:39:14 -04:00
Chris Earle d19ad8b3d1 [TEST][Monitoring] Also set StreamInput version
Sets the StreamInput's version so that it properly deserializes the
incoming message.

Original commit: elastic/x-pack-elasticsearch@e783fbcecc
2018-03-16 14:46:12 -04:00
Alexander Reelsen ff725afe01 Documentation: Fix watcher script condition using params
Original commit: elastic/x-pack-elasticsearch@fe6d819d64
2018-03-16 09:57:29 -07:00
Jay Modi bccf988e9d Run active directory tests against a samba4 fixture (elastic/x-pack-elasticsearch#4067)
This commit adds a Samba4 test fixture that acts as a domain controller
and has the same contents as the cloud active directory instance that
we previously used for tests.

The tests also support reading information from environment variables
so that they can be run against a real active directory instance in our
CI builds.

In addition, this commit also fixes a few issues that surfaced when
making this change. The first is a change in the base DN that is
searched when performing down-level authentication. The base DN is
now the configuration object instead of the domain DN. This change was
required due to the original producing unnecessary referrals, which we
cannot easily follow when running against this test figure. Referrals
cannot easily be followed as they are returned by the ldap server with
an unresolvable DNS name unless the host points to the samba4 instance
for DNS. The port returned in the referral url is the one samba is bound
to, which differs from the port that is forwarded to the host by the
test fixture.

The other issue that is resolved by this change is the addition of
settings that allow specifying non-standard ports for active directory.
This is needed for down-level authentication as we may need to query
the regular port of active directory instead of the global catalog
port as the configuration object is not replicated to the global
catalog.

relates elastic/x-pack-elasticsearch#185
Relates elastic/x-pack-elasticsearch#3800

Original commit: elastic/x-pack-elasticsearch@883c742fba
2018-03-16 10:44:23 -06:00
Lisa Cawley 4c78ede9c1 [DOCS] Update X-Pack installation instructions (elastic/x-pack-elasticsearch#4110)
Original commit: elastic/x-pack-elasticsearch@b3c21b82f3
2018-03-16 09:06:21 -07:00
Chris Earle 485a42dc6a [Monitoring] Stop providing "hkey" addition to license (elastic/x-pack-elasticsearch#4131)
This stops providing the "hkey" (hash) of the license now that the
Monitoring UI no longer uses it (6.3+).

Original commit: elastic/x-pack-elasticsearch@235402fa92
2018-03-16 11:41:24 -04:00
Chris Earle 91401fcb83 [Monitoring] Add "collection_enabled" to usage (elastic/x-pack-elasticsearch#4128)
This adds an indicator to Monitoring's portion of X-Pack usage whether or
not collection is actually enabled. It's no longer enough to have an
exporter defined by default to know if monitoring is actually running.

Original commit: elastic/x-pack-elasticsearch@b2eb881d61
2018-03-16 11:38:18 -04:00
Alexander Reelsen 132ac6ef52 Docs: Clarify outlook email action setup
relates elastic/x-pack-elasticsearch#1815

Original commit: elastic/x-pack-elasticsearch@998fe8fe5e
2018-03-15 15:05:07 -07:00
Alexander Reelsen ab99b82125 Tests: Replace YAML test waiting for watch execution (elastic/x-pack-elasticsearch#4008)
This is the last YAML test, that waits for a watch execution by
specifying some timeout value. This one also gets replaced with a java
test that uses `assertBusy()` and thus is much more likely to succeed.

relates elastic/x-pack-elasticsearch#1513

Original commit: elastic/x-pack-elasticsearch@c2ab8777f4
2018-03-15 14:20:58 -07:00
Alexander Reelsen 7712ab37d1 Documentation: Add username/password parameters for basic auth (elastic/x-pack-elasticsearch#4126)
relates elastic/x-pack-elasticsearch#4070

Original commit: elastic/x-pack-elasticsearch@19d7002d3c
2018-03-15 11:59:15 -07:00
Dimitris Athanasiou c10b2ea631 [ML] Ensure job is not assigned to node that cannot read model_snapshot (elastic/x-pack-elasticsearch#4091)
This adds a minimum compatible version to the model snapshot.
Nodes with a version earlier than that version cannot read
that model snapshot. Thus, such jobs are not assigned to
incompatible nodes.

relates elastic/x-pack-elasticsearch#4077

Original commit: elastic/x-pack-elasticsearch@2ffa6adce0
2018-03-15 17:38:52 +00:00
Alexander Reelsen 92379ca9af Watcher: Ensure usage stats work properly in distributed environment (elastic/x-pack-elasticsearch#4094)
This adds back usage stats by pickybacking on the watcher stats, which
are already running distributed in order to collect and merge watcher
statistics.

In order to be able to track statistics, we need to add information for
each watch in an in-memory data structure that is processed whenever a
usage request is coming in. This processing creates a number of counters
for each node, which then are merged together in the usage stats.

relates elastic/x-pack-elasticsearch#4071

Original commit: elastic/x-pack-elasticsearch@c8bfed288f
2018-03-15 10:28:03 -07:00
David Roberts 783cabbd2f [DOCS] Reflect recent improvements in notes on watch/datafeed privileges (elastic/x-pack-elasticsearch#4116)
Following elastic/x-pack-elasticsearch#3254 security for ML datafeeds has been improved.  The same goes
for watches since elastic/x-pack-elasticsearch#2808.

This change updates a section of the docs that was missed in those changes.
(The majority of the docs changes were made at the appropriate time.)

Original commit: elastic/x-pack-elasticsearch@b3b24ca483
2018-03-15 10:26:56 +00:00
Tim Brooks b121262b2d Fix expiration millis for start_basic (elastic/x-pack-elasticsearch#4124)
This is related to elastic/x-pack-elasticsearch#3877. There was a bug in the PR that introduced
start_basic route. The start basic had an expiration millis that rolled
over into negative numbers. This fixes that issue.

Original commit: elastic/x-pack-elasticsearch@aea9a13d2b
2018-03-14 17:28:36 -06:00
Alexander Reelsen 4109f6e5b7 Watcher: Fix compilation issue
Original commit: elastic/x-pack-elasticsearch@d94d18a12f
2018-03-14 16:20:25 -07:00
Alexander Reelsen c30256e9b5 Watcher: Fix TransformInput toXContent serialization (elastic/x-pack-elasticsearch#4061)
The toXContent serialization of the transform input was broken, which
could lead to the bad case that a watch could be stored with an invalid
toXContent serialization, that could not be read again, when the watch
should either be executed or even just returned by the Get watch API.

relates elastic/x-pack-elasticsearch#4049

Original commit: elastic/x-pack-elasticsearch@f06ad768b8
2018-03-14 15:53:39 -07:00
Lisa Cawley 76e141d390 [DOCS] Split out X-Pack installation substeps (elastic/x-pack-elasticsearch#4125)
Original commit: elastic/x-pack-elasticsearch@df3a4ff045
2018-03-14 14:52:48 -07:00
Boaz Leskes 3d6f4d3c1c remove FullClusterRestartIT.testMonitoring
Per @pickypg : The value of it is to prove that we can continue calling
the same API between versions, which every test we run does already and
does regardless of version.

relates elastic/x-pack-elasticsearch#3068

Original commit: elastic/x-pack-elasticsearch@ceb42d7f47
2018-03-14 21:57:22 +01:00
Nik Everett d1b0067e47 SQL: Ban PrintWriter#println in CLI (elastic/x-pack-elasticsearch#4118)
`PrintWriter#println` doesn't pay attention to the terminal
configuration that we specify so it breaks tests on Windows. If we
instead always use `PrintWriter#print('\n')` then the tests work
properly on windows *and* the CLI actually works properly on Windows.

relates elastic/x-pack-elasticsearch#4109

Original commit: elastic/x-pack-elasticsearch@ac17e691c8
2018-03-14 16:15:32 -04:00
Lee Hinman b14baf4a6f Decouple XContentBuilder from BytesReference (elastic/x-pack-elasticsearch#4119)
* Decouple XContentBuilder from BytesReference

This commit handles the removal of all mentions of BytesReference from
XContentBuilder. This is needed so that we can completely decouple the XContent
code and move it into its own dependency.

This is the x-pack side of https://github.com/elastic/elasticsearch/pull/28972

Original commit: elastic/x-pack-elasticsearch@8ba2e97b26
2018-03-14 13:48:05 -06:00
Tim Brooks a29972c498 Add start_basic action to x-pack client actions
This is related to elastic/x-pack-elasticsearch#3877. This commit adds the start_basic action as a
client action in x-pack.

Original commit: elastic/x-pack-elasticsearch@b89275b92d
2018-03-14 08:24:12 -06:00
Jason Tedor 5c48fc4eaa Adjust main response constructor arguments
This commit adjusts the invocation of the main response constructor as
these were changed upstream to remove a parameter.

Original commit: elastic/x-pack-elasticsearch@846b33c9e9
2018-03-14 07:44:40 -04:00
Tom Veasey 1d525f998c Model memory has increased so we need to update some integration test thresholds (elastic/x-pack-elasticsearch#4033)
Original commit: elastic/x-pack-elasticsearch@f9b77230ce
2018-03-14 10:59:25 +00:00
Tim Brooks 498c110073 Add type parameter to start_trial api (elastic/x-pack-elasticsearch#4102)
This is related to elastic/x-pack-elasticsearch#3877. This commit adds a paramer type to the
start_trial api. This parameter allows the user to pass a type (trial,
gold, or platinum) of license that will be generated. No matter what
type is choosen, you can only generate one per major version.

Original commit: elastic/x-pack-elasticsearch@b42234cbb5
2018-03-13 19:28:11 -06:00
Chris Earle 3c82f24637 [TEST][Monitoring] Remove check for requires_keystore (elastic/x-pack-elasticsearch#4108)
This removes the check for 'requires_keystore' from the plugin output,
which was removed unexpectedly to this test.

Original commit: elastic/x-pack-elasticsearch@80c5c03e20
2018-03-13 13:25:21 -07:00
Jason Tedor fbb752c273 Use Elasticsearch IOUtils (elastic/x-pack-elasticsearch#4105)
This commit replaces the usage of Lucene IOUtils with Elasticsearch
IOUtils, the former of which is now forbidden.

Original commit: elastic/x-pack-elasticsearch@8e0554001f
2018-03-13 12:49:52 -04:00
Nik Everett 5c3b69fe57 SQL: Clean up catch
Use a mutli-catch because it is a little cleaner.

Original commit: elastic/x-pack-elasticsearch@8555e82bca
2018-03-13 10:05:58 -04:00
Nik Everett 47a2f63d5e SQL: Be more careful with break and eof (elastic/x-pack-elasticsearch#4092)
The SQL CLI was being a bit cavalier about `null`, `ctrl-c`, and
`ctrl-d` while reading passwords to the point where it'd halt with
an exception if the user hit `ctrl-d` while typing a password. This
changes it so that the CLI will instead shut down if the user
`ctrl-c`s or `ctrl-d`s while on the password prompt with an
ENOPERM error code.

This also fixes a packaging test failure I caused by a copy and paste
error where the CLI was always enforcing things as though it was reading
a password all the time. This error was causing packaging test failures.

Original commit: elastic/x-pack-elasticsearch@a882c50fc7
2018-03-13 09:00:53 -04:00
Tim Brooks 7f7ac08447 Add api to start basic license (elastic/x-pack-elasticsearch#4083)
This is related to elastic/x-pack-elasticsearch#3877. This commit adds a route /start_basic that
will self generate a basic license. The only validation that is
performed is to check that you do not already have a basic license
installed. Additionally, if you lose features from switching to a basic
license, you must acknowledge the changes.

Original commit: elastic/x-pack-elasticsearch@7b8eeb50b1
2018-03-12 14:39:58 -06:00