Commit Graph

1608 Commits

Author SHA1 Message Date
Alexander Reelsen d4cddc12d0 Tests: Add @AwaitsFix annotation to flaky tests
These contribute to unstable builds, and should be reenabled on a
case by case base and worked on.

Original commit: elastic/x-pack-elasticsearch@a15338415a
2018-01-15 10:28:05 +01:00
Tim Sullivan c9aa4e3d59 [Monitoring] Add additional metric fields to Beats template (elastic/x-pack-elasticsearch#3337)
* [Monitoring] Add uptime.ms field to Beats template

* Add cpu mapping [WIP]

* sort fields

* hierarchical - no dots in fields

* add rss, more event metrics, system cpu and os load

* fix some mapping errors

Original commit: elastic/x-pack-elasticsearch@59cbcb4ee5
2018-01-12 14:26:04 -07:00
Igor Motov aba3f15d81 Add adding ability to associate an ID with tasks (elastic/x-pack-elasticsearch#3500)
X-Pack portion of elastic/elasticsearch#23250

Original commit: elastic/x-pack-elasticsearch@3c9a5d2d08
2018-01-12 15:34:28 -05:00
Jay Modi cec90f452a IndexLifecycleManager checks for index existense and up to date mappings (elastic/x-pack-elasticsearch#3515)
This change makes the IndexLifecycleManager check for both index existence and up to date mappings
on the index prior to executing the provided runnable. Doing this provides a mechanism to make
non-breaking mapping updates to the security index in minor versions.

relates elastic/x-pack-elasticsearch#3462

Original commit: elastic/x-pack-elasticsearch@80f05d83b4
2018-01-12 12:45:52 -07:00
Dimitris Athanasiou 73f8559fca [ML] Update process when filters or calendars are updated (elastic/x-pack-elasticsearch#3385)
Relates elastic/x-pack-elasticsearch#3325

Original commit: elastic/x-pack-elasticsearch@9da4973cda
2018-01-12 17:48:07 +00:00
David Kyle 86e9f63b19 [ML] Expect an array of events in request (elastic/x-pack-elasticsearch#3537)
* Expect an array of events in request

Original commit: elastic/x-pack-elasticsearch@f60bc0b544
2018-01-11 22:16:09 +00:00
jaymode 79a9576596 Test: do not wipe templates from clusters in audit tests
This commit adds both the security index template and the audit index template to the excluded
templates for the RemoteIndexAuditTrailStartingTests and IndexAuditTrailTests. By allowing the
templates to remain in place, we make it easier for subsequent tests in the suite to have their
audit trails started.

Relates elastic/x-pack-elasticsearch#435

Original commit: elastic/x-pack-elasticsearch@6e54de19f9
2018-01-11 14:32:23 -07:00
jaymode 858178602e Test: update TokenAuthIntegTests to modify document to test expiration
This change modifies how we test the deletion of expired documents to be in line with elastic/x-pack-elasticsearch#3468 and
also adds debugging output in case the failures in CI continue.

Relates elastic/x-pack-elasticsearch#2253

Original commit: elastic/x-pack-elasticsearch@979b5357f5
2018-01-11 13:25:07 -07:00
Tim Brooks 0a8e4da3bd Support changing name of core module to server (elastic/x-pack-elasticsearch#3532)
This is related to elastic/elasticsearch#28178. It modifies x-pack to be
compatible with the elasticsearch core module to server name change.

Original commit: elastic/x-pack-elasticsearch@c8d313f882
2018-01-11 11:30:45 -07:00
David Roberts c647d7f38b [ML] Adjust BWC for categorization analyzer config after backporting elastic/x-pack-elasticsearch#3372
Original commit: elastic/x-pack-elasticsearch@27fcd5bdcb
2018-01-11 17:01:37 +00:00
David Kyle 999c1dab48 [ML] Remove extra quotes in field config
Original commit: elastic/x-pack-elasticsearch@d9a7ca5ebf
2018-01-11 15:57:17 +00:00
David Roberts a386b5727e [ML] Increase tokenization flexibility for categorization (elastic/x-pack-elasticsearch#3372)
By moving tokenization for categorization to Java we give users access to considerably more options for tokenizing their log messages prior to using ML to categorize them. Now all Elasticsearch analyzer functionality is available, which opens up the possibility to sensibly categorize non-English log messages.

Relates elastic/machine-learning-cpp#491

Original commit: elastic/x-pack-elasticsearch@5d61b67614
2018-01-11 15:40:13 +00:00
Alexander Reelsen 992a7af126 Watcher: Add support for actions in slack attachments (elastic/x-pack-elasticsearch#3355)
In order to support buttons that can be clicked on within a slack
message, this commits adds support for so called actions within
attachments. This allows to create buttons, that are clicked and execute
a GET request, so actions must be idempotent according to the official
slack documentation.

Official slack documentation is available at
https://api.slack.com/docs/message-attachments#action_fields

Original commit: elastic/x-pack-elasticsearch@29ddc90b01
2018-01-11 11:43:24 +01:00
Albert Zaharovits 3fc17ab918 Audit Event ignore policy (elastic/x-pack-elasticsearch#3005)
This commit introduces audit event filtering policies for the logging audit.
Until now the only way to temper the gush of audit events was to specifically
pick some event types that were reported.
This superposes a way (named policies) to filter events using lucene regexp filters on 
the following event fields: users, realms, roles and indices. The policies are ignore
policies, ie when an event matches it is ignored (not reported).

Original commit: elastic/x-pack-elasticsearch@233f685121
2018-01-11 11:07:46 +02:00
Chris Earle e74f90eba0 [Monitoring] Add Cluster Alert for Nodes Changed (elastic/x-pack-elasticsearch#3504)
This creates a cluster alert that triggers whenever a node is:

- Added
- Removed
- Restarted (aka Removed and Added between collections)

Unlike all previous cluster alerts, this cluster is always instantly resolved because there is no way to otherwise resolve it later (especially if nodes are replaced).

This will require a small change to the UI so that it does not simply ignore resolved alerts.

Original commit: elastic/x-pack-elasticsearch@6340bf7960
2018-01-10 21:29:00 -05:00
David Kyle 4c973c16c2 [ML] Rename event filter params (elastic/x-pack-elasticsearch#3525)
Original commit: elastic/x-pack-elasticsearch@3089193d45
2018-01-10 14:40:32 +00:00
David Kyle d3d965c8ee [ML] Configure autodetect with scheduled events (elastic/x-pack-elasticsearch#3521)
Original commit: elastic/x-pack-elasticsearch@e51b70ec69
2018-01-10 12:02:14 +00:00
David Kyle 66b187fe17 [ML] Account for job groups membership when getting events (elastic/x-pack-elasticsearch#3473)
* Account for job groups membership when getting events

Original commit: elastic/x-pack-elasticsearch@a4185cc460
2018-01-10 11:14:09 +00:00
Dimitris Athanasiou aa25704170 [ML] Add scheduled_events field to bucket (elastic/x-pack-elasticsearch#3513)
Adds `scheduled_events` to the bucket object so
it can read the corresponding field written from the
autodetect process.

Relates elastic/x-pack-elasticsearch#3016

Original commit: elastic/x-pack-elasticsearch@3f35f867f0
2018-01-10 10:57:09 +00:00
Jay Modi cbf1427ff4 Allow license installation with TLS disabled and single-node discovery (elastic/x-pack-elasticsearch#3427)
This change allows production licenses to be installed when TLS is not enabled and the discovery
type is set to single-node.

Relates elastic/x-pack-elasticsearch#3245

Original commit: elastic/x-pack-elasticsearch@73815a3976
2018-01-09 12:32:37 -07:00
Jason Tedor 374ab447f7 Push build.snapshot to BWC builds too
We need to push this flag down to the command line for BWC builds or the
artifacts in release tests will have the wrong version (being considered
snapshots instead of non-snapshots).

Original commit: elastic/x-pack-elasticsearch@279fd02aa7
2018-01-09 12:48:48 -05:00
David Kyle f73a7803ce [ML] Delete calendar events endpoint (elastic/x-pack-elasticsearch#3388)
* Delete calendar events endpoint

Original commit: elastic/x-pack-elasticsearch@70aebfae2c
2018-01-09 11:55:36 +00:00
Yannick Welsch bd3d652901 Fix Gradle wrapper usage on Windows when building BWC (elastic/x-pack-elasticsearch#3509)
Relates elastic/x-pack-elasticsearch#3502

Original commit: elastic/x-pack-elasticsearch@de69017a4f
2018-01-09 11:58:41 +01:00
Jason Tedor e92947e1b9 Use Gradle wrapper when building BWC
This commit modifies the BWC build to invoke the Gradle wrapper. The
motivation for this is two-fold:
 - BWC versions might be dependent on a different version of Gradle than
   the current version of Gradle
 - in a follow-up we are going to need to be able to set JAVA_HOME to a
   different value than the current value of JAVA_HOME

Relates elastic/x-pack-elasticsearch#3502

Original commit: elastic/x-pack-elasticsearch@22062f635b
2018-01-08 21:47:41 -05:00
Alexander Reelsen 1d3c622746 Tests: Fix mockwebserver to parse incoming http bodies as UTF8
By not specifying a charset this used the system charset and
led to breaking tests under windows.

relates elastic/x-pack-elasticsearch#3498

Original commit: elastic/x-pack-elasticsearch@17d415ebbd
2018-01-08 18:29:22 +01:00
Alexander Reelsen f80f1dd268 Tests: Silence test under windows as it fails
Relates elastic/x-pack-elasticsearch#3498

Original commit: elastic/x-pack-elasticsearch@2dda8851e4
2018-01-08 14:37:33 +01:00
Alexander Reelsen 6f2fddc5f6 Watcher: Fix encoding of UTF8 data in HttpClient (elastic/x-pack-elasticsearch#3398)
The HttpClient uses an Apache HTTP client class named StringEntity to
encode a HTTP request body. This one however assumes ISO-8859-1 as its
charset when encoding the string based body to bytes.

This commit switches to a byte array based body, then sets the content
type header and falls back to the old text/plain content type if no
content type header is specified.

relates elastic/x-pack-elasticsearch#3397

Original commit: elastic/x-pack-elasticsearch@d5a6e7f0c7
2018-01-08 09:44:07 +01:00
David Kyle 2eb3f02e40 [ML] Rename SpecialEvents -> ScheduledEvents (elastic/x-pack-elasticsearch#3485)
* Rename SpecialEvents -> ScheduledEvents


Original commit: elastic/x-pack-elasticsearch@4bfc52c435
2018-01-05 16:35:42 +00:00
David Roberts f508e14e06 [TEST] Unmute ML categorization tests
The C++ changes these rely on are now in the snapshots on s3, so these
tests will work.

Relates elastic/machine-learning-cpp#512

Original commit: elastic/x-pack-elasticsearch@cb8f7b31e8
2018-01-05 16:02:35 +00:00
David Roberts d7416d53e0 [ML] Fix categorization integration test following fix to C++ code (elastic/x-pack-elasticsearch#3483)
The categorization integration tests were picking up a bug in the C++
categorization code, but the expected results were set incorrectly to
match the incorrect C++ output.  This change adjusts the expected
results to reflect what is really expected.  It also mutes the tests
to avoid failures while the C++ fix propagates through the build
process.  The tests will be unmuted tomorrow.

Relates elastic/machine-learning-cpp#512

Original commit: elastic/x-pack-elasticsearch@d2b57b6216
2018-01-05 11:48:01 +00:00
David Kyle 46c1272ed3 [ML] Fix broken backwards compatibility of field names (elastic/x-pack-elasticsearch#3463)
Original commit: elastic/x-pack-elasticsearch@613943dfe7
2018-01-04 10:05:16 +00:00
Jason Tedor 3b06254573 Set processors on audit remote client
When creating a transport client for a remote index audit trail, we are
implicitly allowing the construction of this transport client to
initialize the number of processors that Netty thinks are on the
system. Since we never pushed down the number of processors, this will
always default to the number of cores on the machine. If the user has
also set the processors setting, when the server bootstraps it will try
to push the number of processors down to Netty too. If this value does
not match the number of cores, we will fail in bootstrap because we
guard against initializing the number of processors that Netty sees to
different values. Instead, the transport client should inherit the
number of processors too and push this down when it pushes the number of
processors down to Netty. We have to worry about another possibility: an
explicit setting for the number of processors for the transport client
so we require this matches the inherited value.

Relates elastic/x-pack-elasticsearch#3469


Original commit: elastic/x-pack-elasticsearch@032810bb0b
2018-01-04 00:06:16 -05:00
Ryan Ernst b05ee6fcc9 fix silly serialization test
Original commit: elastic/x-pack-elasticsearch@fc934534d7
2018-01-03 13:51:23 -08:00
Ryan Ernst 2e2cce5db5 Fix test usage of PluginInfo in xpack
Original commit: elastic/x-pack-elasticsearch@2c81ed40ff
2018-01-03 12:56:14 -08:00
Tim Vernum 2ad6a3a538 Cleanup the handling for bootstrap passwords (elastic/x-pack-elasticsearch#3470)
Minor refactoring on the reserved realm:
- Removed some duplicated code
- Added in some additional assertions
- Extended some testing
- Removed use of the obsolete "allow_default_passwords" from the test.

Original commit: elastic/x-pack-elasticsearch@584171d2bd
2018-01-03 20:08:33 +10:00
Tim Vernum e8985f4455 [Security] has_privileges.has_all_requested should respect cluster privileges (elastic/x-pack-elasticsearch#3379)
The has_all_requested field in the has_privileges response was not taking the cluster privileges into account

relates elastic/x-pack-elasticsearch#3366

Original commit: elastic/x-pack-elasticsearch@68d2b98670
2017-12-28 08:38:41 +10:00
Albert Zaharovits 3ecc433f43 AD authn: never clear passwords on Bind connections (elastic/x-pack-elasticsearch#3351)
It is unsafe to clear passwords of bind requests if the connection is live
and might be used latter (for eg for group searches). This is a temporary
fix that exposes passwords in memory.

Original commit: elastic/x-pack-elasticsearch@e2e1f1a358
2017-12-27 19:17:07 +02:00
Michael Basnight 220aa734ee Migrate actions to plugin core (elastic/x-pack-elasticsearch#3424)
This commit hacks up the xpack repo in such a way that it moves all
actions that the transport client uses to plugin core. It also moves
any classes that those actions use to plugin core, with a few
exceptions. I tried to split up any classes that pulled in server side
logic into the client, but that was not always 100% possible. Consider
this commit a guide, and since I do not know the codebase for each
plugin, consider it a best guess for what should be moved. A few other
things were merged in below.

This commit extracts the parts of Condition and AlwaysCondition that are
needed by classes that will be moved into the client package. The only
odd thing here is that since there are two parent classes (Condition and
AlwaysCondition) it was not possible to make AlwaysConditionInteral
extend from AlwaysCondition and still parse a ConditionInternal object,
which is what the ConditionFactory expects.

This commit removes the use of internal users in the User class, and
instead moves them to Authentication, where they are used.

[insert obligatory chuck norris karate image here]

ref elastic/x-pack-elasticsearch#2925

Original commit: elastic/x-pack-elasticsearch@42d0b72209
2017-12-27 11:31:16 -05:00
Hendrik Muhs f8d62d0ade [ML] wait for validation of forecast request and send error in case (elastic/x-pack-elasticsearch#3399)
change the forecast API to wait for the validation and return an error if validation fails.

fixes elastic/machine-learning-cpp#489

Original commit: elastic/x-pack-elasticsearch@5c0553b284
2017-12-27 11:25:26 +01:00
Chris Earle 01e3db3740 [Monitoring] Add Cluster Alert Blacklist (elastic/x-pack-elasticsearch#3326)
This adds a traditional blacklist that can be set [dynamically] at the exporter level to block specific cluster alerts (watches) from being created. It goes further and actually removes any watch that has been created already for the current cluster.

Original commit: elastic/x-pack-elasticsearch@1f67bb9501
2017-12-21 11:04:56 -05:00
Chris Earle 76cfdfcef7 [Logstash] Index Management should use auto_expand_replicas (elastic/x-pack-elasticsearch#3373)
This modifies the default template to use auto_expand_replicas so that a one node cluster can stay green, even while using the Logstash management feature.

Original commit: elastic/x-pack-elasticsearch@b1ddf3af6f
2017-12-21 10:58:12 -05:00
David Kyle b98706183e Fix Java 9 & 10 test failures comparing ZonedDateTimes
Original commit: elastic/x-pack-elasticsearch@2767bb3f16
2017-12-21 15:48:05 +00:00
David Kyle 0950a903f2 [ML] Get events for job and from all calendars (elastic/x-pack-elasticsearch#3401)
* Add get events for job and from all calendars

Original commit: elastic/x-pack-elasticsearch@8029ffb898
2017-12-21 14:05:54 +00:00
David Kyle 3f6478e798 Fix timing issue in test
Original commit: elastic/x-pack-elasticsearch@7fe221ef15
2017-12-21 09:19:14 +00:00
Alexander Reelsen 8c99a4dc78 Watcher: Add refresh parameter to index action (elastic/x-pack-elasticsearch#3350)
This allows to configure the refresh parameter to the index action,
which is either applied to the resulting index or bulk request.

Relates elastic/apm-dev#76

Original commit: elastic/x-pack-elasticsearch@aedb6adc66
2017-12-21 10:18:16 +01:00
Hendrik Muhs bf74c77fef [ML] allow forecast only on post 6.1 jobs (elastic/x-pack-elasticsearch#3362)
allow forecast only on post 6.1 jobs

discuss issue: elastic/machine-learning-cpp#494

relates elastic/x-pack-elasticsearch#3219

Original commit: elastic/x-pack-elasticsearch@c6884bc40f
2017-12-21 07:47:32 +01:00
Chris Earle 2507a07ec9 [Watcher] Use auto_expand_replicas on triggered_watches index too (elastic/x-pack-elasticsearch#3371)
In the previous pass, only the ".watch*" templates were impacted. This also should use `auto_expand_replicas`.

Original commit: elastic/x-pack-elasticsearch@5d3a4f1f7b
2017-12-20 17:25:04 -05:00
Chris Earle 362ba75e6b [ML] Use auto_expand_replicas (elastic/x-pack-elasticsearch#3375)
This creates all ML templates using the `index.auto_expand_replicas` set as `"0-1"` so that a single node cluster (e.g., dev) can startup as green.

Original commit: elastic/x-pack-elasticsearch@7a243c5b06
2017-12-20 17:23:36 -05:00
David Kyle e53ac4484c [ML] Calendar event actions (elastic/x-pack-elasticsearch#3365)
* Calendar event actions

* Add page params and date range tests

* Address review comments

* Support POSTing params in the body of a request


Original commit: elastic/x-pack-elasticsearch@22a7e17a8f
2017-12-20 17:39:44 +00:00
Boaz Leskes 5e51422f4d PkiAuthenticationTests & SslIntegrationTests should instead use NetworkAddress's format magic
Original commit: elastic/x-pack-elasticsearch@601e8e774d
2017-12-20 16:05:41 +01:00