Commit Graph

3818 Commits

Author SHA1 Message Date
Alexander Reelsen bb033f1e00 Watcher: Clear out WatchStore on watch index deletion (elastic/elasticsearch#2807)
If someone deletes the watch index (i.e. by deleting all indices), the watcher
in memory store still contains all the watches and tries to execute watches -
which results in exceptions as the watch itself cannot be updated anymore.

In order to minimize this problem (it cant be get rid of completely), we should
act accordingly if the watch index goes missing (either deleted or closed) and
clear out the memory representation of watches in the watchstore as well as trying
to finish all the current executions.

Closes elastic/elasticsearch#2794

Original commit: elastic/x-pack-elasticsearch@12d98cd566
2016-09-07 15:06:03 +02:00
Court Ewing f53fd02da0 Merge pull request elastic/elasticsearch#3305 from w33ble/reporting-fix/whitelisted-headers
Reporting fix/whitelisted headers

Original commit: elastic/x-pack-elasticsearch@2818f32098
2016-09-06 19:08:29 -04:00
Joe Fleming faf98b757c Merge pull request elastic/elasticsearch#3303 from w33ble/reporting-fix/wrap-json-parsing
Reporting fix/wrap json parsing

Original commit: elastic/x-pack-elasticsearch@2d2c3e8dbe
2016-09-06 15:43:33 -07:00
Joe Fleming 45dc0814d5 Merge pull request elastic/elasticsearch#3302 from w33ble/reporting-fix/date-pattern
Fix queue index pattern date

Original commit: elastic/x-pack-elasticsearch@9ab1f502d2
2016-09-06 15:41:58 -07:00
javanna ad5606d1f7 [TEST] don't use null script lang in WatcherUtilsTests
same as elastic/x-pack@9e1e0988c1 but for testSerializeSearchRequest this time.

Original commit: elastic/x-pack-elasticsearch@aa7e3814b6
2016-09-06 23:00:56 +02:00
Nik Everett edb8c12f75 Always use mustache language
Using a null language just defaults to painless which is wrong for
watcher.

Original commit: elastic/x-pack-elasticsearch@48bdc31695
2016-09-06 15:45:23 -04:00
Nik Everett 05222b0d3f Deal with ConcreteShardRequest
It exists now and the intercepters have to deal with it.

Original commit: elastic/x-pack-elasticsearch@fdef0578ed
2016-09-06 15:42:06 -04:00
Martijn van Groningen 36486a7993 Changes for PR: https://github.com/elastic/elasticsearch/pull/20310
Original commit: elastic/x-pack-elasticsearch@9db3d05274
2016-09-06 18:44:41 +02:00
jaymode 7965608add security: make control of logfile audit output consistent with index output
This change moves the logfile audit output from determining what to log based on the
logger level to a enum based configuration that is used by the index output.

A few notable changes were made:

* We alway log all the information we have except for the request body
* The request body is no longer logged by default for REST events; the user needs to
explicitly opt in as there could be sensitive data in the body
* Added a `realm_authentication_failed` event that separates overall authentication
failure from that of an individual realm

Original commit: elastic/x-pack-elasticsearch@343a2bcdd9
2016-09-06 10:50:33 -04:00
Nik Everett 89ce4ebb08 Fix remaining fractional TimeValue issues with watcher
Removes the remaining spots where watcher makes fractional TimeValues.

Closes elastic/elasticsearch#3231

Original commit: elastic/x-pack-elasticsearch@22b0d37ed3
2016-09-06 09:53:22 -04:00
jaymode cd9add5350 test: remove unnecessary verify check
Original commit: elastic/x-pack-elasticsearch@f748c0a565
2016-09-06 08:36:04 -04:00
jaymode 1e6a924e74 security: add support for disabling users
This change adds support for disabling users. Users can be disabled by setting the enabled
property to false and the AuthenticationService will check to make sure that the user is enabled.
If the user is not enabled, this will be audited as an authentication failure.

Also as part of this work, the AnonymousUser was cleaned up to remove having a static instance
that caused issues with tests.

Finally, the poller of users was removed to simplify the code in the NativeUsersStore. In our other
realms we rely on the clear cache APIs and the timeout of the user cache. We should have the
same semantics for the native realm.

Closes elastic/elasticsearch#2172

Original commit: elastic/x-pack-elasticsearch@0820e40183
2016-09-06 07:41:48 -04:00
Chris Earle ec008ec4a9 Looong lines
Original commit: elastic/x-pack-elasticsearch@8c7931c179
2016-09-05 19:05:28 -04:00
Chris Earle 5d5a9afb57 [Monitoring] Use Low Level REST Client for HTTP Exporter
This rewrites the HTTP Exporter to use the REST client underneath. Functionality is improved in resource blocking (templates and pipelines existing) and the majority of the code fundamentall simplified by removing direct HTTP calls.

This is blocked by the SSLService pull request. After that is merged, the I will update this PR to reflect those changes and it could possibly allow us to remove the security privileges required for monitoring.

Original commit: elastic/x-pack-elasticsearch@1ad25f17f8
2016-09-05 15:16:03 -04:00
Alexander Reelsen 41334abda0 Watcher: Reenabling HipChatServiceTests
The authentication token stopped working, deleted it and created a new one
on the hipchat web site. Tests are working now again.

Closes elastic/elasticsearch#3162

Original commit: elastic/x-pack-elasticsearch@a0623af799
2016-09-05 12:27:14 +02:00
Martijn van Groningen 09c7f534b3 added jdocs and rename
Original commit: elastic/x-pack-elasticsearch@8a97b420c1
2016-09-03 11:01:47 +02:00
Joe Fleming b0e4bbb553 Merge pull request elastic/elasticsearch#3304 from w33ble/reporting-fix/sidebar-width
Reporting fix/sidebar width

Original commit: elastic/x-pack-elasticsearch@b02b5bd296
2016-09-02 16:01:52 -07:00
Nik Everett 3c33879018 Mark OldWatchIndices test AwaitsFix
Is freaking out in CI.

Original commit: elastic/x-pack-elasticsearch@3ab43467ed
2016-09-02 18:31:19 -04:00
Martijn van Groningen 565f50dbe5 Remove custom parsing and toXContent logic from TextTemplate and delegate to the Script as much as possible
Original commit: elastic/x-pack-elasticsearch@6d23549dc1
2016-09-02 23:36:07 +02:00
Tim Sullivan 33cdecd39e Merge pull request elastic/elasticsearch#3088 from tsullivan/monitoring-ui-phonehome-pivot-iii
Monitoring UI: Welcome Message and Phone Home Controls from Global DIRECTIVE Notification

Original commit: elastic/x-pack-elasticsearch@165d68b053
2016-09-02 09:43:39 -07:00
Nik Everett 203faaf4f4 2.x backwards compatibility support for watcher
Basic backwards compatibility support for watcher.

Closes elastic/elasticsearch#3230

Relates to elastic/elasticsearch#3231 - this actually should fix all the failures caused
by fractional time values but it does so by being able to parse them.
Being able to parse them is important for 2.x compatibility but 5.0
watches shouldn't produce fractional time values. This fixes the
particular way of making fractional time values mentioned in elastic/elasticsearch#3231
but I expect there are a half dozen more places to fix. The actual
watcher tests are fairly basic.

Original commit: elastic/x-pack-elasticsearch@328717455c
2016-09-02 09:24:46 -04:00
javanna 119bb67967 adapt to elastic/elasticsearch#20288: More info classes to implement Writeable rather than Streamable
Original commit: elastic/x-pack-elasticsearch@22cadf604d
2016-09-02 10:24:15 +02:00
Jack Conradson de4c776f0f Merge pull request elastic/elasticsearch#3272 from jdconrad/deprecate
Deprecate Groovy, Python, and Javascript

Original commit: elastic/x-pack-elasticsearch@e5c1d5a112
2016-09-01 16:30:25 -07:00
Chris Earle bb5ff7ab36 Fix test on Linux; setting name missing 'total.'
Original commit: elastic/x-pack-elasticsearch@de0183b3ef
2016-09-01 18:12:00 -04:00
Jack Conradson 670a57274a Merge branch 'master' into deprecate
Original commit: elastic/x-pack-elasticsearch@c9636bd3f1
2016-09-01 14:53:42 -07:00
Chris Earle 6d40cb63d0 [Monitoring] Collect IO Stats
This ensures that the data exists in 5.0.

Original commit: elastic/x-pack-elasticsearch@bc6e7931d1
2016-09-01 15:45:51 -04:00
Chris Earle b60e8aebd2 [Monitoring] Publish X-Pack Usage with Cluster Info
This publishes X-Pack usage data to the cluster info from the elected master node. This allows phone home to retrieve this data from the index, rather than fetching it live from the connected cluster (thereby not getting it from any n - 1 clusers that are not connceted).

Original commit: elastic/x-pack-elasticsearch@79bfaaaf0b
2016-09-01 15:38:54 -04:00
jaymode 2e010d52e9 security: do not allow built-in user/role names to be defined in the file realm
This change restricts built-in user/role names from passing validation when we are reading or modifying
the files used by this realm.

Closes elastic/elasticsearch#2078

Original commit: elastic/x-pack-elasticsearch@9f6b34f39d
2016-09-01 14:46:38 -04:00
jaymode 74f55bf46e security: add option to specify the certificate validity period
This commit adds the option to specify the number of days that a certificate is valid for.

Closes elastic/elasticsearch#2877

Original commit: elastic/x-pack-elasticsearch@fcf0a28321
2016-09-01 14:21:30 -04:00
jaymode 5d5bd015c0 add a PreBuiltXPackTransportClient
This change adds a transport client that comes preconfigured with the same plugins as the
PreBuiltTransportClient and also adds x-pack.

Closes elastic/elasticsearch#2970

Original commit: elastic/x-pack-elasticsearch@bb60534bd4
2016-09-01 13:49:10 -04:00
Jack Conradson 1d6d20679d Merge branch 'master' into deprecate
Original commit: elastic/x-pack-elasticsearch@afdfa26dc0
2016-09-01 08:52:34 -07:00
Chris Earle 6f115a532d [Monitoring] Remove "agent" package
This removes the "agent" package from org.elasticsearch.xpack.monitoring.agent.*, so that now everything is simply org.elasticsearch.xpack.monitoring.*.

Follow-on work will be refactoring some of the other code, but this is a first step now that it's always the agent (in effect).

Original commit: elastic/x-pack-elasticsearch@14025cb17c
2016-09-01 10:56:27 -04:00
jaymode ad9a7c9b96 Migrate xpack to use the common ssl configuration
This change migrates xpack (security, watcher, and monitoring) to use the common ssl
configuration for the elastic stack. As part of this work, several aspects of how we deal
with SSL has been modified.

From a functionality perspective, an xpack wide configuration for SSL was added and
all of the code that needs SSL uses the SSLService now. The following is a list of all
of the aspects of xpack that can have their own SSL configuration, which are separate
from the xpack wide configuration:

* Transport
* Transport profiles
* HTTP Transport
* Realms
* Monitoring Exporters
* HTTP Client

In terms of the code, some cleanups were made with these changes. SSLConfiguration is
now a concrete class and SSLConfiguration.Custom and SSLConfiguration.Global have been
removed. The validate method on key and trust configurations has been removed and these
classes will now throw exceptions when they are constructed with bad values. The
OptionalSettings helper class has been removed as it was just a file with one line functions
that made the code harder to understand. The SSL configuration and service classes have
been moved from the security source directories to the main xpack source set. The SSLService
now handles more of the configuration of the SSLEngine it returns to prevent callers from
having to handle those aspects. The settings that get registered for SSL have been moved to
XPackSettings.

Also included in this PR is a update to the docs around SSL. This includes a large simplification to
the documentation in that the certificate authority configuration section has been removed and the
process that is documented for generating certificates only includes the CLI tool that we bundle.

Closes elastic/elasticsearch#3104
Closes elastic/elasticsearch#2971
Closes elastic/elasticsearch#3164

Original commit: elastic/x-pack-elasticsearch@5bd9e5ef38
2016-09-01 10:51:41 -04:00
jaymode 54103127d0 fix line length
Original commit: elastic/x-pack-elasticsearch@caca0fc0b7
2016-09-01 09:37:05 -04:00
jaymode c5cde120b7 test: add 2.4.0 bwc index
Original commit: elastic/x-pack-elasticsearch@58bcf3abaf
2016-09-01 09:09:48 -04:00
jaymode 836e1d3a28 security: add a reserved role for the ingest feature in Kibana
Closes elastic/elasticsearch#1667

Original commit: elastic/x-pack-elasticsearch@da2f6d9c4e
2016-09-01 08:09:36 -04:00
Simon Willnauer 3638c8639f [TEST] Don't set create=true if no ID is specified
Original commit: elastic/x-pack-elasticsearch@874def17e8
2016-09-01 10:35:44 +02:00
Jason Tedor d348d4781a Merge pull request elastic/elasticsearch#3266 from jasontedor/log4j2
Introduce Log4j 2

Original commit: elastic/x-pack-elasticsearch@b1cee13ac8
2016-08-31 23:34:45 -04:00
Jason Tedor b29cdc0877 Fix additional exception logging call
This commit modifies an exception logging calls to use a parameterized
message from Log4j.

Original commit: elastic/x-pack-elasticsearch@7b0df7d056
2016-08-31 23:16:07 -04:00
Jason Tedor 7a3d416558 Merge branch 'master' into log4j2
* master:
  only lint .js and .jsx files
  Designating list and count APIs as system APIs
  Inverting logic
  Use system API module function exported by Kibana plugin
  monitoring ui/cluster row: IsClusterSupported helper
  monitoring ui: Initial Test Automation Hooks
  monitoring ui/license: toaster content update
  monitoring ui: onClick handler syntax polish
  monitoring ui/license: updating wording on unsupported cluster toaster
  monitoring ui:fix default min shard replication to `N/A`
  monitoring ui: empty state cleanups
  monitoring ui/license: fix redirect issue with license expiry page + back button
  monitoring ui:cluster listing treatment for clusters w/ invalid and unsupported license
  monitoring ui: remove “health” check for cluster listing
  monitoring ui: show clusters that have had license deleted
  Designate certain API calls as system APIs + treat them as special

Original commit: elastic/x-pack-elasticsearch@95865f89ac
2016-08-31 22:35:02 -04:00
Jack Conradson 9f2dc128e8 Merge branch 'master' into deprecate
Original commit: elastic/x-pack-elasticsearch@1eb5bc7fdd
2016-08-31 19:03:13 -07:00
Jack Conradson 9f2e9bc01b Merge branch 'master' into deprecate
Original commit: elastic/x-pack-elasticsearch@1157ea6367
2016-08-31 17:30:03 -07:00
Jack Conradson 0ca243dc83 Merge branch 'master' into deprecate
Original commit: elastic/x-pack-elasticsearch@25985e9144
2016-08-31 15:57:44 -07:00
Jack Conradson 31c9d258c7 Made deprecation of Groovy, Javascript, and Python more explicit.
Original commit: elastic/x-pack-elasticsearch@04d725e9c1
2016-08-31 15:57:09 -07:00
Jason Tedor cd0f17c372 Merge branch 'master' into log4j2
* master:
  Changes tests to conform with new cluster health API, calling setWaitForNoRelocatingShards(true) instead of setWaitForRelocatingShards(0)

Original commit: elastic/x-pack-elasticsearch@bde6ad8c8a
2016-08-31 16:17:28 -04:00
Ali Beyad 8757c2f6e1 Changes tests to conform with new cluster health API, calling
setWaitForNoRelocatingShards(true) instead of setWaitForRelocatingShards(0)

Original commit: elastic/x-pack-elasticsearch@c7c12fe64c
2016-08-31 11:56:38 -04:00
Jason Tedor a89f0110f4 Merge branch 'master' into log4j2
* master:
  Use releasable locks in NativeRolesStore
  security: limit the size of the role store cache
  security: remove explicit handshake wait in netty4 transport
  test: smoke-test-plugins-ssl no longer relies on logging to start
  kibana monitoring/uuid config key reference update
  Docs: Updated release date for 2.4 in RNs.
  Update README.md
  Build: Add apijar task to assemble so it gets built with other artifacts
  monitoring ui/license: cluster listing status cell treatment for basic/unsupported cluster
  monitoring ui:fix cluster overview when cluster has no indices/shards
  monitoring ui/license: logic cleanup per feedback
  monitoring ui/license: primary cluster asterisk styling
  monitoring ui/license: allow clicking into primary cluster if all are basic
  monitoring ui: add isPrimary property to cluster listing response
  Security: throw exception if we cannot extract indices from an indices request
  Security: add tests for delete and update by query

Original commit: elastic/x-pack-elasticsearch@3cb41739ee
2016-08-31 10:26:30 -04:00
Jason Tedor 82079185c2 Use releasable locks in NativeRolesStore
This commit replaces the use of try/finally blocks to handle safe
locking/unlocking for role cache read/write lock with releasable locks
in try-with-resources blocks.

Relates elastic/elasticsearch#3278

Original commit: elastic/x-pack-elasticsearch@fbd659cd85
2016-08-31 10:18:09 -04:00
jaymode 7d78911082 security: limit the size of the role store cache
Previously the roles store cache was unbounded as it was a just using a ConcurrentHashMap,
which could lead to excessive memory usage in cases where there are a large number of roles
as we tried to eagerly load the roles into the cache if they were not present. The roles store now
loads roles on demand and caches them for a finite period of time.

Additionally, the background polling of roles has been removed to reduce complexity. A best effort
attempt is made to clear the roles cache upon modification and if necessary the cache can be
cleared manually.

See elastic/elasticsearch#1837

Original commit: elastic/x-pack-elasticsearch@450dd779c8
2016-08-31 09:22:25 -04:00
jaymode 06ff97f63d security: remove explicit handshake wait in netty4 transport
Netty 4's SslHandler does not require the application to wait for the handshake to
be completed before data is written. This change removes the explicit wait on each
handshake future.

Original commit: elastic/x-pack-elasticsearch@c19bcebb83
2016-08-31 08:23:51 -04:00