253 Commits

Author SHA1 Message Date
Alexander Reelsen
1aedda3627 X-Pack: Create notification module
In order to move things from watcher to x-pack this created a notification module in x-pack.
This also means that the HTTPClient was moved up and settings have changed from
`xpack.watcher.http` to just `xpack.http`.

Further things done:

* Move http under o.e.xpack.common
* Moved secret service to o.e.xpack.common, initializing in XpackPlugin
* Moved text template to o.e.xpack.common.text
* Moved http client initialization into xpack plugin
* Renamed xpack.watcher.encrypt_sensitive_data setting, moved into Watch class
* Moved script service proxy to common

Original commit: elastic/x-pack-elasticsearch@41eb6e6946
2016-05-04 08:53:29 +02:00
Daniel Mitterdorfer
7eebacc884 Disable HTTP compression by default when HTTPS is enabled.
With elastic/elasticsearchelastic/elasticsearch#7309 we enable HTTP compression by
default. However, this can pose a security risk for HTTPS
traffic (e.g. BREACH attack). Hence, we disable HTTP compression
by default again if HTTPS enabled (note that this still allows the
user to explicitly enable HTTP compression if they want to).

Relates elastic/elaticsearchelastic/elasticsearch#7309

Original commit: elastic/x-pack-elasticsearch@8da100c9a5
2016-05-03 08:54:57 +02:00
Alexander Reelsen
74edbe6332 Watcher: Refactoring, move to org.elasticsearch.xpack
This refactors the org.elasticsearch.watcher over to
org.elasticsearch.xpack.watcher

This also adds all watcher actions to the KnownActionsTests,
as watcher actions had not been taken care of until here.

Original commit: elastic/x-pack-elasticsearch@a046dc7c6a
2016-05-02 10:58:34 +02:00
jaymode
773876caee security: ssl by default on the transport layer
This commit adds the necessary changes to make SSL work on the transport layer by default. A large
portion of the SSL configuration/settings was re-worked with this change. Some notable highlights
include support for PEM cert/keys, reloadable SSL configuration, separate HTTP ssl configuration, and
separate LDAP configuration.

The following is a list of specific items addressed:

* `SSLSettings` renamed to `SSLConfiguration`
* `KeyConfig` and `TrustConfig` abstractions created. These hide the details of how `KeyManager[]` and `TrustManager[]` are loaded. These are also responsible for settings validation (ie keystore password is not null)
* Configuration fallback is changed. Previously any setting would fallback to the "global" value (`xpack.security.ssl.*`). Now a keystore path, key path, ca paths, or truststore path must be specified otherwise the configuration for that key/trust will fallback to the global configuration. In other words if you want to change part of a keystore or truststore in a profile you need to supply all the information. This could be considered breaking if a user relied on the old fallback
* JDK trusted certificates (`cacerts`) are trusted by default (breaking change). This can be disabled via a setting.
* We now monitor the SSL files for changes and enable dynamic reloading of the configuration. This will make it easier for users when they are getting set up with certificates so they do not need to restart every time. This can be disabled via a setting
* LDAP realms can now have their own SSL configurations
* HTTP can now have its own SSL configuration
* SSL is enabled by default on the transport layer only. Hostname verification is enabled as well. On startup if no global SSL settings are present and SSL is configured to be used, we auto generate one based on the default CA that is shipped. This process includes a best effort attempt to generate the subject alternative names.
* `xpack.security.ssl.hostname_verification` is deprecated in favor of `xpack.security.ssl.hostname_verification.enabled`
* added Bouncy Castle info to NOTICE
* consolidated NOTICE and LICENSE files

Closes elastic/elasticsearch#14
Closes elastic/elasticsearch#34
Closes elastic/elasticsearch#1483
Closes elastic/elasticsearch#1933
Addresses security portion of elastic/elasticsearch#673

Original commit: elastic/x-pack-elasticsearch@7c359db90b
2016-04-29 12:50:07 -04:00
jaymode
de48b2426b change how audit user is compared, do not setDaemon, test cleanup
This commit makes a few modifications to the IndexAuditTrail class:

* Use `InternalAuditUser#is` to determine if the principal is the auditor when we have a user
and simply compare `InternalAuditUser#NAME` when only a string principal is available
* Remove the `Thread#setDaemon` call in the QueueConsumer as this thread should be terminated
as part of the shutdown of the node

In terms of tests, there are some issues and changes to how we test certain aspects. The muted tests
were not accurate since the tests immediately checked for the existence of an index and did not poll or
wait and this operation is asynchronous so the index could be created after the exists request was
executed. These tests were removed and a new class was added to test the muted behavior. In these
tests we override the audit trails implementation of a queue, which will set a flag to indicate a message
has been added to the queue. This is a synchronous operation so it can be checked immediately.

The other tests in the IndexAuditTrail tests remain but a few changes have been made to the execution.

* ensureYellow is called for the index we expect to be created before searching for documents
* the remote cluster is only setup at the beginning of the suite rather than before every test to ensure
quicker execution
* the maximum number of shards has been reduced to three since we do not really need up to 10 shards
for a single document

Original commit: elastic/x-pack-elasticsearch@501b6ce9da
2016-04-29 09:08:10 -04:00
jaymode
c39b3ba2fc security: add the proper behavior for the standard license
This change adds the proper behavior for the standard license which is:

* authentication is enabled but only the reserved, native and file realms are available
* authorization is enabled

Features that are disabled:

* auditing
* ip filtering
* custom realms
* LDAP, Active Directory, PKI realms

See elastic/elasticsearch#1263

Original commit: elastic/x-pack-elasticsearch@920c045bf1
2016-04-28 09:33:57 -04:00
jaymode
91943318bf security: cleanup authentication service
This commit removes duplicated code in the authentication service by combining
the authentication logic for rest and transport requests. As part of this we no longer
cache the authentication token since we put the user in the context and serialize the
user.

Additionally we now pass the thread context to the AuthenticationFailureHandler to
restore access to the headers and context.

Original commit: elastic/x-pack-elasticsearch@79e2375a13
2016-04-28 07:59:16 -04:00
jaymode
4f7dad8da2 security: handle null values for full name and email
This commit adds logic so that we properly handle null tokens for full name and
email.

Closes elastic/elasticsearch#1887

Original commit: elastic/x-pack-elasticsearch@e03188c29f
2016-04-28 07:41:27 -04:00
jaymode
f4f156b351 test: add awaits fix to FLS field stats tests
See elastic/elasticsearch#2120

Original commit: elastic/x-pack-elasticsearch@fc7950bf65
2016-04-27 13:55:59 -04:00
Alexander Reelsen
3bbe5916d1 Fix compilation issue
Original commit: elastic/x-pack-elasticsearch@803275d634
2016-04-26 14:03:19 +02:00
jaymode
c7ad6b9872 test: add a simple test for reserved realm authentication
See elastic/elasticsearch#2089

Original commit: elastic/x-pack-elasticsearch@1bede0a206
2016-04-25 07:34:14 -04:00
Martijn van Groningen
b9515357fa Migrated from indexed scripts to store scripts
Original commit: elastic/x-pack-elasticsearch@a0218f1c9e
2016-04-22 13:43:55 +02:00
uboness
5c9d96211f Extended X-Pack Info API with Features Info
- introduced the "Feature Set" notion - graph, security, monitoring, watcher, these are all feature sets
- each feature set can be:
 - `available` - indicates whether this feature set is available under the current license
 - `enabled` - indicates whether this feature set is enabled (note that the feature set can be enabled, yet unavailable under the current license)
- while at it, cleaned up the main modules of watcher, security, monitoring and graph.

Original commit: elastic/x-pack-elasticsearch@5b3e19fe8c
2016-04-20 14:30:48 -07:00
Jay Greenberg
8af3f91eb5 Merge pull request elastic/elasticsearch#2044 from PhaedrusTheGreek/group_search_noattrs
Change some LDAP searches to NOATTRS to avoid unnecessary results

Original commit: elastic/x-pack-elasticsearch@60c41af5a6
2016-04-20 09:29:38 -04:00
jaymode
659439841e test: adapt to removal of setting
Original commit: elastic/x-pack-elasticsearch@5f195001b9
2016-04-19 14:31:06 -04:00
PhaedrusTheGreek
962729bd3b Changed LDAP searches to NOATTRS in order to avoid returning unecessary
data in searches where only getDn() is done on results

Original commit: elastic/x-pack-elasticsearch@5ce64235a1
2016-04-19 10:47:27 -04:00
Martijn van Groningen
e24d09b54e test: allow percolate api to fail when the percolator field can't be found
Original commit: elastic/x-pack-elasticsearch@3343c9dc3a
2016-04-19 14:11:53 +02:00
Martijn van Groningen
0c7dff4fa7 security: Deal with upstream percolator changes.
From now on, if field level security and percolator is used then the percolator field needs to be included in the allowed fields.

Original commit: elastic/x-pack-elasticsearch@7d39b5caf6
2016-04-19 11:23:04 +02:00
Ryan Ernst
7275d48bbd Remove XContentBuilderString
This is the xplugins side of elastic/elasticsearchelastic/elasticsearch#17833

Original commit: elastic/x-pack-elasticsearch@2400192775
2016-04-18 14:37:17 -07:00
jaymode
e66a6871c0 security: fix initialization of server sets in ldap session factories
The SessionFactory construction was calling the `ldapServers` method in the constructor,
which was fine for all of the session factories except for the ActiveDirectorySessionFactory.
The ActiveDirectorySessionFactory overrides the ldapServers method and use class variables
that are initialized in its constructor so the value was always null.

This change moves setup to an init method for objects that depend on variables set during
construction.

Closes elastic/elasticsearch#2011

Original commit: elastic/x-pack-elasticsearch@07c15ce171
2016-04-18 07:22:21 -04:00
uboness
8aa48ffaff Introduced the X-Pack Info API
- Removed Shield's Info API
- Removed Watcher's Info API

Closes elastic/elasticsearch#2014

Original commit: elastic/x-pack-elasticsearch@6910cb1d6e
2016-04-17 13:38:19 +02:00
Robert Muir
350ccaad43 Merge pull request elastic/elasticsearch#2025 from elastic/fieldsecurity-points
field-level security should filter points

Original commit: elastic/x-pack-elasticsearch@5422fe610d
2016-04-15 11:19:24 -04:00
jaymode
98a308352a security: resolve date match expressions for authorization
Elasticsearch supports the concept of date match expressions for index names and
the authorization service was trying to authorize the names without resolving them
to their concrete index names. This change now resolves these names

Closes elastic/elasticsearch#1983

Original commit: elastic/x-pack-elasticsearch@3c6baa8e83
2016-04-15 08:49:20 -04:00
Robert Muir
8bcc280539 field-level security should filter points
Original commit: elastic/x-pack-elasticsearch@5a8739a2bd
2016-04-14 18:28:49 -04:00
Colin Goodheart-Smithe
2dc8a720c2 Fix Eclipse Compile error in ReservedRealmTests
The eclipse compiler errors on this class because "the method containsInAnyOrder(T...) of type Matchers is not applicable as the formal varargs element type T is not accessible here". This is because the first common superclass of `XPackUser` and `KibanaUser` is `ReservedUser` which is package protected and not available to this test class. This change casts to `User` so the error does not occur in Eclipse.

Original commit: elastic/x-pack-elasticsearch@be8fa82720
2016-04-14 14:30:06 +01:00
Martijn van Groningen
5f7220dea4 Fix compile errors due to upstream changes in HasChild- and HasParentQueryBuilder
Original commit: elastic/x-pack-elasticsearch@9945e89b6e
2016-04-14 14:46:08 +02:00
Alexander Reelsen
847287278b Tests: Adapting to Version changes in core
Original commit: elastic/x-pack-elasticsearch@89e9cf427d
2016-04-13 11:43:54 +02:00
Daniel Mitterdorfer
3fd3adef4c Limit request size on HTTP level
With this commit we limit the size of all in-flight requests on
HTTP level. The size is guarded by the same circuit breaker that
is also used on transport level. Similarly, the size that is used
is HTTP content length.

Relates elastic/elasticsearchelastic/elasticsearch#16011

Original commit: elastic/x-pack-elasticsearch@318b7a4a8a
2016-04-13 10:39:49 +02:00
Daniel Mitterdorfer
0d0e2b432c Limit request size on transport level
With this commit we limit the size of all in-flight requests on
transport level. The size is guarded by a circuit breaker and is
based on the content size of each request.

By default we use 100% of available heap meaning that the parent
circuit breaker will limit the maximum available size. This value
can be changed by adjusting the setting

network.breaker.inflight_requests.limit

Relates elastic/elasticsearchelastic/elasticsearch#16011

Original commit: elastic/x-pack-elasticsearch@d1c43fe8d9
2016-04-13 10:39:04 +02:00
Adrien Grand
26e1535eee Fix compilation as a result of elastic/elasticsearchelastic/elasticsearch#16268.
Original commit: elastic/x-pack-elasticsearch@4a334d7f7d
2016-04-11 18:06:48 +02:00
Adrien Grand
5b57727b34 Replace usage of settingsBuilder with just builder.
Original commit: elastic/x-pack-elasticsearch@fe038bbc49
2016-04-08 18:09:02 +02:00
Chris Earle
daa875db11 Remove hostname from NetworkAddress.format (x-plugins side)
This removes the old usage of NetworkAddress.formatAddress in favor of the updated version, which is just
the method renamed to NetworkAddress.format (replacing the old version of that method).

There is no impact to x-plugins beyond making the build work because all places were currently using that
method variant already.

Original commit: elastic/x-pack-elasticsearch@05f0dcfa90
2016-04-07 17:29:14 -04:00
jaymode
52b6fc54b8 test: explicitly initialize anonymous user in ReservedRealmTests
Original commit: elastic/x-pack-elasticsearch@46ce5c03a1
2016-04-07 10:54:47 -04:00
jaymode
8049a82953 security: add support for main action
This commit adds support for the change in elasticsearch where the `/` rest
endpoint now delegates to an action and can be authorized.

Original commit: elastic/x-pack-elasticsearch@8ef38ce50f
2016-04-07 09:25:21 -04:00
jaymode
b56e2f3bca test: reset anonymous after test to avoid messing with other tests
Closes elastic/elasticsearch#1956

Original commit: elastic/x-pack-elasticsearch@9b57d295c8
2016-04-07 06:12:02 -04:00
jaymode
931c67b49c security: add type argument to fix compile
Original commit: elastic/x-pack-elasticsearch@81acbd2e29
2016-04-06 19:09:29 -04:00
jaymode
d08446e221 security: add reserved roles and users
This commit adds reserved or built-in user and role support to x-pack. The reserved roles
cannot be modified by users. The reserved users also cannot be modified with the exception
of changing the password for a user.

In order to change the password for a user, a new API has been added. This API only supports
changing passwords for native and reserved users.

To support allowing a user to change their own password, a default role has been added to grant
access. This default role only grants access to user operations that pertain to the user that is
being authorized. In other words, the default role grants `joe` the ability to change their own password
but does not allow them to change the password of a different user.

Additionally, the authenticate API was made a transport action and is granted by the default role.

Closes elastic/elasticsearch#1727
Closes elastic/elasticsearch#1185
Closes elastic/elasticsearch#1158

Original commit: elastic/x-pack-elasticsearch@1a6689d90f
2016-04-06 18:23:18 -04:00
jaymode
f888082ce6 security: remove the use of shield in settings
This commit migrates all of the `shield.` settings to `xpack.security.` and makes changes to
use the new Settings infrastructure in core.

As a cleanup, this commit also renames Shield to Security since this class is only in master
and will not affect 2.x.

See elastic/elasticsearch#1441

Original commit: elastic/x-pack-elasticsearch@a5a9798b1b
2016-04-06 14:00:46 -04:00
jaymode
9031cee432 security: allow indices monitor actions to access the security index
This commit allows authorized users to monitor the security index. This fixes an issue
with the _cat/indices api, which resolves the concrete indices using the cluster state
and then makes a indices stats request. Without this change, the api fails with an
authorization exception because it is specifically requesting the security index and the
user is not the internal user.

Closes elastic/elasticsearch#1895

Original commit: elastic/x-pack-elasticsearch@070a389833
2016-04-06 12:59:15 -04:00
Alexander Reelsen
366498eca4 Tests: Fix systemkeytool tests
One test was missing the check if posix permissions are supported by the file system.
As it does not make sense to not run 50% of the tests in 50% of the cases, the
logic to configure which capabilities a in-memory FS should have has been moved
into each test.

Original commit: elastic/x-pack-elasticsearch@59a32ea26d
2016-04-05 14:08:03 +02:00
Alexander Reelsen
323f80216d Security: Fix systemkey CLI tool
When called without arguments, systemkey tool returned with an AIOOE.
This fixes the issue, but also ports over the tests to jimfs, so they
can actually run, as the security manager is always enabled and thus the
tests never ran before.

Closes elastic/elasticsearch#1926

Original commit: elastic/x-pack-elasticsearch@887b681607
2016-04-05 11:46:20 +02:00
jaymode
4036ce97c1 shield: do not use ThreadPool#scheduleWithFixedDelay for pollers
This commit makes the user and roles poller use a self rescheduling runnable to schedule the
next run of the poller rather than using scheduleWithFixedDelay. This is done because the
pollers perform blocking I/O operations and everything using that thread pool method runs on
the schedule thread and because of this, in certain situations this can lead to a deadlock which
will prevent the cluster from forming.

Original commit: elastic/x-pack-elasticsearch@9fd0748c8c
2016-04-01 21:25:16 -04:00
Chris Earle
3126fcb856 Improved tests with better error message
Original commit: elastic/x-pack-elasticsearch@cb79988dc3
2016-04-01 14:20:03 -04:00
Chris Earle
86ed96b83b Adding support for STANDARD license
Original commit: elastic/x-pack-elasticsearch@1671d8ade3
2016-04-01 12:49:05 -04:00
Chris Earle
55b9569f7b Removing isPaid, allFeaturesEnabled, and isActive methods from enums.
Original commit: elastic/x-pack-elasticsearch@8b8c7792c7
2016-04-01 12:49:05 -04:00
Chris Earle
5e81beabf9 Simplifying License Checks
Too many places are checking for enumerations when they're really more interested in a "higher" level of
information. This will help with the forthcoming addition of the STANDARD operation mode as well.

Original commit: elastic/x-pack-elasticsearch@2799c27e19
2016-04-01 12:49:05 -04:00
jaymode
d6cab8b9f1 security: read correct file when listing users
Original commit: elastic/x-pack-elasticsearch@dca906abba
2016-04-01 06:30:34 -04:00
javanna
02751ffff8 Merge branch 'master' into enhancement/discovery_node_one_getter
Original commit: elastic/x-pack-elasticsearch@cf4c5bc630
2016-03-30 17:26:02 +02:00
javanna
52ad574827 Remove DiscoveryNode#id in favour of existing DiscoveryNode#getId
Original commit: elastic/x-pack-elasticsearch@64951de2f9
2016-03-30 17:22:59 +02:00
Nik Everett
0531dd8b88 Switch from getRandom to random
This is a reaction to
https://github.com/elastic/elasticsearch/pull/17394
which handled a long standing TODO in core.

Original commit: elastic/x-pack-elasticsearch@76425300a2
2016-03-30 08:58:31 -04:00