This commit catches up with master which removed the shard level injector
and changed the logic how the SearcherWrapper works and is installed.
The way we now install it is via a package private onModule(IndexModule) call.
There is no public API for this anymore.
The wrapper also doesn't need to watch the IndexShards state since now it will only
be used when the shard is STARTED or RECOVERED.
Original commit: elastic/x-pack-elasticsearch@42b9eeef3d
This commit removes all uses of com.google.common.net.InetAddresses
across the codebase. This is one of the few remaining steps in the
eventual removal of Guava as a dependency.
Relates elastic/elasticsearchelastic/elasticsearch#13224
Original commit: elastic/x-pack-elasticsearch@f6a0ed1395
This commit removes the use of com.google.common.cache.Cache,
com.google.common.cache.CacheBuilder and
com.google.common.cache.CacheLoader.
Relates elastic/elasticsearchelastic/elasticsearch#13224, elastic/elasticsearchelastic/elasticsearch#13717
Original commit: elastic/x-pack-elasticsearch@04c254ee95
Some tests are still using a node builder for nodes that need to load Shield, which is not currently
supported by elasticsearch. This changes some to use a MockNode and awaits fix a tribe test
that was previously testing nothing.
Original commit: elastic/x-pack-elasticsearch@57d0c902b6
This updates our test certificates to have more subject alternative names and adds a script
to make regeneration easier in the future. Also added are the private keys in PEM format and
a .p12 version of the keystore. In the future we may want to start supporting PKCS12 keystores
and this will be a starting point for that.
Original commit: elastic/x-pack-elasticsearch@92c94c9f28
These files are generated automatically with eclipse:eclipse so they shouldn't
be tracked in the repository.
Original commit: elastic/x-pack-elasticsearch@3869ddebfa
This also stops the remote IndexAuditTrail services prior to wiping the cluster to prevent possible
writes after a template has been deleted.
Original commit: elastic/x-pack-elasticsearch@30ef226250
This change updates the IPFilter to always allow traffic from the bound addresses of the node
even if they have been explicitly disabled. This behavior can be disabled through a setting but
that could be dangerous if the blocking rule is added via a persistent setting stored in the cluster
state.
Closeselastic/elasticsearch#487
Original commit: elastic/x-pack-elasticsearch@4c1cf9455f
When we receive a message and cannot decode the signed user header, we should audit
this as a tampered request. The request may not be tampered if nodes are configured
differently in terms of system keys, but this at least provides some evidence of the bad
request on the nodes that receives it.
Closeselastic/elasticsearch#647
Original commit: elastic/x-pack-elasticsearch@2f3411d9f9
This extracts an interface for methods that a caching realm must implement. The idea for this
interface is to allow for custom realms to integrate easily with the clear cache api. With this change
a custom realm can implement this API and the clear cache action can call the evict or evictAll
methods on the realm.
Closeselastic/elasticsearch#544
Original commit: elastic/x-pack-elasticsearch@62b0029689
This commit removes all uses of com.google.common.primitives.Ints
across the codebase.
Relates elastic/elasticsearchelastic/elasticsearch#13224
Original commit: elastic/x-pack-elasticsearch@9500282387
This commit keeps the behavior the same and does not exclude the index audit trail template from
being wiped, that will be done in a future commit.
Original commit: elastic/x-pack-elasticsearch@1ac9e22923
This commit removes all uses of com.google.common.base.Joiner across
the codebase.
Relates elastic/elasticsearchelastic/elasticsearch#13224
Original commit: elastic/x-pack-elasticsearch@f69b2addca
The IndexAuditTrail was checking the logger level of the class before adding the request
class to the message, which was an error. The logger level should not control anything
about the IndexAuditTrail.
Closeselastic/elasticsearch#546
Original commit: elastic/x-pack-elasticsearch@35ee8ab453
In 2.0, plugins cannot specify mandatory settings, they can only specify a default additional
set of settings. For tribe nodes, we require shield to be enabled and be a mandatory plugin.
If the settings specified by the user conflict with this, we now throw an exception and fail
startup.
Closeselastic/elasticsearch#426
Original commit: elastic/x-pack-elasticsearch@db4d6d7923
This commit removes all uses of com.google.common.collect.Iterables
across the codebase.
Relates elastic/elasticsearchelastic/elasticsearch#13224
Original commit: elastic/x-pack-elasticsearch@ca517de412
This adds the release notes to the documentation for Shield. Note, two new features do not
have links as the documentation for these are still pending.
Original commit: elastic/x-pack-elasticsearch@e66df5cf14
The SSL randomization can cause issues when running this test multiple times to look for an issue
since the suite cluster may have a different SSL value than the remote cluster during subsequent
runs.
Closeselastic/elasticsearch#607
Original commit: elastic/x-pack-elasticsearch@8ba3140c52
This commit cuts over to StandardCharset vs. guavas Charsets, removes
obsolete uses of Collections2 / Function and replaces all LoadingCaches
with simple CHM#loadIfAbsent
Original commit: elastic/x-pack-elasticsearch@7d1d607e9e
The IndexAuditTrail was not setting the appropriate user header on requests to see if the index exists and
the mapping is updated. This did not fail in tests because we set shield.user, but fails during a normal
installation.
Closeselastic/elasticsearch#626
Original commit: elastic/x-pack-elasticsearch@3771612b20
Remove use of com.google.common.collect.Sets.
This commit removes all uses of com.google.common.collect.Sets. This is
one of many steps in the eventual removal of Guava as a dependency.
Relates elastic/elasticsearchelastic/elasticsearch#13224
Original commit: elastic/x-pack-elasticsearch@5276ed9723
This commit removes all uses of com.google.common.collect.Maps. This is
one of many steps in the eventual removal of Guava as a dependency.
Relates elastic/elasticsearchelastic/elasticsearch#13224
Original commit: elastic/x-pack-elasticsearch@3708fc0c60
I fixed a couple more warnings and added suppressions, so that when
elastic/elasticsearchelastic/elasticsearch#13410 lands, x-plugins will not break.
Original commit: elastic/x-pack-elasticsearch@8a19b2b71b
Today, if the roles.yml file does not exist the roles that are defined programmatically are not
available. This is incorrect because the reserved roles should always be available and not depend
on the parsing of the file. This change ensures that the reserved roles are made available even
when the roles.yml file is missing.
Closeselastic/elasticsearch#602
Original commit: elastic/x-pack-elasticsearch@ee2fd2ddbf
This change removes the override of finalize in SecuredString to resolve a issue where
the char[] can be cleared by the call in the finalize method but the char array is still being used.
The specific issue that occurs is in the BCrypt usage of the SecuredString. A character is concatenated
and then the utf8Bytes method is called. In most cases, the proper bytes are returned but occasionally
the byte array is returned with only zeroes. This occurs under load and/or memory pressure and can be
provoked by running BCryptTests with a small heap (12 - 16 megabytes) and the SecuredString
implementation with the overridden finalize method.
Closeselastic/elasticsearch#589
Original commit: elastic/x-pack-elasticsearch@fb6430ea9d
This change adds a new permission that allows authorized users to execute a request as
another user. The flow is as follows:
1. The user making the request is authenticated
2. The user that is being impersonated is looked up
3. The requesting user is authorized for the privilege to run as the specified user
4. The impersonated user is then authorized for the given request
Additionally, the auditing has been updated to support this capability and indicates when a
user has been granted the ability to run as another user and then also indicates both the user
who is being impersonated and the requesting user when actions are granted/denied.
Closeselastic/elasticsearch#17
Original commit: elastic/x-pack-elasticsearch@00e5a6169b
For the RemoteIndexAuditTrailStartingTests we start a remote cluster and index into the test cluster and after
the test we stop the remote cluster but indexing operations can still be ongoing in the test cluster and thus cause
locked shards causing lots of noise in CI. We simply skip this check for these tests.
Closeselastic/elasticsearch#598
Original commit: elastic/x-pack-elasticsearch@e678f82f29
This change makes the environment optional for the ClientSSLService, which is used for
Transport Clients and also used for LDAP client connections. Since we use the ClientSSLService
for LDAP connections, we still need the environment when running as a node under the
security manager to resolve paths from the configuration directory.
Closeselastic/elasticsearch#573
Original commit: elastic/x-pack-elasticsearch@862fafffe3
Instead of wrapping the IndexSearcher and applying the role query during the rewrite, the role query gets applied in a custom filtered reader that applies the query via the live docs.
The big advantage is that DLS is being applied in all document based APIs instead of just the _search and _percolate APIs.
In order to better deal with the cost of converting the role query to a bitset, the bitsets are cached in the bitset filter cache
and if the role query bitset is sparse the role query and main query will execute in a leapfrog manner to make executing queries faster.
If the role query bitset isn't sparse, we fallback to livedocs.
Closeselastic/elasticsearch#537
Original commit: elastic/x-pack-elasticsearch@330b96e1f2
This commit removes and now forbids all uses of
com.google.common.base.Strings across the codebase.
For uses of com.google.common.base.Strings.isNullOrEmpty, use
org.elasticsearch.common.Strings.isNullOrEmpty.
For uses of com.google.common.base.Strings.padStart use
org.elasticsearch.common.Strings.padStart.
For uses of com.google.common.base.Strings.nullToEmpty use
org.elasticsearch.common.Strings.coalesceToEmpty.
Relates elastic/elasticsearchelastic/elasticsearch#13224
Original commit: elastic/x-pack-elasticsearch@d6e349e654
This commit removes and now forbids all uses of
com.google.common.base.Predicate and com.google.common.base.Predicates
across the codebase. This is one of the many steps in the eventual
removal of Guava as a dependency. This was enabled by
elastic/elasticsearchelastic/elasticsearch#13314.
Relates elastic/elasticsearchelastic/elasticsearch#13224, elastic/elasticsearchelastic/elasticsearch#13349
Original commit: elastic/x-pack-elasticsearch@968b81849f
The InternalTestCluster checks after the tests run if there are any locked shards, which is still
possible because the IndexAuditTrail is running and auditing events. This change makes the
test stop the IndexAuditTrail if they are running in the local cluster.
Original commit: elastic/x-pack-elasticsearch@f7fb564525
The ShieldWrappingCluster was added to prevent deletion of the IndexAuditTrail template when
running tests since the wipe() method will delete all templates. However, the cluster wrapping
is problematic with indexRandom() and causes noise in CI. This change removes the
ShieldWrappingCluster and changes the IndexAuditTrail to recreate the index template if it is
removed.
Closeselastic/elasticsearch#562
Original commit: elastic/x-pack-elasticsearch@0403ea7cef
This commit adds a test to ensure that all request handlers are wrapped
by ProfileSecuredRequestHandler.
Original commit: elastic/x-pack-elasticsearch@26473d0ddc
Currently we copy the authorization header from every rest request to the action request. This is not
necessary because the user associated with each request is copied into the context and then if the
request leaves the node, the user will be serialized into a string and attached as a header.
This commit removes the copying of the authorization header as it is not necessary and by not copying
it, we limit the amount of copies we make of this sensitive information.
Original commit: elastic/x-pack-elasticsearch@4e5ba4b4aa
This commit adds document and field level security to Shield.
Field level security can be enabled by adding the `fields` option to a role in the `role.yml` file.
For example:
```yaml
customer_care:
indices:
'*':
privileges: read
fields:
- issue_id
- description
- customer_handle
- customer_email
- customer_address
- customer_phone
```
The `fields` list is an inclusive list of fields that controls what fields should be accessible for that role. By default all meta fields (_uid, _type, _source, _ttl etc) are also included, otherwise ES or specific features stop working. The `_all` field if configured, isn't included by default, since that actually contains data from all the other fields. If the `_all` field is required then this needs to be added to the `fields` list in a role. In the case of the content of the `_source` field and `_field_names` there is special filtering in place so that only the content relevant for the role are being returned.
If no `fields` is specified then field level security is disabled for that role and all fields in an index are accessible.
Field level security can be setup per index group.
Field level security is implemented at the Lucene level by wrapping a directory index reader and hides fields away that aren't in the `field` list defined with the role of the current user. It as if the other fields never existed.
* Any `realtime` read operation from the translog is disabled. Instead this operations fall back to the Lucene index, which makes these operations compatible with field level security, but there aren't realtime.
* If user with role A executes first and the result gets cached and then a user with role B executes the same query results from the query executed with role A would be returned. This is bad and therefore the query cache is disabled.
* For the same reason the request cache is also disabled.
* The update API is blocked. An update request needs to be executed via a role that doesn't have field level security enabled.
Document level security can be enabled by adding the `query` option to a role in the `role.yml` file:
```yaml
customer_care:
indices:
'*':
privileges: read
query:
term:
department_id: 12
```
Document level security is implemented as a filter that filters out documents there don't match with the query. This is like index aliases, but better, because the role query is embedded on the lowest level possible in ES (Engine level) and on all places the acquire an IndexSearcher the role query will always be included. While alias filters are applied at a higher level (after the searcher has been acquired)
Document level security can be setup per index group.
Right now like alias filters the document level security isn't applied on all APIs. Like for example the get api, term vector api, which ignore the alias filter. These apis do acquire an IndexSearcher, but don't use the IndexSearcher itself and directly use the index reader to access the inverted index and there for bypassing the role query. If it is required to these apis need document level security too the the implementation for document level security needs to change.
Closeselastic/elasticsearch#341
Original commit: elastic/x-pack-elasticsearch@fac085dca6
PreProcessModule was an alternate way to customize another module's
behavior inside plugins. The preferred (and only in the future) way to
do this is with onModule in the plugin itself. This change moves the
only two remaining users of PreProcessModule to do so in their
respective plugins. The use case was adding roles for shield
authorization, but these roles were really static, so there was no
reason they could not be configured up front.
Original commit: elastic/x-pack-elasticsearch@e67ac2dcb6
This adds the extension points necessary to enable a user to write a elasticsearch plugin
that can integrate with Shield and add a custom authentication realm. For the most part,
the work here just exposes the existing interfaces we have been using for Realms and
factories to create realms. An additional interface was added to allow for a custom
authentication failure handler to be used. This was needed to support use cases like SSO
and Kerberos where additional headers may need to be sent to the user or a different
HTTP response code would need to be sent.
Relates to elastic/elasticsearch#24
Original commit: elastic/x-pack-elasticsearch@13442e5919
SpawnModules will be going away very soon as part of
elastic/elasticsearchelastic/elasticsearch#12783. This change removes its use from all
x-plugins.
Most spawnmodules uses here were to either collect a number of modules
into one (so the modules were just moved up into the plugin itself), or
to spawn a module which interacted with an extension point from ES. This
change moves those, as well as most uses of PreProcessModule, to use
onModule.
Original commit: elastic/x-pack-elasticsearch@6430e35379
Jason Tedor
Nik Everett
Areek Zillur
jaymode
debadair
Simon Willnauer
javanna
Robert Muir
Lee Hinman
Martijn van Groningen
uboness
Ryan Ernst
Adrien Grand
Britta Weber
David Pilato
Boaz Leskes