The IndexAuditTrail was not setting the appropriate user header on requests to see if the index exists and
the mapping is updated. This did not fail in tests because we set shield.user, but fails during a normal
installation.
Closeselastic/elasticsearch#626
Original commit: elastic/x-pack-elasticsearch@3771612b20
Remove use of com.google.common.collect.Sets.
This commit removes all uses of com.google.common.collect.Sets. This is
one of many steps in the eventual removal of Guava as a dependency.
Relates elastic/elasticsearchelastic/elasticsearch#13224
Original commit: elastic/x-pack-elasticsearch@5276ed9723
This commit removes all uses of com.google.common.collect.Maps. This is
one of many steps in the eventual removal of Guava as a dependency.
Relates elastic/elasticsearchelastic/elasticsearch#13224
Original commit: elastic/x-pack-elasticsearch@3708fc0c60
I fixed a couple more warnings and added suppressions, so that when
elastic/elasticsearchelastic/elasticsearch#13410 lands, x-plugins will not break.
Original commit: elastic/x-pack-elasticsearch@8a19b2b71b
Today, if the roles.yml file does not exist the roles that are defined programmatically are not
available. This is incorrect because the reserved roles should always be available and not depend
on the parsing of the file. This change ensures that the reserved roles are made available even
when the roles.yml file is missing.
Closeselastic/elasticsearch#602
Original commit: elastic/x-pack-elasticsearch@ee2fd2ddbf
This change removes the override of finalize in SecuredString to resolve a issue where
the char[] can be cleared by the call in the finalize method but the char array is still being used.
The specific issue that occurs is in the BCrypt usage of the SecuredString. A character is concatenated
and then the utf8Bytes method is called. In most cases, the proper bytes are returned but occasionally
the byte array is returned with only zeroes. This occurs under load and/or memory pressure and can be
provoked by running BCryptTests with a small heap (12 - 16 megabytes) and the SecuredString
implementation with the overridden finalize method.
Closeselastic/elasticsearch#589
Original commit: elastic/x-pack-elasticsearch@fb6430ea9d
This change adds a new permission that allows authorized users to execute a request as
another user. The flow is as follows:
1. The user making the request is authenticated
2. The user that is being impersonated is looked up
3. The requesting user is authorized for the privilege to run as the specified user
4. The impersonated user is then authorized for the given request
Additionally, the auditing has been updated to support this capability and indicates when a
user has been granted the ability to run as another user and then also indicates both the user
who is being impersonated and the requesting user when actions are granted/denied.
Closeselastic/elasticsearch#17
Original commit: elastic/x-pack-elasticsearch@00e5a6169b
For the RemoteIndexAuditTrailStartingTests we start a remote cluster and index into the test cluster and after
the test we stop the remote cluster but indexing operations can still be ongoing in the test cluster and thus cause
locked shards causing lots of noise in CI. We simply skip this check for these tests.
Closeselastic/elasticsearch#598
Original commit: elastic/x-pack-elasticsearch@e678f82f29
This change makes the environment optional for the ClientSSLService, which is used for
Transport Clients and also used for LDAP client connections. Since we use the ClientSSLService
for LDAP connections, we still need the environment when running as a node under the
security manager to resolve paths from the configuration directory.
Closeselastic/elasticsearch#573
Original commit: elastic/x-pack-elasticsearch@862fafffe3
Instead of wrapping the IndexSearcher and applying the role query during the rewrite, the role query gets applied in a custom filtered reader that applies the query via the live docs.
The big advantage is that DLS is being applied in all document based APIs instead of just the _search and _percolate APIs.
In order to better deal with the cost of converting the role query to a bitset, the bitsets are cached in the bitset filter cache
and if the role query bitset is sparse the role query and main query will execute in a leapfrog manner to make executing queries faster.
If the role query bitset isn't sparse, we fallback to livedocs.
Closeselastic/elasticsearch#537
Original commit: elastic/x-pack-elasticsearch@330b96e1f2
This commit removes and now forbids all uses of
com.google.common.base.Strings across the codebase.
For uses of com.google.common.base.Strings.isNullOrEmpty, use
org.elasticsearch.common.Strings.isNullOrEmpty.
For uses of com.google.common.base.Strings.padStart use
org.elasticsearch.common.Strings.padStart.
For uses of com.google.common.base.Strings.nullToEmpty use
org.elasticsearch.common.Strings.coalesceToEmpty.
Relates elastic/elasticsearchelastic/elasticsearch#13224
Original commit: elastic/x-pack-elasticsearch@d6e349e654
This commit removes and now forbids all uses of
com.google.common.base.Predicate and com.google.common.base.Predicates
across the codebase. This is one of the many steps in the eventual
removal of Guava as a dependency. This was enabled by
elastic/elasticsearchelastic/elasticsearch#13314.
Relates elastic/elasticsearchelastic/elasticsearch#13224, elastic/elasticsearchelastic/elasticsearch#13349
Original commit: elastic/x-pack-elasticsearch@968b81849f
The InternalTestCluster checks after the tests run if there are any locked shards, which is still
possible because the IndexAuditTrail is running and auditing events. This change makes the
test stop the IndexAuditTrail if they are running in the local cluster.
Original commit: elastic/x-pack-elasticsearch@f7fb564525
The ShieldWrappingCluster was added to prevent deletion of the IndexAuditTrail template when
running tests since the wipe() method will delete all templates. However, the cluster wrapping
is problematic with indexRandom() and causes noise in CI. This change removes the
ShieldWrappingCluster and changes the IndexAuditTrail to recreate the index template if it is
removed.
Closeselastic/elasticsearch#562
Original commit: elastic/x-pack-elasticsearch@0403ea7cef
This commit adds a test to ensure that all request handlers are wrapped
by ProfileSecuredRequestHandler.
Original commit: elastic/x-pack-elasticsearch@26473d0ddc
Currently we copy the authorization header from every rest request to the action request. This is not
necessary because the user associated with each request is copied into the context and then if the
request leaves the node, the user will be serialized into a string and attached as a header.
This commit removes the copying of the authorization header as it is not necessary and by not copying
it, we limit the amount of copies we make of this sensitive information.
Original commit: elastic/x-pack-elasticsearch@4e5ba4b4aa
This commit adds document and field level security to Shield.
Field level security can be enabled by adding the `fields` option to a role in the `role.yml` file.
For example:
```yaml
customer_care:
indices:
'*':
privileges: read
fields:
- issue_id
- description
- customer_handle
- customer_email
- customer_address
- customer_phone
```
The `fields` list is an inclusive list of fields that controls what fields should be accessible for that role. By default all meta fields (_uid, _type, _source, _ttl etc) are also included, otherwise ES or specific features stop working. The `_all` field if configured, isn't included by default, since that actually contains data from all the other fields. If the `_all` field is required then this needs to be added to the `fields` list in a role. In the case of the content of the `_source` field and `_field_names` there is special filtering in place so that only the content relevant for the role are being returned.
If no `fields` is specified then field level security is disabled for that role and all fields in an index are accessible.
Field level security can be setup per index group.
Field level security is implemented at the Lucene level by wrapping a directory index reader and hides fields away that aren't in the `field` list defined with the role of the current user. It as if the other fields never existed.
* Any `realtime` read operation from the translog is disabled. Instead this operations fall back to the Lucene index, which makes these operations compatible with field level security, but there aren't realtime.
* If user with role A executes first and the result gets cached and then a user with role B executes the same query results from the query executed with role A would be returned. This is bad and therefore the query cache is disabled.
* For the same reason the request cache is also disabled.
* The update API is blocked. An update request needs to be executed via a role that doesn't have field level security enabled.
Document level security can be enabled by adding the `query` option to a role in the `role.yml` file:
```yaml
customer_care:
indices:
'*':
privileges: read
query:
term:
department_id: 12
```
Document level security is implemented as a filter that filters out documents there don't match with the query. This is like index aliases, but better, because the role query is embedded on the lowest level possible in ES (Engine level) and on all places the acquire an IndexSearcher the role query will always be included. While alias filters are applied at a higher level (after the searcher has been acquired)
Document level security can be setup per index group.
Right now like alias filters the document level security isn't applied on all APIs. Like for example the get api, term vector api, which ignore the alias filter. These apis do acquire an IndexSearcher, but don't use the IndexSearcher itself and directly use the index reader to access the inverted index and there for bypassing the role query. If it is required to these apis need document level security too the the implementation for document level security needs to change.
Closeselastic/elasticsearch#341
Original commit: elastic/x-pack-elasticsearch@fac085dca6
PreProcessModule was an alternate way to customize another module's
behavior inside plugins. The preferred (and only in the future) way to
do this is with onModule in the plugin itself. This change moves the
only two remaining users of PreProcessModule to do so in their
respective plugins. The use case was adding roles for shield
authorization, but these roles were really static, so there was no
reason they could not be configured up front.
Original commit: elastic/x-pack-elasticsearch@e67ac2dcb6
This adds the extension points necessary to enable a user to write a elasticsearch plugin
that can integrate with Shield and add a custom authentication realm. For the most part,
the work here just exposes the existing interfaces we have been using for Realms and
factories to create realms. An additional interface was added to allow for a custom
authentication failure handler to be used. This was needed to support use cases like SSO
and Kerberos where additional headers may need to be sent to the user or a different
HTTP response code would need to be sent.
Relates to elastic/elasticsearch#24
Original commit: elastic/x-pack-elasticsearch@13442e5919
SpawnModules will be going away very soon as part of
elastic/elasticsearchelastic/elasticsearch#12783. This change removes its use from all
x-plugins.
Most spawnmodules uses here were to either collect a number of modules
into one (so the modules were just moved up into the plugin itself), or
to spawn a module which interacted with an extension point from ES. This
change moves those, as well as most uses of PreProcessModule, to use
onModule.
Original commit: elastic/x-pack-elasticsearch@6430e35379
Today the XContent building of the response for the ClearRealmsCacheResponse is broken and causes
an exception to be thrown. This fixes the building of the response and adds tests that call the HTTP
endpoint and do a basic check on the response.
Closeselastic/elasticsearch#390
Original commit: elastic/x-pack-elasticsearch@8ad9dae4ea
This commit changes the groupId to the above mentioned one
so that S3 uploads will end up in the right bucket. This will
allow the Elasticsearch plugin manager to install the commercial
plugins like
```
bin/plugin install {watcher,shield,license,marvel}
```
like the official ones.
Original commit: elastic/x-pack-elasticsearch@642f1f006a
Squashed commit of the following:
commit 9e1eee30eca33e96fc0010300331207ae609a825
Author: Robert Muir <rmuir@apache.org>
Date: Thu Aug 6 06:11:15 2015 -0400
fix shield integration tests with master changes
TODO: this all needs cleanup
commit elastic/x-pack@098f8994a6
Author: debadair <deb.adair@elasticsearch.com>
Date: Wed Aug 5 13:09:17 2015 -0700
Added missing description and edited the table.
commit elastic/x-pack@f1782adb40
Author: Martijn van Groningen <martijn.v.groningen@gmail.com>
Date: Wed Jul 15 14:08:27 2015 +0200
There is no need to check if the primary shards of the history indices are started, since we don't load watch records any more during the Watcher startup process.
commit elastic/x-pack@689fdcd00b
Author: Martijn van Groningen <martijn.v.groningen@gmail.com>
Date: Fri Jul 17 22:44:52 2015 +0200
Use custom metadata to remember that Watcher stopped via an explicit stop api call.
Also expose WatcherMetaData in stats api
commit elastic/x-pack@abaa42d608
Author: Tanguy Leroux <tlrx.dev@gmail.com>
Date: Wed Aug 5 11:31:42 2015 +0200
Marvel: Add Cluster Stats timeout
commit elastic/x-pack@1be4701a4b
Author: uboness <uboness@gmail.com>
Date: Wed Aug 5 00:20:05 2015 +0200
[watcher] Remove default timezone usage
Closeselastic/elasticsearch#387
commit elastic/x-pack@88f588d97b
Merge: elastic/x-pack@c49a303elastic/x-pack@bb12e9a
Author: Robert Muir <rcmuir@gmail.com>
Date: Tue Aug 4 17:11:52 2015 -0400
Merge pull request elastic/elasticsearch#386 from rmuir/shield_integ_update
adapt shield to integration tests changes
commit elastic/x-pack@bb12e9aa8d
Author: Robert Muir <rmuir@apache.org>
Date: Tue Aug 4 16:17:23 2015 -0400
adapt shield to integration tests changes
we don't use ports 9200/9300 by default for integration tests anymore.
they easily conflict with the user running/debugging es on their machine.
Original commit: elastic/x-pack-elasticsearch@17a474df2c
we don't use ports 9200/9300 by default for integration tests anymore.
they easily conflict with the user running/debugging es on their machine.
Original commit: elastic/x-pack-elasticsearch@3ff7491459
Currently, we attach the zip assembly for all plugins but Shield when deploying. This is problematic because
we want to track the downloads of the zips through our downloads service. This commit changes the
behavior to not attach the zip assembly when deploying publicly.
The source files were also being attached even for public deployments and this commit adds a property to
stop that for public deployments and a check to ensure that no sources or zip file is attached when deploying
publicly.
Additionally, the default profile overrides the distribution management of the parent pom(s) so that a deploy
without a profile specified will not work. Without this change issuing a deploy command on a machine that
has credentials for sonatype's repositories could have deployed the artifacts to the sonatype OSS repository.
Closeselastic/elasticsearch#321
Original commit: elastic/x-pack-elasticsearch@37a0a6c312
We can just run these during the integration test phase: there is
no benefit in running them during `mvn test` too.
Original commit: elastic/x-pack-elasticsearch@4b275920e2
With elastic/elasticsearchelastic/elasticsearch#12623 base test classes were renamed
to use "TestCase" suffix. This updates x-plugins to reflect those
name changes. It also renames some tests that were marked
with @Slow (which was forbidden with elastic/elasticsearchelastic/elasticsearch#12617 and
elastic/elasticsearch elastic/elasticsearch#12618) to use the IT suffix to run
under `mvn verify`.
Original commit: elastic/x-pack-elasticsearch@05ffe2f202
homeFile() is removed and should not be used, we need to cleanup,
but this is just a rote change to get builds green.
Original commit: elastic/x-pack-elasticsearch@05d0fb4a7c
Prior to this commit, we were InetAddress.getLocalHost() to get the hostname and host
address when auditing. This is different than how we report the node's hostname and host
address in other places where we use NetworkUtils. This caused false failures to be seen
with the IndexAuditTrail tests. This commit switches the audit trails to use the NetworkUtils
methods.
Closeselastic/elasticsearch#285
Original commit: elastic/x-pack-elasticsearch@c0bd7e94f6
Today, if a LDAP server is down and the LDAP realm uses the user search mechanism this will prevent the
node from starting up. This is not ideal because users can still authenticate with another realm if it is
configured. This change tries to create the connection pool on initialization but if it fails, creation will retried
on each attempted authentication until the server is available again.
Closeselastic/elasticsearch#107
Original commit: elastic/x-pack-elasticsearch@f2ccf858ff
The integration tests do not have any node attributes so the body that is returned is empty
and causes the test to fail.
Original commit: elastic/x-pack-elasticsearch@ad846796f1
Currently, we use the local cluster state when determining if the index audit trail can be
started. This is correct when we are logging to the same cluster but is incorrect when we
log to a remote cluster. Instead we should try to initialize the client and get the remote
cluster's state.
This also changes the enqueue method to stop throwing an exception on failing to add a
message to the queue. The exception was unnecessary and causing hard to read logs.
It is now replaced with a simple warn log message.
Closeselastic/elasticsearch#317
Original commit: elastic/x-pack-elasticsearch@238e9159b3
Currently, we attach the internal audit user to all requests. This is incorrect for requests that
need to be sent to a remote cluster. For these cases, we should require a user to be defined
to access the remote cluster if it is protected by Shield.
Additionally, the origin_address field for rest request fields is formatted differently than other
address fields. This changes the field to only be the remote address.
Closeselastic/elasticsearch#278Closeselastic/elasticsearch#279
Original commit: elastic/x-pack-elasticsearch@a5f86b1974
In order to correctly copy the license plugin version, we have
to enable useBaseVersion in order to strip the snapshot version
number.
Original commit: elastic/x-pack-elasticsearch@025da92c35
This removes the parent from x-dev-tools because we won't be publishing this artifact to sonatype's repository.
Original commit: elastic/x-pack-elasticsearch@013efb50e9
In the documentation reorgantization for Shield, the type of the active directory realm changed
and is not valid. Users are configuring Shield incorrectly and running into issues because of
this. This changes the realm type to the correct value.
Closeselastic/elasticsearch#291
Original commit: elastic/x-pack-elasticsearch@7f8b65eda7
jaymode
Lee Hinman
Jason Tedor
Martijn van Groningen
debadair
uboness
Ryan Ernst
Adrien Grand
Areek Zillur
Britta Weber
David Pilato
Boaz Leskes
Simon Willnauer
Alexander Reelsen
Robert Muir
Tanguy Leroux
Suyog Rao
Clinton Gormley