As we are not dependent on any versions any more and we never exposed
an version type functionality, nor documented it nor tested it, we
should remove this for the next major release.
Note, this just removes the ability to set those options, which we ignored
anyway in the transport action.
Original commit: elastic/x-pack-elasticsearch@3830203f50
This commit stops the index audit trail from executing blocking calls on the cluster
state update thread. Blocking calls were executed when indexing to a remote cluster
to get that cluster's state and also possibly put a template and mappings.
Closeselastic/elasticsearch#3989
Original commit: elastic/x-pack-elasticsearch@a8c0269fad
Ensure that tests never sleep, but run using timewarping,
so that sleeping can be removed from the test code.
Original commit: elastic/x-pack-elasticsearch@40fc3814f9
This prevents a possible NPE when sending emails, as some host have
a perfectly fine internet connection, but cannot resolve their localhost.
In addition I also removed a EmailService.send() method that was only used
in tests and thus not needed.
Closeselastic/elasticsearch#3227
Original commit: elastic/x-pack-elasticsearch@d2e29b4c92
The clock interface is merely used to create mock clocks and inject them into tests.
We can do this with the java8 based java.time.Clock class as well, so there is no need
to keep this interface.
Original commit: elastic/x-pack-elasticsearch@ae30dc29ca
The base test case class has been adapted to assert that no warn nor
error messages are logged to the Log4j status logger. An issue that
exists in x-pack uncovered by this change is that the mock appender in
the capturing logger was never started. This commit addresses this
issue.
Relates elastic/elasticsearch#3988
Original commit: elastic/x-pack-elasticsearch@8fc0b99bae
Make them do things slightly less frequently and wait longer
to make sure everything is shut down. Sometimes it takes a while
but it does get there.
Original commit: elastic/x-pack-elasticsearch@76fc4acf47
This explicitly shuts down monitoring at the end of every index
which should help prevent the shutdown problems we were seeing.
Original commit: elastic/x-pack-elasticsearch@2be0ee0df8
We only need it to be so fast when starting up. We can slow it
down once it has started so shutdown isn't problematic.
Original commit: elastic/x-pack-elasticsearch@052d64402f
* [Monitoring] Use Suffix with Marvel Index alias
Using a suffix allows a non-aliased Monitoring index get created for _today_. Otherwise the time-based index for today cannot be created.
Original commit: elastic/x-pack-elasticsearch@92e7695a2c
These tests were failing for me locally with this reproduction
line about 80% of the time:
```
gradle :x-plugins:elasticsearch:integTest -Dtests.seed=660D249EDCC648E5 -Dtests.class=org.elasticsearch.xpack.monitoring.OldMonitoringIndicesBackwardsCompatibilityIT -Dtests.method="testOldIndexes" -Dtests.security.manager=true -Dtests.jvms=12 -Dtests.locale=vi-VN -Dtests.timezone=Arctic/Longyearbyen
```
This was due to the ten second collection interval lining up with the
10 second sleep in `assertBusy`. Eventually the collection interval
lines up in such a way that it take more than ten seconds total to
create the alias for the monitoring index. 😢
This fixes that by dropping the interval to 100ms, making the test
no longer fail with that seed and succeed much more quickly. 😄
Relates to elastic/elasticsearch#3951
Original commit: elastic/x-pack-elasticsearch@b5dfa6ef7d
Those tests ran 35 seconds on my local notebook. By not using a sleep based
scripting engine and some other tweaks the time is down to 13 seconds.
Also renamed the class to remove the `Slow` prefix.
Original commit: elastic/x-pack-elasticsearch@5289fe8dab
Moves the tests for aliases starting with `-` into the backwards
compatibility tests because we can no longer create such aliases.
Original commit: elastic/x-pack-elasticsearch@3639fe4d46
* [Monitoring UI] Show Replica Count not Replication Factor in Overview
This changes it to only show the replica count as `total - primaries` rather than showing the replication factor, which is particularly unhelpful when different indices have different replica counts.
Original commit: elastic/x-pack-elasticsearch@552f94bf8f
`action.destructive_requires_name` setting was ignored by the security plugin as wildcards got expanded and resolved in the plugin before es core could actually check if the operation was supposed to be allowed or not. We are discussing how we could perform the check earlier in es core, but anyways it is good to perform the same check in the security plugin just to make sure.
Closeselastic/elasticsearch#3689
Original commit: elastic/x-pack-elasticsearch@3414cb3471
The old handling was not thread safe, as it used to replace volatile
objects in the code. This implementation uses a concurrent hashmap
to easily allow adding/removing schedules without having to replace
whole objects
Original commit: elastic/x-pack-elasticsearch@0aa618b372
When parsing chain inputs there were possibilities to write invalid
JSON that resulting in losing the order of the inputs without any
exception being thrown.
This commit makes the parsing more strict.
Closeselastic/elasticsearch#3736
Original commit: elastic/x-pack-elasticsearch@963641ee2b
Currently security always parses the permissions filters with a shard id equal
to `0` even if the query is executed on a different shard. Also it does not
protect against queries that may rely on the current timestamp even though we
don`t currently have ways to make sure that all shards use a consistent
timestamp.
Sibling of elastic/elasticsearchelastic/elasticsearch#21196.
Original commit: elastic/x-pack-elasticsearch@cab47f2ed2
The execution time of the trigger tests was extremely slow, because it
really waited until executions happened. This uses the mock clock to
advance in time manually.
This also allows to remove the bad apples annotation and make sure that
the schedule engine tests for both implementations are run all the time.
Relates elastic/elasticsearch#1007
Original commit: elastic/x-pack-elasticsearch@f9436f506f
Refactored ScriptType to clean up some of the variable and method names. Added more documentation. Deprecated the 'in' ParseField in favor of 'stored' to match the indexed scripts being replaced by stored scripts.
Original commit: elastic/x-pack-elasticsearch@d7c7bd7362
My current assumption is, that creating the templates is not
yet finished (as this is async), so that we need to add
another check that the templates have been added before
continuing.
Relates elastic/elasticsearch#3892
Original commit: elastic/x-pack-elasticsearch@3880d200a1
Today we have the same madness in two places and no dedicated test. This
change moves the real madness into a single place and adds a test for it
to make sure it actually works and isn't just crazy.
Original commit: elastic/x-pack-elasticsearch@dabf5fdd63
We used to be very lenient with all kinds of exceptions related to the
`.security` index. Yet, sometimes in tests the index is not yet there but
transport clients already pinging the node this causes issues and transport
clients disconnect. Now if the index is not present we simply return no role.
Original commit: elastic/x-pack-elasticsearch@60948d0c2a
TransportGetRolesAction optimizes for single role case while this
optimization can be simply inside the NativeRoleStore and being
way more contained.
Original commit: elastic/x-pack-elasticsearch@c43d8ba341
`TransportGetUsersAction` does some funky blocking calls even though
it's specifying `SAME` as the thread-pool indicating that it's fast or
forking off quickly. Both might not be true today. This change adds
async support to the methods it calls without breaking the existing
Realm interface. Yet, we might need to do this down the road.
Original commit: elastic/x-pack-elasticsearch@d0959f87f3
This changes adds a special value for negative role lookups so that we can avoid scenarios
where we overload the cluster due to continually trying to load non-existing roles as is often
the case when `unmapped_groups_as_roles` is used with the active directory realm.
Relates elastic/elasticsearch#3530
Original commit: elastic/x-pack-elasticsearch@62567b4c22
* security: restore the correct user when switching to the system user
For internal actions where we need to switch to the SystemUser, we should always restore the proper
context after execution. We were restoring an empty context for actions executed by the SystemUser
in the SecurityServerTransportInterceptor.
In order to accomplish this, a few changes have been made. Both the SecurityServerTransportInterceptor
and the SecurityActionFilter delegate to `SecurityContext#executeAsUser` when a user switch is necessary.
Tests were added for this method to ensure that the consumer is executed as the correct user and the proper
user is restored.
While working on this, a few other cleanups were made:
* SecurityContext can never have a null CryptoService, so a null check was removed
* We no longer replace the user with the system user when the system user is already associated with the request
* The security transport interceptor checks the license state and if auth is not allowed, delegate and return
* The security transport interceptor sendWithUser method now requires authentication to be present or a hard
exception is thrown.
* The TransportFilters integration test has been deleted. This was integration test that relied on the ability to
get instances from a node and trace the execution. This has been replaced by additional unit tests in
ServerTransportFilterTests
Closeselastic/elasticsearch#3845
Original commit: elastic/x-pack-elasticsearch@d8bcb59cb7
This change removes the blocking notion from fetching the roles
from a remote index. This also removes the blocking client calls
that can potentially deadlock a request if executed on the transport
thread.
Relates to elastic/elasticsearch#3790
Original commit: elastic/x-pack-elasticsearch@c2eda39043
This change now installs a signed license that has been generated at runtime so the
BWC tests can run without hitting licensing issues. The x-pack BWC tests pull in the
full cluster state, which contains the trial license from when the indices and state
was generated. After the trial license period and grace period issues arise with the
tests.
Closeselastic/elasticsearch#3858
Original commit: elastic/x-pack-elasticsearch@1c79e874e5
Today when a request is executed with InternalClient the thread context might
be lost if another component like security exchanges it by executing an async call
or an internal action. This can be a serious security problem since if the async
call executes as the system user all subsequent calls made by the response
thread will also execute as the system user instead.
Original commit: elastic/x-pack-elasticsearch@80682f338d
This commit changes the permissions of the files generated by the certgen tool to 600 (like syskeygen does)
Original commit: elastic/x-pack-elasticsearch@bca74e9c92
The calls made by the native users and roles store use the internal xpack user to make the request
and this user has a built-in role that has a single instance. A bug was introduced when fixing the logic
for applying the anonymous role to all users in elastic/elasticsearch#3716. The anonymous role was now being added to
the xpack user, even though the additional role would have no effect as this user is a superuser.
When the anonymous role is applied to the xpack user and exists as a native role or doesn't exist
at all, we run into a deadlock since we wind up querying for the role as a user that also has the
anonymous role.
This change special cases the XPackUser when getting the collection of roles so that the only role
applied to this user is the superuser role.
Closeselastic/elasticsearch#3822
Original commit: elastic/x-pack-elasticsearch@e3093904f1
This commit removes the dependency on the briks automatons library and instead uses the lucene
version. Shield was originally implemented using the lucene version, but issues arose with supporting
multiple versions of elasticsearch and API changes, so we moved to using the briks library.
x-pack and elasticsearch are always the same version so we can use the lucene version of the
automatons and remove the briks library. This also brings with it protection from huge automatons
that we did not have before.
Original commit: elastic/x-pack-elasticsearch@e3f34b6b55
This changes updates the unboundid ldapsdk to the latest version to stay up to date
with their releases.
Original commit: elastic/x-pack-elasticsearch@b9e4f7f062
This change simplifies the creation of Actions and Transformations.
It moves all instantiation away from guice into straight forward
constructor based initialization.
Original commit: elastic/x-pack-elasticsearch@3c0bca2bea
A commit in core removed the UUID parameter from the
ClusterStatsResponse constructor. This commit adjusts x-plugins to this.
Original commit: elastic/x-pack-elasticsearch@6f2f26168e
This change is a first step towards a real abstraction on top of all the
notification services. There are a bunch of followup changes coming for this
that will remove most of the classes in here but this is a first small step
to actually have a notification service interface.
Original commit: elastic/x-pack-elasticsearch@e14abf8a8b
This commit changes the logging to only log if we actually loaded the system key, otherwise
the message is misleading as the key file may not even exist but we output that it was
loaded.
Original commit: elastic/x-pack-elasticsearch@0af7953c64
Instead of using the long running and long blocking single polling HTTP attachment for our reporting,
we should use the async API provided by kibana. The new workflow (all blocking and in a single watch)
looks like this:
1. An initial request is sent to trigger the report generation, which returns a path
2. This path is used to continuously check if the report is done (then it is sent back) or kibana sends another HTTP error code, which will result in watcher to sleep for another interval until the report is finally returned.
Features include configurable interval time and retry count, so that the total amount of waiting can be tweaked into two directions.
This is what the reporting type looks like right now
```
{
"my-attachment":{
"reporting":{
"url":"http://www.example.org/my-dashboard",
"retries":6, // optional, default 40
"interval":"1s", // optional, default 15s
"auth":{
"basic":{
"username":"foo",
"password":"secret"
}
}
}
}
}
```
The interval/retries can also be configured via settings.
Note, that this is just a temporal workaround until the watcher execution can execute in an asynchronous fashion.
Closeselastic/elasticsearch#3524
Original commit: elastic/x-pack-elasticsearch@d1eaa856b9
The `.triggered-watches`, `.watches` and `.security` indices should load
as early as possible, and not wait for other indices (especially not
for time-based indices, that are old).
This commit adds an index.priority to the template for those indices.
The values 1000, 900 and 800 were chosen rather arbitrary, mainly we
did not want to go with 10, because it was used in the sample documentation.
Security should always be loaded first, because we might need this index for
other operations.
Any administrator can still change all the values in the indices, but this
cares for better defaults.
Original commit: elastic/x-pack-elasticsearch@6ed0fb7975
As discussed in #elastic/elasticsearch-migration/79 supporting aliases for watcher allows
the migration plugin to work.
This adds the relevent checks in the WatchStore and the TriggeredWatchStore that aliases are
supported, as the current assumption was always to just load an index.
Also, this rarely sets those indices as aliases in all the integration tests, so that this
case gets tested.
Note: The new WatchStoreUtils.getConcreteIndex() method will be put into core, as this is a
useful helper for others.
Original commit: elastic/x-pack-elasticsearch@4a98af691d
This change reduces the Condition infrastructure to a single interface called
`Condition` this interface is used to produce and parse requests but also
encapsulates the executable condition. The per class Result, Factory and Executable
are removed and replaced by a single class containing all logic.
Original commit: elastic/x-pack-elasticsearch@2870dff7ad
Watcher does not require any unique build info anymore, as all is put into
the MANIFEST.MF file during the build.
Also the xpack-properties is unused now and can be deleted.
Original commit: elastic/x-pack-elasticsearch@62f121c979
ExecutableActions is really an unnecessary abstraction on top of
List and Map. This commit remove the class and all its usage.
Original commit: elastic/x-pack-elasticsearch@b938499fcf
The system user gets used to put mappings for an index during recovery from local shards, which
is how the shrink index process works. The system user previously had this privilege in 2.x as
we did not have the ThreadContext and dynamic mapping updates would be done by the system user;
with the ThreadContext, these mapping updates are done by the actual user so this privilege
was removed from the SystemUser.
Closeselastic/elasticsearch#3766
Original commit: elastic/x-pack-elasticsearch@cd5d7bea53
The security indices resolver checks through an assertion that shard level requests always have their wildcard expressions resolved. Index names that start with `-` or `+` though shouldn't be considered wild card expressions. Up to 6.x there can be indices with names starting with `-` or `+` and we have to take that into account.
Also moved from assertion to explicit exception so we can also test it better.
Original commit: elastic/x-pack-elasticsearch@a520bbf247
If we create index test1 and alias test1-alias, and tests configure access for test* for some users, this is going to cause problems when verifying exclusions like -test2, as the index itself gets excluded but the alias that points to it doesn't. That is expected behaviour, with this commit we modify the way aliases are named to use a prefix rather than a suffix (e.g. from test1-alias to alias-test1).
Changed also the way aliases creation is randomized.
Original commit: elastic/x-pack-elasticsearch@7f9877e858
missing `.get()` :) the create index request was never sent. The indices were being automatically created when indexing a document into them.
Original commit: elastic/x-pack-elasticsearch@129d69c88e
The checkNodeStats method in this test checks for many fields in every documents of all bwc indices, but some fields like disk_threshold_enabled have been removed in 5.x. This commit changes the method so that it checks for the right fields in the right version.
closeselastic/elasticsearch#3672
Original commit: elastic/x-pack-elasticsearch@c95209cc3b
This commit responds to an API change in core migrating from
EsExecutors#boundedNumberOfProcessors to EsExecutors#numberOfProcessors.
Original commit: elastic/x-pack-elasticsearch@87d6fad971
extracted loading of authorized indices and aliases to separate class (AuthorizedIndices) with reduced dependencies. Allows also to lazily load authorized indices the first time they are required, and reuse them if they are needed again later. Removes AuthzService dependency in indices resolver.
Removed array of resolvers in authorization service as we support only one. Removed IndicesAndAliasesResolver interface and rename DefaultIndicesAndAliasesResolver to IndicesAndAliasesResolver.
Original commit: elastic/x-pack-elasticsearch@a267fefa07
FieldAndDocumentLevelSecurityRequestInterceptor really support intercepting only subclasses of IndicesRequests, we shouldn't have logic that is never used around intercepting CompositeIndicesRequest. Also we can guarantee at compile time, using generics, that only supported subclasses are intercepted through it, no need to verify that at runtime.
Original commit: elastic/x-pack-elasticsearch@6ab6e2d50e
Eagerly authorizing CompositeIndicesRequests allowed the security plugin to fail fast up until now, but it makes it very hard to reason about each specific item in a multi items request. Either all items fail, or none do. We would rather want to adopt a similar behaviour to es core, where individual items fail without affecting other items that are part of the same request. We can rely on the fact that es core always authorizes both main action and every subaction too, and skip authorization for the main action. By subaction we mean either all sub search requests in msearch, as well as each shard level get in mget or shard level bulk request for bulk.
BulkRequestInterceptor was converted to intercept BulkShardRequests rather than BulkRequest as that is where bulk is authorized after this change.
Split IndicesAndAliasesResolverIntegrationTests into ReadActionsTests and WriteActionsTests as they require different set of permissions, lots of tests added.
Explicitly listing the composite actions makes sure that the actions that can bypass security are known, somebody adding a similar action must to add it to the list, so we know it doesn't happen by mistake. At this point the CompositeIndicesRequest can be used as a marker interface only (it is not really needed but can be used to verify that composite actions use a request that implements such interface).
Given that we don't authorize composite actions based on their indices anymore, but only their sub-requests which implement IndicesRequest, printing out the indices names in the audit log for requests like bulk and msearch is confusing. Removed support for that.
Authorize composite indices actions based on their name only, their indices will be authorized at the sub-request/shard level
Rather than simply granting bulk, mget, msearch etc. and relying on authorization at the sub-request/shard level, we check that the current user can at least execute the action. This justifies the grant line that gets written in the audit log, the action is potentially possible without looking at the indices. Each specific item will fail or succeed later and will yield its own specific audit log entry.
Original commit: elastic/x-pack-elasticsearch@4570caf019
Like es core does in TransportIndicesExistsAction, we should only consider expandWildcardsOpen and expandWildcardsClosed out of the indices options passed in with IndicesExistsRequest. ignore_unavailable and allow_no_indices should always be considered both true, to prevent the request from throwing exception as it is supposed to return true or false, no exceptions.
Original commit: elastic/x-pack-elasticsearch@daa274b3fd
Supporting allowNoIndices means that the security plugin has a behaviour much more similar to vanilla es when dealing with wildcard expressions that match no indices, or empty clusters. The default for most request is to allow no indices, but security plugin could only disallow no indices all the time up until now.
The technical problem was that when anything gets resolved to an empty set of indices, we couldn't let that go through to es core, as that would become resolved to all indices by es core, which would be a security hole. We have now found a way though to replace an empty set of indices with something that es core will for sure resolve to no indices, so we can let the request through. We simply replace empty indices with '-*'.
Multi apis requests (e.g. _msearch) have yet to be fixed, as all their indices end up in the same bucket while they should each be authorized separately, so that every specific item can fail or be let through.
Original commit: elastic/x-pack-elasticsearch@0f67a0bfea
For all the requests that support multiple indices and wildcards, hence implementing IndicesRequest.Replaceable, we replace the wildcard expressions with the explicit names of the authorized indices they match. _all or empty indices is treated as a wildcard expression. We can also honour the ignore_unavailable option by going over all the explicit names and filter out the non authorized ones when ignore_unavailable is set to true. If ignore_unavailable is set to false, we leave everything as-is, which will cause an authorization exception to be thrown if only one of those explicit indices is not authorized for the current user.
This is the first step towards resolving elastic/elasticsearch#1250. The remaining issue is that in case we are left with no indices after stripping out the ones that the user is not authorized for, we throw an authorization exception rather than returning an empty response. That will require honouring the allow_no_indices option, which will also change the behaviour when a cluster is empty.
Relates to elastic/elasticsearch#1250
Original commit: elastic/x-pack-elasticsearch@e4ca940d05
The superuser role is the only user assignable role that grants access to the .security index, but when
resolving wildcards the index was not getting resolved. The resolution of indices and aliases explicitly
excludes the .security index for users that are not the internal user without checking if the user has the
superuser role. This commit adds a check in for the superuser role.
Original commit: elastic/x-pack-elasticsearch@02ee0a8740
The role cache was previously using the wrong time unit for its expire after write time; the
value passed to the cache was milliseconds instead of nanoseconds.
Original commit: elastic/x-pack-elasticsearch@65f7b08763
Alexander Reelsen
Yannick Welsch
Ryan Ernst
jaymode
Jason Tedor
Nik Everett
Chris Earle
Jay Modi
Clinton Gormley
Luca Cavanna
Boaz Leskes
Adrien Grand
Jack Conradson
Simon Willnauer
Simon Willnauer
Tanguy Leroux
javanna
Areek Zillur