This change allows reads of our native users and roles when the template version has not been updated to
match the current version. This is useful for rolling upgrades where the nodes are also being actively
queried and/or indexed into. Without this, we can wreak havoc on a cluster by causing exceptions during
replication, which leads to shard failures. On nodes that match the version defined in the template,
write operations are allowed since we know that we are backwards compatible in terms of format but we
may have added new fields and shouldn't index them until the mappings and template have been updated.
As part of this, the rolling upgrade tests from core were used as the basis for a very basic set of tests
for doing a rolling upgrade with x-pack.
Closeselastic/elasticsearch#4126
Original commit: elastic/x-pack-elasticsearch@9be518ef00
Currently, a tribe node ignored underlying cluster licenses
due to inablity to select an appropriate license from multiple
licenses. Now that tribe node supports merging custom metadata
(elasticsearch#elastic/elasticsearch#21552), we can enable license support in tribe
node.
Now, tribe node chooses license with the highest operation
mode from underlying cluster licenses. This commit also
adds integration tests for licensing to verify that:
- autogenerated trial license propagates to tribe node
- tribe node chooses the highest operation mode license
- removing a license from underlying cluster license is
removed from tribe
closeselastic/elasticsearch#3212
Original commit: elastic/x-pack-elasticsearch@b5c003decd
This commit changes the toXContent() method so that it generates content that can be parsed again by the parse() method. Before this commit, the authentication of the HTTP request is rendered as:
{ "auth": {"username": "foo", "password": "bar" } }
but the parsing method expects the authentication type to be a root object:
{ "auth": { "basic" : {"username": "foo", "password": "bar" } } }
The toXContent method has been adapted to include the type of authentication in the generated content.
Original commit: elastic/x-pack-elasticsearch@b740466109
This removes the mock webserver trying to bind to a port range, just
try to bind to any port.
Also removed unneeded wrapper class in the process.
Original commit: elastic/x-pack-elasticsearch@19b6ac393a
When a fatal error is thrown on the network layer, such an error never
makes its way to the uncaught exception handler. This prevents the node
from being torn down if an out of memory error or other fatal error is
thrown while handling HTTP or transport traffic. This commit adds logic
to ensure that such errors bubble their way up to the uncaught exception
handler, even though Netty tries really hard to swallow everything.
Original commit: elastic/x-pack-elasticsearch@f76757674f
This commit aims to fix the regularly failing test OldMonitoringIndicesBackwardsCompatibilityTests. It enables/disables monitoring data collection using time interval "-1" as well as stopping AgentService instances directly and checks multiple times if indices are still present at the end of the test.
closeselastic/elasticsearch#3999
Original commit: elastic/x-pack-elasticsearch@8ac785061a
* Record audit trail even if indicies-resolver throws exception
If the IndicesAndAliasesResolver throws an exception, treat is as "accessDenied" for the purpose of the audit-trail.
This can occur when an index request has a wildcard that doesn't match (and "allowNoIndices" is false)
Closeselastic/elasticsearch#3719
Original commit: elastic/x-pack-elasticsearch@ca6567e5ed
RemoteIndexAuditTrailStartingTests has been failing quite often due to several
race conditions. This change simplifies the test (one node per cluster, local and
remote) and fixes some issues in IndexAuditTrail. The interrupt based shutdown has
been removed from the `QueueConsumer` and an additional health call has been added
to trigger another start attempt if the first one failed. This stabilize the test
that now run in 1 second rather than 1 minute.
Original commit: elastic/x-pack-elasticsearch@a1c8131cd2
This commit responds to a chance in core which modified the interface
for ThreadPool#executor and ThreadPool#generic to return an executor
service rather than an executor.
Relates elastic/elasticsearch#4107
Original commit: elastic/x-pack-elasticsearch@7c7c6a3b90
This is a followup from elastic/elasticsearchelastic/elasticsearch#21590 and needs to be
committed first or at the same time since netty_3 is removed
Original commit: elastic/x-pack-elasticsearch@131d74dd6b
If these tests were in `:core` then then `IT` would be an appropriate
suffix, but in xplugins they should have `Tests`. This moves them
from the `integTest` task to the `test` task which is a good idea
because it lets them run in parallel with the other tests. Naming
them `IT` means that they are run not in parallel and they are run
with a running instance of Elasticsearch which they don't need.
This cuts the build from 15.5 minutes to 13.5 minutes for me.
Original commit: elastic/x-pack-elasticsearch@7b4bcf8bc5
This commit fixes a compilation issue introduced by the merge of
sequence numbers branch into master. This merge introduced a new
constructor parameter for ShardStats.
Original commit: elastic/x-pack-elasticsearch@b234b8ee50
This commit migrates the authentication service to an asynchronous model where we use listeners
instead of blocking and waiting for the authentication to return. This is the first part of making
authentication asynchronous as we still have blocking I/O inside of realms.
See elastic/elasticsearch#3790
Original commit: elastic/x-pack-elasticsearch@9339af4af8
In the authentication service, we currently only copy the username and roles of the
user that was authenticated but we should instead preserve all of their information
in the newly created user object. This change does that through the user of a new
constructor in the user class that takes in both users.
Closeselastic/elasticsearch#3877
Original commit: elastic/x-pack-elasticsearch@7455078841
If a authenticated user with run as permission attempts to run as an unknown user, the unknown
user will be assigned the default role and anonymous role if enabled. This change prevents this
from happening as we require the run as user to have been looked up by a realm.
Closeselastic/elasticsearch#3878
Original commit: elastic/x-pack-elasticsearch@034f44757d
The internal test cluster now auto manages min_master_nodes ( see https://github.com/elastic/elasticsearch/pull/21458 ). This requires some code changes but also changes the timings of forming a cluster. This has had a funny side effect where the master is no longer always the first node to be started in the cluster. This caused issues with watcher tests which freeze time.
Original commit: elastic/x-pack-elasticsearch@1e5ea8ae94
This change mocks the ClusterService and sets it up so that when remote indexing, we see the
current node as the master.
Original commit: elastic/x-pack-elasticsearch@1f26b3df1f
Template test is easy to fix. I have no idea what is up with the
dynamic index name test. It is finding results on the wrong time.
Original commit: elastic/x-pack-elasticsearch@535676164c
Today when a message is not fully read on a response, we log (among
other details) the handler name. Unfortunately, if the handler is a
wrapper, all that we see is
o.e.x.s.t.TransportService$ContextRestoreResponseHandler@7446ba18
completely losing the offending handler. This commit adds an override
for TransportService$ContextRestoreResponseHandler#toString so that the
underlying offender can be discovered.
Relates elastic/elasticsearch#4043
Original commit: elastic/x-pack-elasticsearch@3eb1003d5b
This removes all guice module classes from the watcher codebase, so that guice is
only used for the transport and rest actions, but nowhere else in the codebase.
Also it ensures, that only ticker/schedule are valid trigger engine options.
Original commit: elastic/x-pack-elasticsearch@400ba24c33
There was an unused and unconverted xpack.watcher.plugin.version
in the watch history index template, which resulted in this template output:
```
"watch_history_1" : {
"order" : 2147483647,
"template" : ".watcher-history-1*",
"settings" : {
... ,
"xpack" : {
"watcher" : {
"plugin" : {
"version" : "${xpack.watcher.plugin.version}"
},
"template" : {
"version" : "1"
}
}
}
}
```
As everything is one plugin now, this can be safely removed.
Original commit: elastic/x-pack-elasticsearch@c5c0bcaaaa
This change adds the restoration of the original context inside the listeners and handlers where
we restore another context. This prevents us from polluting the context of the thread that called
the listener and leaving around a different user in the thread context.
Original commit: elastic/x-pack-elasticsearch@0f30363ef7
As we are not dependent on any versions any more and we never exposed
an version type functionality, nor documented it nor tested it, we
should remove this for the next major release.
Note, this just removes the ability to set those options, which we ignored
anyway in the transport action.
Original commit: elastic/x-pack-elasticsearch@3830203f50
This commit stops the index audit trail from executing blocking calls on the cluster
state update thread. Blocking calls were executed when indexing to a remote cluster
to get that cluster's state and also possibly put a template and mappings.
Closeselastic/elasticsearch#3989
Original commit: elastic/x-pack-elasticsearch@a8c0269fad
Ensure that tests never sleep, but run using timewarping,
so that sleeping can be removed from the test code.
Original commit: elastic/x-pack-elasticsearch@40fc3814f9
This prevents a possible NPE when sending emails, as some host have
a perfectly fine internet connection, but cannot resolve their localhost.
In addition I also removed a EmailService.send() method that was only used
in tests and thus not needed.
Closeselastic/elasticsearch#3227
Original commit: elastic/x-pack-elasticsearch@d2e29b4c92
The clock interface is merely used to create mock clocks and inject them into tests.
We can do this with the java8 based java.time.Clock class as well, so there is no need
to keep this interface.
Original commit: elastic/x-pack-elasticsearch@ae30dc29ca
The base test case class has been adapted to assert that no warn nor
error messages are logged to the Log4j status logger. An issue that
exists in x-pack uncovered by this change is that the mock appender in
the capturing logger was never started. This commit addresses this
issue.
Relates elastic/elasticsearch#3988
Original commit: elastic/x-pack-elasticsearch@8fc0b99bae
Make them do things slightly less frequently and wait longer
to make sure everything is shut down. Sometimes it takes a while
but it does get there.
Original commit: elastic/x-pack-elasticsearch@76fc4acf47
This explicitly shuts down monitoring at the end of every index
which should help prevent the shutdown problems we were seeing.
Original commit: elastic/x-pack-elasticsearch@2be0ee0df8
We only need it to be so fast when starting up. We can slow it
down once it has started so shutdown isn't problematic.
Original commit: elastic/x-pack-elasticsearch@052d64402f
* [Monitoring] Use Suffix with Marvel Index alias
Using a suffix allows a non-aliased Monitoring index get created for _today_. Otherwise the time-based index for today cannot be created.
Original commit: elastic/x-pack-elasticsearch@92e7695a2c
These tests were failing for me locally with this reproduction
line about 80% of the time:
```
gradle :x-plugins:elasticsearch:integTest -Dtests.seed=660D249EDCC648E5 -Dtests.class=org.elasticsearch.xpack.monitoring.OldMonitoringIndicesBackwardsCompatibilityIT -Dtests.method="testOldIndexes" -Dtests.security.manager=true -Dtests.jvms=12 -Dtests.locale=vi-VN -Dtests.timezone=Arctic/Longyearbyen
```
This was due to the ten second collection interval lining up with the
10 second sleep in `assertBusy`. Eventually the collection interval
lines up in such a way that it take more than ten seconds total to
create the alias for the monitoring index. 😢
This fixes that by dropping the interval to 100ms, making the test
no longer fail with that seed and succeed much more quickly. 😄
Relates to elastic/elasticsearch#3951
Original commit: elastic/x-pack-elasticsearch@b5dfa6ef7d
Those tests ran 35 seconds on my local notebook. By not using a sleep based
scripting engine and some other tweaks the time is down to 13 seconds.
Also renamed the class to remove the `Slow` prefix.
Original commit: elastic/x-pack-elasticsearch@5289fe8dab
Moves the tests for aliases starting with `-` into the backwards
compatibility tests because we can no longer create such aliases.
Original commit: elastic/x-pack-elasticsearch@3639fe4d46
* [Monitoring UI] Show Replica Count not Replication Factor in Overview
This changes it to only show the replica count as `total - primaries` rather than showing the replication factor, which is particularly unhelpful when different indices have different replica counts.
Original commit: elastic/x-pack-elasticsearch@552f94bf8f
`action.destructive_requires_name` setting was ignored by the security plugin as wildcards got expanded and resolved in the plugin before es core could actually check if the operation was supposed to be allowed or not. We are discussing how we could perform the check earlier in es core, but anyways it is good to perform the same check in the security plugin just to make sure.
Closeselastic/elasticsearch#3689
Original commit: elastic/x-pack-elasticsearch@3414cb3471
The old handling was not thread safe, as it used to replace volatile
objects in the code. This implementation uses a concurrent hashmap
to easily allow adding/removing schedules without having to replace
whole objects
Original commit: elastic/x-pack-elasticsearch@0aa618b372
When parsing chain inputs there were possibilities to write invalid
JSON that resulting in losing the order of the inputs without any
exception being thrown.
This commit makes the parsing more strict.
Closeselastic/elasticsearch#3736
Original commit: elastic/x-pack-elasticsearch@963641ee2b
Currently security always parses the permissions filters with a shard id equal
to `0` even if the query is executed on a different shard. Also it does not
protect against queries that may rely on the current timestamp even though we
don`t currently have ways to make sure that all shards use a consistent
timestamp.
Sibling of elastic/elasticsearchelastic/elasticsearch#21196.
Original commit: elastic/x-pack-elasticsearch@cab47f2ed2
The execution time of the trigger tests was extremely slow, because it
really waited until executions happened. This uses the mock clock to
advance in time manually.
This also allows to remove the bad apples annotation and make sure that
the schedule engine tests for both implementations are run all the time.
Relates elastic/elasticsearch#1007
Original commit: elastic/x-pack-elasticsearch@f9436f506f
Refactored ScriptType to clean up some of the variable and method names. Added more documentation. Deprecated the 'in' ParseField in favor of 'stored' to match the indexed scripts being replaced by stored scripts.
Original commit: elastic/x-pack-elasticsearch@d7c7bd7362
My current assumption is, that creating the templates is not
yet finished (as this is async), so that we need to add
another check that the templates have been added before
continuing.
Relates elastic/elasticsearch#3892
Original commit: elastic/x-pack-elasticsearch@3880d200a1
Today we have the same madness in two places and no dedicated test. This
change moves the real madness into a single place and adds a test for it
to make sure it actually works and isn't just crazy.
Original commit: elastic/x-pack-elasticsearch@dabf5fdd63
We used to be very lenient with all kinds of exceptions related to the
`.security` index. Yet, sometimes in tests the index is not yet there but
transport clients already pinging the node this causes issues and transport
clients disconnect. Now if the index is not present we simply return no role.
Original commit: elastic/x-pack-elasticsearch@60948d0c2a
TransportGetRolesAction optimizes for single role case while this
optimization can be simply inside the NativeRoleStore and being
way more contained.
Original commit: elastic/x-pack-elasticsearch@c43d8ba341
`TransportGetUsersAction` does some funky blocking calls even though
it's specifying `SAME` as the thread-pool indicating that it's fast or
forking off quickly. Both might not be true today. This change adds
async support to the methods it calls without breaking the existing
Realm interface. Yet, we might need to do this down the road.
Original commit: elastic/x-pack-elasticsearch@d0959f87f3
This changes adds a special value for negative role lookups so that we can avoid scenarios
where we overload the cluster due to continually trying to load non-existing roles as is often
the case when `unmapped_groups_as_roles` is used with the active directory realm.
Relates elastic/elasticsearch#3530
Original commit: elastic/x-pack-elasticsearch@62567b4c22
* security: restore the correct user when switching to the system user
For internal actions where we need to switch to the SystemUser, we should always restore the proper
context after execution. We were restoring an empty context for actions executed by the SystemUser
in the SecurityServerTransportInterceptor.
In order to accomplish this, a few changes have been made. Both the SecurityServerTransportInterceptor
and the SecurityActionFilter delegate to `SecurityContext#executeAsUser` when a user switch is necessary.
Tests were added for this method to ensure that the consumer is executed as the correct user and the proper
user is restored.
While working on this, a few other cleanups were made:
* SecurityContext can never have a null CryptoService, so a null check was removed
* We no longer replace the user with the system user when the system user is already associated with the request
* The security transport interceptor checks the license state and if auth is not allowed, delegate and return
* The security transport interceptor sendWithUser method now requires authentication to be present or a hard
exception is thrown.
* The TransportFilters integration test has been deleted. This was integration test that relied on the ability to
get instances from a node and trace the execution. This has been replaced by additional unit tests in
ServerTransportFilterTests
Closeselastic/elasticsearch#3845
Original commit: elastic/x-pack-elasticsearch@d8bcb59cb7
This change removes the blocking notion from fetching the roles
from a remote index. This also removes the blocking client calls
that can potentially deadlock a request if executed on the transport
thread.
Relates to elastic/elasticsearch#3790
Original commit: elastic/x-pack-elasticsearch@c2eda39043
Jay Modi
Areek Zillur
Tanguy Leroux
Alexander Reelsen
Jason Tedor
Tim Vernum
Simon Willnauer
Adrien Grand
Nik Everett
jaymode
Boaz Leskes
Ryan Ernst
Jack Conradson
Yannick Welsch
Chris Earle
Clinton Gormley
Luca Cavanna
Simon Willnauer