The new MockWebServer assumed that responses are inserted at the beginning
and removed later on. This was not thread safe.
Also this fixes a bug in the HttpExporterIT where there was no wait time
for a bulk request, even though the request execution is asynchronous.
Closeselastic/elasticsearch#4335
Original commit: elastic/x-pack-elasticsearch@11f31f68bd
In order to prepare to the distributed watch execution, this commit
removes the in memory watch store.
Whenever a watch is needed now, a get request is executed and the parsing
is done. This happens when
* Put
* Get
* Ack
* Activate/Deactivate
* Execute
Note: This also means there are no usage stats currently regarding
the watch count, because we would need to execute a query. This would
require the usage stats to be async, see elastic/elasticsearch#3569
Another advantage is, that there is no dirty flag in the watch itself
needed anymore, because the watch is always the latest. Also write
operations store immediately and dont leave anything in memory.
Also ActionListener.wrap() was used a lot instead of more verbose anonmyous
inner classes.
Original commit: elastic/x-pack-elasticsearch@c47465b47c
Modified the definition and loading of settings in Security to provide early detection and failure of invalid (unrecognised or syntactically invalid) settings.
Also consolidates the number of places where settings were defined.
Each realm now defines its supported settings.
This is facilitated for custom realms via a new "getRealmSettings" method on XPackExtension.
The realm group setting performs validation of the child settings with reference to the "type".
For backwards compatibility reasons, realm types that have no configuration defined, will be accepted during validation, but may fail at realm creation time.
All SSL settings have been centralised into SSLConfigurationSettings, which supports a variable "prefix" to accommodate the multiple places we define SSL config.
HTTP Proxy settings are explicitly defined rather than being a generic group.
Where possible all security settings now reference a Setting object, and there are less magic strings scattered in the code.
Closes: elastic/elasticsearch#3965
Original commit: elastic/x-pack-elasticsearch@2c76a137a9
In the triggered watch store a list of integers was returned to keep track
which watches had been successfully stored and thus should be executed.
This means, that an arraylist, plus autoboxing/unboxing needs to be done for
all the triggered watches. This data structure can easily be replaced with a BitSet,
resulting in much less objects being created or parsed - also it's a bit faster.
Original commit: elastic/x-pack-elasticsearch@e9fba67e34
This change removes the deprecated methods from the realm class. These methods include
blocking authentication and lookup and the lookup supported method.
Original commit: elastic/x-pack-elasticsearch@cff21e21ee
With this commit we enable the Jackson feature 'STRICT_DUPLICATE_DETECTION'
by default. This ensures that JSON keys are always unique. While this has
a performance impact, benchmarking has indicated that the typical drop in
indexing throughput is around 1 - 2%.
As a last resort, we allow users to still disable strict duplicate checks
by setting `-Des.json.strict_duplicate_detection=false` which is
intentionally undocumented.
Relates elastic/elasticsearchelastic/elasticsearch#19614
Original commit: elastic/x-pack-elasticsearch@cced57b884
The latest release of the mock web server requires more security permissions, and
we dont need all the functionality anyway.
This introduces a small MockWebServer using the JDK internal HttpServer, yet fullfilling
all our needs and supporting SSL as well for testing.
The MockWebServer allows to enqueue responses and also requires you to enqueue as many responses
as you requests will be executed - there is no fallback at the moment.
SSL is also supported by passing an SSL context - for which the TestsSSLService is needed, which
makes the required methods public.
Original commit: elastic/x-pack-elasticsearch@55f4a172a2
Now when a cluster gets updated with a new license,
the license uid and type will be logged as info. Making
it easier to verify the current license mode of the cluster.
Original commit: elastic/x-pack-elasticsearch@c205cb42ae
There were some test failures as LicenseServiceClusterTests was not properly setting minimum_master_nodes while having autoMinMasterNodes disabled.
Original commit: elastic/x-pack-elasticsearch@2030db2424
A companion PR to https://github.com/elastic/elasticsearch/pull/21846 where the above methods were removed. See ES PR for details.
With the concurrent starting the issues with licenses and time freeze became more apparent and I had to apply my suggestion to only freeze time once the license has been applied (as opposed to freeze on node start up). Since this also means that a node that starts up after the cluster time has been frozen need to also immediately freeze, it felt natural to use a `ServiceDisruptionScheme`. Although the name doesn't really make sense here, it all has all the logic we need.
Original commit: elastic/x-pack-elasticsearch@5641742f60
This commit migrates the actual authentication calls that are performed by internal realms
to be asynchronous so that we do not execute blocking calls on network threads. The realms
that used LDAP were especially bad as they can issue many different requests and/or open
multiple connections per authentication attempt.
The LdapRealm now uses the ThreadPool to fork a thread for authentication. This is necessary
because a LDAP bind operation is not allowed to be performed asynchronously and must be done
in a blocking manner. After the bind request has completed, all other requests can be done
asynchrnously. The LdapUtils class now provides asynchronous methods for searching and also
includes the handling of referral following as automated following is not supported by
UnboundID when using the asynchronous calls.
Original commit: elastic/x-pack-elasticsearch@dfb259e498
* [Watcher] HistoryStore should hide secrets
Some watcher actions like Jira stores the whole HTTP request in case of a failure. This is very helpful when something goes wrong, but it has to hide the password used for Basic authentication otherwise it will appear in the watch record in the .watcher-history index. In general, I would consider better to always hide secrets when storing something in the history index.
This commit removes the XContentParams "hide secrets" set at the JiraIssue so that it is now set at the HistoryStore level.
It also fixes test failures that can be reproduced with:
gradle :x-plugins:elasticsearch:test -Dtests.seed=BA730F93E726AECC -Dtests.class=org.elasticsearch.xpack.notification.jira.JiraIssueTests -Dtests.method="testToXContent" -Dtests.security.manager=true -Dtests.locale=es -Dtests.timezone=Asia/Choibalsan
Original commit: elastic/x-pack-elasticsearch@d686b3b3ad
The local exporter previously fired off asynchronous requests every time a cluster state was
observed that did not contain all of the required items for monitoring. This change adds a
flag so that monitoring can wait for the pending requests to complete before retrying. This
will reduce the number of duplicated log messages as well.
Additionally, the log message for adding modern aliases now contains the name of the indices.
Closeselastic/elasticsearch#3756
Original commit: elastic/x-pack-elasticsearch@727a0adfbe
* Convert security rest filter to rest handler wrapper
This is the xpack side of elastic/elasticsearchelastic/elasticsearch#21905
Original commit: elastic/x-pack-elasticsearch@38bfa771b6
This commit removes some unnecessary code: collectors do not need to extend AbstractLifecycleComponent and do not need to be started/stopped. The extra
interface Collector is removed and AbstractCollector renamed to Collector.
Original commit: elastic/x-pack-elasticsearch@75893d66e3
Transform and condition scripts were only compiled on its initial creation, so
when a new watch is created or when the master node loads all the watches. However
changing a script (like a stored one) did not lead to any changes in the in memory
watch store and thus the old script was executed again.
We do however have a mechanism in Elasticsearch's ScriptService that already does some
caching, and should reuse that one.
Closeselastic/elasticsearch#4237
Original commit: elastic/x-pack-elasticsearch@477548e237
This commit improves the output of the migrate tool in cases when there are errors parsing entries
in the roles or users files. This is done through the use of a logger that delegates its output to
the terminal.
Additionally, the `-c` option has been removed. This option was used to set the configuration directory
but this should be handled one way only and that is through the use of the `-Epath.conf` setting.
Closeselastic/elasticsearch#3757Closeselastic/elasticsearch#3758
Original commit: elastic/x-pack-elasticsearch@811e367766
Currently we have these logs for integration tests only.
This adds the following log at the start:
```
logger.info("[{}]: before test", getTestName());
```
and this is logged at the end, but before any clean up done in sub classes
```
logger.info("[{}]: after test", getTestName());
```
Original commit: elastic/x-pack-elasticsearch@2ca7296665
It may take a few clicks before the templates are added.
Waiting for started doesn't guarantee that the index templates have been added.
Original commit: elastic/x-pack-elasticsearch@ee94b740a8
This commit adds tests for DNS only hostname verification. This is a followup of elastic/elasticsearchelastic/elasticsearch#21828, which fixes issues with this type of hostname verification
due to some addresses losing the host information.
Original commit: elastic/x-pack-elasticsearch@8a63bb113d
When we encounter a composite request, we authorize at first without looking at the indices, to see whether the action can be executed at all. We then rely on the action to delegate to an inner action per sub-request, which will be authorized based on the indices it refers to. The first step works great for the simulate mode of search template, as it doesn't involve any index. The second step will make sure that when search template involves a search, it will be authorized as a normal search request would, based on the indices it reads from.
Note that the wildcard expansion happens now on the search side, it doesn't have to happen when executing the first authorization step, hence SearchTemplateRequest doesn't have to implement IndicesRequest, only SearchRequest has to (which it does already).
Closeselastic/elasticsearch#4171
Original commit: elastic/x-pack-elasticsearch@d586bd90cb
This commit applies the same fix merged in elastic/elasticsearch#4179 for XDocsClientYamlTestSuiteIT. It adds a waitForSecurityTemplate() method in order to wait for the security-index-template to be created by the SecurityTemplateService.
Original commit: elastic/x-pack-elasticsearch@1476f30e2d
This commit moves the iteration of realms for authentication and user lookup to
be done in an asynchronous fashion. The existing blocking methods have been deprecated
to allow custom realm implementors time to switch. All internal realms implement the
asynchronous methods.
This PR is another step toward the full migration to async authentication, but does not
complete the work. Additional work is needed for the LDAP realms, which make blocking
network calls. These blocking calls will be handled in a follow-up PR.
See elastic/elasticsearch#3790
Original commit: elastic/x-pack-elasticsearch@a65a9b2bb4
If the HTTP response is an invalid one, it is still logged as success.
This commit changes the behaviour, that if the response status code is
set to -1 (which means it could not be interpreted), than an IOException
is thrown and thus the execution will be marked as a failure.
Closeselastic/elasticsearch#4152
Original commit: elastic/x-pack-elasticsearch@5736fbe3c0
* Do not try bind to port range but use free ephemeral port
* Start a new email server in all tests, do not use static one
* Remove selection of username/password, as it was static anyway
* Remove Listener.Handle class, as it is not needed, when not running in static context
Original commit: elastic/x-pack-elasticsearch@8816cc25f6
OldMonitoringIndicesBackwardsCompatibilityTests fails because it waits for more shards stats to be collected but that can only work if new indices are created in the meanwhile.
Original commit: elastic/x-pack-elasticsearch@003c28cf93
This commit updates the watch_history.json file so that it includes mappings for the new Jira action. It also update the JiraIssue format so that it now includes the name of the account used to create the Jira issue. It also update the REST tests to check that Jira action result are searchable and hide the user's password.
Original commit: elastic/x-pack-elasticsearch@75888f7748
This commit adds tests to ensure that user lookup in caching realms works as expected. An unclear
contract in the Cache#computeIfAbsent method allowed for null values to be returned from this method
even if there should have been exception reported to the loader. This has been fixed in the cache
implementation and we add tests to verify that the caching of user lookups is done properly under
concurrent operations.
Closeselastic/elasticsearch#4054
Original commit: elastic/x-pack-elasticsearch@41567c6ed9
This change allows reads of our native users and roles when the template version has not been updated to
match the current version. This is useful for rolling upgrades where the nodes are also being actively
queried and/or indexed into. Without this, we can wreak havoc on a cluster by causing exceptions during
replication, which leads to shard failures. On nodes that match the version defined in the template,
write operations are allowed since we know that we are backwards compatible in terms of format but we
may have added new fields and shouldn't index them until the mappings and template have been updated.
As part of this, the rolling upgrade tests from core were used as the basis for a very basic set of tests
for doing a rolling upgrade with x-pack.
Closeselastic/elasticsearch#4126
Original commit: elastic/x-pack-elasticsearch@9be518ef00
Currently, a tribe node ignored underlying cluster licenses
due to inablity to select an appropriate license from multiple
licenses. Now that tribe node supports merging custom metadata
(elasticsearch#elastic/elasticsearch#21552), we can enable license support in tribe
node.
Now, tribe node chooses license with the highest operation
mode from underlying cluster licenses. This commit also
adds integration tests for licensing to verify that:
- autogenerated trial license propagates to tribe node
- tribe node chooses the highest operation mode license
- removing a license from underlying cluster license is
removed from tribe
closeselastic/elasticsearch#3212
Original commit: elastic/x-pack-elasticsearch@b5c003decd
This commit changes the toXContent() method so that it generates content that can be parsed again by the parse() method. Before this commit, the authentication of the HTTP request is rendered as:
{ "auth": {"username": "foo", "password": "bar" } }
but the parsing method expects the authentication type to be a root object:
{ "auth": { "basic" : {"username": "foo", "password": "bar" } } }
The toXContent method has been adapted to include the type of authentication in the generated content.
Original commit: elastic/x-pack-elasticsearch@b740466109
This removes the mock webserver trying to bind to a port range, just
try to bind to any port.
Also removed unneeded wrapper class in the process.
Original commit: elastic/x-pack-elasticsearch@19b6ac393a
When a fatal error is thrown on the network layer, such an error never
makes its way to the uncaught exception handler. This prevents the node
from being torn down if an out of memory error or other fatal error is
thrown while handling HTTP or transport traffic. This commit adds logic
to ensure that such errors bubble their way up to the uncaught exception
handler, even though Netty tries really hard to swallow everything.
Original commit: elastic/x-pack-elasticsearch@f76757674f
This commit aims to fix the regularly failing test OldMonitoringIndicesBackwardsCompatibilityTests. It enables/disables monitoring data collection using time interval "-1" as well as stopping AgentService instances directly and checks multiple times if indices are still present at the end of the test.
closeselastic/elasticsearch#3999
Original commit: elastic/x-pack-elasticsearch@8ac785061a
* Record audit trail even if indicies-resolver throws exception
If the IndicesAndAliasesResolver throws an exception, treat is as "accessDenied" for the purpose of the audit-trail.
This can occur when an index request has a wildcard that doesn't match (and "allowNoIndices" is false)
Closeselastic/elasticsearch#3719
Original commit: elastic/x-pack-elasticsearch@ca6567e5ed
RemoteIndexAuditTrailStartingTests has been failing quite often due to several
race conditions. This change simplifies the test (one node per cluster, local and
remote) and fixes some issues in IndexAuditTrail. The interrupt based shutdown has
been removed from the `QueueConsumer` and an additional health call has been added
to trigger another start attempt if the first one failed. This stabilize the test
that now run in 1 second rather than 1 minute.
Original commit: elastic/x-pack-elasticsearch@a1c8131cd2
This commit responds to a chance in core which modified the interface
for ThreadPool#executor and ThreadPool#generic to return an executor
service rather than an executor.
Relates elastic/elasticsearch#4107
Original commit: elastic/x-pack-elasticsearch@7c7c6a3b90
This is a followup from elastic/elasticsearchelastic/elasticsearch#21590 and needs to be
committed first or at the same time since netty_3 is removed
Original commit: elastic/x-pack-elasticsearch@131d74dd6b
If these tests were in `:core` then then `IT` would be an appropriate
suffix, but in xplugins they should have `Tests`. This moves them
from the `integTest` task to the `test` task which is a good idea
because it lets them run in parallel with the other tests. Naming
them `IT` means that they are run not in parallel and they are run
with a running instance of Elasticsearch which they don't need.
This cuts the build from 15.5 minutes to 13.5 minutes for me.
Original commit: elastic/x-pack-elasticsearch@7b4bcf8bc5
This commit fixes a compilation issue introduced by the merge of
sequence numbers branch into master. This merge introduced a new
constructor parameter for ShardStats.
Original commit: elastic/x-pack-elasticsearch@b234b8ee50
This commit migrates the authentication service to an asynchronous model where we use listeners
instead of blocking and waiting for the authentication to return. This is the first part of making
authentication asynchronous as we still have blocking I/O inside of realms.
See elastic/elasticsearch#3790
Original commit: elastic/x-pack-elasticsearch@9339af4af8
In the authentication service, we currently only copy the username and roles of the
user that was authenticated but we should instead preserve all of their information
in the newly created user object. This change does that through the user of a new
constructor in the user class that takes in both users.
Closeselastic/elasticsearch#3877
Original commit: elastic/x-pack-elasticsearch@7455078841
If a authenticated user with run as permission attempts to run as an unknown user, the unknown
user will be assigned the default role and anonymous role if enabled. This change prevents this
from happening as we require the run as user to have been looked up by a realm.
Closeselastic/elasticsearch#3878
Original commit: elastic/x-pack-elasticsearch@034f44757d
The internal test cluster now auto manages min_master_nodes ( see https://github.com/elastic/elasticsearch/pull/21458 ). This requires some code changes but also changes the timings of forming a cluster. This has had a funny side effect where the master is no longer always the first node to be started in the cluster. This caused issues with watcher tests which freeze time.
Original commit: elastic/x-pack-elasticsearch@1e5ea8ae94
This change mocks the ClusterService and sets it up so that when remote indexing, we see the
current node as the master.
Original commit: elastic/x-pack-elasticsearch@1f26b3df1f
Nik Everett
Jason Tedor
Alexander Reelsen
Daniel Mitterdorfer
Luca Cavanna
Tim Vernum
javanna
Simon Willnauer
Jay Modi
Ryan Ernst
Tanguy Leroux
Areek Zillur
Yannick Welsch
jaymode
Jim Ferenczi
Boaz Leskes
Simon Willnauer
Martijn van Groningen
Adrien Grand