We frequently have support requests to diagnose LDAP realm problems.
One of the tools that would be useful in those cases is to be able to turn on trace logging and be able to see the LDAP searches and their results
Original commit: elastic/x-pack-elasticsearch@632d8e4f19
This commit marks ActionThrottleTests#testFailingActionDoesGetThrottled
as awaits fix as this test fails reliably.
Original commit: elastic/x-pack-elasticsearch@465c48b603
Changes required are for two reasons:
1) SearchResponse is a self contained valid object which doesn't need to be wrapped in a new object anymore
2) RestToXContentBuilderListener requires ToXContentObject, hence GraphExploreResponse needs to be moved over
Original commit: elastic/x-pack-elasticsearch@12277d0220
This commit renames the current AgentService into MonitoringService and changes the way it works: it was previously based on thread sleeping and it now use thread scheduling instead.
At every given time interval, a MonitoringExecution is executed. It first checks if monitoring data can be collected and if so it will collect data from Collectors and then export the data using the Exporters. There are cases where the data cannot be collected: when the service is stopping, when the interval has been set to -1 after the MonitoringExecution has been scheduled, or when the previous data collection is not yet terminated. In this last case, MonitoringExecution will still be executed at the given interval but will not collect any data.
All tasks are executed on the generic thread pool.
closeselastic/elasticsearch#2866
Original commit: elastic/x-pack-elasticsearch@d37b4d3731
Watcher: Use Apache HttpClient for internal Watcher HttpClient
The current implementation based on URLConnection has several drawbacks.
* If server returned HTTP header but then got stuck, no timeout would help, the connection remained stuck
* GET requests with a body were not supported, the method was silently changed to POST
* More complex handling of input/error stream handling, the body could not be read from a single input stream
NOTE: This is a BWC breaker. From now on every part of the URL needs to be encoded properly before it is configured in the requeust builder. This requires an upgrade of all watches.
Closeselastic/elasticsearch#1141
Original commit: elastic/x-pack-elasticsearch@bbc8f85dd8
An integration test was failing due to the possibility of the
.watch index being an alias (which only happens rarely).
Original commit: elastic/x-pack-elasticsearch@d53a4ebdd4
Watcher: Remove async code in sync code block
When removing the watch store this code snippet sneaked in.
A call to get a watch in order to find out if it exists was async, but
the code checking the result was called immediately afterwards without
waiting for the result, thus always using the default value.
This also removes some unused code in TriggeredWatchStore.
Original commit: elastic/x-pack-elasticsearch@c47e70bf8a
Add `logstash`, in addition to `kibana` type mappings if they are missing from the _existing_ .monitoring-data-2 indices. This allows existing indices to be upgraded in place to support the Logstash monitoring feature added in 5.2 without having to delete the .monitoring-data-2 index.
Original commit: elastic/x-pack-elasticsearch@864343a116
* [Monitoring] BWC Allow New Types to be added to .monitoring-data-2 index
This adds the new Logstash (and technically Kibana) types to the index mapping if they're not already there. Pre-existing indices will be blocked from creating new types in the index due to the index setting. The index setting cannot be flipped without opening/closing the index, so manually adding the new types is easier.
Original commit: elastic/x-pack-elasticsearch@e85e800335
Currently, roles making use of field or document level security are still applied when
the license level does not enable field and document level security. There is no indication
that these roles are not being applied so it is misleading to users. This change prevents
these roles for applying to authorization and also adds a transient metadata to the response
that indicates which features of a role is unlicensed.
Additionally, this PR prevents the addition or modification of roles to include field or
document level security.
Closeselastic/elasticsearch#2472
Original commit: elastic/x-pack-elasticsearch@c9455958f5
The interval value set is "minimumShouldMatch" and also the documented query dsl
parameter is `minimum_should_match` so we can deprecate
`minimum_number_should_match` and its getters/setters in 5.x and remove them in 6.
Original commit: elastic/x-pack-elasticsearch@53ce64b7b3
* [Monitoring] Use Cgroup Data from _nodes/stats
This adds Cgroup data from _nodes/stats.
- This also adds 5m and 15m from system load, which are sent from ES, but
were not recorded. Kibana does record/report those values though.
Original commit: elastic/x-pack-elasticsearch@8e04452c60
This PR adds UI for visualizing Logstash internal metrics with contributions from @ph, @jsvd and @pickypg
Changes include:
* x-pack component for Logstash which installs a monitoring exporter plugin to Logstash core. This periodically ships
monitoring data to the monitoring bulk API endpoint.
* Adds xpack.monitoring.* setting to logstash.yml when x-pack is installed.
* UI changes to graph Logstash monitoring data. Logstash processes are called nodes. They send separate monitoring info
which gets aggregated and displayed at cluster level and also at individual node level.
* Adds gradle build support for Logstash x-pack which can be controled via `xpack.logstash.build=false`
Fixeselastic/elasticsearch#4169
Original commit: elastic/x-pack-elasticsearch@f58ef406c1
When migrating realms to use the new settings for SSL, the removal of the
hostname_verification setting was missed in addition to actually using the
ssl.verification_mode setting. This change deprecates the
hostname_verification and enables the use of the ssl.verification_mode setting.
Original commit: elastic/x-pack-elasticsearch@ea24e83fd3
This PR changes how we use roles and how we look at the roles of a user. Previously we looked up each role individually, parsed each into their own `Role` object, and had a wrapper that essentially served as an iterator over the roles. The same pattern was also followed for the permissions that composed a role (ClusterPermission, IndicesPermission, and RunAsPermission). This resulted in a lot of code that was hard to follow and could be inefficient.
Now, we look up the roles for a user in bulk and only get the RoleDescriptor for each role. Once all role descriptors have been retrieved, we build a single Role that represents the user's permissions and we also cache this combination for better performance as authorization can happen many times for a single top level request as we authorize the top level request and any sub requests, which could be a large number in the case of shard requests.
This change also enabled a large cleanup of our permission and privilege classes, which should reduce the footprint of what needs to be followed. Some of the notable changes are:
* Consolidation of GeneralPrivilege and AbstractAutomatonPrivilege into the Privilege class
* The DefaultRole class has been removed and the permissions it provided were moved into the AuthorizationService
* The GlobalPermission class was removed as there is a single role that represents a user's permissions
* The Global inner classes for the various permissions were removed
* The Core inner class was removed and ClusterPermission, IndexPermission, RunAsPermission became final classes instead of interfaces
* The Permission interface has been removed. The isEmpty() method defined by this interface is not needed as we can simply evaluate the permission to get the same effect
* The ClusterPermission#check method only takes the action name again
* The AutomatonPredicate class was removed and replaced by Automatons#predicate
* IndicesAccessControl objects no longer need to be merged when evaluating permissions
* MergedFieldPermissions has been removed
* The Name class that was used to hold an array of strings has been removed and replaced with the use of a Set
* Privilege resolution is more efficient by only combining automata once
Other items:
* NativeRolesStore no longer does caching, so the RoleAndVersion class could be removed
* FileRolesStore doesn't need to be an AbstractLifecycleComponent
Relates elastic/elasticsearch#4327
Original commit: elastic/x-pack-elasticsearch@c1901bc82e
When attempting to a get a watch that does not exist, the listener is called once inside
an if statement and the execution of the method continues as if the watch existed. This
causes failures to happen including a NPE. This commit wraps the execution in a if-else
to prevent this.
Original commit: elastic/x-pack-elasticsearch@27f09852e7
We are trying to get rid of guava, so it doesn't make much sense to use it as a dependency unless we really need to.
Original commit: elastic/x-pack-elasticsearch@e2e2d58fba
Switches custom cluster state components from PROTO-based de-serialization to named objects based de-serialization
Original commit: elastic/x-pack-elasticsearch@e1be8597ee
In https://github.com/elastic/elasticsearch/pull/21964, index
and delete operations are executed as single item bulk requests
internally. This means index and delete operations use the
bulk transport endpoints (indices:data/write/bulk[s][p] and
indices:data/write/bulk[s][r]).
This PR adds bulk transport endpoint to 'write' and 'delete'
index privilages and adds index and delete action as composite
actions to delay the authentication to the shard level.
Original commit: elastic/x-pack-elasticsearch@2305fc9ca0
This commit adapts to the change of the method name
ESTestCase#randomPostivieLong to ESTestCase#randomNonNegativeLong.
Original commit: elastic/x-pack-elasticsearch@689429cb54
This "super registry" will eventually replace things like
`IndiciesQueriesRegistry` but for now it is just another thing
to plumb across requests.
Original commit: elastic/x-pack-elasticsearch@da26a42b36
When the index action is used to do some bulk indexing, the single
items of the response were not checked to have been indexed successful.
This could lead to NPEs due to an index response being null when the index
operation had failed. The action was still logged as a success though.
This commit only returns SUCCESS for the action, if all items were indexed
successfully. If all items failed, the result will be FAILED as well. Lastly
the result status PARTIAL_FAILURE is used if there were successful and unsuccessful
index operations.
Additionally some minor cleanups happened, like changing equals/hashcode.
Closeselastic/elasticsearch#4416
Original commit: elastic/x-pack-elasticsearch@692687e1af
Removing the WatchLockService could result in duplication of wids, because of a wrong
call to replace underscores with dashes. As UUIDs.createBase64UUID() can contain underscores
but they are kind of reserved in the Wid class due to handling of watch ids, this just uses
the toString() representation of a random UUID.
Closeselastic/elasticsearch#4422
Original commit: elastic/x-pack-elasticsearch@dceb01ae5e
This test fails due to a changing cluster state(?). The
test checks that a local exporter is ready and then continues.
However, during the test, we see output similar to:
skipping exporter [_local] as it isn't ready yet
Which indicates that the cluster state has changed and the
exporter does not return a bulk anymore. Hence, the test is
failing although at one point in time it returned a bulk.
By enabling trace logging we should be able to find out
what's going on.
Original commit: elastic/x-pack-elasticsearch@d7e2200dd9
The watch lock service is not really needed, as there is already
a data structure that has information about the currently executing
watches, that can be consulted before executed.
This change will now check, if there is already a watch running with
the current id. If there is not, execution will happen as usual. If
there is however, than a watch record will be created, stating that
the watch is currently being executed - which means that it is either
being executed or in the list of planned executions.
This way users can check in the watch history, if a watch has been executed
more often than it should.
In order to easily search for this, a new execution state called
`NOT_EXECUTED_ALREADY_QUEUED` has been added.
Original commit: elastic/x-pack-elasticsearch@867acec3c3
In LoggingAuditTrail and the audit-event filtering, we distinguish between single-realm authentication failures ("realm_authentication_failure") and global failures ("authentication_failure").
Update the message output of IndexAuditTrail to reflect this distinction.
Original commit: elastic/x-pack-elasticsearch@56802ae2df
With this commit we enable the Jackson feature 'STRICT_DUPLICATE_DETECTION'
by default for all XContent types (not only JSON).
We have also changed the name of the system property to disable this feature
from `es.json.strict_duplicate_detection` to the now more appropriate name
`es.xcontent.strict_duplicate_detection`.
Relates elastic/elasticsearchelastic/elasticsearch#19614
Relates elastic/elasticsearchelastic/elasticsearch#22073
Original commit: elastic/x-pack-elasticsearch@c92e2a6c55
This PR removes all leniency in the conversion of Strings to booleans: "true"
is converted to the boolean value `true`, "false" is converted to the boolean
value `false`. Everything else raises an error.
Original commit: elastic/x-pack-elasticsearch@6400f18911
* Build: Convert xplugins to use new extra projects setup
This change makes the gradle initialization for xplugins look in the
correct location for elasticsearch, which is now as a sibling of an
elasticsearch-extra directory, with x-plugins as a child of the extra
directory.
The elasticsearch side of this change is
elastic/elasticsearchelastic/elasticsearch#21773. This change will enable renaming x-plugins
to x-pack, see elastic/elasticsearch#3643.
Original commit: elastic/x-pack-elasticsearch@09398aea5a
The new MockWebServer assumed that responses are inserted at the beginning
and removed later on. This was not thread safe.
Also this fixes a bug in the HttpExporterIT where there was no wait time
for a bulk request, even though the request execution is asynchronous.
Closeselastic/elasticsearch#4335
Original commit: elastic/x-pack-elasticsearch@11f31f68bd
In order to prepare to the distributed watch execution, this commit
removes the in memory watch store.
Whenever a watch is needed now, a get request is executed and the parsing
is done. This happens when
* Put
* Get
* Ack
* Activate/Deactivate
* Execute
Note: This also means there are no usage stats currently regarding
the watch count, because we would need to execute a query. This would
require the usage stats to be async, see elastic/elasticsearch#3569
Another advantage is, that there is no dirty flag in the watch itself
needed anymore, because the watch is always the latest. Also write
operations store immediately and dont leave anything in memory.
Also ActionListener.wrap() was used a lot instead of more verbose anonmyous
inner classes.
Original commit: elastic/x-pack-elasticsearch@c47465b47c
Modified the definition and loading of settings in Security to provide early detection and failure of invalid (unrecognised or syntactically invalid) settings.
Also consolidates the number of places where settings were defined.
Each realm now defines its supported settings.
This is facilitated for custom realms via a new "getRealmSettings" method on XPackExtension.
The realm group setting performs validation of the child settings with reference to the "type".
For backwards compatibility reasons, realm types that have no configuration defined, will be accepted during validation, but may fail at realm creation time.
All SSL settings have been centralised into SSLConfigurationSettings, which supports a variable "prefix" to accommodate the multiple places we define SSL config.
HTTP Proxy settings are explicitly defined rather than being a generic group.
Where possible all security settings now reference a Setting object, and there are less magic strings scattered in the code.
Closes: elastic/elasticsearch#3965
Original commit: elastic/x-pack-elasticsearch@2c76a137a9
In the triggered watch store a list of integers was returned to keep track
which watches had been successfully stored and thus should be executed.
This means, that an arraylist, plus autoboxing/unboxing needs to be done for
all the triggered watches. This data structure can easily be replaced with a BitSet,
resulting in much less objects being created or parsed - also it's a bit faster.
Original commit: elastic/x-pack-elasticsearch@e9fba67e34
This change removes the deprecated methods from the realm class. These methods include
blocking authentication and lookup and the lookup supported method.
Original commit: elastic/x-pack-elasticsearch@cff21e21ee
With this commit we enable the Jackson feature 'STRICT_DUPLICATE_DETECTION'
by default. This ensures that JSON keys are always unique. While this has
a performance impact, benchmarking has indicated that the typical drop in
indexing throughput is around 1 - 2%.
As a last resort, we allow users to still disable strict duplicate checks
by setting `-Des.json.strict_duplicate_detection=false` which is
intentionally undocumented.
Relates elastic/elasticsearchelastic/elasticsearch#19614
Original commit: elastic/x-pack-elasticsearch@cced57b884
The latest release of the mock web server requires more security permissions, and
we dont need all the functionality anyway.
This introduces a small MockWebServer using the JDK internal HttpServer, yet fullfilling
all our needs and supporting SSL as well for testing.
The MockWebServer allows to enqueue responses and also requires you to enqueue as many responses
as you requests will be executed - there is no fallback at the moment.
SSL is also supported by passing an SSL context - for which the TestsSSLService is needed, which
makes the required methods public.
Original commit: elastic/x-pack-elasticsearch@55f4a172a2
Now when a cluster gets updated with a new license,
the license uid and type will be logged as info. Making
it easier to verify the current license mode of the cluster.
Original commit: elastic/x-pack-elasticsearch@c205cb42ae
There were some test failures as LicenseServiceClusterTests was not properly setting minimum_master_nodes while having autoMinMasterNodes disabled.
Original commit: elastic/x-pack-elasticsearch@2030db2424
A companion PR to https://github.com/elastic/elasticsearch/pull/21846 where the above methods were removed. See ES PR for details.
With the concurrent starting the issues with licenses and time freeze became more apparent and I had to apply my suggestion to only freeze time once the license has been applied (as opposed to freeze on node start up). Since this also means that a node that starts up after the cluster time has been frozen need to also immediately freeze, it felt natural to use a `ServiceDisruptionScheme`. Although the name doesn't really make sense here, it all has all the logic we need.
Original commit: elastic/x-pack-elasticsearch@5641742f60
This commit migrates the actual authentication calls that are performed by internal realms
to be asynchronous so that we do not execute blocking calls on network threads. The realms
that used LDAP were especially bad as they can issue many different requests and/or open
multiple connections per authentication attempt.
The LdapRealm now uses the ThreadPool to fork a thread for authentication. This is necessary
because a LDAP bind operation is not allowed to be performed asynchronously and must be done
in a blocking manner. After the bind request has completed, all other requests can be done
asynchrnously. The LdapUtils class now provides asynchronous methods for searching and also
includes the handling of referral following as automated following is not supported by
UnboundID when using the asynchronous calls.
Original commit: elastic/x-pack-elasticsearch@dfb259e498
* [Watcher] HistoryStore should hide secrets
Some watcher actions like Jira stores the whole HTTP request in case of a failure. This is very helpful when something goes wrong, but it has to hide the password used for Basic authentication otherwise it will appear in the watch record in the .watcher-history index. In general, I would consider better to always hide secrets when storing something in the history index.
This commit removes the XContentParams "hide secrets" set at the JiraIssue so that it is now set at the HistoryStore level.
It also fixes test failures that can be reproduced with:
gradle :x-plugins:elasticsearch:test -Dtests.seed=BA730F93E726AECC -Dtests.class=org.elasticsearch.xpack.notification.jira.JiraIssueTests -Dtests.method="testToXContent" -Dtests.security.manager=true -Dtests.locale=es -Dtests.timezone=Asia/Choibalsan
Original commit: elastic/x-pack-elasticsearch@d686b3b3ad
The local exporter previously fired off asynchronous requests every time a cluster state was
observed that did not contain all of the required items for monitoring. This change adds a
flag so that monitoring can wait for the pending requests to complete before retrying. This
will reduce the number of duplicated log messages as well.
Additionally, the log message for adding modern aliases now contains the name of the indices.
Closeselastic/elasticsearch#3756
Original commit: elastic/x-pack-elasticsearch@727a0adfbe
* Convert security rest filter to rest handler wrapper
This is the xpack side of elastic/elasticsearchelastic/elasticsearch#21905
Original commit: elastic/x-pack-elasticsearch@38bfa771b6
This commit removes some unnecessary code: collectors do not need to extend AbstractLifecycleComponent and do not need to be started/stopped. The extra
interface Collector is removed and AbstractCollector renamed to Collector.
Original commit: elastic/x-pack-elasticsearch@75893d66e3
Tim Vernum
Jason Tedor
javanna
Tim B
Tanguy Leroux
Alexander Reelsen
Chris Earle
Chris Earle
Jay Modi
Daniel Mitterdorfer
Christoph Büscher
Luca Cavanna
jaymode
Adrien Grand
Igor Motov
Yannick Welsch
Nik Everett
Areek Zillur
Ryan Ernst
Daniel Mitterdorfer
Simon Willnauer
Boaz Leskes
Jim Ferenczi