In order to prevent exceptions to bubble up to the thread pool exception
handler, this properly wraps all the calls for the watcher service
within an executor into an AbstractRunnable to catch and log a possible
exception.
relates elastic/x-pack-elasticsearch#3854
Original commit: elastic/x-pack-elasticsearch@c0b39e6b5b
* X-Pack-Security: Improve error messaging during setup password
Changes are done to improve logging and exception messages when
setup passwords. Now it checks whether
- X-Pack plugin is available
- X-Pack Security is available
- X-Pack Security is enabled
by invoking /_xpack query and throws a specific error message.
1. Modifications to simple http client to return http response.
2. Changes in SetupPasswordTool for throwing appropriate exceptions.
2. Unit tests
relates elastic/x-pack-elasticsearch#3476
Original commit: elastic/x-pack-elasticsearch@eb0c5a19c5
We were missing a notification for when a job is updated. This is
useful so users know that there's been changes which could justify
a change in the job behaviour.
In addition, having those notifications allows our integrations
tests to know when the update was processed which avoids having
to use `sleep()` with its instabilities.
Original commit: elastic/x-pack-elasticsearch@0b4eda2232
Consolidates type handling into DataType, makes DataType available to
JDBC by moving to sql-proto and removes support for all parameter types
that cannot be handled by the server.
Original commit: elastic/x-pack-elasticsearch@b8024f5c46
This commit introduces a new watcher setting to defer starting watcher
until it has been called with the API for the first time. This is
primarily useful in testing environments, as this ensures that watcher
does not try to reload itself because of starting first and then
creating watcher indices.
In addition the undocumented and unused option
xpack.watcher.start_immediately has been removed.
Relates elastic/x-pack-elasticsearch#3854
Original commit: elastic/x-pack-elasticsearch@2b55aec4ad
This fixes a regression introduced in Elasticsearch 6.0, when switching
from HttpURLConnection to the Apache HTTP Client.
In the old implementation there was no way to specify if you wanted to use HTTP
or HTTPS for your proxy, only HTTP. If people needed to use HTTPs, they
could just use the CONNECT feature of the proxy.
The new implementation used the scheme of the request that was about to
be sent out as the proxy scheme to be used. So if the request was HTTPS
but the proxy server was HTTP this created a problem.
This commit changes the default scheme to be just HTTP, so that then the
standard CONNECT procecure is taken care off.
Without a real proxy server this is super hard to test. I have verified
this with the following test against a tinyproxy running on port 8888,
but I do not have a great idea how to test this in a unit testable way using a real proxy.
Original commit: elastic/x-pack-elasticsearch@f68e72d8f1
The watcher index uses auto expanding of replicas nowadays, so there is
no need to set the number of replicas.
Also ensuring that all watcher indices are green on startup has been moved
to one call, in order to prevent indices being moved around after adding
the triggered watches or watch history index.
Original commit: elastic/x-pack-elasticsearch@dbe04adf64
* Add fields to `.logstash`'s mapping in template
This "makes room" in the index for pipeline settings and node groups. Due to this change, users will be able to specify settings and node groups for a pipeline via the Centralized Config Management UI in Kibana. Logstash will only retrieve pipelines associated with the node group specified via the `xpack.management.group.id` setting in `logstash.yml`. For the retrieved pipelines, Logstash will apply any (optionally) specified pipeline settings before (re)loading the pipelines.
* Making field name more explicit + adding multi field for better search
Original commit: elastic/x-pack-elasticsearch@2df101f0b1
This adds the `HEADERS_SETTING` as setting reported by the Monitoring plugin, which was changed in 6.2.
It also adds an IT to ensure that it is not missed in the future (existing integration tests were passing in the Settings and not being validated by the cluster!).
Original commit: elastic/x-pack-elasticsearch@d6f0b4b503
* SQL: fix name of Arithmetic functions
Remove id from name of Arithmetic function and improve name of Neg
Original commit: elastic/x-pack-elasticsearch@4f3e8d6a2d
Since elastic/x-pack-elasticsearch#3254 security headers have been stored in datafeed cluster state
to allow the datafeed to run searches using the credentials of the user
who created/updated it. As a result the parser was changed to read the
"headers" field so that cluster state could be reloaded. However, this
meant that datafeed configs could be submitted with a "headers" field.
No security loophole arose from this, as subsequent code overwrites the
contents of any supplied headers. But it could be confusing that an
erroneously supplied field did not cause a parse failure as it usually
would.
This change makes the config parser for datafeeds reject a "headers"
field. Now only the metadata parser used for reloading cluster state
will read a "headers" field.
Original commit: elastic/x-pack-elasticsearch@afa503275f
java.time features it's own halted clock, called a fixed clock, we can
use that one.
On top of that the watcher xcontent parser does not need a clock at all,
just a timestamp when parsing happened.
Original commit: elastic/x-pack-elasticsearch@2061aeffe1
The api jar was added for xpack extensions. However, extensions have
been removed in favor of using SPI, and the individual xpack jars like
core and security are published to enable this. This commit removes the
api jar, and switches the transport client to use the core jar (which
the api jar was just a rename of).
Original commit: elastic/x-pack-elasticsearch@58e069e66c
There were a number of leftover unnecessary elements in the module
build.gradle files that were holdovers from copying the original plugin
build.gradle. This commit removes these elements.
Original commit: elastic/x-pack-elasticsearch@08babbd520
Calling start() when already in the STARTING state doesn't do anything, so the component
gets stuck in STARTING state forever.
Also: wait on the required index name not just the cluster.
Also: added more logging to help diagnose such issues (either in RemoteIndexAuditTrailStartingTests or production)
Original commit: elastic/x-pack-elasticsearch@fb81214fe7
The old home made sense before x-pack was split. The new home lines up
with where security keeps its scripts.
The jar file wasn't being included any more so this re-adds that.
Original commit: elastic/x-pack-elasticsearch@d3ec941397
This commit adds special handling for null values when building a DistinguishedNamePredicate for
role mapping. Previously this would have resulted in an exception from the unboundid ldapsdk as
the DN is invalid.
relates elastic/x-pack-elasticsearch#3787
Original commit: elastic/x-pack-elasticsearch@9386dae03e
This allows any datetime function to be present in `EXTRACT` which feels
more consistent. `EXTRACT(FOO FROM bar)` is now just sugar for
`FOO(bar)`. This is *much* simpler to explain in the documentation then
"these 10 fields are supported by extract and they are the same as this
subset of the datetime functions."
The implementation of this is a little simpler then the old way. Instead
of resolving the function in the parser we create an
`UnresolvedFunction` that looks *almost* just like what we'd create for
a single argument function and resolve the function in the `Analyzer`.
This feels like a net positive as it allows us to group `EXTRACT`
resolution failures with other function resolution failures.
This also creates `UnresolvedFunctionTests` and
`UnresolvedAttributeTests`. I had to create `UnresolvedFunctionTests`
because `UnreolvedFunction` now has three boolean parameters which is
incompatible with the generic `NodeSubclassTests`'s requirement that all
ctor parameters be unique. I created `UnresolvedAttributeTests` because
I didn't want `UnresolvedFunctionTests` to call `NodeSubclassTests` and
figured that we'd want `UnresolvedAttributeTest` eventually and now felt
like as good a time as any.
Added a
Original commit: elastic/x-pack-elasticsearch@358aada308
If any of the follow take place on security index, then any cached role mappings
are potentially invalid and the associated realms need to clear any cached users.
- Index recovers from red
- Index is deleted
- Index becomes out-of-date / not-out-of-date
Original commit: elastic/x-pack-elasticsearch@1bcd86fcd4
The HTTP Exporter in Monitoring allowed users to set a timeout parameters
for the requests. When set, this was setting the `master_timeout` query parameter
in Bulk Requests. The problem is that Bulk Requests do not support this type of
timeout.
Original commit: elastic/x-pack-elasticsearch@9be194006e
This adds support to allow different schemes in a proxy being used
compared to what the actual request requires. So if your proxy runs via
HTTP, but the endpoint you want to connect to uses HTTPS, this is now
possible to configure the proxy explicitely.
Also a small unit test for parsing this has been added.
relates elastic/x-pack-elasticsearch#3596
Original commit: elastic/x-pack-elasticsearch@176f7cdf0e
Tests have been failing because of out of order cluster state processing
or because of stopping/starting was still in progress. Current tests do
not do further stop/start tries after the first leading to potential
being stuck in an unwanted state. This commit removes the methods
checking for the state being started or stopped in favor of using the
stopWatcher/startWatcher methods which now check if the desired state is
reached and otherwise issue another start/stop command.
Original commit: elastic/x-pack-elasticsearch@97b3232a6a
The notifier is scheduled to run once per second. Currently,
it simply polls for the next update in the queue. However,
when there are multiple updates queued up, there is no
reason to wait for subsequent runs in order to execute the
rest of the updates.
This commit changes the notifier to drain the queue each time
it runs. It then serially executes the updates.
relates elastic/x-pack-elasticsearch#3769
Original commit: elastic/x-pack-elasticsearch@7a433c17f2
This change fixes the skip check for the EllipticCurveSSLTests. The skip check that is in the test
was added to proctect against failures on JVMs that do not support EC ciphers such as the packaged
openjdk in some linux distributions. The old skip check did not execute until the cluster was up
so the test would still fail with errors such as no cipher suites in common. This change moves the
check into a before class method that checks availability.
Original commit: elastic/x-pack-elasticsearch@5550ca4e1e
The WatcherLifeCycleService is responsible for deciding if watches need
to be reloaded while running. In order to do this, the service stores
the currently local shard allocation ids in a List.
This data structure however was not properly updated all the time, when
it should have been - for example when a master node is not available.
This lead to unintended reloads, even though there was no change in the
allocated shards. This in turn lead to unwanted executions and unwanted
loading of triggered watches.
This commit should also fix one of the more nasty ongoing test failures,
where the test returns with an exception that only parts of watcher have
been started. The AbstractWatcherIntegrationTestCase now properly waits
until watcher is started before starting the test case itself.
Original commit: elastic/x-pack-elasticsearch@097f12a900
Changes the behaviour of the role mapping API to perform a "DistinguishedNameMatch"
when the field is a DN. This is achieved by moving the responsibility for defining
the matching rules from the expression to the data (ExpressionModel)
Because the role mapping API is used within the SAML realm, which may or may not be
using DNs, this implementation assumes that the "dn" and "groups" should be
compared as DNs if they parse as a DN.
For SAML this behaviour will generally do the right thing, as members of the "groups"
field might be DNs (if the data is sourced from an LDAP directory) but often will not be.
Original commit: elastic/x-pack-elasticsearch@3a4dfbba79
Consolidates handling of JDBC types conversion into a single file that should simplify maintaining consistency between type handling. Also separates the types that are handled as part of Elasticsearch output and types that are handled as user-supplied parameters.
relates elastic/x-pack-elasticsearch#3556
Original commit: elastic/x-pack-elasticsearch@d251fce66b
This commit fixes SamlSpMetadataBuilderTests#testBuildFullMetadata failures on windows due to
differing newline endings.
Original commit: elastic/x-pack-elasticsearch@3181c96e46
Monitoring creates indices concurrently to the test execution. In that
case ensureYellow might not be enough and will cause test-failures when shards
are not active etc. This change uses a new method added in core to ensure shards
are not initializing anymore.
relates elastic/x-pack-elasticsearch#2672
Relates to elastic/elasticsearch#28416
Original commit: elastic/x-pack-elasticsearch@661e87f2ee
Introduce system commands as alternative to meta HTTP endpoints
Pass in cluster name
Use 'BASE TABLE' instead of 'INDEX' when describing a table to stick
with the SQL terminology
Original commit: elastic/x-pack-elasticsearch@600312b8f7
In order to more easily integrate xpack once it moves into the
elasticsearch repo, references to the existing x-pack-elasticsearch need
to be reduced. This commit introduces a few helper "methods" available
to any project within xpack (through gradle project extension
properties, as closures). All refeerences to project paths now use these
helper methods, except for those pertaining to bwc, which will be
handled in a followup.
Original commit: elastic/x-pack-elasticsearch@850668744c
Now that ML binaries are public, there is no longer a need to use the s3
client to access the bucket, since creds are not needed. This commit
also moves the cpp snapshot project under the ml module, since it is
specific to that and does not need to clutter the plugin dir.
Original commit: elastic/x-pack-elasticsearch@51e77da4ac
This change removes the XPackExtension mechanism in favor of
SecurityExtension that can be loaded via SPI and doesn't need
another (duplicate) plugin infrastructure
Original commit: elastic/x-pack-elasticsearch@f39e62a040
The EmailSecretsIntegrationTests did not properly wait for watcher to be
stopped before starting it again.
This test failed a few times per week across master and 6.x branches.
Original commit: elastic/x-pack-elasticsearch@cf20d58b0b
The ML snapshot artifacts bucket is now public, so we no longer need to
grab generated s3 creds from vault. This makes the download task run
noticably faster.
Original commit: elastic/x-pack-elasticsearch@e680e55f3d
This change moves watcher's Cron class in the schedule package of xpack-core so that it can be used by other projects (rollup).
Original commit: elastic/x-pack-elasticsearch@f0aa32ccc2
This commit reenables running ITs in xpack by adding an internalClusterTest to xpack modules that contain ESIntegTestCase tests. The new task allows us to run these independently of rest integ tests, which are disabled for xpack modules because installing the bundled plugins directly is not quite the same as installing via the meta plugin. Some tests (ML) are moved to their own qa module to accommodate the need for a real cluster. A couple tests (monitoring and upgrade) have been marked as AwaitsFix.
Commits that have been folded into this commit:
* Move ML IT tests to qa/ml-native-tests
* Add internalClusterTest task and disable rest integ tests for xpack
modules. Also tweak ML tests and get upgrade tests working
* Adding the keystore and security back to the ml native tests
* Fixing native integ test
* Fix last ML test, add awaits fix to monitoring and upgrade tests
* cleanup PR
* fix checkstyle
Original commit: elastic/x-pack-elasticsearch@3c0ed6fd3b
This commit moves the exception classes that SQL uses to follow the
Elasticsearch convention. In the places where varargs were used, the
`LoggerMessageFormat` (`{}`) standard is used instead.
In also removes on Exception - `ExecutionException` since it seemed to not ever
be beefed up, it can be re-added later if needed.
This removes the varargs version of `ClientException` to push the formatting
back on the caller, since `ClientException` cannot depend on Elasticsearch for
formatting the arguments.
There were also a couple of places where we incorrectly passed the Throwable
cause as a var-arg and were unintentionally swallowing it since `String.format`
discards unused arguments.
Relates to elastic/x-pack-elasticsearch#2880
Original commit: elastic/x-pack-elasticsearch@5f5d580e57
This change simply changes the wording of the message that is returned to the user when
installation of a license is attempted with security enabled and TLS disabled. The term
"production" has been removed as it means something different to users.
See elastic/x-pack-elasticsearch#2636
Original commit: elastic/x-pack-elasticsearch@9739c72d66
The commit switches the CreateTokenRequest and InvalidateTokenRequest to throw an
IllegalArgumentException when the version the request is being serialized to does not support the
request. This fixes test failures due to the use of the AssertingTransportInterceptor testing
serialization with versions prior to 6.2. The IAE is an indication to the
AssertingTranpsortInterceptor that the request does not support the version.
Original commit: elastic/x-pack-elasticsearch@c73abf1bc0
By moving these into the ML module:
1. The classes are in the same module as their unit tests
2. We can extend the JobProvider in the future with functionality
that is not in core
Original commit: elastic/x-pack-elasticsearch@610a89a3d9
This commit changes the token service to use a scroll based approach when finding all tokens by
the realm. Without this, we may only find a few tokens and leave some active that need to be
invalidated.
relates elastic/x-pack-elasticsearch#3688
Original commit: elastic/x-pack-elasticsearch@20e97b6aae
The security manager permissions were copied wholesale from pre-split
X-Pack. However, this grants unnecessary permissions to the child
plugins. This commit is a simple attempt at removing permissions that
are not needed in the child plugins.
Relates elastic/x-pack-elasticsearch#3651
Original commit: elastic/x-pack-elasticsearch@8325ed83d7
Some tests seem to be pretty flaky due to concurrent watcher restarts.
This change makes sure we never restart watcher once the node is shutting down.
Original commit: elastic/x-pack-elasticsearch@f0bed7269b
x-pack makes extensive use of group settings in its exporters as well as
its notification services. This does not cope well with settings being
marked as filtered.
This replaces a fair share of group settings (unfortunately not all of
them) with affix key settings, allowing for better filtering and
registering of settings.
Original commit: elastic/x-pack-elasticsearch@968bc8c2ee
Fixes an issue in the test where the class can be constructed but fails to
replace its children due to validation of the number of elements in the list.
Original commit: elastic/x-pack-elasticsearch@3199318d9c
This commit moves the source file in x-pack-core to a org.elasticsearch.xpack.core package. This is to prevent issues where we have compile-time success reaching through packages that will cross module boundaries at runtime (due to being in different classloaders). By moving these to a separate package, we have compile-time safety. Follow-ups can consider build time checking that only this package is defined in x-pack-core, or sealing x-pack-core until modules arrive for us.
Original commit: elastic/x-pack-elasticsearch@232e156e0e
This reverts commit elastic/x-pack-elasticsearch@f91c401a60 due to
failing tests, like
./gradlew :x-pack-elasticsearch:plugin:watcher:test -Dtests.seed=AE30350FCE96D26D -Dtests.class=org.elasticsearch.xpack.watcher.watch.WatchTests -Dtests.method="testParserSelfGenerated" -Dtests.security.manager=true -Dtests.locale=ja-JP -Dtests.timezone=EET
Original commit: elastic/x-pack-elasticsearch@e45d79d643
At present the PersistentTasksService is created inside the ML plugin.
This is undesirable, as other plugins will use persistent tasks in the
near future.
This change refactors the startup code so that the PersistentTasksService
no longer needs to be passed to any constructors for ML components.
A future change will still be required to actually move the initialization
of the PersistentTasksClusterService, PersistentTasksService and
PersistentTasksExecutorRegistry out of the ML plugin, but following this
change it should be fairly simple.
Original commit: elastic/x-pack-elasticsearch@3c2a8e020e
The HaltedClock was a leftover from moving over from our own Clock
implementation to a java.time one. java.time already has a fixed clock,
this one is not needed.
Original commit: elastic/x-pack-elasticsearch@f91c401a60
Adds documentation for all of the date time functions using the new
cli-like format extracted from the csv spec. In the process of doing
this I noticed that the `WEEK` function isn't exposed as a function.
This exposes it for consistency.
Relates to elastic/x-pack-elasticsearch#2898
Original commit: elastic/x-pack-elasticsearch@0459b24cb9
This change adds SPI loading for XPackExtensions that allows to extend
XPack via an ordinary plugin. This can co-exist with the existin
extension mechanism for the time being.
Original commit: elastic/x-pack-elasticsearch@bf02b56dee
This commit re-enables thirdPartyAudit for x-pack core. Previously, when
xpack was a single plugin, it transitively picked up httpcore-nio
through the elasticsearch rest client. Now that xpack core does not
depend on the rest client, httpcore-nio must be added as a dependency.
Additionally, commons-logging was previously handled through the rest
client, but now xpack depends directly on this, thus excludes must be
added for the pesky missing classes there.
This commit also cleans up unnecessary parts of plugin/build.gradle no
longer necessary.
Original commit: elastic/x-pack-elasticsearch@70e936bdc3
This commit fixes failures in SAMLRealmTests#testReadIdpMetadataFromHttps. The main test failure is
caused by the test relying too much on timing with a sleep; the sleep has been removed and replaced
with an assert busy. The second issue that was fixed in the test, is the test makes two requests to
the webserver but only one request is queued in the MockWebServer leading to a NPE.
relates elastic/x-pack-elasticsearch#3667
Original commit: elastic/x-pack-elasticsearch@6c7076c990
The invalid token test has been failing due to the way the test expects an exception to be
returned. Recent changes allowed the exception to be returned via a listener but the test
was not adapted and the code in the token service was not properly handling these exceptions
when they were returned via a listener.
relates elastic/x-pack-elasticsearch#3630
Original commit: elastic/x-pack-elasticsearch@1cf2cc0427
We are relying on the fact that :x-pack-elasticsearch:plugin:core is
configured before these. This happens when these projects are
alphabetically after core, but is not the case when they are not
(ccr). To address this, we simplfy be explicit about evaluation order in
all sub-plugins.
Relates elastic/x-pack-elasticsearch#3663
Original commit: elastic/x-pack-elasticsearch@755f84258b
In many cases we use the `ShardOperationFailedException` interface to abstract an exception that can only be of one type, namely `DefaultShardOperationException`. There is no need to use the interface in such cases, the concrete type should be used instead. That has the additional advantage of simplifying parsing such exceptions back from rest responses for the high-level REST client.
Original commit: elastic/x-pack-elasticsearch@b2259afcbf
Also, removes check for whether a job-to-remove exists
and replaces it with a check of whether a job-to-remove
is already present in the calendar. This allows to
remove a job that may no longer exists and it improves
feedback for the case that an existing job is removed from
a calendar that doesn't contain it.
relates elastic/x-pack-elasticsearch#3620
Original commit: elastic/x-pack-elasticsearch@3ea39be1b6
This commit moves mostly security tests that were left in plugin/src. It
also moves the CompositeTestingXPackPlugin into tribe license tests qa
project, which was the only remaining use. This class needs to be
removed, as it has unavoidable problems with dependency conflicts (ie
guava) between security and watcher, which it pulls in both into the
test classpath.
Original commit: elastic/x-pack-elasticsearch@756209e010
This commit removes a leftover println that was added while debugging
the execution of the API JAR task.
Original commit: elastic/x-pack-elasticsearch@37c2e8fe5b
These were copied wholesale from the pre-split X-Pack
descriptor. However, only ML has a native controller. This removes the
plugin installation asking multiple times to approve the existence of a
native controlled for every bundled plugin.
Relates elastic/x-pack-elasticsearch#3650
Original commit: elastic/x-pack-elasticsearch@4fca606243
Introduces:
- SAML Realm
- REST & Transport actions to support SAML single signon / signout
- Tests for above
- More XML than you ever wanted to see.
Original commit: elastic/x-pack-elasticsearch@b0fe7bb652
With plugins of a meta-plugin now loaded in separate classloaders, we
should not be loading all classes in all plugins of a meta-plugin when
executing scripts. This is particularly problematic in the case of
security extensions where the install plugin extension command would be
running with the classpath of all plugins. However, if there is JAR hell
in this classpath, installation would fail. This is not realistic though
since the plugins are run in separate classloaders. To fix this, for the
scripts of a plugin, we only set the classpath to include the JARs for
that plugin and the JARs of core. This leads us to the introduction of
plugin-specific env scripts.
Relates elastic/x-pack-elasticsearch#3649
Original commit: elastic/x-pack-elasticsearch@543df37eed
Some imports were changed in 6.x to address line-length issues
there. This commit pulls the same changes to master to keep the branches
consistent to simplify backports.
Original commit: elastic/x-pack-elasticsearch@190f9d41f5
This is no longer needed for general X-Pack since the tests will not be
depending on ML. We move this class to an ML specific directory and
remove the dependency from other tests.
Original commit: elastic/x-pack-elasticsearch@9b287f7460
I went to write some docs for datetime functions that look like:
```
SELECT YEAR(CAST('2018-01-19T10:23:27Z' AS TIMESTAMP)) as year;
year
2018
```
because I figured they'd be pretty easy to read because they didn't
require any knowledge of a data set. But it turns out that constant
folding doesn't work properly for date time functions because they don't
actually apply the extraction.
Original commit: elastic/x-pack-elasticsearch@aa9c66b2c7
This commit splits the transport implementations into components that
can be used client-side (in the transport client) and server-side (in
the server). This enables removing security as a dependency for the
transport client.
Relates elastic/x-pack-elasticsearch#3635
Original commit: elastic/x-pack-elasticsearch@e480eb7eb2
Since we don't do any serialization of Cursor classes from the Transport
side (it is treated entirely as a string), we don't actually need to tell anyone
about SQL's writeables
Original commit: elastic/x-pack-elasticsearch@ad2c10e327
This is related to elastic/elasticsearch#elastic/x-pack-elasticsearch#28275. It modifies x-pack to
support the changes in channel contexts. Additionally, it simplifies
the SSLChannelContext by relying on some common work between it and
BytesChannelContext.
Original commit: elastic/x-pack-elasticsearch@8a8fcce050
This is the next step in removing the top level sql directory.
I named the directory `sql-cli` instead of `cli` because that puts it at
the maven coordinates `org.elasticsearch.plugin:sql-cli` instead of
`org.elasticsearch.plugin:cli`.
Relates to elastic/x-pack-elasticsearch#3363
Original commit: elastic/x-pack-elasticsearch@d41a57a136
This commit makes ML snapshot downloading happen less often. It does
that by first moving the download location to a directory outside the
destructive power of gradle clean, and then also uses the md5 of the zip
to compare to that found in s3. This allows us to do a cheap HEAD
request to find if the file has changed.
Original commit: elastic/x-pack-elasticsearch@cd8b00fd31
This moves SQL's server project into `plugin:sql` without modifying how the integration is performed. I know that it is not correct with regards to the x-pack modularization but I think it is a good first step.
Original commit: elastic/x-pack-elasticsearch@2f40d02e4d
This commit adds the ability to refresh tokens that have been obtained by the API using a refresh
token. Refresh tokens are one time use tokens that are valid for 24 hours. The tokens may be used
to get a new access and refresh token if the refresh token has not been invalidated or
already refreshed.
relates elastic/x-pack-elasticsearch#2595
Original commit: elastic/x-pack-elasticsearch@23435eb815
This is related to elastic/x-pack-elasticsearch#3246. This commit adds a SSL/TLS layer to the nio
work implemented in the SSLChannelContext and SSLDriver classes.
This work is used to build up a SecurityNioTransport implementation.
This transport does yet offer feature parity with our normal security
transport. It mainly offers SSL/TLS security.
Original commit: elastic/x-pack-elasticsearch@d0e0484418
The cluster state listener used by watch now have two additional checks.
First, when no master node exists in the cluster state, watcher will
stop and the indexing listener will not try to trigger any new watch.
Second, when there is a global cluster write level block, it would not
be possible to update the watches index or write into the watcher
history, so the listener can bail at that case as well.
In addition this also changes the log level from debug to info when
watcher is stopped. It turned out that there are zero insights when or
if watcher is stopped when normal logging is activated. This makes it
super hard for support to know when watcher is stopped or started at all
due to shards being moved around.
Original commit: elastic/x-pack-elasticsearch@5e9ce24380
* [Security] Handle cache expiry in token service
The keyCache on TokenService.KeyAndCache has a 60 minute expiry.
If the token service was idle for more than 60 minutes, the current
key would be expired and it would then fail to generate user tokens.
Original commit: elastic/x-pack-elasticsearch@fd98130a27
* Security Realms: Predictable ordering for realms
To have predictable ordering of realms, by having secondary
sorting on realm name resulting in stable and consistent documentation.
Documentation update describing how ordering of realms is determined.
Testing done by adding unit test for the change, ran gradle clean check locally.
relates elastic/x-pack-elasticsearch#3403
Original commit: elastic/x-pack-elasticsearch@98c42a8c51
We now separate the compiler Java home from the time runtime Java home
(the one that is used to compile class files versus the one that is used
to run tests). This commit adapts x-pack-elasticsearch to this change.
Relates elastic/x-pack-elasticsearch#3477
Original commit: elastic/x-pack-elasticsearch@bdb096e21c
When events are searched to be passed to the autodetect process, they
are currently calculated based on the latest record timestamp, when
a job opens, and `now` when the process is updated.
This commit changes both to be consistent and based on the earliest
valid timestamp for the job. The earliest valid timestamp is the
latest record timestamp minus the job latency.
Relates elastic/x-pack-elasticsearch#3016
Original commit: elastic/x-pack-elasticsearch@7f882ea053
* [Monitoring] Add uptime.ms field to Beats template
* Add cpu mapping [WIP]
* sort fields
* hierarchical - no dots in fields
* add rss, more event metrics, system cpu and os load
* fix some mapping errors
Original commit: elastic/x-pack-elasticsearch@59cbcb4ee5
This change makes the IndexLifecycleManager check for both index existence and up to date mappings
on the index prior to executing the provided runnable. Doing this provides a mechanism to make
non-breaking mapping updates to the security index in minor versions.
relates elastic/x-pack-elasticsearch#3462
Original commit: elastic/x-pack-elasticsearch@80f05d83b4
This commit adds both the security index template and the audit index template to the excluded
templates for the RemoteIndexAuditTrailStartingTests and IndexAuditTrailTests. By allowing the
templates to remain in place, we make it easier for subsequent tests in the suite to have their
audit trails started.
Relates elastic/x-pack-elasticsearch#435
Original commit: elastic/x-pack-elasticsearch@6e54de19f9
By moving tokenization for categorization to Java we give users access to considerably more options for tokenizing their log messages prior to using ML to categorize them. Now all Elasticsearch analyzer functionality is available, which opens up the possibility to sensibly categorize non-English log messages.
Relates elastic/machine-learning-cpp#491
Original commit: elastic/x-pack-elasticsearch@5d61b67614
In order to support buttons that can be clicked on within a slack
message, this commits adds support for so called actions within
attachments. This allows to create buttons, that are clicked and execute
a GET request, so actions must be idempotent according to the official
slack documentation.
Official slack documentation is available at
https://api.slack.com/docs/message-attachments#action_fields
Original commit: elastic/x-pack-elasticsearch@29ddc90b01
This commit introduces audit event filtering policies for the logging audit.
Until now the only way to temper the gush of audit events was to specifically
pick some event types that were reported.
This superposes a way (named policies) to filter events using lucene regexp filters on
the following event fields: users, realms, roles and indices. The policies are ignore
policies, ie when an event matches it is ignored (not reported).
Original commit: elastic/x-pack-elasticsearch@233f685121
This creates a cluster alert that triggers whenever a node is:
- Added
- Removed
- Restarted (aka Removed and Added between collections)
Unlike all previous cluster alerts, this cluster is always instantly resolved because there is no way to otherwise resolve it later (especially if nodes are replaced).
This will require a small change to the UI so that it does not simply ignore resolved alerts.
Original commit: elastic/x-pack-elasticsearch@6340bf7960
Adds a mode parameter to all SQL-related requests. The mode parameter is used for license checks as well as to define the response content. For now only two modes are supported plain (default) and jdbc. We will add other modes in the future as we add more clients.
Relates elastic/x-pack-elasticsearch#3419
Original commit: elastic/x-pack-elasticsearch@b49ca38d4b
We need to push this flag down to the command line for BWC builds or the
artifacts in release tests will have the wrong version (being considered
snapshots instead of non-snapshots).
Original commit: elastic/x-pack-elasticsearch@279fd02aa7
This commit modifies the BWC build to invoke the Gradle wrapper. The
motivation for this is two-fold:
- BWC versions might be dependent on a different version of Gradle than
the current version of Gradle
- in a follow-up we are going to need to be able to set JAVA_HOME to a
different value than the current value of JAVA_HOME
Relates elastic/x-pack-elasticsearch#3502
Original commit: elastic/x-pack-elasticsearch@22062f635b
The HttpClient uses an Apache HTTP client class named StringEntity to
encode a HTTP request body. This one however assumes ISO-8859-1 as its
charset when encoding the string based body to bytes.
This commit switches to a byte array based body, then sets the content
type header and falls back to the old text/plain content type if no
content type header is specified.
relates elastic/x-pack-elasticsearch#3397
Original commit: elastic/x-pack-elasticsearch@d5a6e7f0c7
The categorization integration tests were picking up a bug in the C++
categorization code, but the expected results were set incorrectly to
match the incorrect C++ output. This change adjusts the expected
results to reflect what is really expected. It also mutes the tests
to avoid failures while the C++ fix propagates through the build
process. The tests will be unmuted tomorrow.
Relates elastic/machine-learning-cpp#512
Original commit: elastic/x-pack-elasticsearch@d2b57b6216
When creating a transport client for a remote index audit trail, we are
implicitly allowing the construction of this transport client to
initialize the number of processors that Netty thinks are on the
system. Since we never pushed down the number of processors, this will
always default to the number of cores on the machine. If the user has
also set the processors setting, when the server bootstraps it will try
to push the number of processors down to Netty too. If this value does
not match the number of cores, we will fail in bootstrap because we
guard against initializing the number of processors that Netty sees to
different values. Instead, the transport client should inherit the
number of processors too and push this down when it pushes the number of
processors down to Netty. We have to worry about another possibility: an
explicit setting for the number of processors for the transport client
so we require this matches the inherited value.
Relates elastic/x-pack-elasticsearch#3469
Original commit: elastic/x-pack-elasticsearch@032810bb0b
Minor refactoring on the reserved realm:
- Removed some duplicated code
- Added in some additional assertions
- Extended some testing
- Removed use of the obsolete "allow_default_passwords" from the test.
Original commit: elastic/x-pack-elasticsearch@584171d2bd
* SQL: Move shared REST client methods to shared-client
This commit is a preliminary step for moving JDBC to the REST client. It extracts the common REST clients from CLI and moves it to shared-client. This will allow us to move to the 5 project setup: rest-proto, shared-client, server, jdbc, cli with the following dependencies:
server <-- rest-proto
shared-client <-- rest-proto
jdbc <-- shared-client
cli <-- shared-client
Relates to elastic/x-pack-elasticsearch#3419
Original commit: elastic/x-pack-elasticsearch@2e6a134de0
It is unsafe to clear passwords of bind requests if the connection is live
and might be used latter (for eg for group searches). This is a temporary
fix that exposes passwords in memory.
Original commit: elastic/x-pack-elasticsearch@e2e1f1a358
This commit hacks up the xpack repo in such a way that it moves all
actions that the transport client uses to plugin core. It also moves
any classes that those actions use to plugin core, with a few
exceptions. I tried to split up any classes that pulled in server side
logic into the client, but that was not always 100% possible. Consider
this commit a guide, and since I do not know the codebase for each
plugin, consider it a best guess for what should be moved. A few other
things were merged in below.
This commit extracts the parts of Condition and AlwaysCondition that are
needed by classes that will be moved into the client package. The only
odd thing here is that since there are two parent classes (Condition and
AlwaysCondition) it was not possible to make AlwaysConditionInteral
extend from AlwaysCondition and still parse a ConditionInternal object,
which is what the ConditionFactory expects.
This commit removes the use of internal users in the User class, and
instead moves them to Authentication, where they are used.
[insert obligatory chuck norris karate image here]
ref elastic/x-pack-elasticsearch#2925
Original commit: elastic/x-pack-elasticsearch@42d0b72209
Alexander Reelsen
Adrien Grand
Yogesh Gaikwad
Michael Basnight
Andy Bristol
Dimitris Athanasiou
David Kyle
Ryan Ernst
Igor Motov
Tim Brooks
Martijn van Groningen
Chris Earle
Shaunak Kashyap
Nik Everett
Costin Leau
Ioannis Kakavas
Lee Hinman
Albert Zaharovits
markharwood
David Roberts
Tim Vernum
Tim Vernum
Jay Modi
Lee Hinman
Costin Leau
David Turner
Tal Levy
Simon Willnauer
jaymode
Jim Ferenczi
Jason Tedor
Tanguy Leroux
Luca Cavanna
Tim Sullivan
Albert Zaharovits
Yannick Welsch