Consolidates type handling into DataType, makes DataType available to
JDBC by moving to sql-proto and removes support for all parameter types
that cannot be handled by the server.
Original commit: elastic/x-pack-elasticsearch@b8024f5c46
This commit introduces a new watcher setting to defer starting watcher
until it has been called with the API for the first time. This is
primarily useful in testing environments, as this ensures that watcher
does not try to reload itself because of starting first and then
creating watcher indices.
In addition the undocumented and unused option
xpack.watcher.start_immediately has been removed.
Relates elastic/x-pack-elasticsearch#3854
Original commit: elastic/x-pack-elasticsearch@2b55aec4ad
This fixes a regression introduced in Elasticsearch 6.0, when switching
from HttpURLConnection to the Apache HTTP Client.
In the old implementation there was no way to specify if you wanted to use HTTP
or HTTPS for your proxy, only HTTP. If people needed to use HTTPs, they
could just use the CONNECT feature of the proxy.
The new implementation used the scheme of the request that was about to
be sent out as the proxy scheme to be used. So if the request was HTTPS
but the proxy server was HTTP this created a problem.
This commit changes the default scheme to be just HTTP, so that then the
standard CONNECT procecure is taken care off.
Without a real proxy server this is super hard to test. I have verified
this with the following test against a tinyproxy running on port 8888,
but I do not have a great idea how to test this in a unit testable way using a real proxy.
Original commit: elastic/x-pack-elasticsearch@f68e72d8f1
The watcher index uses auto expanding of replicas nowadays, so there is
no need to set the number of replicas.
Also ensuring that all watcher indices are green on startup has been moved
to one call, in order to prevent indices being moved around after adding
the triggered watches or watch history index.
Original commit: elastic/x-pack-elasticsearch@dbe04adf64
* Add fields to `.logstash`'s mapping in template
This "makes room" in the index for pipeline settings and node groups. Due to this change, users will be able to specify settings and node groups for a pipeline via the Centralized Config Management UI in Kibana. Logstash will only retrieve pipelines associated with the node group specified via the `xpack.management.group.id` setting in `logstash.yml`. For the retrieved pipelines, Logstash will apply any (optionally) specified pipeline settings before (re)loading the pipelines.
* Making field name more explicit + adding multi field for better search
Original commit: elastic/x-pack-elasticsearch@2df101f0b1
This adds the `HEADERS_SETTING` as setting reported by the Monitoring plugin, which was changed in 6.2.
It also adds an IT to ensure that it is not missed in the future (existing integration tests were passing in the Settings and not being validated by the cluster!).
Original commit: elastic/x-pack-elasticsearch@d6f0b4b503
* SQL: fix name of Arithmetic functions
Remove id from name of Arithmetic function and improve name of Neg
Original commit: elastic/x-pack-elasticsearch@4f3e8d6a2d
Since elastic/x-pack-elasticsearch#3254 security headers have been stored in datafeed cluster state
to allow the datafeed to run searches using the credentials of the user
who created/updated it. As a result the parser was changed to read the
"headers" field so that cluster state could be reloaded. However, this
meant that datafeed configs could be submitted with a "headers" field.
No security loophole arose from this, as subsequent code overwrites the
contents of any supplied headers. But it could be confusing that an
erroneously supplied field did not cause a parse failure as it usually
would.
This change makes the config parser for datafeeds reject a "headers"
field. Now only the metadata parser used for reloading cluster state
will read a "headers" field.
Original commit: elastic/x-pack-elasticsearch@afa503275f
java.time features it's own halted clock, called a fixed clock, we can
use that one.
On top of that the watcher xcontent parser does not need a clock at all,
just a timestamp when parsing happened.
Original commit: elastic/x-pack-elasticsearch@2061aeffe1
The api jar was added for xpack extensions. However, extensions have
been removed in favor of using SPI, and the individual xpack jars like
core and security are published to enable this. This commit removes the
api jar, and switches the transport client to use the core jar (which
the api jar was just a rename of).
Original commit: elastic/x-pack-elasticsearch@58e069e66c
There were a number of leftover unnecessary elements in the module
build.gradle files that were holdovers from copying the original plugin
build.gradle. This commit removes these elements.
Original commit: elastic/x-pack-elasticsearch@08babbd520
Calling start() when already in the STARTING state doesn't do anything, so the component
gets stuck in STARTING state forever.
Also: wait on the required index name not just the cluster.
Also: added more logging to help diagnose such issues (either in RemoteIndexAuditTrailStartingTests or production)
Original commit: elastic/x-pack-elasticsearch@fb81214fe7
The old home made sense before x-pack was split. The new home lines up
with where security keeps its scripts.
The jar file wasn't being included any more so this re-adds that.
Original commit: elastic/x-pack-elasticsearch@d3ec941397
This commit adds special handling for null values when building a DistinguishedNamePredicate for
role mapping. Previously this would have resulted in an exception from the unboundid ldapsdk as
the DN is invalid.
relates elastic/x-pack-elasticsearch#3787
Original commit: elastic/x-pack-elasticsearch@9386dae03e
This allows any datetime function to be present in `EXTRACT` which feels
more consistent. `EXTRACT(FOO FROM bar)` is now just sugar for
`FOO(bar)`. This is *much* simpler to explain in the documentation then
"these 10 fields are supported by extract and they are the same as this
subset of the datetime functions."
The implementation of this is a little simpler then the old way. Instead
of resolving the function in the parser we create an
`UnresolvedFunction` that looks *almost* just like what we'd create for
a single argument function and resolve the function in the `Analyzer`.
This feels like a net positive as it allows us to group `EXTRACT`
resolution failures with other function resolution failures.
This also creates `UnresolvedFunctionTests` and
`UnresolvedAttributeTests`. I had to create `UnresolvedFunctionTests`
because `UnreolvedFunction` now has three boolean parameters which is
incompatible with the generic `NodeSubclassTests`'s requirement that all
ctor parameters be unique. I created `UnresolvedAttributeTests` because
I didn't want `UnresolvedFunctionTests` to call `NodeSubclassTests` and
figured that we'd want `UnresolvedAttributeTest` eventually and now felt
like as good a time as any.
Added a
Original commit: elastic/x-pack-elasticsearch@358aada308
If any of the follow take place on security index, then any cached role mappings
are potentially invalid and the associated realms need to clear any cached users.
- Index recovers from red
- Index is deleted
- Index becomes out-of-date / not-out-of-date
Original commit: elastic/x-pack-elasticsearch@1bcd86fcd4
The HTTP Exporter in Monitoring allowed users to set a timeout parameters
for the requests. When set, this was setting the `master_timeout` query parameter
in Bulk Requests. The problem is that Bulk Requests do not support this type of
timeout.
Original commit: elastic/x-pack-elasticsearch@9be194006e
This adds support to allow different schemes in a proxy being used
compared to what the actual request requires. So if your proxy runs via
HTTP, but the endpoint you want to connect to uses HTTPS, this is now
possible to configure the proxy explicitely.
Also a small unit test for parsing this has been added.
relates elastic/x-pack-elasticsearch#3596
Original commit: elastic/x-pack-elasticsearch@176f7cdf0e
Tests have been failing because of out of order cluster state processing
or because of stopping/starting was still in progress. Current tests do
not do further stop/start tries after the first leading to potential
being stuck in an unwanted state. This commit removes the methods
checking for the state being started or stopped in favor of using the
stopWatcher/startWatcher methods which now check if the desired state is
reached and otherwise issue another start/stop command.
Original commit: elastic/x-pack-elasticsearch@97b3232a6a
The notifier is scheduled to run once per second. Currently,
it simply polls for the next update in the queue. However,
when there are multiple updates queued up, there is no
reason to wait for subsequent runs in order to execute the
rest of the updates.
This commit changes the notifier to drain the queue each time
it runs. It then serially executes the updates.
relates elastic/x-pack-elasticsearch#3769
Original commit: elastic/x-pack-elasticsearch@7a433c17f2
This change fixes the skip check for the EllipticCurveSSLTests. The skip check that is in the test
was added to proctect against failures on JVMs that do not support EC ciphers such as the packaged
openjdk in some linux distributions. The old skip check did not execute until the cluster was up
so the test would still fail with errors such as no cipher suites in common. This change moves the
check into a before class method that checks availability.
Original commit: elastic/x-pack-elasticsearch@5550ca4e1e
The WatcherLifeCycleService is responsible for deciding if watches need
to be reloaded while running. In order to do this, the service stores
the currently local shard allocation ids in a List.
This data structure however was not properly updated all the time, when
it should have been - for example when a master node is not available.
This lead to unintended reloads, even though there was no change in the
allocated shards. This in turn lead to unwanted executions and unwanted
loading of triggered watches.
This commit should also fix one of the more nasty ongoing test failures,
where the test returns with an exception that only parts of watcher have
been started. The AbstractWatcherIntegrationTestCase now properly waits
until watcher is started before starting the test case itself.
Original commit: elastic/x-pack-elasticsearch@097f12a900
Changes the behaviour of the role mapping API to perform a "DistinguishedNameMatch"
when the field is a DN. This is achieved by moving the responsibility for defining
the matching rules from the expression to the data (ExpressionModel)
Because the role mapping API is used within the SAML realm, which may or may not be
using DNs, this implementation assumes that the "dn" and "groups" should be
compared as DNs if they parse as a DN.
For SAML this behaviour will generally do the right thing, as members of the "groups"
field might be DNs (if the data is sourced from an LDAP directory) but often will not be.
Original commit: elastic/x-pack-elasticsearch@3a4dfbba79
Consolidates handling of JDBC types conversion into a single file that should simplify maintaining consistency between type handling. Also separates the types that are handled as part of Elasticsearch output and types that are handled as user-supplied parameters.
relates elastic/x-pack-elasticsearch#3556
Original commit: elastic/x-pack-elasticsearch@d251fce66b
This commit fixes SamlSpMetadataBuilderTests#testBuildFullMetadata failures on windows due to
differing newline endings.
Original commit: elastic/x-pack-elasticsearch@3181c96e46
Monitoring creates indices concurrently to the test execution. In that
case ensureYellow might not be enough and will cause test-failures when shards
are not active etc. This change uses a new method added in core to ensure shards
are not initializing anymore.
relates elastic/x-pack-elasticsearch#2672
Relates to elastic/elasticsearch#28416
Original commit: elastic/x-pack-elasticsearch@661e87f2ee
Introduce system commands as alternative to meta HTTP endpoints
Pass in cluster name
Use 'BASE TABLE' instead of 'INDEX' when describing a table to stick
with the SQL terminology
Original commit: elastic/x-pack-elasticsearch@600312b8f7
In order to more easily integrate xpack once it moves into the
elasticsearch repo, references to the existing x-pack-elasticsearch need
to be reduced. This commit introduces a few helper "methods" available
to any project within xpack (through gradle project extension
properties, as closures). All refeerences to project paths now use these
helper methods, except for those pertaining to bwc, which will be
handled in a followup.
Original commit: elastic/x-pack-elasticsearch@850668744c
Now that ML binaries are public, there is no longer a need to use the s3
client to access the bucket, since creds are not needed. This commit
also moves the cpp snapshot project under the ml module, since it is
specific to that and does not need to clutter the plugin dir.
Original commit: elastic/x-pack-elasticsearch@51e77da4ac
This change removes the XPackExtension mechanism in favor of
SecurityExtension that can be loaded via SPI and doesn't need
another (duplicate) plugin infrastructure
Original commit: elastic/x-pack-elasticsearch@f39e62a040
The EmailSecretsIntegrationTests did not properly wait for watcher to be
stopped before starting it again.
This test failed a few times per week across master and 6.x branches.
Original commit: elastic/x-pack-elasticsearch@cf20d58b0b
The ML snapshot artifacts bucket is now public, so we no longer need to
grab generated s3 creds from vault. This makes the download task run
noticably faster.
Original commit: elastic/x-pack-elasticsearch@e680e55f3d
This change moves watcher's Cron class in the schedule package of xpack-core so that it can be used by other projects (rollup).
Original commit: elastic/x-pack-elasticsearch@f0aa32ccc2
This commit reenables running ITs in xpack by adding an internalClusterTest to xpack modules that contain ESIntegTestCase tests. The new task allows us to run these independently of rest integ tests, which are disabled for xpack modules because installing the bundled plugins directly is not quite the same as installing via the meta plugin. Some tests (ML) are moved to their own qa module to accommodate the need for a real cluster. A couple tests (monitoring and upgrade) have been marked as AwaitsFix.
Commits that have been folded into this commit:
* Move ML IT tests to qa/ml-native-tests
* Add internalClusterTest task and disable rest integ tests for xpack
modules. Also tweak ML tests and get upgrade tests working
* Adding the keystore and security back to the ml native tests
* Fixing native integ test
* Fix last ML test, add awaits fix to monitoring and upgrade tests
* cleanup PR
* fix checkstyle
Original commit: elastic/x-pack-elasticsearch@3c0ed6fd3b
This commit moves the exception classes that SQL uses to follow the
Elasticsearch convention. In the places where varargs were used, the
`LoggerMessageFormat` (`{}`) standard is used instead.
In also removes on Exception - `ExecutionException` since it seemed to not ever
be beefed up, it can be re-added later if needed.
This removes the varargs version of `ClientException` to push the formatting
back on the caller, since `ClientException` cannot depend on Elasticsearch for
formatting the arguments.
There were also a couple of places where we incorrectly passed the Throwable
cause as a var-arg and were unintentionally swallowing it since `String.format`
discards unused arguments.
Relates to elastic/x-pack-elasticsearch#2880
Original commit: elastic/x-pack-elasticsearch@5f5d580e57
Adrien Grand
Ryan Ernst
Igor Motov
Tim Brooks
Martijn van Groningen
Alexander Reelsen
Chris Earle
Shaunak Kashyap
Nik Everett
Michael Basnight
Costin Leau
Ioannis Kakavas
Lee Hinman
Albert Zaharovits
David Kyle
markharwood
David Roberts
Tim Vernum
Tim Vernum
Jay Modi
Lee Hinman
Costin Leau
David Turner
Dimitrios Athanasiou
Tal Levy
Simon Willnauer
jaymode
Jim Ferenczi