Register SQL as an xpackModule
Specify group for SQL QA to disambiguate projects (otherwise due to an
old Gradle bug (https://github.com/gradle/gradle/issues/847) any
subprojects under SQL QA will not be able to refer to SQL xpackModule
Co-authored-by: Alpar Torok <torokalpar@gmail.com>
There have been at least two PRs trying to fix the spelling of "lazi" because it
isn't very clear from the example that the english analyzer will stem each token
in the example. This adds a short description of the analysis process to make
this clearer.
Relates to #31797
If a get alias api call requests a specific alias pattern then
indices not having any matching aliases should not be included in the response.
This is a second attempt to fix this (first attempt was #28294).
The reason that the first attempt was reverted is because when xpack
security is enabled then index expression (like * or _all) are resolved
prior to when a request is processed in the get aliases transport action,
then `MetaData#findAliases` can't know whether requested all where
requested since it was already expanded in concrete alias names. This
change replaces aliases(...) replaceAliases(...) method on AliasesRequests
class and leave the aliases(...) method on subclasses. So there is a distinction
between when xpack security replaces aliases and a user setting aliases via
the transport or high level http client.
Closes#27763
This is a followup to #31537. It makes a number of changes requested by
a review that came after the PR was merged. These are mostly cleanups
and doc improvements.
Only the shards that receive the bulk request will be affected by
`refresh`. Imagine a `_bulk?refresh=wait_for` request with three
documents in it that happen to be routed to different shards in an index
with five shards. The request will only wait for those three shards to
refresh. The other two shards of that make up the index do not
participate in the `_bulk` request at all.
Relates to #31819
Fixes 2 issues that together cause errors during index creation
with geo_shapes that use the term strategy. The term strategy changes
the default for points_only parameter, but this wasn't taken into
account during serialization. So, setting the term strategy would add
`"points_only": true` to serialization. At the same time if the term
strategy would also cause the `points_only` setting to be not marked as
a processed element during parsing, which would cause index creation to
fail with the error: `Mapping definition for [location] has unsupported`
`parameters: [points_only : true]`.
Fixes#31707
At the end of every `ESRestTestCase` we clean the cluster which includes
deleting all of the templates. If xpack is installed it'll automatically
recreate a few templates every time they are removed. Which is slow.
This change stops the cleanup from removing the xpack templates. It cuts
the time to run the docs tests more than in half and it probably saves a
bit more time on other tests as well.
Removes support for storing scripts without the usual json around the
script. So You can no longer do:
```
POST _scripts/<templatename>
{
"query": {
"match": {
"title": "{{query_string}}"
}
}
}
```
and must instead do:
```
POST _scripts/<templatename>
{
"script": {
"lang": "mustache",
"source": {
"query": {
"match": {
"title": "{{query_string}}"
}
}
}
}
}
```
This improves error reporting when you attempt to store a script but don't
quite get the syntax right. Before, there was a good chance that we'd
think of it as a "raw" template and just store it. Now we won't do that.
Nice.
It seems that java 11 tightened some validations with regard to
time formats. The random instance creator was setting an odd
time format to the data description which is invalid when run
with java 11. This commit changes it to a valid format.
Job persistent tasks with stale allocation IDs used to always be
considered as OPENING jobs in the ML job node allocation decision.
However, FAILED jobs are not relocated to other nodes, which leads
to them blocking up the nodes they failed on after node restarts.
FAILED jobs should not restrict how many other jobs can open on a
node, regardless of whether they are stale or not.
Closes#31794
Job updates or changes to calendars or filters may
result into updating the job process if it has been
running. To preserve the order of updates, process
updates are queued through the UpdateJobProcessNotifier
which is only running on the master node. All actions
performing such updates must run on the master node.
However, the CRUD actions for calendars and filters
are not master node actions. They have been submitting
the updates to the UpdateJobProcessNotifier even though
it might have not been running (given the action was
run on a non-master node). When that happens, the update
never reaches the process.
This commit fixes this problem by ensuring the notifier
runs on all nodes and by ensuring the process update action
gets the resources again before updating the process
(instead of having those resources passed in the request).
This ensures that even if the order of the updates
gets messed up, the latest update will read the latest
state of those resource and the process will get back
in sync.
This leaves us with 2 types of updates:
1. updates to the job config should happen on the master
node. This is because we cannot refetch the entire job
and update it. We need to know the parts that have been changed.
2. updates to resources the job uses. Those can be handled
on non-master nodes but they should be re-fetched by the
update process action.
Closes#31803
* Upgrade bouncycastle
Required to fix
`bcprov-jdk15on-1.55.jar; invalid manifest format `
on jdk 11
* Downgrade bouncycastle to avoid invalid manifest
* Add checksum for new jars
* Update tika permissions for jdk 11
* Mute test failing on jdk 11
* Add JDK11 to CI
* Thread#stop(Throwable) was removed
http://mail.openjdk.java.net/pipermail/core-libs-dev/2018-June/053536.html
* Disable failing tests #31456
* Temprorarily disable doc tests
To see if there are other failures on JDK11
* Only blacklist specific doc tests
* Disable only failing tests in ingest attachment plugin
* Mute failing HDFS tests #31498
* Mute failing lang-painless tests #31500
* Fix backwards compatability builds
Fix JAVA version to 10 for ES 6.3
* Add 6.x to bwx -> java10
* Prefix out and err from buildBwcVersion for readability
```
> Task :distribution:bwc:next-bugfix-snapshot:buildBwcVersion
[bwc] :buildSrc:compileJava
[bwc] WARNING: An illegal reflective access operation has occurred
[bwc] WARNING: Illegal reflective access by org.codehaus.groovy.reflection.CachedClass (file:/home/alpar/.gradle/wrapper/dists/gradle-4.5-all/cg9lyzfg3iwv6fa00os9gcgj4/gradle-4.5/lib/groovy-all-2.4.12.jar) to method java.lang.Object.finalize()
[bwc] WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.reflection.CachedClass
[bwc] WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
[bwc] WARNING: All illegal access operations will be denied in a future release
[bwc] :buildSrc:compileGroovy
[bwc] :buildSrc:writeVersionProperties
[bwc] :buildSrc:processResources
[bwc] :buildSrc:classes
[bwc] :buildSrc:jar
```
* Also set RUNTIME_JAVA_HOME for bwcBuild
So that we can make sure it's not too new for the build to understand.
* Align bouncycastle dependency
* fix painles array tets
closes#31500
* Update jar checksums
* Keep 8/10 runtime/compile untill consensus builds on 11
* Only skip failing tests if running on Java 11
* Failures are dependent of compile java version not runtime
* Condition doc test exceptions on compiler java version as well
* Disable hdfs tests based on runtime java
* Set runtime java to minimum supported for bwc
* PR review
* Add comment with ticket for forbidden apis
The ack watch action has a check for currently executed watches, to make
sure that currently running watches cannot be acknowledged. This check
only checked on the coordinating node for watches being executed, but should
have checked the whole cluster using a WatcherStatsRequest, which is
being switched to in this commit.
As SecureSetting is extended from Setting, you can easily accidentally
use `SecureSetting.simpleString()` to read a secure setting instead of
`SecureSetting.secureString()`. This commit changes this behaviour in
some watcher notification services.
There is at most one model size stats document per bucket, but
during lookback a job can churn through many buckets very quickly.
This can lead to many cluster state updates if established model
memory needs to be updated for a given model size stats document.
This change rate limits established model memory updates to one
per job per 5 seconds. This is done by scheduling the updates 5
seconds in the future, but replacing the value to be written if
another model size stats document is received during the waiting
period. Updating the values in arrears like this means that the
last value received will be the one associated with the job in the
long term, whereas alternative approaches such as not updating the
value if a new value was close to the old value would not.
Today, `AmazonS3Fixture` returns 403 on attempts to access any inappropriate
bucket, whether known or otherwise. In fact, S3 reports 404 on nonexistent
buckets and 403 on inaccessible ones. This change enhances `AmazonS3Fixture` to
distinguish these cases.
Today TransportService is tightly coupled with Transport since it
requires an instance of TransportService in order to receive responses
and send requests. This is mainly due to the Request and Response handlers
being maintained in TransportService but also because of the lack of a proper
callback interface.
This change moves request handler registry and response handler registration into
Transport and adds all necessary methods to `TransportConnectionListener` in order
to remove the `TransportService` dependency from `Transport`
Transport now accepts one or more `TransportConnectionListener` instances that are
executed sequentially in a blocking fashion.
Add hard limit to the number of items
a filter may have. This serves to protect
from excessive overhead due to the filters
taking too much memory or lookups becoming
too expensive.
This change adds stats about forecasts, to the jobstats api as well as xpack/_usage. The following
information is collected:
_xpack/ml/anomaly_detectors/{jobid|_all}/_stats:
- total number of forecasts
- memory statistics (mean/min/max)
- runtime statistics
- record statistics
- counts by status
_xpack/usage
- collected by job status as well as overall (_all):
- total number of forecasts
- number of jobs that have at least 1 forecast
- memory, runtime, record statistics
- counts by status
Fixes#31395
Adds a Minio fixture to run the S3 repository tests against Minio. Also collapses the single qa
subproject into the s3-repository project, which simplifies the code structure (having it all in one
place) and helps to avoid having too many Gradle subprojects.
This completes the removal of Painless Type. The new data structures in the definition are a map of names (String) to Java Classes and a map of Java Classes to Painless Structs. The names to Java Classes map can contain a 2 to 1 ratio of names to classes depending on whether or not a short (imported) name is used. The Java Classes to Painless Structs is 1 to 1 always where the Java Class name must match the Painless Struct name. This should lead a significantly simpler type system in Painless moving forward since the Painless Type only held redundant information since Painless does not support generics.
Previously the call to register a listener for settings updates was in
each individual service, rather than in the notification service
itself. This change ensures that each child of the notification service
gets registered with the settings update consumer.
Piotr Prądzyński
Martijn van Groningen
Costin Leau
Christoph Büscher
hanbj
Yannick Welsch
Alan Woodward
Jim Ferenczi
Alpar Torok
Tim Brooks
Nik Everett
Igor Motov
Sohaib Iftikhar
Mayya Sharipova
Dimitris Athanasiou
David Roberts
Vladimir Dolzhenko
Alexander Reelsen
Costin Leau
David Turner
Simon Willnauer
Hendrik Muhs
Lee Hinman
Lisa Cawley
Jack Conradson
Armin Braun
Michael Basnight