The computed field contained a list of all aggs that were computed
for this particular rollup doc. It was used to help filter to the
correct rollup job/set of jobs.
But this functionality was never perfect, and has been obsoleted by
validating the rollup caps while searching. So we can remove the
computed field and save a bunch of space (since they were quite bulky)
Original commit: elastic/x-pack-elasticsearch@455644488f
This commit changes the combination of multiple automatons representing
a pattern so that the result of each step is minimal. Previously, the
code unioned the automata and performed the minimization operation
after all of the automata had been combined. This resulted in patterns
with lots of overlap causing a TooComplexToDeterminizeException even
though the end result could be a automaton that is total. Minimizing
the automata as we go, allows us to build an automata that could not
previously be built at the cost of additional operations. Automata are
typically cached in the security code, so the net performance impact
should be minimal.
Original commit: elastic/x-pack-elasticsearch@b59fe8d690
This commit fixes several issues with the current implementation of
starting & stopping watcher
1. The WatcherServiceResponse was always returning a message, that the
request was acknowledged, completely independent from the fact if it was
or not.
2. A new cluster state instance was always returned, regardless if the
state had changed or not (which is explicitely mentioned in the
javadocs to check for this)
3. The AckedClusterStateUpdateTask now returns a proper WatcherServiceResponse
4. A failure now gets logged
Relates elastic/x-pack-elasticsearch#4225 (this is just a hunch for now)
Original commit: elastic/x-pack-elasticsearch@f4c1749f95
This PR adds logic to ensure that the fields (and field types) configured
in the Rollup Job are present in the index/indices specified by the job's
index pattern. If a field is missing, or is not aggregatable, it
will throw an exception before the job is created.
This is important for user-friendliness, because otherwise the user
only discovers an issue with mapping when the job is started and
fails to rollup correctly (and only really noticeable by looking at logs,
since it's a runtime failure).
Original commit: elastic/x-pack-elasticsearch@686cd03072
This avoid setting the payload to `null` when sending it to the remote
monitoring cluster. The payload will be GCed when this overall object is
GCed, which should be very soon anyway.
Original commit: elastic/x-pack-elasticsearch@04f63c4150
When a watch is acknowledged, while it is also being executed, the
acknowledgment information can get lost. The reason for this is the
fact, that the execution writes the watch status inside of the watch
regardless, if other writes happened inbetween to make sure the
execution state is caught.
This commit checks the current executions in the execution service and
aborts the API call, if the specified watch ID can be found in those.
Note, this does not prevent this issue fully, as a watch could be
triggered, while the acknowledgement update is running, but it does
reduce the surface area of this problem. In order to properly solve
this, indexing the watch status as part of a watch would need to be
changed.
relates elastic/x-pack-elasticsearch#4003
Original commit: elastic/x-pack-elasticsearch@d7e218b2ac
Many users take the JSON from an PUT watch API and put it into the
execute watch API and then start to wonder why there is an error, as
they forget to wrap the watch inside a 'watch' field.
This commit adds a better error message in this case hinting at the user
to add a 'watch' field.
Original commit: elastic/x-pack-elasticsearch@5b56b4abad
This removes the `assert` that has been causing some very unexpected test
failures due to unexpected threading issues.
Some of the code changed and became async, so it is no longer guaranteed
that the same thread calls `doFlush` and `doClose`. We could similarly
make the field `volatile`, but since this `assert` is not really helping
anything it's easier to remove it.
Original commit: elastic/x-pack-elasticsearch@ba39de605f
If the license key specified by the system property license.key does not
exist, Gradle does not care. Gradle should care, so this commit makes it
care.
Original commit: elastic/x-pack-elasticsearch@afc0a1443c
This commit fixes an issue in the security nio transport tests where
renegotiation was not actually being tested. We were not waiting on the
handshake complete latches. This commit fixes this issue.
Original commit: elastic/x-pack-elasticsearch@47bebc5c13
* Add test matrix axis files for periodic java testing
* Add properties file defining java versions to use
Original commit: elastic/x-pack-elasticsearch@d679b9ab3e
Serialization assertions in ElasticsearchAssertions, a transport
interceptor that used them, and a plugin that added that interceptor
were removed from the test framework. This test case no longer needs to
exclude them from its plugins.
Original commit: elastic/x-pack-elasticsearch@07e5c58983
If a job is deleted and then GetJobs API is immediately called,
it is possible for a job to be returned in the response. This is likely
due to the GetJobs API being executed on a node with a slightly
stale cluster state which shows the job as still existing.
So we delegate to the master node so the list of jobs/tasks is current.
After routing to the master, we need to check if the rollup job
is in the PersistentTask's CS. A job can be acknowledged canceled,
removed from the CS, but the allocated task is still alive. So we
first check the CS to make sure it's really there before going to the
allocated task to get the status.
As extra precaution, when running local to the task, we also make
sure the task isn't canceled before including it in the response.
relates elastic/x-pack-elasticsearch#4041
Original commit: elastic/x-pack-elasticsearch@3b6fb65e12
All ML objects stored in internal indices are currently parsed
strictly. This means unknown fields lead to parsing failures.
In turn, this means we cannot add new fields in any of those
objects (e.g. bucket, record, calendar, etc.) as it is not
backwards compatible.
This commit changes this by introducing lenient parsing when
it comes to reading those objects from the internal indices.
Note we still use strict parsing for the objects we read from
the c++ process, which is nice as it guarantees we would detect
if any of the fields were renamed on one side but not the other.
Also note that even though this is going in from 6.3, we cannot
introduce new fields until 7.0.
relates elastic/x-pack-elasticsearch#4232
Original commit: elastic/x-pack-elasticsearch@3f95d3c7b9
This commit sets the order of the audit log template to 1000 instead of
using the max value. This will allow a user to define a template that
adds an alias.
Original commit: elastic/x-pack-elasticsearch@2267322755
Adds a SecureSetting option for the "bind_password" in LDAP/AD realms
and deprecates the non-secure version.
LDAP bind passwords should now be configured with the setting
`xpack.security.authc.realms.REALM_NAME.secure_bind_password`
in the elasticsearch keystore.
Original commit: elastic/x-pack-elasticsearch@1a0cebd77e
- Changes in CertUtils to add algorithm parameter to
generateSignedCertificates
- Changes in Tests to randomly pick signature algorithms
- Changes in Tests to randomly pick encryption algorithms
relates elastic/x-pack-elasticsearch#3983
Original commit: elastic/x-pack-elasticsearch@d1b5f3a166
If a user has roles that grant access to a large number of disparate
index patterns, then the resulting Automaton can become large and
too costly to determinise. This happens rarely, and is usually a sign
of a poorly implemented security model, so we have no immediate plans
to change the implementation. However the resulting error message is
not clear and does not provide sufficient information for users to
resolve the underlying problem.
This commit catches the underlying exception and provides a more
specific error message, with DEBUG logging of the offending index
patterns.
Original commit: elastic/x-pack-elasticsearch@532be70efc
All logging audit settings are update-able via cluster settings
update API (prefix.emit_node_host_address,
prefix.emit_node_host_name, prefix.emit_node_name, events.include,
events.exclude).
Original commit: elastic/x-pack-elasticsearch@96adbd0ae2
`doSaveState` can be invoked on different types of failure. Some of
these failures are recoverable (e.g. search exception) which just cause
the job to reset until the next trigger time. Other exceptions might
be caused by an Abort request.
Previously `doSaveState` assumed that the indexer state would be
INDEXING, STOPPED or STARTED and asserted that. But if we are ABORTING
it failed the assertion, and in production would try to persist
that aborting state which is not needed (and may complicate matters later).
This commit removes the assertion and only tries to persist if we
are not aborting. If we're aborting, we just invoke the next handler
which is likely an onFailure handler.
Relates to elastic/x-pack-elasticsearch#4243
Original commit: elastic/x-pack-elasticsearch@3643b7c0e4
While it makes sense to apply auto-chunking in order to limit
the time range of the search for previewing datafeeds without aggs,
the same is not the case when aggs are used. In contrary, we should
do the preview the same way it would be if the datafeed run, as this
can reveal problems with regard to the datafeed configuration.
In addition, by default datafeeds with aggs have a manual chunking config
that limits the cost of each search. So, setting the chunking to auto
in those cases may lead to the datafeed preview failing even though
actually running the datafeed would work fine.
Original commit: elastic/x-pack-elasticsearch@79e317efb2
The ML open_job and start_datafeed endpoints start persistent tasks and
wait for these to be successfully assigned before returning. Since the
setup sequence is complex they do a "fast fail" validation step on the
coordinating node before the setup sequence. However, this leads to the
possibility of the "fast fail" validation succeeding and the eventual
persistent task assignment failing due to other changes during the setup
sequence. Previously when this happened the endpoints would time out,
which in the case of the open_job action takes 30 minutes by default.
The start_datafeed endpoint has a shorter default timeout of 20 seconds,
but in both cases the result of a timeout is an unfriendly HTTP 500
status.
This change adjusts the criteria used to wait for the persistent tasks to
be assigned to account for the possibility of assignment failure and, if
this happens, return an error identical to what the "fast fail"
validation would have returned. Additionally in this case the unassigned
persistent task is cancelled, leaving the system in the same state as if
the "fast fail" validation had failed.
Original commit: elastic/x-pack-elasticsearch@16916cbc13
Fixes an inconsistency bug in which `LdapSession`s built by
`LdapUserSearchSessionFactory` are different if the factory is
configured to use a connection pool or not. The bind status of the
connection, or the connection(s) from the pool, passed through to
the newly minted `LdapSession` are now identical. Connections are
bind to the bind_dn configuration entry in the realm config.
Original commit: elastic/x-pack-elasticsearch@094af063ea
If there are multiple jobs that are all the "best" (e.g. share the
best interval) we have no way of knowing which is actually the best.
Unfortunately, we cannot just filter for all the jobs in a single
search because their doc_counts can potentially overlap.
To solve this, we execute an msearch-per-job so that the results
stay isolated. When rewriting the response, we iteratively
unroll and reduce the independent msearch responses into a single
"working tree". This allows us to intervene if there are
overlapping buckets and manually choose a doc_count.
Job selection is found by recursively descending through the aggregation
tree and independently pruning the list of valid job caps in each branch.
When a leaf node is reached in the branch, the remaining jobs are
sorted by "best'ness" (see comparator in RollupJobIdentifierUtils for the
implementation) and added to a global set of "best jobs". Once
all branches have been evaluated, the final set is returned to the
calling code.
Job "best'ness" is, briefly, the job(s) that have
- The largest compatible date interval
- Fewer and larger interval histograms
- Fewer terms groups
Note: the final set of "best" jobs is not guaranteed to be minimal,
there may be redundant effort due to independent branches choosing
jobs that are subsets of other branches.
Related changes:
- We have to include the job's ID in the rollup doc's
hash, so that different jobs don't overwrite the same summary
document.
- Now that we iteratively reduce the agg tree, the agg framework
injects empty buckets while we're working. In most cases this
is harmless, but for `avg` aggs the empty bucket is a SumAgg while
any unrolled versions are converted into AvgAggs... causing a cast
exception. To get around this, avg's are renamed to
`{source_name}.value` to prevent a conflict
- The job filtering has been pushed up into a query filter, since it
applies to the entire msearch rather than just individual agg components
- We no longer add a filter agg clause about the date_histo's interval, because
that is handled by the job validation and pruning.
Original commit: elastic/x-pack-elasticsearch@995be2a039
Lisa Cawley
Zachary Tong
Jay Modi
Alan Woodward
Albert Zaharovits
Alexander Reelsen
Zachary Tong
Chris Earle
Dimitris Athanasiou
Jason Tedor
Lee Hinman
Adrien Grand
Tim Brooks
David Leatherman
Aaron Bull Schaefer
Igor Motov
Andy Bristol
David Kyle
Tim Vernum
Yogesh Gaikwad
Ioannis Kakavas
David Roberts