The ML open_job and start_datafeed endpoints start persistent tasks and
wait for these to be successfully assigned before returning. Since the
setup sequence is complex they do a "fast fail" validation step on the
coordinating node before the setup sequence. However, this leads to the
possibility of the "fast fail" validation succeeding and the eventual
persistent task assignment failing due to other changes during the setup
sequence. Previously when this happened the endpoints would time out,
which in the case of the open_job action takes 30 minutes by default.
The start_datafeed endpoint has a shorter default timeout of 20 seconds,
but in both cases the result of a timeout is an unfriendly HTTP 500
status.
This change adjusts the criteria used to wait for the persistent tasks to
be assigned to account for the possibility of assignment failure and, if
this happens, return an error identical to what the "fast fail"
validation would have returned. Additionally in this case the unassigned
persistent task is cancelled, leaving the system in the same state as if
the "fast fail" validation had failed.
Original commit: elastic/x-pack-elasticsearch@16916cbc13
Fixes an inconsistency bug in which `LdapSession`s built by
`LdapUserSearchSessionFactory` are different if the factory is
configured to use a connection pool or not. The bind status of the
connection, or the connection(s) from the pool, passed through to
the newly minted `LdapSession` are now identical. Connections are
bind to the bind_dn configuration entry in the realm config.
Original commit: elastic/x-pack-elasticsearch@094af063ea
If there are multiple jobs that are all the "best" (e.g. share the
best interval) we have no way of knowing which is actually the best.
Unfortunately, we cannot just filter for all the jobs in a single
search because their doc_counts can potentially overlap.
To solve this, we execute an msearch-per-job so that the results
stay isolated. When rewriting the response, we iteratively
unroll and reduce the independent msearch responses into a single
"working tree". This allows us to intervene if there are
overlapping buckets and manually choose a doc_count.
Job selection is found by recursively descending through the aggregation
tree and independently pruning the list of valid job caps in each branch.
When a leaf node is reached in the branch, the remaining jobs are
sorted by "best'ness" (see comparator in RollupJobIdentifierUtils for the
implementation) and added to a global set of "best jobs". Once
all branches have been evaluated, the final set is returned to the
calling code.
Job "best'ness" is, briefly, the job(s) that have
- The largest compatible date interval
- Fewer and larger interval histograms
- Fewer terms groups
Note: the final set of "best" jobs is not guaranteed to be minimal,
there may be redundant effort due to independent branches choosing
jobs that are subsets of other branches.
Related changes:
- We have to include the job's ID in the rollup doc's
hash, so that different jobs don't overwrite the same summary
document.
- Now that we iteratively reduce the agg tree, the agg framework
injects empty buckets while we're working. In most cases this
is harmless, but for `avg` aggs the empty bucket is a SumAgg while
any unrolled versions are converted into AvgAggs... causing a cast
exception. To get around this, avg's are renamed to
`{source_name}.value` to prevent a conflict
- The job filtering has been pushed up into a query filter, since it
applies to the entire msearch rather than just individual agg components
- We no longer add a filter agg clause about the date_histo's interval, because
that is handled by the job validation and pruning.
Original commit: elastic/x-pack-elasticsearch@995be2a039
This changes `_xpack/monitoring/_bulk` to fundamentally behave in the same
way as `_bulk` and never return 202 when data is ignored (something
`_bulk` cannot do). Instead, anyone interested will have to inspect the
returned response for the ignored flag.
Original commit: elastic/x-pack-elasticsearch@07254a006d
The credentials now get injected via environment variables, so that
external services can pull those.
As soon as the specified environment variables are set, the tests are run. No need to check for the @Network annotation
This also introduces new secret store settings for the secure settings in order to be sure to not leak them in the configuration files, that get dumped.
Relates elastic/x-pack-elasticsearch#3800
Original commit: elastic/x-pack-elasticsearch@a2cfb9cb86
Now that security is not enabled by default for a trial license, the
packaging tests are failing because they expect security to be
enabled. This commit adds enabling security in all instances started
during the packaging tests.
Original commit: elastic/x-pack-elasticsearch@9838393ecb
* [Monitoring/Beats] Add new CPU fields, remove old CPU fields
* use long instead of double for cpu counters
* time => time.ms
Original commit: elastic/x-pack-elasticsearch@244b08a574
The HTTPClient in watcher always appended a question mark at the end of
an URL, regardless if parameters were used or not. This commit adds a
check to only pass valid parameters to the URI construction.
Original commit: elastic/x-pack-elasticsearch@184f8f441c
When running SQL REST queries, a client can ask (through Accept header) for
the data to be returned in CSV or TSV format in addition to plain text,
json & co.
Original commit: elastic/x-pack-elasticsearch@12d87b3033
The documentation mentions that the xpack.watcher.encrypt_sensitive_data
setting needs to be set in the keystore. This is wrong however, it needs
to be set in the standard elasticsearch yaml file.
relates elastic/x-pack-elasticsearch#4195
Original commit: elastic/x-pack-elasticsearch@613d63da85
In order to deal with the most anticipated scenario, when datafeed
frequency is greater than the query_delay, we add the query_delay
to the frequency in order to determine the next time we will trigger
a real-time run. For example, if frequency is 10s and query_delay 1s,
we make sure to trigger the real-time run at a 10s + 1s = 11s offset.
However, this is not correct in the case the frequency is less or
equal to the query_delay. For example, if frequency is 1s and
query_delay is 10s. we would also end up triggering at 11s offset.
But the right behaviour would be to trigger every second while
ensuring we are searching for up to 10seconds ago.
This commit fixes this issue.
relates elastic/x-pack-elasticsearch#4167
Original commit: elastic/x-pack-elasticsearch@f605885167
If a user specifies an 'Authorization' header in an HTTPRequest we,
which might be needed instead of using HTTP Basic Auth due to using
Bearer Authentication, then in case of an failure, the request gets
logged including that Authorization header.
In addition, each implementation of a sent message for jira/hipchat/slack filters
out special fields when a HTTP request is written in case of a failed
response in order to not leak secret data.
Relates elastic/x-pack-elasticsearch#3800
Original commit: elastic/x-pack-elasticsearch@66efdd9b36
This change disables security for trial licenses unless security is
explicitly enabled in the settings. This is done to facilitate users
getting started and not having to deal with some of the complexities
involved in getting security configured. In order to do this and avoid
disabling security for existing users that have gold or platinum
licenses, we have to disable security after cluster formation so that
the license can be retrieved.
relates elastic/x-pack-elasticsearch#4078
Original commit: elastic/x-pack-elasticsearch@96bdb889fc
This commit moves the dev key into core and renames to make it clear it
is for snapshots, and makes the production key a required parameter of
release builds.
Original commit: elastic/x-pack-elasticsearch@ea299bd5a2
Changes are done in SetupPasswordTool to load the keystore
and set security settings to load password protected keys in SSL configuration.
Check for the verification mode and appropriately use hostname verifier.
Close the keystore after setup-password is complete.
Unit test for CommandLineHttpClient
TODO: TLS documentation needs to be fixed, which will be taken up as a separate
fix due to documentation refactoring in progress.
relates elastic/x-pack-elasticsearch#3760
Original commit: elastic/x-pack-elasticsearch@72e8666e21
This commit adds the setting to use a trial license in the core rest
tests with security project. This is needed so that security is
actually enabled during these tests.
Original commit: elastic/x-pack-elasticsearch@b08bdaeb6f
Aggregations may return `NaN`, and the comparison code would return `true` if this result was passed to a `gte` or `lte` condition.
Original commit: elastic/x-pack-elasticsearch@3b16ae6675
SAML is only available on a Platinum license. If you try and use SAML
on a Gold license, then the error message is misleading - it gives the
equivalent of "cannot find saml realm".
This change adds a standard license error of
"current license is non-compliant for [saml]"
if SAML rest actions are used when SAML is not licensed.
Original commit: elastic/x-pack-elasticsearch@7c0e26d58e
Ioannis Kakavas
David Roberts
Albert Zaharovits
Andy Bristol
Lee Hinman
Zachary Tong
Dimitris Athanasiou
Chris Earle
Alexander Reelsen
Jim Ferenczi
David Turner
Jason Tedor
Tim Sullivan
Costin Leau
javanna
Zachary Tong
David Kyle
Christoph Büscher
Jay Modi
Tim Brooks
Ryan Ernst
Yogesh Gaikwad
Lee Hinman
Lisa Cawley
jaymode
Alan Woodward
Tim Vernum