On Windows, log4cxx always writes to stderr in UTF-16, and we get the
logs from C++ to Java by redirecting stderr to our named pipe. Hence
the log handler in Java needs to tolerate the log stream it's reading
being either UTF-16 (for Windows) or UTF-8 (for other platforms).
Fixeselastic/machine-learning-cpp#385
Original commit: elastic/x-pack-elasticsearch@89237d7125
This kind of sucks, because we shouldn't have to wait that long for tests to run.
But they're failing CI with some regularity, and we rely on these integration tests.
Original commit: elastic/x-pack-elasticsearch@3f4acb2a32
This is a forward-port of elastic/x-pack-elasticsearch/pull/2921.
original commit message:
Before this commit, a cluster with security enabled and backed by
native-realm user permissions allowed rolled upgrades to clusters without
upgrading the `.security` index. This resulted in the newly established
6.0 cluster not able to register the native-realm users previously established
in the `.security` index. In order to fix this, one would have to rely on file-based
users to re-configure and upgrade the `.security` index. Since this state is easily
avoidable with an upgrade, this commit rejects the joining of upgraded nodes without
upgrading the security index beforehand.
modifications:
Test with 7.x vs 6.x nodes.
Original commit: elastic/x-pack-elasticsearch@56f81bfb20
This commit updates the logic for determining which branch to use to make it consistent with the
logic in elasticsearch. This change means that testing BWC within the same major picks the correct
branch.
Original commit: elastic/x-pack-elasticsearch@2d75d15c41
This adds a rolling upgrade test for X-Pack monitoring. It works by using the `_xpack/monitoring/_bulk` endpoint to send arbitrary data, then verify that it exists.
This forces a few things to happen, thereby testing the behavior:
1. The templates must exist.
2. The elected master node must be "ready" to work (hence the first
point).
3. The same "system_api_version" is accepted by every version of ES.
Original commit: elastic/x-pack-elasticsearch@012e5738bb
Improving logging for unexpected autodetect termination (crash, oom). Output to the log pipe not conforming to the json log output format are treated as fatal error and logged, so that the crash as well as a proper error message if available gets logged.
Original commit: elastic/x-pack-elasticsearch@ae5d792d3f
This change should have been made in elastic/x-pack-elasticsearch#2913. Now we hold the process
context lock throughout the job close procedure, the timeout for trying
to lock it should be the timeout used for job open/close rather than the
timeout for connecting named pipes.
Original commit: elastic/x-pack-elasticsearch@79672b0825
This change removes the xpack plugin's dependency on the tribe module, which is not a published
artifact. For the most part this just involves moving some test classes around, but for the
security and tribe integration the usage of constant settings was removed and replaced with the
string names. This is a bit unfortunate, but a test was added in a QA project that depends on tribe
that will alert us if a new setting is added that we need to be aware of.
relates elastic/x-pack-elasticsearch#2656
Original commit: elastic/x-pack-elasticsearch@649a8033e4
This change fixes the check for the version of the security template after the template updater was
changed to only run on the master node in elastic/elasticsearch#27294. Additionally, the wait time
for the cluster to have a yellow status has been increased to account for delayed shards and slower
machines.
Original commit: elastic/x-pack-elasticsearch@a2e72bed12
I imagine this needless indirection arose from accepting the wrong
IntelliJ suggestion for an import.
Original commit: elastic/x-pack-elasticsearch@54d7e854d3
The 5.6 Upgrade API will reindex .security to .security-6 and create a .security alias.
But the 6.0 default was to create a .security-v6 index and a .security alias if none existed (e.g. fresh x-pack install)
Having two different index names based on the method of install/upgrade complicates the code and testing, so we're unifying on the .security-6 index name that already exists in the wild.
Original commit: elastic/x-pack-elasticsearch@d78f569c5f
When simultaneous close requests were made for the same job it was possible
that one of the requests would inappropriately log error messages about the
job having failed. This change prevents that problem, whilst continuing to
adhere to the requirement that close requests for already closing jobs do not
return until the close request that is doing the work completes.
relates elastic/x-pack-elasticsearch#2912
Original commit: elastic/x-pack-elasticsearch@513b7fa1d6
The default internal XPack user no longer has access to the security index, but it should have read-only access to the audit log so that watches can be triggered based on audit events (but cannot write audit records)
Original commit: elastic/x-pack-elasticsearch@5c37720dad
This commit changes the handling of exceptions when retrieving roles from the native roles store.
Previously, exceptions would have caused the request to terminate and the exception would be
sent back to the user. This makes for a bad experience when a cluster hasn't been upgraded to the
latest index format and anonymous access is enabled with a native role as all requests without
preemptive basic authentication would result in an exception. The change here is to allow the
request to continue processing. Once the security index is up to date, the roles cache is cleared
so that the native roles can be picked up.
relates elastic/x-pack-elasticsearch#2686
Original commit: elastic/x-pack-elasticsearch@ef5149140f
Certgen was generating "other name" SANs without the explicit [0] tag that is required.
This was masked by the fact that the JRE X.509 classes always wrap the "other name" name-value in a [0] tag (even if it already has one)
Also switched to a UTF8 String from an IA5 string to match the configuration being used for testing in openssl.
Original commit: elastic/x-pack-elasticsearch@1b87964ec7
This is the X-Pack side of elastic/elasticsearch#27235. To force people
who construct an Environment object in production code to think about the
correct setting of configPath there is no longer a single argument
constructor in the Environment class. Instead there is a factory method
in the test framework to replace it. Having this in the test framework
ensures that there is no way to use it in production code.
Original commit: elastic/x-pack-elasticsearch@4860e92d90
We should not be constructing a temporary Environment object in production
code. This currently isn't causing any problems, but it might in the future
if elastic/elasticsearch#27144 or something similar is ever merged. Instead
the master Environment of the node should always be used.
Original commit: elastic/x-pack-elasticsearch@6276a54a45
This adds the data necessary to add a warning to the alerts UI representing each cluster when xpack.security.transport.tls.enabled is not set to true for a trial licensed cluster running with
xpack.security.enabled.
Original commit: elastic/x-pack-elasticsearch@28fe8bad76
This adds details about the shards and the health of the index. By adding these stats directly to the document, the UI can avoid many aggregations and enable better searching and sorting against indices.
Original commit: elastic/x-pack-elasticsearch@f38ae5ce69
This commit removes the FAILED state for the IndexAuditTrail so that we always try to keep starting
the service. Previously, on any exception during startup we moved to a failed state and never tried
to start again. The users only option was to restart the node. This was problematic in the case of
large clusters as there could be common timeouts of cluster state listeners that would cause the
startup of this service to fail.
Additionally, the logic in the IndexAuditTrail to update the template on the current cluster has
been removed and replaced with the use of the TemplateUpgradeService. However, we still need to
maintain the ability to determine if a template on a remote cluster should be PUT. To avoid always
PUTing the template, the version field has been added so it only needs to be PUT once on upgrade.
Finally, the default queue size has been increased as this is another common issue that users hit
with high traffic clusters.
relates elastic/x-pack-elasticsearch#2658
Original commit: elastic/x-pack-elasticsearch@27e2ce7223
Adding this field enables a very simple mechanism for detecting node changes in the cluster state via Watcher (and other mechanisms). The next step is to add the cluster alert that uses it.
Original commit: elastic/x-pack-elasticsearch@1eacc25cff
This commit adds a new `certutil` command and deprecates the `certgen` command.
The new certuil consists of sub commands that are (by default) are simpler to use than the old monolithic command, but still support all the previous behaviours.
Original commit: elastic/x-pack-elasticsearch@3f57687da9
The execution state of a watch did not differentiate between failures of
the execution like a broken painless script and a thread pool rejection.
This adds an own state, which allows to aggregate on such data in the
watch history, which should ease debugging issues a bit.
Original commit: elastic/x-pack-elasticsearch@351e64e14d
For the purpose of getting this API consumed by our UI, returning
overall buckets that match the job's largest `bucket_span` can
result in too much data. The UI only ever displays a few buckets
in the swimlane. Their span depends on the time range selected and
the screen resolution, but it will only ever be a relatively
low number.
This PR adds the ability to aggregate overall buckets in a user
specified `bucket_span`. That `bucket_span` may be equal or
greater to the largest job's `bucket_span`. The `overall_score`
of the result overall buckets is the max score of the
corresponding overall buckets with a span equal to the job's
largest `bucket_span`.
The implementation is now chunking the bucket requests
as otherwise the aggregation would fail when too many buckets
are matching.
Original commit: elastic/x-pack-elasticsearch@981f7a40e5
If a bulk update references aliases rather than concrete indices,
it is possible that a single shard level request could have multiple distinct "index names", potentially including "date math".
Those names will resolve to the same concrete index, but they might have different privileges.
Original commit: elastic/x-pack-elasticsearch@34cfd11df8
This commit adds a new interval_ms field to the monitoring documents.
This field indicates the current collection interval for Elasticsearch or
external monitored systems. The value is indexed as a long.
Related to elastic/x-pack-elasticsearch#212
Original commit: elastic/x-pack-elasticsearch@2ceb20455c
This change fixes an incorrect check for a missing password setting for the PKI realm. The check
only allowed the secure setting to be used for the PkiRealm password even though the legacy setting
is still valid. This change fixes the check.
Relates elastic/x-pack-elasticsearch#2487
Original commit: elastic/x-pack-elasticsearch@a4524c2c05
This commit changes the MonitoringIt and XPackRestIT tests so that the
disableMonitoring() method now use the XPack Usage API in order to check
that the monitoring exporters are correctly disabled. It checks at the
beginning of the tests (all exporters must be disabled before running
the test) and also at the end of the test.
This commit also fixes a bug in MonitoringIT where the Bulk thread pool
active queue was wrongly extracted from the response's map, forcing the
test to always wait for 30sec.
relates elastic/x-pack-elasticsearch#2459
Original commit: elastic/x-pack-elasticsearch@2d349e840f
Provides more verbose messaging around errors and possible causes when the tool aborts.
This change is primarily focused on errors connecting to the Elasticsearch node when TLS is enabled on the HTTP connection.
Original commit: elastic/x-pack-elasticsearch@aa8f7c6143
When certgen configuration was read from an input file (`-in` option) validation errors were collected but never reported. Depending on the type of error this may have caused the tool to exit with an internal error (e.g. NPE).
Validation is now applied after parsing the file and if errors are found the tool exits.
Original commit: elastic/x-pack-elasticsearch@b2262ed1d7
ML uses node attributes to ensure that the master node knows how many
ML jobs may be allocated to each node. This change prevents a user
messing up the way these attributes are used by setting them differently
using node.attr.* entries in their elasticsearch.yml.
This covers the "very short term" change outlined in elastic/x-pack-elasticsearch#2649
Original commit: elastic/x-pack-elasticsearch@9c381801d9
Adds the GET overall_buckets API.
The REST end point is: GET
/_xpack/ml/anomaly_detectors/job_id/results/overall_buckets
The API returns overall bucket results. An overall bucket
is a summarized bucket result over multiple jobs.
It has the `bucket_span` of the longest job's `bucket_span`.
It also has an `overall_score` that is the `top_n` average of the
max anomaly scores per job.
relates elastic/x-pack-elasticsearch#2693
Original commit: elastic/x-pack-elasticsearch@ba6061482d
The execution state is kind of a global indicator if a watch has been
running successfully and is used by the watcher UI.
However this field is only stored in the watch history but not part of
the watch status, thus it is not available everywhere. In order to
simplify the watcher UI this commit also adds the field to the
watch status which is stored together with the watch.
It is stored under the `status.execution_state` field as `status.state`
is already taken. This is also reflects with the name of the java class.
The WatchStatus class does not contain serialization checks, as this is
intended to be backported to 6.x, where those checks will be added.
Once the backport is done, the old execution state field can be fully
deleted from the master branch in another commit (syncing with Kibana
folks required).
relates elastic/x-pack-elasticsearch#2385
* fix doc tests
Original commit: elastic/x-pack-elasticsearch@26e8f99571
The path of a JIRA endpoint used to be fixed. This commit allows the
path to be dynamic, so that users can deploy their JIRA instance under
an arbitrary prefix.
Original commit: elastic/x-pack-elasticsearch@7702505114
This adds a dynamic setting, which defaults to `false` currently, that can be used to delete all `.watcher-history*` indices that match the same age requirements as Monitoring indices.
Original commit: elastic/x-pack-elasticsearch@8ca3bdbca3
Do not execute bind on on the LDAP reader thread
Each LDAP connection has a single associated thread, executing the handlers for async requests; this is managed by the LDAP library. The bind operation is blocking for the connection. It is a deadlock to call bind, if on the LDAP reader thread for the same connection, because waiting for the bind response blocks the thread processing responses (for this connection).
This will execute the bind operation (and the subsequent runnable) on a thread pool after checking for the conflict above.
Closes: elastic/x-pack-elasticsearch#2570, elastic/x-pack-elasticsearch#2620
Original commit: elastic/x-pack-elasticsearch@404a3d8737
Uses the appropriate overload of `generateRandomStringArray` to disallow empty arrays from being returned.
Original commit: elastic/x-pack-elasticsearch@2596653ca1
Since the transport ssl enabled setting is usable in 6.x again, this change adds back the value to
the xpack security usage so that it can be included in phone home data.
Original commit: elastic/x-pack-elasticsearch@52f6176df0
The upgrade API adds a "type" field to role mapping documents.
The parser would reject these docs due to an unexpected field. We now ignore the "type" field instead.
Original commit: elastic/x-pack-elasticsearch@538f5adab2
Those classes used to be elasticsearch-agnostic wrappers
of the query parameters. However, we now do not need that
layer of abstraction. Instead we can make those builders
own the building of the SearchSourceBuilder, which
simplifies the JobProvider and makes them reusable.
Original commit: elastic/x-pack-elasticsearch@b079cce1d6
Since we are authorising on a single shard of a single index,
and there are only 3 possible actions that an item might represent,
we can test which items are authorised with a maximum of 3 permission
evaluations, regardless of how many items are actually in the shard
request. Previously we would test them all independently which had
a much higher overhead for large bulk requests.
Relates: elastic/x-pack-elasticsearch#2369
Original commit: elastic/x-pack-elasticsearch@aceacf0aa3
A number of REST requests require a body but did not explicitly validate for it.
This would typically cause a NPE if they were called with no body.
Original commit: elastic/x-pack-elasticsearch@863ac89429
The test also used the timewarp trigger for watches to be executed, but it is sufficient to just call the execute watch API to make this test faster.
Original commit: elastic/x-pack-elasticsearch@3a4165f72c
When getting a single bucket, the get_buckets API can take a timestamp
either in the body or in the URL. Prior to this change, if a timestamp
was specified in the URL but a body not containing a timestamp was specified
(either empty or containing other parameters like exclude_interim or sort)
then it would cause a bad_request exception. This in turn causes problems
for clients that cannot send a body when GETting and always send a body when
POSTing.
This change fixes get_buckets to always read any timestamp in the URL, even
when a body is sent.
relates elastic/x-pack-elasticsearch#2515
Original commit: elastic/x-pack-elasticsearch@5c23dd972e
If a job close is requested after a job was opened but before
its process was launched, the job close returns successfully
without doing anything. The result is that the process hangs
around. This has been causing test failures as documented
int elastic/x-pack-elasticsearch#2360 and elastic/x-pack-elasticsearch#1270.
This commit fixes this problem by refactoring the
AutodetectProcessManager. It introduces a state pattern
to make clear the states of the process and it uses locking
to ensure a close waits for the job process to be created.
relates elastic/x-pack-elasticsearch#1270
Original commit: elastic/x-pack-elasticsearch@ff858bd136
The AuthenticationService#testInvalidToken would cause a suite timeout in the case of an exception
due to a incorrect stream size as the latch was never counted down. This fixes the missing latch
countdown.
relates elastic/x-pack-elasticsearch#2615
Original commit: elastic/x-pack-elasticsearch@e838e6e912
Checking the size of the map doesn't make sense when each filter is
checked independently right after.
Original commit: elastic/x-pack-elasticsearch@58e5d3401d
This is related to elastic/x-pack-elasticsearch#1941.
Currently we support self-generating either a basic or trial license at
cluster startup. With the addition of the basic option, it is possible
that a user would choose to self-generate and eventually register a
basic license.
This commit allows a user to upgrade to a 30-day trial license if they
have not already utilized this 30-day trial license before. Additionally
it adds a get route to check if the user is eligible to upgrade. This
route will allow kibana to implement a cleaner UI.
Original commit: elastic/x-pack-elasticsearch@7f19b33a08
This removes the creation and handling of the Beats monitoring template and its data until we actually expect to support it (most likely 6.2 - 6.3).
Original commit: elastic/x-pack-elasticsearch@2dc8abbb37
David Roberts
Tim Vernum
Tal Levy
jaymode
Chris Earle
Hendrik Muhs
Jay Modi
David Kyle
Christoph Büscher
Tim Vernum
Simon Willnauer
Chris Earle
Nik Everett
Albert Zaharovits
Tanguy Leroux
Martijn van Groningen
Nhat
Alexander Reelsen
Dimitris Athanasiou
Jason Tedor
Lee Hinman
Nhat
Tim Brooks