This cleans up logging, when starting several elasticsearch instances,
as otherwise you cannot see, which node emits this log message.
Original commit: elastic/x-pack-elasticsearch@c8c2819d86
When a watch is executed, it sends an update request to the watch to
udpate its status.
This update request also updates the status.state field, which contains
information, if the watch is active. If the watch gets executed, and
during execution a watch gets disabled, then the current execution will
set the watch back to active.
This commit fixes the current behaviour and never changes the state of
a watch when updating the status after executing, allowing
activate/deactivate calls to work as expected, regardless if a watch
is being executed.
This will fix not only the current behaviour but also some flaky tests.
Original commit: elastic/x-pack-elasticsearch@ca69109ecb
It is really hard to debug some issues with watcher, when only the
e.getMessage() is returned as failure reasons instead of the whole
stack trace.
This commit gets rid of ExceptionsHelper.detailedMessage(e) and always
returns the whole exception.
This commit also extends the watch history to have all fields named
error be treated like an object to be sure they do not get
indexed. No matter where it's placed in the hierarchy
In addition a few Field interface classes were removed, that only contained parse fields.
relates elastic/x-pack-elasticsearch#1816
Original commit: elastic/x-pack-elasticsearch@b2ce680139
This commit adds the max_running_jobs setting from elasticsearch.yml
into a node attribute called ml.max_open_jobs. Previously there was
an assumption that max_running_jobs would be the same for all nodes in
the cluster. However, during a rolling cluster restart where the value
of the setting is being changed this clearly cannot be the case, and
would cause unexpected/unpredictable limits to be used during the period
when different nodes had different settings.
For backwards compatibility, if another node in the cluster has not added
its setting for max_running_jobs to the cluster state then the old
(flawed but better than nothing) approach is applied, i.e. assume the
remote node's setting for max_running_jobs is equal to that of the node
deciding the job allocation.
Relates elastic/x-pack-elasticsearch#2185
Original commit: elastic/x-pack-elasticsearch@1e62b89183
Validating job groups during parsing results into
the validation error being wrapped into a parse
exception. The UI then does not display the cause of the
error. Finally, it is conceptually not a parse error, so
it belongs outside the parsing phase.
Original commit: elastic/x-pack-elasticsearch@a03f002bdc
Only unit tests were broken. Production ML code was always terminating
bulk requests with newlines.
Original commit: elastic/x-pack-elasticsearch@96ed06fed3
If one of the old watcher templates does not exist when we try
to delete it, the upgrade should just continue.
Original commit: elastic/x-pack-elasticsearch@6a52bad329
This removes the `IndicesStatsCollector` and, instead, it reuses the superset version of the call from the `IndexStatsCollector`.
On clusters with a large number of indices, this should actually help a good amount in reducing wasted calls and memory allocation without any difference in the output.
Original commit: elastic/x-pack-elasticsearch@93b09878e4
This commit re-enables the OpenLDAP tests that were previously running against a one-off instance
in AWS but now run against a vagrant fixture. There were some IntegTests that would run against the
OpenLDAP instance randomly but with this change they no longer run against OpenLDAP. This is ok as
the functionality that is tested by these has coverage elsewhere.
relates elastic/x-pack-elasticsearch#1823
Original commit: elastic/x-pack-elasticsearch@ac9bc82297
record_count is no longer written to new results, but is still tolerated
for backwards compatibility. However, in the backwards compatibility case
the results index must already contain the required mapping. There's no
need to add this mapping to newly created results indices.
Original commit: elastic/x-pack-elasticsearch@e586f3ba96
Fix TemplateTransformMappingTests to work, even if date rolls over
during execution.
Reenable test in BootStrapTests, was forgotten.
Remove the SecurityF/MonitoringF/WatcherF classes, as there is a gradle
command to easily start elasticsearch with xpack
Remove HasherBenchmark, as it is not a test and relies on RandomContext
that is not available anymore (also I think a JMH benchmark would be
needed here).
Remove ManualPublicSmtpServersTester, was not usable anymore.
Remove OldWatcherIndicesBackwardsCompatibilityTests, now in dedicated
rolling upgrade tests.
Remove unused EvalCron class.
Original commit: elastic/x-pack-elasticsearch@100fa9e9b0
This commit enhances job/datafeed APIs that support acting
on multiple jobs/datafeeds at once so that they accept
expressions that may contain comma-separated lists or
wildcard patterns.
The APIs that are enhances are:
- get jobs API
- get job stats API
- close job API
- get datafeeds API
- get datafeed stats API
- stop datafeed API
relates elastic/x-pack-elasticsearch#1876
Original commit: elastic/x-pack-elasticsearch@45a1139d97
This change makes the setup password tool build the default URL from the settings provided by the
environment. This will ease the amount of work a user would have to do in order to run the tool as
http vs https will be selected automatically and the port/host will as well.
Original commit: elastic/x-pack-elasticsearch@79affe4a79
This commit moves the reading of the token passphrase to the creation of the bootstrap check to
avoid issues with the secure settings keystore already being closed and thus causing issues during
startup.
Original commit: elastic/x-pack-elasticsearch@bba1cc832d
* Don’t set detector field names to empty strings
* Check summary count field and categorisation field names are not empty strings
* Check a detector has a by field when using multivariate by fields
Original commit: elastic/x-pack-elasticsearch@461be12f9f
If the native controller dies or is killed then requests to open jobs
now immediately return with an error that says what the problem is.
The error that is logged also now clearly records the problem.
Previously open job requests would time out if the native controller
was not running, and logged errors were not easy to understand without
in-depth knowledge of the ML code.
relates elastic/x-pack-elasticsearch#2140
Original commit: elastic/x-pack-elasticsearch@fc7f074d4a
This commit adds detection of support for AES 256 bit ciphers and enables their use when the JVM
supports them. For OpenJDK, this is often the case without any changes but for the Oracle JVM, the
unlimited policy file needs to be installed. In order to simplify the work a user would need to do
we can detect this support and automatically enable the AES 256 bit versions of the ciphers we
already enable.
Original commit: elastic/x-pack-elasticsearch@5f23b18a1e
Replacing integration tests with rest tests and unit tests, thus removing integration tests that require start of a new cluster. Removing unused testing methods
Original commit: elastic/x-pack-elasticsearch@265966d80c
The deprecated handlers should have been removed earlier, but are now
going to to away finally.
Also the watcher restart action has been removed, mainly because users
should not blindly restart, but always make sure, that watcher is
stopped correctly before restarting. This had been removed from the
transport action previously.
Original commit: elastic/x-pack-elasticsearch@78a5ec3c05
This flaky tests was using sleep, latches and a custom script plugin,
causing issues with stopping/starting tests.
This removes the integration tests and replaces it with a unit test.
Also removed a couple of unused ctor/method parameters as cleanup.
relates elastic/x-pack-elasticsearch#1639
Original commit: elastic/x-pack-elasticsearch@2a42faf2db
As all the triggering is now done on the shards, people can use
bulk, *-by-query and regular index/delete operations on the watcher
index and do not need to use the dedicated APIs.
We can now remove the long existing rest operation hijacking code.
Original commit: elastic/x-pack-elasticsearch@08f4f4c3de
This commit fixes the handling of some exceptions when we attempt to create the security index and
alias. The issue here is provoked by a test that is currently muted with an AwaitsFix,
GroupMappingTests, which will be unmuted in another change.
Original commit: elastic/x-pack-elasticsearch@55f6b656cb
This commit proposes removing an unnecessary directory push/pop from the
X-Pack scripts. It is not clear exactly why these were added, the
original change was almost three years ago in
elastic/x-pack@ea9ba7cdd0 but
unfortunately the commit message does not elucidate the exact the
problem, nor is there an associated pull request. This change has
propogated into all of the X-Pack scripts yet still the reasons are
unclear. The little that we can glean from the commit message is that
there was a problem with the default paths if the script was executed
outside of the Elasticsearch home. It seems that such issues have been
addressed by the recent introduction of elasticsearch-env so maybe we
can simplify these scripts here?
Relates elastic/x-pack-elasticsearch#2125
Original commit: elastic/x-pack-elasticsearch@9548c47743
When invoking the elasticsearch-env.bat or x-pack-env.bat batch scripts
on Windows, if these scripts exits due to an error (e.g., Java can not
be found, or the wrong version of Java is found), then the script
exits. Sadly, on Windows, this does not also terminate the caller,
instead returning control. This means we have to explicitly exit so that
is what we do in this commit.
Relates elastic/x-pack-elasticsearch#2126
Original commit: elastic/x-pack-elasticsearch@18645db62c
This allows 6.0+ monitoring clusters to be used out of the box with CCS for extended read-only access.
Original commit: elastic/x-pack-elasticsearch@2b1e4ca4e4
The full cluster restart tests are currently geared towards the 5.6 -> 6.0 upgrade and have some
issues when the versions are changed to 6.x -> 7.0. One issue is a real code issue in that the
security code always expects the mappings to have the same version as the version of the node, but
we no longer update the mappings on the security index during a rolling upgrade. We know look at
the index format to determine if the index is up to date.
Original commit: elastic/x-pack-elasticsearch@14c1c72ff6
This commit removes the use of a now removed --path.conf command-line
flag from the retrieve roles and users tests.
Original commit: elastic/x-pack-elasticsearch@30d5f5f648
This commit fixes the users tool command tests which were broken because
of a guard added that es.path.conf is set. We do not want to set this
system property in tests so instead we override createEnv where the
problematic guard exists.
Original commit: elastic/x-pack-elasticsearch@78b757695b
This commit fixes the list X-Pack extension command tests which were
broken because of a guard added that es.path.conf is set. We do not want
to set this system property in tests so instead we override createEnv
where the problematic guard exists.
Original commit: elastic/x-pack-elasticsearch@b1bc4ddcb8
This commit fixes some failing CLI tests. The failure here is that a
guard against the system property es.path.conf was added yet these tests
were not adapted for this change. This commit implements this adapation
which overrides the createEnv method where the problematic guard is
invoked. We do this to avoid having to set es.path.conf in tests.
Original commit: elastic/x-pack-elasticsearch@20e1724823
This commit responds to an upstream change which removes the --path.conf
command-line flag and instead uses the replacement mechanism for setting
the configuration path via the system property es.path.conf.
Relates elastic/x-pack-elasticsearch#2113
Original commit: elastic/x-pack-elasticsearch@4fefbffecb
This commit moves the creation of the CryptoService to the createComponents method so that bootstrap
checks have been checked before the crypto service is instantiated. The cryptoservice was changed to
expect that the bootstrap check has passed before being instantiated in elastic/x-pack-elasticsearch#1831.
Original commit: elastic/x-pack-elasticsearch@cf11cf4782
This commit is fixing an issue with the build. Currently we have a
scenario where a call to build a transport client is not using the
security client. This modifies the settings source to ensure that the
security client is used.
Original commit: elastic/x-pack-elasticsearch@2d6ea1f4e4
This commit improves the error message in 6x if the security index has
not been upgraded, and warns the user that the native realm will not be
functional until the upgrade API is run.
Original commit: elastic/x-pack-elasticsearch@710b7634b4
This change will provide a exception with a message to any callers of the empty trust manager for
better visibility into issues.
Original commit: elastic/x-pack-elasticsearch@c8241aea98
Since change elastic/x-pack-elasticsearch@f796949 authentication is not allowed to respond with null, it must be AuthenticationResult.notHandled()
- Fixes 1 case where the native realm would respond null if the user was not found
- Fixes some edge cases in the LDAP realm.
Original commit: elastic/x-pack-elasticsearch@bc739a1d40
In logstash parlance there really is no pipeline.name, its pipeline.id
This also removes support for deprecated gauge types `text` and `boolean` we will be removing those soon in logstash
This also renames `counters` to `long_counters` to be more explicit and for consistency with the gauge type. Also, if we ever decide to add other types of counters this will be more clear
Original commit: elastic/x-pack-elasticsearch@8f44a94579
The user/role was added to simplify they setup surrounding Beats monitoring, but Beats monitoring has been delayed until the UI work is begun.
Original commit: elastic/x-pack-elasticsearch@1c0c85562e
When security initialization fails during startup today we throw an
Error. This triggers the uncaught exception handler immediately killing
the node. While the node is going to die either way, we should not be
triggering the killer and in general we should avoid using Throwables
that extend Error.
Relates elastic/x-pack-elasticsearch#2035
Original commit: elastic/x-pack-elasticsearch@546f7f9002
This commit removes a legacy check for an unsupported environment
variable. This environment variable has not been supported since 1.x so
it is safe to stop checking for the existence of this setting.
Relates elastic/x-pack-elasticsearch#2048
Original commit: elastic/x-pack-elasticsearch@023230fa9e
Today we explicitly export the HOSTNAME variable from scripts. This is
probably a relic from the days when the scripts were not run on bash but
instead assume a POSIX-compliant shell only where HOSTNAME is not
guaranteed to exist. Yet, bash guarantees that HOSTNAME is set so we do
not need to set it in scripts. This commit removes this legacy.
Relates elastic/x-pack-elasticsearch#2047
Original commit: elastic/x-pack-elasticsearch@7b833e061c
This commit fixes the croneval script to respect the CONF_DIR
environment variable used to locate the configuration directory.
Original commit: elastic/x-pack-elasticsearch@79974947f9
This commit fixes an issue with the usage of the environment variable
ES_HOME in the certgen script; the script was missing the use of $ to
obtain the value of the environment variable ES_HOME.
Relates elastic/x-pack-elasticsearch#2046
Original commit: elastic/x-pack-elasticsearch@63128db0eb
With the new template and mapping update mechanisms introduced as part
of the Upgrade API work, the NativeRealmMigrator is no longer needed or
used. This commit removes the NativeRealmMigrator code and the
associated tests for it.
Original commit: elastic/x-pack-elasticsearch@5d2d7a582c
This is related to elastic/x-pack-elasticsearch#1778. This commit adds a setting
(xpack.license.self_generated.type) which allows the user to specify
what type of license will be self-generated on node startup. The allowed
types are basic or trial.
Original commit: elastic/x-pack-elasticsearch@0a16a59e10
To achieve backwards compatibility the easiest way is
to restore the old watcher stats, which are supposed to run
on the master node only.
The distributed watcher stats have been moved under the statsdist
package and the action name has been changed as well.
This way there is no need to have a serialization BWC layer,
we can just call different actions.
Note: With the current approach developers still need to change
their java applications if they try to receive watcher stats,
as by default we are now using the distributed stats in the
watcher client.
Original commit: elastic/x-pack-elasticsearch@49b3a45452
Alexander Reelsen
Colin Goodheart-Smithe
David Roberts
Dimitris Athanasiou
Chris Earle
Tim Sullivan
Jay Modi
David Kyle
Tim Brooks
Luca Cavanna
Ryan Ernst
Jason Tedor
Lisa Cawley
Yannick Welsch
Martijn van Groningen
Igor Motov
Simon Willnauer
Tim Vernum
Michael Basnight
Tal Levy
jaymode
Ali Beyad
Andrew Cholakian
Jack Conradson
Adrien Grand
markharwood