Commit Graph

260 Commits

Author SHA1 Message Date
Jason Tedor e2c8f8dfa3
Rename ILM package to ilm (#44725)
This commit renames the ILM package from indexlifecycle to ilm. We have
all come to know index lifecycle management as ILM, the APIs and
settings use ilm, and it would be nice of the package did too. This
commit makes that change.
2019-07-23 16:46:38 +09:00
Ioannis Kakavas 3714cb63da Allow parsing the value of java.version sysprop (#44017)
We often start testing with early access versions of new Java
versions and this have caused minor issues in our tests
(i.e. #43141) because the version string that the JVM reports
cannot be parsed as it ends with the string -ea.

This commit changes how we parse and compare Java versions to
allow correct parsing and comparison of the output of java.version
system property that might include an additional alphanumeric
part after the version numbers
 (see [JEP 223[(https://openjdk.java.net/jeps/223)). In short it 
handles a version number part, like before, but additionally a 
PRE part that matches ([a-zA-Z0-9]+).

It also changes a number of tests that would attempt to parse
java.specification.version in order to get the full version
of Java. java.specification.version only contains the major
version and is thus inappropriate when trying to compare against
a version that might contain a minor, patch or an early access
part. We know parse java.version that can be consistently
parsed.

Resolves #43141
2019-07-22 20:14:56 +03:00
Tal Levy c8a8915b27
migrate rollup/monitoring/graph/watcher actions to Writeable (#44464) (#44538)
this commit migrates leftover actions from a few x-pack plugins
to the new Writeable.Reader infrastructure.

relates #34389.
2019-07-18 08:42:56 -07:00
Ryan Ernst 17c4b2b839
Convert MasterNodeRequest to implement Writeable.Reader (#44452) (#44513)
This commit converts all MasterNodeRequest subclasses to fullfill
Writeable.Reader constructors.

relates #34389
2019-07-17 18:01:29 -07:00
Jason Tedor 39c5f98de7
Introduce test issue logging (#44477)
Today we have an annotation for controlling logging levels in
tests. This annotation serves two purposes, one is to control the
logging level used in tests, when such control is needed to impact and
assert the behavior of loggers in tests. The other use is when a test is
failing and additional logging is needed. This commit separates these
two concerns into separate annotations.

The primary motivation for this is that we have a history of leaving
behind the annotation for the purpose of investigating test failures
long after the test failure is resolved. The accumulation of these stale
logging annotations has led to excessive disk consumption. Having
recently cleaned this up, we would like to avoid falling into this state
again. To do this, we are adding a link to the test failure under
investigation to the annotation when used for the purpose of
investigating test failures. We will add tooling to inspect these
annotations, in the same way that we have tooling on awaits fix
annotations. This will enable us to report on the use of these
annotations, and report when stale uses of the annotation exist.
2019-07-18 05:33:33 +09:00
Jake Landis eb7d43f4cf
Log write failures for watcher history document. (#44129) (#44357)
The failure is correctly getting propagated, this commit adds support to
explicitly look for .watch-history failures using the same logging strategy
as triggered watch failures.
2019-07-16 08:48:09 -05:00
Ryan Ernst e0b82e92f3
Convert BaseNode(s) Request/Response classes to Writeable (#44301) (#44358)
This commit converts all BaseNodeResponse and BaseNodesResponse
subclasses to implement Writeable.Reader instead of Streamable.

relates #34389
2019-07-15 18:07:52 -07:00
Ryan Ernst 59658daef9
Separate streamable based master node actions (#44313)
This commit creates new base classes for master node actions whose
response types still implement Streamable. This simplifies both finding
remaining classes to convert, as well as creating new master node
actions that use Writeable for their responses.

relates #34389
2019-07-15 09:20:20 -07:00
Ryan Ernst 1dcf53465c Reorder HandledTransportAction ctor args (#44291)
This commit moves the Supplier variant of HandledTransportAction to have
a different ordering than the Writeable.Reader variant. The Supplier
version is used for the legacy Streamable, and currently having the
location of the Writeable.Reader vs Supplier in the same place forces
using casts of Writeable.Reader to select the correct super constructor.
This change in ordering allows easier migration to Writeable.Reader.

relates #34389
2019-07-12 13:45:09 -07:00
Ryan Ernst c6efb9be2a Convert ReplicationResponse to Writeable (#43953)
This commit convers ReplicationResponse and all its subclasses to
support Writeable.Reader as a constructor.

relates #34389
2019-07-10 12:45:10 -07:00
Alexander Reelsen 9077c4402f Watcher: Allow to execute actions for each element in array (#41997)
This adds the ability to execute an action for each element that occurs
in an array, for example you could sent a dedicated slack action for
each search hit returned from a search.

There is also a limit for the number of actions executed, which is
hardcoded to 100 right now, to prevent having watches run forever.

The watch history logs each action result and the total number of actions
the were executed.

Relates #34546
2019-07-03 11:28:50 +02:00
Jake Landis eb73bed40d
7x watcher backport testfixes (#43848)
* fix org.elasticsearch.xpack.watcher.test.integration.RejectedExecutionTests (#41777)

This commit un-mutes org.elasticsearch.xpack.watcher.test.integration.RejectedExecutionTests
which was failing intermittently due to a logic bug. It is not possible to use the real
Watcher scheduler (which is needed for this test) and reliabliby count the .triggered-watches
since current count of documents in the .triggered-watches index is based on the timing of the
scheduler and the ability to delete based on the Watcher and Write thread pools.

This commit simply removes the .triggered-watch check and relies soley on the .watcher-history
index as an indication that operations that can occur when the Watcher threadpool is rejecting.

closes #41734

* fix unlikely bug that can prevent Watcher from restarting (#42030)

The bug fixed here is unlikely to happen. It requires ES to be started with
ILM disabled, Watcher enabled, and Watcher explicitly stopped and restarted.
Due to template validation Watcher does not fully start and can result in a
partially started state. This is an unlikely scenerio outside of the testing
framework.

Note - this bug was introduced while the test that would have caught it was
muted. The test remains muted since the underlying cuase of the random failures
has not been identified. When this test is un-muted it will now work.
2019-07-02 12:16:06 -05:00
Albert Zaharovits 5e17bc5dcc
Consistent Secure Settings #40416
Introduces a new `ConsistentSecureSettingsValidatorService` service that exposes
a single public method, namely `allSecureSettingsConsistent`. The method returns
`true` if the local node's secure settings (inside the keystore) are equal to the
master's, and `false` otherwise. Technically, the local node has to have exactly
the same secure settings - setting names should not be missing or in surplus -
for all `SecureSetting` instances that are flagged with the newly introduced
`Property.Consistent`. It is worth highlighting that the `allSecureSettingsConsistent`
is not a consensus view across the cluster, but rather the local node's perspective
in relation to the master.
2019-06-29 23:26:17 +03:00
Ryan Ernst 5b4089e57e
Remove nodeId from BaseNodeRequest (#43658)
TransportNodesAction provides a mechanism to easily broadcast a request
to many nodes, and collect the respones into a high level response. Each
node has its own request type, with a base class of BaseNodeRequest.
This base request requires passing the nodeId to which the request will
be sent. However, that nodeId is not used anywhere. It is private to the
base class, yet serialized to each node, where the node could just as
easily find the nodeId of the node it is on locally.

This commit removes passing the nodeId through to the node request
creation, and guards its serialization so that we can remove the base
request class altogether in the future.
2019-06-27 18:45:14 -07:00
Martijn van Groningen 101cf384ba
Replace Streamable w/ Writable in AcknowledgedResponse and subclasses (backport 7.x) (#43525)
This commit replaces usages of Streamable with Writeable for the
AcknowledgedResponse and its subclasses, plus associated actions.

Note that where possible response fields were made final and default
constructors were removed.

This is a large PR, but the change is mostly mechanical.

Relates to #34389
Backport of #43414
2019-06-24 13:47:37 +02:00
Jason Tedor 1f1a035def
Remove stale test logging annotations (#43403)
This commit removes some very old test logging annotations that appeared
to be added to investigate test failures that are long since closed. If
these are needed, they can be added back on a case-by-case basis with a
comment associating them to a test failure.
2019-06-19 22:58:22 -04:00
Jason Tedor 5bc3b7f741
Enable node roles to be pluggable (#43175)
This commit introduces the possibility for a plugin to introduce
additional node roles.
2019-06-13 15:15:48 -04:00
Ryan Ernst 172cd4dbfa Remove description from xpack feature sets (#43065)
The description field of xpack featuresets is optionally part of the
xpack info api, when using the verbose flag. However, this information
is unnecessary, as it is better left for documentation (and the existing
descriptions describe anything meaningful). This commit removes the
description field from feature sets.
2019-06-11 09:22:58 -07:00
James Baiera 1300183001
NullPointerException when creating a watch with Jira action (#41922) (#42081) (#42873)
NullPointerException when secured_url does not use proper scheme in jira action. 
This commit will handle Expection and display proper message.
2019-06-05 16:03:07 -04:00
Jason Tedor aad1b3a2a0
Fix version parsing in various tests (#42871)
This commit fixes the version parsing in various tests. The issue here is that
the parsing was relying on java.version. However, java.version can contain
additional characters such as -ea for early access builds. See JEP 233:

Name                            Syntax
------------------------------  --------------
java.version                    $VNUM(\-$PRE)?
java.runtime.version            $VSTR
java.vm.version                 $VSTR
java.specification.version      $VNUM
java.vm.specification.version   $VNUM

Instead, we want java.specification.version.
2019-06-04 18:22:20 -04:00
Mark Vieira e44b8b1e2e
[Backport] Remove dependency substitutions 7.x (#42866)
* Remove unnecessary usage of Gradle dependency substitution rules (#42773)

(cherry picked from commit 12d583dbf6f7d44f00aa365e34fc7e937c3c61f7)
2019-06-04 13:50:23 -07:00
Mark Vieira c1816354ed
[Backport] Improve build configuration time (#42674) 2019-05-30 10:29:42 -07:00
Ryan Ernst a49bafc194
Split document and metadata fields in GetResult (#38373) (#42456)
This commit makes creators of GetField split the fields into document fields and metadata fields. It is part of larger refactoring that aims to remove the calls to static methods of MapperService related to metadata fields, as discussed in #24422.
2019-05-23 14:01:07 -07:00
Jay Modi dbbdcea128
Update ciphers for TLSv1.3 and JDK11 if available (#42082)
This commit updates the default ciphers and TLS protocols that are used
when the runtime JDK supports them. New cipher support has been
introduced in JDK 11 and 12 along with performance fixes for AES GCM.
The ciphers are ordered with PFS ciphers being most preferred, then
AEAD ciphers, and finally those with mainstream hardware support. When
available stronger encryption is preferred for a given cipher.

This is a backport of #41385 and #41808. There are known JDK bugs with
TLSv1.3 that have been fixed in various versions. These are:

1. The JDK's bundled HttpsServer will endless loop under JDK11 and JDK
12.0 (Fixed in 12.0.1) based on the way the Apache HttpClient performs
a close (half close).
2. In all versions of JDK 11 and 12, the HttpsServer will endless loop
when certificates are not trusted or another handshake error occurs. An
email has been sent to the openjdk security-dev list and #38646 is open
to track this.
3. In JDK 11.0.2 and prior there is a race condition with session
resumption that leads to handshake errors when multiple concurrent
handshakes are going on between the same client and server. This bug
does not appear when client authentication is in use. This is
JDK-8213202, which was fixed in 11.0.3 and 12.0.
4. In JDK 11.0.2 and prior there is a bug where resumed TLS sessions do
not retain peer certificate information. This is JDK-8212885.

The way these issues are addressed is that the current java version is
checked and used to determine the supported protocols for tests that
provoke these issues.
2019-05-20 09:45:36 -04:00
Ryan Ernst 6fd8924c5a Switch run task to use real distro (#41590)
The run task is supposed to run elasticsearch with the given plugin or
module. However, for modules, this is most realistic if using the full
distribution. This commit changes the run setup to use the default or
oss as appropriate.
2019-05-06 12:34:07 -07:00
Christoph Büscher bf1a70f242 Mute RejectedExecutionTests#testHistoryAndTriggeredOnRejection 2019-05-02 11:50:04 +02:00
Jake Landis 7ac3283509
Fix Watcher deadlock that can cause in-abilty to index documents. (#41418) (#41684)
This commit removes the usage of the `BulkProcessor` to write history documents
and delete triggered watches on a `EsRejectedExecutionException`. Since the
exception could be handled on the write thread, the write thread can be blocked
waiting on watcher threads (due to a synchronous method). This is problematic
since those watcher threads can be blocked waiting on write threads.

This commit also moves the handling of the exception to the generic threadpool
to avoid submitting write requests from the write thread pool.

fixes #41390
2019-04-30 16:13:24 -05:00
Yogesh Gaikwad c0d40ae4ca
Remove deprecated stashWithOrigin calls and use the alternative (#40847) (#41562)
This commit removes the deprecated `stashWithOrigin` and
modifies its usage to use the alternative.
2019-04-28 21:25:42 +10:00
Christoph Büscher 52495843cc [Docs] Fix common word repetitions (#39703) 2019-04-25 20:47:47 +02:00
Gordon Brown 66366d0307
Extract template management from Watcher (#41169)
This commit extracts the template management from Watcher into an
abstract class, so that templates and lifecycle policies can be managed
in the same way across multiple plugins. This will be useful for SLM, as
well as potentially ILM and any other plugins which need to manage index
templates.
2019-04-17 13:42:36 -06:00
Martijn van Groningen f56b2ecb37
Remove xpack dependencies from qa rest modules (#41134) (7.x backport) (#41202)
This commit removes xpack dependencies of many xpack qa modules.
(for some qa modules this will require some more work)

The reason behind this change is that qa rest modules should not depend
on the x-pack plugins, because the plugins are an implementation detail and
the tests should only know about the rest interface and qa cluster that is
being tested.

Also some qa modules rely on xpack plugins and hlrc (which is a valid
dependency for rest qa tests) creates a cyclic dependency and this is
something that we should avoid.  Also Eclipse can't handle gradle cyclic
dependencies (see #41064).

* don't copy xpack-core's plugin property into the test resource of qa
modules. Otherwise installing security manager fails, because it tries
to find the XPackPlugin class.
2019-04-15 19:14:43 +02:00
Christoph Büscher 2980a6c70f Clarify some ToXContent implementations behaviour (#41000)
This change adds either ToXContentObject or ToXContentFragment to classes
directly implementing ToXContent currently. This helps in reasoning about
whether those implementations output full xcontent object or just fragments.

Relates to #16347
2019-04-15 09:42:08 +02:00
Gordon Brown c8bc4ab003
Improve Watcher test framework resiliency (#40658)
It is possible for the watches tracked by ScheduleTriggerEngineMock to
get out of sync with the Watches in the ScheduleTriggerEngine
production code, which can lead to watches failing to run.

This commit:

1. Changes TimeWarp to try to run the watch on all schedulers, rather than stopping after one which claims to have the watch registered. This reduces the impact of desynchronization between the mocking code and the backing production code.
2. Makes ScheduleTriggerEngineMock respect pauses of execution again. This is necessary to prevent duplicate watch invocations due to the above change.
3. Tweaks how watches are registered in ScheduleTriggerEngineMock to prevent race conditions due to concurrent modification.
4. Tweaks WatcherConcreteIndexTests to use TimeWarp instead of waiting for watches to be triggered, as TimeWarp is more reliable and accomplishes the same goal.
2019-04-12 16:53:50 -06:00
Lee Hinman 7b190609ab
(7.x) Use environment settings instead of state settings for Watcher config (#41158)
Backport of (#41087)

* Use environment settings instead of state settings for Watcher config

Prior to this we used the settings from cluster state to see whether ILM was
enabled of disabled, however, these settings don't accurately reflect the
`xpack.ilm.enabled` setting in `elasticsearch.yml`.

This commit changes to using the `Environment` settings, which correctly reflect
the ILM enabled setting.

Resolves #41042
2019-04-12 12:15:44 -06:00
Mark Vieira 1287c7d91f
[Backport] Replace usages RandomizedTestingTask with built-in Gradle Test (#40978) (#40993)
* Replace usages RandomizedTestingTask with built-in Gradle Test (#40978)

This commit replaces the existing RandomizedTestingTask and supporting code with Gradle's built-in JUnit support via the Test task type. Additionally, the previous workaround to disable all tasks named "test" and create new unit testing tasks named "unitTest" has been removed such that the "test" task now runs unit tests as per the normal Gradle Java plugin conventions.

(cherry picked from commit 323f312bbc829a63056a79ebe45adced5099f6e6)

* Fix forking JVM runner

* Don't bump shadow plugin version
2019-04-09 11:52:50 -07:00
Jay Modi f34663282c
Update apache httpclient to version 4.5.8 (#40875)
This change updates our version of httpclient to version 4.5.8, which
contains the fix for HTTPCLIENT-1968, which is a bug where the client
started re-writing paths that contained encoded reserved characters
with their unreserved form.
2019-04-05 13:48:10 -06:00
Martijn van Groningen 809a5f13a4
Make -try xlint warning disabled by default. (#40833)
Many gradle projects specifically use the -try exclude flag, because
there are many cases where auto-closeable resource ignore is never
referenced in body of corresponding try statement. Suppressing this
warning specifically in each case that it happens using
`@SuppressWarnings("try")` would be very verbose.

This change removes `-try` from any gradle project and adds it to the
build plugin. Also this change removes exclude flags from gradle projects
that is already specified in build plugin (for example -deprecation).

Relates to #40366
2019-04-05 08:02:26 +02:00
Alexander Reelsen 6ac307d468 Replace javax activation with jakarta activation (#40247)
The eclipse foundation has taken over the javax.activation dependency, which
resulted in a naming change of the dependency.
2019-04-04 10:28:24 +02:00
David Roberts 324fef0548
[TEST] Mute WatchMetadataTests.testWatchMetadata
Due to https://github.com/elastic/elasticsearch/issues/40631
2019-04-03 13:56:33 +02:00
David Roberts 54b0ff8e33 [TEST] Mute WebhookHttpsIntegrationTests.testHttps
Due to https://github.com/elastic/elasticsearch/issues/35503
2019-04-01 14:20:01 +01:00
Yannick Welsch 8b9a006740 Mute testHttpInput
Relates #40587
2019-03-28 14:05:51 +01:00
Yannick Welsch ddc385d874 Mute WatchAckTests.testAckAllActions
Relates to #35506
2019-03-28 12:13:37 +01:00
Like d57b7a2d5b Migrate Streamable to Writeable for WatchStatus (#37390) 2019-03-27 07:18:35 -07:00
Benjamin Trent 22e688a709
Muting watcher tests for issue #35503 (#40527)
* Muting watcher tests for issue #35503

* blocking the two troublsome suites
2019-03-27 09:16:06 -05:00
Julie Tibshirani 25954a8dd3 Stop clearing all watches in watcher integration tests. (#39724) 2019-03-26 13:14:33 -07:00
Alexander Reelsen c46dd6ad08 Replace java mail with jakarta mail (#40088)
The eclipse foundation has taken over the javax mail dependency, which
resulted in a naming change of the dependency.
2019-03-19 09:56:44 +01:00
Jason Tedor 73a672b8dd
Fix Watcher stats class cast exception (#39821)
The watcher stats implementation tries to look at all queued watches
before preparing the result. We want to cast these to a
WatchExecutionTask to extract the context to prepare the stats for
queued watches. The problem is that not all tasks on the watcher queue
were WatchExecutionTask. This is because a manually executed watch was
not even at all wrapped in a WatchExecutionTask. Moreover, we were using
ExecutorService#submit(Runnable) which would wrap the Runnable in a
FutureTask<?>. This commit addresses this by using a WatchExecutionTask,
and also using ExecutorService#execute(Runnable) so that no wrapping
occurs. This will let us continue with the assumption that all queued
tasks are WatchExecutionTasks.
2019-03-08 14:52:10 -05:00
Jake Landis e0abc3ce96
Remove the index type from internal watcher indexes (#39761) (#39853)
This commit removes the "doc" type from watcher internal indexes.
The template still carries the "_doc" type since that is needed for
the internal representation.

This impacts the .watches, .triggered-watches, and .watch-history indexes.

External consumers do not need any changes since all external calls
go through the _watcher API, and should not interact with the the .index directly.

Relates #38637
2019-03-08 12:46:36 -06:00
Gordon Brown eb288a6f85
Use any index specified by .watches for Watcher (#39541) (#39708)
Previously, Watcher only attached its listener to indices that started
with the prefix `.watches`, which causes Watcher to silently fail to
schedule newly created Watches if the `.watches` alias is redirected to
an index that does not start with `.watches`.

Watcher now attaches the listener to all indices, so that Watcher can
respond to changes in which index has the `.watches` alias.

Also adjusts the tests to randomly use non-prefixed concrete indices 
for .watches and .triggered_watches.
2019-03-05 11:45:34 -07:00
Lee Hinman ad8228aec9
Use non-ILM template setting up watch history template & ILM disabled (#39420)
Backport of #39325

When ILM is disabled and Watcher is setting up the templates and policies for
the watch history indices, it will now use a template that does not have the
`index.lifecycle.name` setting, so that indices are not created with the
setting.

This also adds tests for the behavior, and changes the cluster state used in
these tests to be real instead of mocked.

Resolves #38805
2019-02-27 11:11:19 -07:00
Lee Hinman 7b8178c839
Remove Hipchat support from Watcher (#39374)
* Remove Hipchat support from Watcher (#39199)

Hipchat has been shut down and has previously been deprecated in
Watcher (#39160), therefore we should remove support for these actions.

* Add migrate note
2019-02-25 15:08:46 -07:00
Julie Tibshirani 29243f7001 Avoid using TimeWarp in TransformIntegrationTests. (#39277)
This commit makes `TransformIntegrationTests` into a standard integration test, as
opposed to using `TimeWarp`, which registers the mock component
`ScheduleEngineTriggerMock` to trigger watches.

The simplification may help with flakiness we've observed `TimeWarp, as in #37882.
2019-02-21 18:02:44 -08:00
Tim Vernum 4aa50ed348
Resolve concurrency with watcher trigger service (#39164)
The watcher trigger service could attempt to modify the perWatchStats
map simultaneously from multiple threads. This would cause the
internal state to become inconsistent, in particular the count()
method may return an incorrect value for the number of watches.

This changes replaces the implementation of the map with a
ConcurrentHashMap so that its internal state remains consistent even
when accessed from mutiple threads.

Backport of: #39092
2019-02-20 19:18:00 +11:00
Julie Tibshirani f5b28ca69d Enable test logging for TransformIntegrationTests#testSearchTransform.
There is already fairly detailed debug logging in the watcher framework, which
should hopefully help debug the failure.

Relates to #37882.
2019-02-19 18:15:34 -08:00
Lee Hinman 60c1dcde88 Only flush Watcher's bulk processor if Watcher is enabled (#38803)
When shutting down Watcher, the `bulkProcessor` is null if watcher has been
disabled in the configuration. This protects the flush and close calls with a
check for watcher enabled to avoid a NullPointerException

Resolves #38798
2019-02-13 16:13:53 -07:00
Przemyslaw Gomulka 542ee5f46a
Format Watcher.status.lastChecked and lastMetCondition (#38788) backport#38626
Change the formatting for Watcher.status.lastCheck and lastMetCondition
to be the same as Watcher.status.state.timestamp. These should all have
only millisecond precision
closes #38619
backport #38626
2019-02-13 08:33:53 +01:00
Przemyslaw Gomulka 7e178aa4a7
Enable IndexActionTests and WatcherIndexingListenerTests Backport #38738
fix tests to use clock in milliseconds precision in watcher code
make sure the date comparison in string format is using same formatters
some of the code was modified in #38514 possibly because of merge conflicts

closes #38581
Backport #38738
2019-02-12 13:05:44 +01:00
Przemyslaw Gomulka ba9a4d13e1
mute Failing tests related to logging and joda-java migration backport(#38704)(#38710)
the tests awaits fix from #38693 and #38705 and #38581
2019-02-11 13:15:12 +01:00
Przemyslaw Gomulka 0e5a734e7e
Fix HistoryIntegrationTests timestamp comparison #38565 Backport#38505
When the millisecond part of a timestamp is 0 the toString
representation in java-time is omitting the millisecond part (joda was
not). The Search response is returning timestamps formatted with
WatcherDateTimeUtils, therefore comparisons of strings should be done
with the same formatter

relates #27330
BackPort #38505
2019-02-11 08:50:21 +01:00
Christoph Büscher 779673c792 Mute failing WatchStatusIntegrationTests (#38621)
Relates to #38619
2019-02-08 13:56:47 +01:00
Jason Tedor fdf6b3f23f
Add 7.1 version constant to 7.x branch (#38513)
This commit adds the 7.1 version constant to the 7.x branch.

Co-authored-by: Andy Bristol <andy.bristol@elastic.co>
Co-authored-by: Tim Brooks <tim@uncontended.net>
Co-authored-by: Christoph Büscher <cbuescher@posteo.de>
Co-authored-by: Luca Cavanna <javanna@users.noreply.github.com>
Co-authored-by: markharwood <markharwood@gmail.com>
Co-authored-by: Ioannis Kakavas <ioannis@elastic.co>
Co-authored-by: Nhat Nguyen <nhat.nguyen@elastic.co>
Co-authored-by: David Roberts <dave.roberts@elastic.co>
Co-authored-by: Jason Tedor <jason@tedor.me>
Co-authored-by: Alpar Torok <torokalpar@gmail.com>
Co-authored-by: David Turner <david.turner@elastic.co>
Co-authored-by: Martijn van Groningen <martijn.v.groningen@gmail.com>
Co-authored-by: Tim Vernum <tim@adjective.org>
Co-authored-by: Albert Zaharovits <albert.zaharovits@gmail.com>
2019-02-07 16:32:27 -05:00
David Turner 5a3c452480
Align docs etc with new discovery setting names (#38492)
In #38333 and #38350 we moved away from the `discovery.zen` settings namespace
since these settings have an effect even though Zen Discovery itself is being
phased out. This change aligns the documentation and the names of related
classes and methods with the newly-introduced naming conventions.
2019-02-06 11:34:38 +00:00
Przemyslaw Gomulka df4eb0485d
Enable CronEvalToolTest.testEnsureDateIsShownInRootLocale (#38394)
The test is now expected to be always passing no matter what the random
locale is. This is fixed with using jdk ZoneId.systemDefault() in both
the test and CronEvalTool

closes #35687
2019-02-05 17:48:47 +01:00
David Turner 3b2a0d7959
Rename no-master-block setting (#38350)
Replaces `discovery.zen.no_master_block` with `cluster.no_master_block`. Any
value set for the old setting is now ignored.
2019-02-05 08:47:56 +00:00
David Turner 2d114a02ff
Rename static Zen1 settings (#38333)
Renames the following settings to remove the mention of `zen` in their names:

- `discovery.zen.hosts_provider` -> `discovery.seed_providers`
- `discovery.zen.ping.unicast.concurrent_connects` -> `discovery.seed_resolver.max_concurrent_resolvers`
- `discovery.zen.ping.unicast.hosts.resolve_timeout` -> `discovery.seed_resolver.timeout`
- `discovery.zen.ping.unicast.hosts` -> `discovery.seed_addresses`
2019-02-05 08:46:52 +00:00
Przemyslaw Gomulka 9b64558efb
Migrating from joda to java.time. Watcher plugin (#35809)
part of the migrating joda time work. Migrating watcher plugin to use JDK's java-time

refers #27330
2019-02-04 15:08:31 +01:00
Boaz Leskes f6e06a2b19 Adapt minimum versions for seq# powered operations in Watch related requests and UpdateRequest (#38231)
After backporting #37977, #37857 and #37872
2019-02-01 20:37:16 -05:00
Boaz Leskes b11732104f
Move watcher to use seq# and primary term for concurrency control (#37977)
* move watcher to seq# occ

* top level set

* fix parsing and missing setters

* share toXContent for PutResponse and rest end point

* fix redacted password

* fix username reference

* fix deactivate-watch.asciidoc have seq no references

* add seq# + term to activate-watch.asciidoc

* more doc fixes
2019-01-30 20:14:59 -05:00
Jay Modi 54dbf9469c
Update httpclient for JDK 11 TLS engine (#37994)
The apache commons http client implementations recently released
versions that solve TLS compatibility issues with the new TLS engine
that supports TLSv1.3 with JDK 11. This change updates our code to
use these versions since JDK 11 is a supported JDK and we should
allow the use of TLSv1.3.
2019-01-30 14:24:29 -07:00
Luca Cavanna 2325fb9cb3
Remove test only SearchShardTarget constructor (#37912)
Remove SearchShardTarget test only constructor and replace all the usages with calls to the other constructor that accepts a ShardId.
2019-01-29 14:58:11 +01:00
Like 6ed35fbb94 Support merge nested Map in list for JIRA configurations (#37634)
This commit allows JIRA API fields that require a list of key/value 
pairs (maps), such as JIRA "components" to use use template snippets 
(e.g. {{ctx.payload.foo}}). Prior to this change the templated value 
(not the de-referenced value) would be sent via the API and error. 

Closes #30068
2019-01-28 18:01:09 -06:00
Jake Landis 99b75a9bdf
deprecate types for watcher (#37594)
This commit adds deprecation warnings for index actions
and search actions when executed via watcher. Unit and 
integration tests updated accordingly. 

relates #35190
2019-01-28 13:46:43 -06:00
Julie Tibshirani 455f223c3a Mute TransformIntegrationTests#testSearchTransform
Tracked in #37882.
2019-01-25 11:12:45 -08:00
Martijn Laarman dfecb256cb
Exit batch files explictly using ERRORLEVEL (#29583)
* Exit batch files explictly using ERRORLEVEL

This makes sure the exit code is preserved when calling the batch
files from different contexts other than DOS

Fixes #29582

This also fixes specific error codes being masked by an explict

exit /b 1

causing the useful exitcodes from ExitCodes to be lost.

* fix line breaks for calling cli to match the bash scripts

* indent size of bash files is 2, make sure editorconfig does the same for bat files

* update indenting to match bash files

* update elasticsearch-keystore.bat indenting

* Update elasticsearch-node.bat to exit outside of endlocal
2019-01-25 16:44:33 +01:00
Lee Hinman 427bc7f940
Use ILM for Watcher history deletion (#37443)
* Use ILM for Watcher history deletion

This commit adds an index lifecycle policy for the `.watch-history-*` indices.
This policy is automatically used for all new watch history indices.

This does not yet remove the automatic cleanup that the monitoring plugin does
for the .watch-history indices, and it does not touch the
`xpack.watcher.history.cleaner_service.enabled` setting.

Relates to #32041
2019-01-23 10:18:08 -07:00
Albert Zaharovits 5308746270
Remove Watcher Account "unsecure" settings (#36736)
Removes all sensitive settings (passwords, auth tokens, urls, etc...) for
watcher notifications accounts. These settings were deprecated (and
herein removed) in favor of their secure sibling that is set inside the
elasticsearch keystore. For example:
`xpack.notification.email.account.<id>.smtp.password`
is no longer a valid setting, and it is replaced by
`xpack.notification.email.account.<id>.smtp.secure_password`
2019-01-20 12:51:24 +02:00
Jay Modi f3edbe2911
Security: remove SSL settings fallback (#36846)
This commit removes the fallback for SSL settings. While this may be
seen as a non user friendly change, the intention behind this change
is to simplify the reasoning needed to understand what is actually
being used for a given SSL configuration. Each configuration now needs
to be explicitly specified as there is no global configuration or
fallback to some other configuration.

Closes #29797
2019-01-14 14:06:22 -07:00
Alexander Reelsen bbd093059f
Add whitelist to watcher HttpClient (#36817)
This adds a configurable whitelist to the HTTP client in watcher. By
default every URL is allowed to retain BWC. A dynamically configurable
setting named "xpack.http.whitelist" was added that allows to
configure an array of URLs, which can also contain simple regexes.

Closes #29937
2019-01-11 09:22:47 +01:00
Alpar Torok 3d66764660 Mute watcher SingleNodeTests
Tracking:  #36782
2019-01-10 12:23:29 +02:00
Christophe Bismuth 9602d794c6 Separate out validation of groups of settings (#34184)
Today, a setting can declare that its validity depends on the values of other
related settings. However, the validity of a setting is not always checked
against the correct values of its dependent settings because those settings'
correct values may not be available when the validator runs.

This commit separates the validation of a settings updates into two phases,
with separate methods on the `Setting.Validator` interface. In the first phase
the setting's validity is checked in isolation, and in the second phase it is
checked again against the values of its related settings. Most settings only
use the first phase, and only the few settings with dependencies make use of
the second phase.
2019-01-07 16:12:58 +00:00
Alpar Torok a7c3d5842a
Split third party audit exclusions by type (#36763) 2019-01-07 17:24:19 +02:00
Tanguy Leroux 6347461146
Rename ClusterBlocks.hasGlobalBlock methods (#36941)
As suggested in #36775, this pull request renames the following methods:

ClusterBlocks.hasGlobalBlock(int)
ClusterBlocks.hasGlobalBlock(RestStatus)
ClusterBlocks.hasGlobalBlock(ClusterBlockLevel)

to something that better reflects the property of the ClusterBlock that is searched for:

ClusterBlocks.hasGlobalBlockWithId(int)
ClusterBlocks.hasGlobalBlockWithStatus(RestStatus)
ClusterBlocks.hasGlobalBlockWithLevel(ClusterBlockLevel)
2019-01-07 09:42:27 +01:00
Michael Basnight 7cbf03c001
Scripting: Remove deprecated params.ctx (#36848)
When the script contexts were created in 6, the use of params.ctx was
deprecated. This commit cleans up that code and ensures that params.ctx
is null in both watcher script contexts.

Relates: #34059
2018-12-20 21:30:24 -06:00
Michael Basnight d00780d00c
Watcher: Remove unused local variable in doExecute (#36655) 2018-12-20 13:43:10 -06:00
Alpar Torok e9ef5bdce8
Converting randomized testing to create a separate unitTest task instead of replacing the builtin test task (#36311)
- Create a separate unitTest task instead of Gradle's built in 
- convert all configuration to use the new task 
- the  built in task is now disabled
2018-12-19 08:25:20 +02:00
Alexander Reelsen 00521a5b36
Watcher: Ensure all internal search requests count hits (#36697)
In previous commits only the stored toXContent version of a search
request was using the old format. However an executed search request was
already disabling hit counts. In 7.0 hit counts will stay enabled by
default to allow for proper migration.

Closes #36177
2018-12-18 10:11:39 +01:00
Boaz Leskes e356b8cb95
Add doc's sequence number + primary term to GetResult and use it for updates (#36680)
This commit adds the last sequence number and primary term of the last operation that have
modified a document to `GetResult` and uses it to power the Update API.

Relates #36148 
Relates #10708
2018-12-17 15:22:13 +01:00
Albert Zaharovits 6f038997e1
Watcher accounts constructed lazily (#36656)
This fixes two bugs about watcher notifications:
* registering accounts that had only secure settings was not possible before;
these accounts are very much practical for Slack and PagerDuty integrations.
* removes the limitation that, for an account with both secure and cluster settings,
the admin had to first change/add the secure settings and only then add the
dependent dynamic cluster settings. The reverse order would trigger a
SettingsException for an incomplete account.

The workaround is to lazily instantiate account objects, hoping that when accounts
are instantiated all the required settings are in place. Previously, the approach
was to greedily validate all the account settings by constructing the account objects,
even if they would not ever be used by actions. This made sense in a world where
all the settings were set by a single API. But given that accounts have dependent
settings (that must be used together) that have to be changed using different APIs
(POST _nodes/reload_secure_settings and PUT _cluster/settings), the settings group
would technically be in an invalid state in between the calls.
This fix builds account objects, and validates the settings, when they are
needed by actions.
2018-12-17 14:45:55 +02:00
Tal Levy cd1bec3a06
[refactor] add Environment in BootstrapContext (#36573)
There are certain BootstrapCheck checks that may need access environment-specific
values. Watcher's EncryptSensitiveDataBootstrapCheck passes in the node's environment
via a constructor to bypass the shortcoming in BootstrapContext. This commit
pulls in the node's environment into BootstrapContext.

Another case is found in #36519, where it is useful to check the state of the
data-path. Since PathUtils.get and Paths.get are forbidden APIs, we rely on
the environment to retrieve references to things like node data paths.

This means that the BootstrapContext will have the same Settings used in the
Environment, which currently differs from the Node's settings.
2018-12-12 21:07:21 -08:00
Ryan Ernst a0da390df2
Scripting: Switch watcher to use joda bwc time objects (#35966)
This commit converts the watcher execution context to use the joda
compat java time objects. It also again removes the joda methods from
the painless whitelist.
2018-12-10 17:29:25 -08:00
Michael Basnight b5b6e37a60
Deprecate X-Pack centric watcher endpoints (#36218)
This commit is part of our plan to deprecate and ultimately remove the use of
_xpack in the REST APIs.

Relates #35958
2018-12-08 12:57:16 -06:00
Jim Ferenczi 18866c4c0b
Make hits.total an object in the search response (#35849)
This commit changes the format of the `hits.total` in the search response to be an object with
a `value` and a `relation`. The `value` indicates the number of hits that match the query and the
`relation` indicates whether the number is accurate (in which case the relation is equals to `eq`)
or a lower bound of the total (in which case it is equals to `gte`).
This change also adds a parameter called `rest_total_hits_as_int` that can be used in the
search APIs to opt out from this change (retrieve the total hits as a number in the rest response).
Note that currently all search responses are accurate (`track_total_hits: true`) or they don't contain
`hits.total` (`track_total_hits: true`). We'll add a way to get a lower bound of the total hits in a
follow up (to allow numbers to be passed to `track_total_hits`).

Relates #33028
2018-12-05 19:49:06 +01:00
Jim Ferenczi e179fd1274
Add support for rest_total_hits_as_int in watcher (#36035)
This change adds the support for rest_total_hits_as_int
in the watcher search inputs. Setting this parameter in the request
will transform the search response to contain the total hits as
a number (instead of an object).
Note that this parameter is currently a noop since #35849 is not
merged.

Closes #36008
2018-11-30 18:02:37 +01:00
Jim Ferenczi 5c7b2c5f9b
HLRC: Add get watch API (#35531)
This changes adds the support for the get watch API in the high level rest client.
2018-11-30 11:02:46 +01:00
Albert Zaharovits 04ee05f199
Fix Watcher NotificationService's secure settings (#35610)
The NotificationService (base class for SlackService, HipchatService ...) has both dynamic
cluster settings and SecureSettings and builds the clients (Account) that are used to comm
with external services. This commit fixes an important bug about updating/reloading any
of these settings (both Secure and dynamic cluster). Briefly the bug is due to the fact that
both the secure settings as well as the dynamic node scoped ones can be updated
independently, but when constructing the clients some of the settings might not be visible.
2018-11-29 21:34:16 +02:00
Jay Modi 2061eeb122
Remove use of AbstractComponent in xpack (#35394)
This commit removes the use of AbstractComponent in xpack where it was
still being extended. It has been replaced with explicit logger
declarations.

See #34488
2018-11-27 11:28:26 -07:00
Alexander Reelsen a615ab454a
Watcher: Only trigger a watch if new or schedule/changed (#35908)
The trigger engine did always create a new schedule data structure, when
the watcher indexing listener called an add. However the indexing
listener also called add, when the watch status was updated. This means,
that upon a watch status update the watch got retriggered, potentially
waiting a defined interval from the watch status update onwards, instead
of waiting from the last run.

This commit only updates the schedule in the trigger engine, if it
actually has changed, otherwise the existing schedule will not be
touched. This has two results

1. If a watch is updated by an execution, the existing interval will not
be touched (meaning the scheduled time will not move forward).
2. If a watch is updated by a user, but the schedule is not changed, it
will not be reset from the update (for example starting to count from 5
minutes again, if the interval was set to 5 minutes).

Furthermore some minor cleanups were applied, making variables final in
the ctor, preventing double creation of variables.
2018-11-27 16:15:37 +01:00
David Roberts 2ee25c858d Mute CronEvalToolTests.testEnsureDateIsShownInRootLocale
Due to https://github.com/elastic/elasticsearch/issues/35687
2018-11-19 09:35:29 +00:00
Ryan Ernst 3c03975ce3
Watcher: Use joda method to get local TZ (#35608)
This commit switches from using java util's default timezone method to
using joda. The former can cause problems when the string representation
of the timezone is unknown to joda.

closes #35518
2018-11-16 15:46:27 -08:00
Alexander Reelsen 9f3effd690
Watcher: Ignore system locale/timezone in croneval CLI tool (#33215)
The elasticsearch-croneval CLI tool uses local dates to display when
something gets triggered the next time. This is very confusing.

This commit ensures, that UTC and local timezone times will be written
out.

The output looks like this and contains localized dates for each trigger
date as well as for `now`.

Now is [Tue, 28 Aug 2018 17:23:51 +0000] in UTC, local time is [ᏔᎵᏁ, 28 ᎦᎶ 2018 12:23:51 -0500]
Here are the next 10 times this cron expression will trigger:
1.	Mon, 2 Jan 2040 11:00:00 +0000
	ᏉᏅᎯ, 2 ᎤᏃ 2040 06:00:00 -0500
2.      ...

This also removes an old outstanding TODO to use the jopt parsing to
cast the count to an integer instead of doing it ourselves.
2018-11-07 09:03:35 +01:00