Commit Graph

106 Commits

Author SHA1 Message Date
Michael Basnight 1bf205f862 Test: Fix a second case of bad watch creation
There was still a case with a null text that allowed for 0 attachments
to be created. This commit ensures that greater than zero are created
if the text is null. Otherwise, it uses the same logic to create 0 to 3
random attachments.

Closes #31948
2018-07-13 09:33:29 -05:00
Tim Vernum c662565f29
Access SSL contexts using names instead of Settings (#30953)
Historically we have loaded SSL objects (such as SSLContext,
SSLIOSessionStrategy) by passing in the SSL settings, constructing a
new SSL configuration from those settings and then looking for a
cached object that matches those settings.

The primary issue with this approach is that it requires a fully
configured Settings object to be available any time the SSL context
needs to be loaded. If the Settings include SecureSettings (such as
passwords for keys or keystores) then this is not true, and the cached
SSL object cannot be loaded at runtime.

This commit introduces an alternative approach of naming every cached
ssl configuration, so that it is possible to load the SSL context for
a named configuration (such as "xpack.http.ssl"). This means that the
calling code does not need to have ongoing access to the secure
settings that were used to load the configuration.

This change also allows monitoring exporters to use SSL passwords
from secure settings, however an exporter that uses a secure SSL setting
(e.g. truststore.secure_password) may not have its SSL settings updated
dynamically (this is prevented by a settings validator).
Exporters without secure settings can continue to be defined and updated
dynamically.
2018-07-13 16:40:09 +10:00
Christoph Büscher 332c134517 [TEST] Mute SlackMessageTests.testTemplateRender 2018-07-12 22:41:35 +02:00
Michael Basnight d76293f990
Test: fix null failure in watcher test (#31968)
A new commit was merged that does not allow a null attachement &&
text. This is valid for the slack API, as it does not allow this, but
our unit tests did. This commit fixes the broken unit test.

Closes #31948
2018-07-11 09:37:48 -05:00
Yannick Welsch dadf96a840 [TEST] Mute SlackMessageTests.testTemplateRender 2018-07-11 11:12:52 +02:00
albendz 8ec33b742d Watcher: Slack message empty text (#31596)
Slack accepts an empty text or attachments, but not both. This commit
ensures that both are not empty when creating a watch.

Closes #30071

Replacing old pull request: #31288
2018-07-10 09:47:00 -05:00
Alexander Reelsen 3aa0de9e10
Watcher: Add ssl.trust email account setting (#31684)
In order to allow users to specify hosts, where SSL is always trusted,
this setting is exposed. Otherwise the system keystore needs to be
configured properly.

For more info see https://javaee.github.io/javamail/docs/api/com/sun/mail/smtp/package-summary.html
2018-07-09 09:44:06 +02:00
Sohaib Iftikhar 40b822c878 Scripting: Remove support for deprecated StoredScript contexts (#31394)
Removes support for storing scripts without the usual json around the
script. So You can no longer do:
```
POST _scripts/<templatename>
{
    "query": {
        "match": {
            "title": "{{query_string}}"
        }
    }
}
```

and must instead do:
```
POST _scripts/<templatename>
{
    "script": {
        "lang": "mustache",
        "source": {
            "query": {
                "match": {
                    "title": "{{query_string}}"
                }
            }
        }
    }
}
```

This improves error reporting when you attempt to store a script but don't
quite get the syntax right. Before, there was a good chance that we'd
think of it as a "raw" template and just store it. Now we won't do that.
Nice.
2018-07-05 09:30:08 -04:00
Alexander Reelsen 0a2ef59c41
Watcher: Fix check for currently executed watches (#31137)
The ack watch action has a check for currently executed watches, to make
sure that currently running watches cannot be acknowledged. This check
only checked on the coordinating node for watches being executed, but should
have checked the whole cluster using a WatcherStatsRequest, which is
being switched to in this commit.
2018-07-04 17:55:53 +02:00
Alexander Reelsen 4328470dd8
Watcher: Ensure correct method is used to read secure settings (#31753)
As SecureSetting is extended from Setting, you can easily accidentally
use `SecureSetting.simpleString()` to read a secure setting instead of
`SecureSetting.secureString()`. This commit changes this behaviour in
some watcher notification services.
2018-07-04 15:46:59 +02:00
Michael Basnight e65115ae5a
Consolidate watcher setting update registration (#31762)
Previously the call to register a listener for settings updates was in
each individual service, rather than in the notification service
itself. This change ensures that each child of the notification service
gets registered with the settings update consumer.
2018-07-03 11:29:39 -05:00
Alexander Reelsen 8f2feb8414
Watcher: Fix chain input toXcontent serialization (#31721)
The xcontent parameters were not passed to the xcontent serialization
of the chain input for each chain. This could lead to wrongly stored 
watches, which did not contain passwords but only their redacted counterparts, when an input inside of a chain input contained a password.
2018-07-02 20:35:26 +02:00
Alexander Reelsen a72dc9e8fc
Watcher: Remove never executed code (#31135)
The removed code snippet was never executed, as the version was never set and
thus always -1, after parsing the watch. With the changes done in
c9d77d20fd this logic would not have
worked correctly anyway.
2018-06-26 16:14:40 +02:00
Alexander Reelsen bb1d4aaf17
Watcher: Fix put watch action (#31524)
If no version is specified when putting a watch, the index API should be
used instead of the update API, so that the whole watch gets overwritten
instead of being merged with the existing one.

Merging only happens when a version is specified, so that credentials can be omitted, which is important for the watcher UI.
2018-06-25 18:25:34 +02:00
Christoph Büscher 86ab3a2d1a
Reduce number of raw types warnings (#31523)
A first attempt to reduce the number of raw type warnings, 
most of the time by using the unbounded wildcard.
2018-06-25 15:59:03 +02:00
Ryan Ernst 7a150ec06d
Core: Combine doExecute methods in TransportAction (#31517)
TransportAction currently contains 2 doExecute methods, one which takes
a the task, and one that does not. The latter is what some subclasses
implement, while the first one just calls the latter, dropping the given
task. This commit combines these methods, in favor of just always
assuming a task is present.
2018-06-22 15:03:01 -07:00
Ryan Ernst 4f9332ee16
Core: Remove ThreadPool from base TransportAction (#31492)
Most transport actions don't need the node ThreadPool. This commit
removes the ThreadPool as a super constructor parameter for
TransportAction. The actions that do need the thread pool then have a
member added to keep it from their own constructor.
2018-06-21 11:25:26 -07:00
Ryan Ernst 401800d958
Core: Remove index name resolver from base TransportAction (#31002)
Most transport actions don't need to resolve index names. This commit
removes the index name resolver as a super constructor parameter for
TransportAction. The actions that do need the resolver then have a
member added to keep the resolver from their own constructor.
2018-06-19 17:06:09 -07:00
Jason Tedor 0bfd18cc8b
Revert upgrade to Netty 4.1.25.Final (#31282)
This reverts upgrading to Netty 4.1.25.Final until we have a cleaner
solution to dealing with the object cleaner thread.
2018-06-12 19:26:18 -04:00
Aditya Dhulipala 72f8199279 Validate xContentType in PutWatchRequest. (#31088)
Trying to post a new watch without any body currently results in a 
NullPointerException. This change fixes that by validating that 
Post and Put requests always have a body.

Closes #30057
2018-06-12 09:53:36 +02:00
Jason Tedor 563141c6c9
Upgrade to Netty 4.1.25.Final (#31232)
This commit upgrades us to Netty 4.1.25. This upgrade is more
challenging than past upgrades, all because of a new object cleaner
thread that they have added. This thread requires an additional security
permission (set context class loader, needed to avoid leaks in certain
scenarios). Additionally, there is not a clean way to shutdown this
thread which means that the thread can fail thread leak control during
tests. As such, we have to filter this thread from thread leak control.
2018-06-11 16:55:07 -04:00
Tanguy Leroux bf58660482
Remove all unused imports and fix CRLF (#31207)
The X-Pack opening and the recent other refactorings left a lot of 
unused imports in the codebase. This commit removes them all.
2018-06-11 15:12:12 +02:00
Jason Tedor e481b860a1
Enable engine factory to be pluggable (#31183)
This commit enables the engine factory to be pluggable based on index
settings used when creating the index service for an index.
2018-06-07 17:01:06 -04:00
Jason Tedor 8be1361579
Adjust indentation in CLI scripts
This commit adjusts the indentation in the CLI scripts to give a clear
visual indication that the line being indented is a continuation of the
previous line.
2018-06-06 22:52:50 -04:00
Jason Tedor 01b5a46c24
Pass main class by environment variable on Windows (#31156)
A previous refactoring of the CLI scripts migrated all of the CLI tools
to shell to a common script, elasticsearch-cli. This approach is fine in
Bash where it is easy to tear arguments apart but it doesn't work so
well on Windows where quoting is insane. To avoid having to tear the
arguments apart to separate the first argument to elasticsearch-cli from
the remaining arguments, we instead choose a strategy where we can avoid
tearing the arguments apart. To do this, we will instead pass the main
class by an environment variable and then we can pass the arguments
straight through. This will let us avoid awful quoting issues on
Windows. This is the Windows side of that effort and the Bash side was
in a previous commit.
2018-06-06 21:57:58 -04:00
Jason Tedor 95795c8935
Pass main class by environment variable (#31149)
A previous refactoring of the CLI scripts migrated all of the CLI tools
to shell to a common script, elasticsearch-cli. This approach is fine in
Bash where it is easy to tear arguments apart but it doesn't work so
well on Windows where quoting is insane. To avoid having to tear the
arguments apart to separate the first argument to elasticsearch-cli from
the remaining arguments, we instead choose a strategy where we can avoid
tearing the arguments apart. To do this, we will instead pass the main
class by an environment variable and then we can pass the arguments
straight through. This will let us avoid awful quoting issues on
Windows. This is the non-Windows side of that effort and the Windows
side will be in a follow-up.
2018-06-06 21:56:52 -04:00
Martijn van Groningen 735d0e671a
Make PreBuiltAnalyzerProviderFactory plugable via AnalysisPlugin and
move `finger_print`, `pattern` and `standard_html_strip` analyzers
to analysis-common module. (both AnalysisProvider and PreBuiltAnalyzerProvider)

Changed PreBuiltAnalyzerProviderFactory to extend from PreConfiguredAnalysisComponent and
changed to make sure that predefined analyzers are always instantiated with the current
ES version and if an instance is requested for a different version then delegate to PreBuiltCache.
This is similar to the behaviour that exists today in AnalysisRegistry.PreBuiltAnalysis and
PreBuiltAnalyzerProviderFactory. (#31095)

Relates to #23658
2018-06-06 07:40:21 +02:00
Christoph Büscher 3f87c79500
Change ObjectParser exception (#31030)
ObjectParser should throw XContentParseExceptions, not IAE. A dedicated parsing
exception can includes the place where the error occurred.

Closes #30605
2018-06-04 20:20:37 +02:00
Jason Tedor 6fb1e4a759
Fix handling of percent-encoded spaces in Windows batch files (#31034)
If you invoke elasticsearch-plugin (or any other CLI script on Windows)
with a path that has a percent-encoded space (or any other
percent-encoded character) because the CLI scripts now shell into a
common shell script (elasticsearch-cli) the percent-encoded space ends
up being interpreted as a parameter. For example passing install --batch
file:/c:/encoded%20%space/analysis-icu-7.0.0.zip to elasticsearch-plugin
leads to the %20 being interpreted as %2 followed by a zero. Here, the
%2 is interpreted as the second parameter (--batch) and the
InstallPluginCommand class ends up seeing
file:/c/encoded--batch0space/analysis-icu-7.0.0.zip as the path which
will not exist. This commit addresses this by escaping the %* that is
used to pass the parameters to the common CLI script so that the common
script sees the correct parameters without the %2 being substituted.
2018-06-01 15:00:41 -04:00
Nik Everett 283d1bb2cf Watcher: Give test a little more time
Changes watcher's integration tests to wait 30 seconds when starting
watcher rather than 10 seconds because this build failed when starting
took 12 seconds:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.3+periodic/222/console
2018-05-31 18:11:16 -04:00
Igor Motov 1da26a42cb [CI] Mute HttpSecretsIntegrationTests#testWebhookAction test
Tracked by #30094
2018-05-30 14:29:06 -04:00
Yannick Welsch 39c4f89f9b
Move Watcher versioning setting to meta field (#30832)
The .watcher-history-* template is currently using a plugin-custom index setting xpack.watcher.template.version,
which prevents this template from being installed in a mixed OSS / X-Pack cluster, ultimately
leading to the situation where an X-Pack node is constantly spamming an OSS master with (failed)
template updates. Other X-Pack templates (e.g. security-index-template or security_audit_log)
achieve the same versioning functionality by using a custom _meta field in the mapping instead.
This commit switches the .watcher-history-* template to use the _meta field instead.
2018-05-24 15:16:20 +02:00
Yannick Welsch 8145a820c2
Only allow x-pack metadata if all nodes are ready (#30743)
Enables a rolling restart from the OSS distribution to the x-pack based distribution by preventing
x-pack code from installing custom metadata into the cluster state until all nodes are capable of
deserializing this metadata.
2018-05-23 11:41:23 +02:00
Jason Tedor 0fc22de336
Reduce CLI scripts to one-liners on Windows (#30772)
This commit reduces the Windows CLI scripts to one-liners by moving all
of the redundant logic to an elasticsearch-cli script. This commit is
only the Windows side, a previous commit covered the Linux side.
2018-05-22 14:34:29 -04:00
Jason Tedor 3ce2297dc1
Reduce CLI scripts to one-liners (#30759)
This commit reduces the Linux CLI scripts to one-liners by moving all of
the redundant logic to an elasticsearch-cli script. This commit is only
the Linux side, a follow-up will do this for Windows too.
2018-05-21 16:37:57 -04:00
Alexander Reelsen b5a793b569
Tests: Fail if test watches could not be triggered (#30392)
Watcher tests now always fail hard when watches that were 
tried to be triggered in a test using the trigger() method, 
but could not because they were not found on any of the 
nodes in the cluster.
2018-05-18 08:51:43 +02:00
Ryan Ernst 30e0f33d16 Silence sleep based watcher test
see https://github.com/elastic/elasticsearch/issues/30699
2018-05-17 13:04:10 -07:00
Ryan Ernst a4c9c2fa2a
Make xpack modules instead of a meta plugin (#30589)
This commit removes xpack from being a meta-plugin-as-a-module.
It also fixes a couple tests which were missing task dependencies, which
failed once the gradle execution order changed.
2018-05-16 15:35:57 -07:00
Colin Goodheart-Smithe a75b8adce5
Refactors ClientHelper to combine header logic (#30620)
* Refactors ClientHelper to combine header logic

This change removes all the `*ClientHelper` classes which were
repeating logic between plugins and instead adds
`ClientHelper.executeWithHeaders()` and
`ClientHelper.executeWithHeadersAsync()` methods to centralise the
logic for executing requests with stored security headers.

* Removes Watcher headers constant
2018-05-16 11:38:24 +01:00
Alexander Reelsen 6ec5e4408c
Watcher: Ensure secrets integration tests also run triggered watch (#30478)
When the encrpytion of sensitive date is enabled, test that a
scheduled watch is executed as expected and produces the correct value
from a secret in the basic auth header.
2018-05-16 10:32:22 +02:00
Alexander Reelsen ed2d427c48
Watcher: Remove TriggerEngine.getJobCount() (#30395)
Since adding back the per-watch statistics, we do not need to access
every trigger engine implementation to get the current total job count.
This commit removes the unused methods to do so.
2018-05-11 10:04:58 +02:00
Alexander Reelsen f00890ee38
Watcher: Increase HttpClient parallel sent requests (#30130)
The HTTPClient used in watcher is based on the apache http client. The
current client is using a lot of defaults - which are not always
optimal. Two of those defaults are the maximum number of total
connections and the maximum number of connections to a single route.

If one of those limits is reached, the HTTPClient waits for a connection
to be finished thus acting in a blocking fashion. In order to prevent
this when many requests are being executed, we increase the limit of
total connections as well as the connections per route (a route is
basically an endpoint, which also contains proxy information, not
containing an URL, just hosts).

On top of that an additional option has been set to evict
long running connections, which can potentially be reused after some
time. As this requires an additional background thread, this required
some changes to ensure that the httpclient is closed properly. Also the
timeout for this can be configured.
2018-05-09 09:37:47 +02:00
Alexander Reelsen 71ebed6371
Watcher: Mark watcher as started only after loading watches (#30403)
Starting watcher should wait for the watcher to be started before
marking the status as started, which is now done via a callback.

Also, reloading watcher could set the execution service to paused. This could
lead to watches not being executed, when run in tests. This fix does not
change the paused flag in the execution service, just clears out the
current queue and executions.

Closes #30381
2018-05-07 14:19:01 +02:00
Alexander Reelsen 0e6cbbd811
Watcher: Ensure trigger service pauses execution (#30363)
When the watcher service pauses execution due to a cluster state update,
the trigger service and its engines also need to pause properly instead
of keeping going. This is also important when the .watches index is 
deleted, so that watches don't stay in a triggered mode.
2018-05-04 09:02:04 +02:00
Alexander Reelsen 2c38d12e23
Watcher: Make start/stop cycle more predictable and synchronous (#30118)
The current implementation starts/stops watcher using an executor. This
can result in our of order operations.

This commit reduces those executor calls to an absolute minimum in order
to be able to do state changes within the cluster state listener method,
which runs in sequence.

When a state change occurs that forces the watcher service to pause
(like no watcher index, no master node, no local shards), the service is
now in a paused state.

Pausing is a super lightweight operation, which marks the
ExecutionService as paused and waits for the currently executing watches
to finish in the background via an executor. The same applies for
stopping, the potentially long running operation is outsourced in to an
executor, as waiting for executed watches is decoupled from the current
state.

The only other long running operation is starting, where watches need to
be loaded. This is also done via an executor, but has an additional
protection by checking the cluster state version it was started with. If
another cluster state version was trying to load the watches, then this
loading will not take effect.

This PR also cleans up some unused states, like the a simple boolean in
the HistoryStore/TriggeredWatchStore marking it as started or stopped,
as this can now be caught in the execution service.

Another advantage of this approach is the fact, that now only triggered
watches are not getting executed, while watches that are run via the
Execute Watch API will still be executed regardless if watcher is
stopped or not.

Lastly the TickerScheduleTriggerEngine thread now only starts on data nodes.
2018-05-03 09:47:12 +02:00
Ryan Ernst fb0aa562a5
Network: Remove http.enabled setting (#29601)
This commit removes the http.enabled setting. While all real nodes (started with bin/elasticsearch) will always have an http binding, there are many tests that rely on the quickness of not actually needing to bind to 2 ports. For this case, the MockHttpTransport.TestPlugin provides a dummy http transport implementation which is used by default in ESIntegTestCase.

closes #12792
2018-05-02 11:42:05 -07:00
Michael Basnight 62a9b8909e
Remove RepositoriesMetaData variadic constructor (#29569)
The variadic constructor was only used in a few places and the
RepositoriesMetaData class is backed by a List anyway, so just using a
List will make it simpler to instantiate it.
2018-05-01 15:02:06 -05:00
Boaz Leskes 4a537ef03c
Bulk operation fail to replicate operations when a mapping update times out (#30244)
Starting with the refactoring in https://github.com/elastic/elasticsearch/pull/22778 (released in 5.3) we may fail to properly replicate operation when a mapping update on master fails. If a bulk
operations needs a mapping update half way, it will send a request to the master before continuing 
to index the operations. If that request times out or isn't acked (i.e., even one node in the cluster 
didn't process it within 30s), we end up throwing the exception and aborting the entire bulk. This is 
a problem because all operations that were processed so far are not replicated any more to the 
replicas.  Although these operations were never "acked" to the user (we threw an error) it cause the 
local checkpoint on the replicas to lag (on 6.x) and the primary and replica to diverge. 

This PR does a couple of things:
1) Most importantly, treat *any* mapping update failure as a document level failure, meaning only 
    the relevant indexing operation will fail.
2) Removes the mapping update callbacks from `IndexShard.applyIndexOperationOnPrimary` and 
    similar methods for simpler execution. We don't use exceptions any more when a mapping 
    update was successful.

I think we need to do more work here (the fact that a single slow node can prevent those mappings 
updates from being acked and thus fail operations is bad), but I want to keep this as small as I can 
(it is already too big).
2018-05-01 08:15:02 +02:00
Zachary Tong fee000a37f
[TEST] Redirect links to new locations (#30179)
We had a number of awaitsFix links that weren't updated after the xpack
merge.

Where possible I changed the links to the new locations, but in some
circumstances the original ticket was closed (suggesting the awaitsfix
should be removed) or was otherwise unclear the status.
2018-04-27 09:24:46 -07:00
Alexander Reelsen 707ba28d48
Watcher: Ensure mail message ids are unique per watch action (#30112)
Email message IDs are supposed to be unique. In order to guarantee this,
we need to take the action id of a watch action into account as well,
not just the watch id from the watch execution context. This prevents
that two actions from the same watch execution end up with the same
message id.
2018-04-27 08:55:25 +02:00
Jason Tedor d99d0fa669 Add distribution type to startup scripts
This commit adds the distribution type to the startup scripts so that we
can discern from log output and the main response the type of the
distribution (deb/rpm/tar/zip).
2018-04-20 15:34:01 -07:00
Ryan Ernst d8636d0fe3 Rename syskeygen
This commit renames syskeygen to elasticsearch-syskeygen
2018-04-20 15:34:01 -07:00
Ryan Ernst 6dbb7be067 Rename croneval
This commit renames croneval to elasticsearch-croneval
2018-04-20 15:34:00 -07:00
Jason Tedor e64e6d8996 Add distribution flavor to startup scripts
This commit adds the distribution flavor (default versus oss) to the
build process which is passed through the startup scripts to
Elasticsearch. This change will be used to customize the message on
attempting to install/remove x-pack based on the distribution flavor.
2018-04-20 15:33:58 -07:00
Ryan Ernst fab5e21e7d Build: Split distributions into oss and default
This commit makes x-pack a module and adds it to the default
distrubtion. It also creates distributions for zip, tar, deb and rpm
which contain only oss code.
2018-04-20 15:33:57 -07:00
Ryan Ernst 2efd22454a Migrate x-pack-elasticsearch source to elasticsearch 2018-04-20 15:29:54 -07:00