Commit Graph

6315 Commits

Author SHA1 Message Date
Tim Vernum 5056d4e3df Revert unintended changes to syskeygen from elastic/x-pack-elasticsearch#1831 (elastic/x-pack-elasticsearch#2055)
During the development of elastic/x-pack-elasticsearch#1831 (elastic/x-pack-elasticsearch@5ac95c6) there was an intention to replace `SystemKeyTool` with `EncKeyTool`.
The java change was reverted and never committed to master, but the script change was accidentally left in place.

Also removes redundant "properties" variable (refer `elastic/x-pack-elasticsearch@b0a3b89`)

Original commit: elastic/x-pack-elasticsearch@f240479748
2017-07-21 12:51:25 +10:00
Jason Tedor 470e81ce64 Use elasticsearch-env
This commit utilizes the elasticsearch-env script that is added in core
Elasticsearch for significantly simplifying the scripts used in x-pack.

Relates elastic/x-pack-elasticsearch#2049

Original commit: elastic/x-pack-elasticsearch@8ef041d077
2017-07-21 09:39:46 +09:00
Andrew Cholakian c8e42a0db6 Update logstash mappings to use pipeline.id + correct metric types (elastic/x-pack-elasticsearch#2045)
In logstash parlance there really is no pipeline.name, its pipeline.id

This also removes support for deprecated gauge types `text` and `boolean` we will be removing those soon in logstash

This also renames `counters` to `long_counters` to be more explicit and for consistency with the gauge type. Also, if we ever decide to add other types of counters this will be more clear

Original commit: elastic/x-pack-elasticsearch@8f44a94579
2017-07-20 14:30:37 -05:00
Deb Adair 30f2b3f90e [DOCS] Fixing cross doc links.
Original commit: elastic/x-pack-elasticsearch@3333055f42
2017-07-20 11:25:02 -07:00
Deb Adair 3ace57d512 [DOCS] Updates to make GS minidoc build.
Original commit: elastic/x-pack-elasticsearch@04c168e653
2017-07-20 11:24:57 -07:00
Chris Earle e0fe6da303 [Security] Remove beats_system User and Role until needed (elastic/x-pack-elasticsearch#2042)
The user/role was added to simplify they setup surrounding Beats monitoring, but Beats monitoring has been delayed until the UI work is begun.

Original commit: elastic/x-pack-elasticsearch@1c0c85562e
2017-07-20 12:24:15 -04:00
Tim Brooks a0fd423db1 Update documentation for bootstrap password work (elastic/x-pack-elasticsearch#2031)
This is related to elastic/x-pack-elasticsearch#1217. The commit adds documenation describing how to
use the bootstrap password and setup-password tool.

Original commit: elastic/x-pack-elasticsearch@1bad8ddb4d
2017-07-20 11:23:20 -05:00
Jack Conradson e007fee9fb remove lang url parameter from stored script requests (elastic/x-pack-elasticsearch#2029)
Original commit: elastic/x-pack-elasticsearch@1044c3ba53
2017-07-20 08:51:27 -07:00
Jason Tedor 5ecbbbd46d Use plain old exception if security init fails
When security initialization fails during startup today we throw an
Error. This triggers the uncaught exception handler immediately killing
the node. While the node is going to die either way, we should not be
triggering the killer and in general we should avoid using Throwables
that extend Error.

Relates elastic/x-pack-elasticsearch#2035

Original commit: elastic/x-pack-elasticsearch@546f7f9002
2017-07-21 00:26:22 +09:00
Simon Willnauer eb5631b981 Followup for elastic/elasticsearch#25791
Original commit: elastic/x-pack-elasticsearch@806de5670f
2017-07-20 16:45:48 +02:00
Jason Tedor 3f08fad603 Remove CONF_FILE check
This commit removes a legacy check for an unsupported environment
variable. This environment variable has not been supported since 1.x so
it is safe to stop checking for the existence of this setting.

Relates elastic/x-pack-elasticsearch#2048

Original commit: elastic/x-pack-elasticsearch@023230fa9e
2017-07-20 22:42:21 +09:00
Jason Tedor 814c1a21f9 Stop exporting HOSTNAME from scripts
Today we explicitly export the HOSTNAME variable from scripts. This is
probably a relic from the days when the scripts were not run on bash but
instead assume a POSIX-compliant shell only where HOSTNAME is not
guaranteed to exist. Yet, bash guarantees that HOSTNAME is set so we do
not need to set it in scripts. This commit removes this legacy.

Relates elastic/x-pack-elasticsearch#2047

Original commit: elastic/x-pack-elasticsearch@7b833e061c
2017-07-20 22:28:01 +09:00
Alexander Reelsen dce13b87c3 Watcher: Remove stream serialization version checks, happen in 5.x (elastic/x-pack-elasticsearch#2002)
Those checks were moved into 5.x in elastic/x-pack-elasticsearch#2001

Original commit: elastic/x-pack-elasticsearch@7664bf35c5
2017-07-20 14:48:08 +02:00
Jason Tedor 581293aed5 Fix croneval script to respect CONF_DIR
This commit fixes the croneval script to respect the CONF_DIR
environment variable used to locate the configuration directory.

Original commit: elastic/x-pack-elasticsearch@79974947f9
2017-07-20 19:49:28 +09:00
Jason Tedor 5fe584472e Fix certgen script usage of ES_HOME
This commit fixes an issue with the usage of the environment variable
ES_HOME in the certgen script; the script was missing the use of $ to
obtain the value of the environment variable ES_HOME.

Relates elastic/x-pack-elasticsearch#2046

Original commit: elastic/x-pack-elasticsearch@63128db0eb
2017-07-20 16:02:21 +09:00
Jason Tedor 6552c9a5f7 Remove use of ES_INCLUDE
This commit removes all uses of ES_INCLUDE as this functionality has
been removed from core Elasticsearch in favor of a dedicated include
script for establishing the environment.

Relates elastic/x-pack-elasticsearch#2046

Original commit: elastic/x-pack-elasticsearch@92f8470e44
2017-07-20 15:41:59 +09:00
Tim Vernum 776f7cec65 [TEST] Improve SSLTrustRestrictionsTests (elastic/x-pack-elasticsearch#2015)
Use assertBusy rather than a fixed sleep time to wait for SSL reloading.

relates elastic/x-pack-elasticsearch#2007

Original commit: elastic/x-pack-elasticsearch@c8f789c327
2017-07-20 13:10:27 +10:00
Ali Beyad fe9d99daed Removes NativeRealmMigrator since its no longer needed (elastic/x-pack-elasticsearch#2028)
With the new template and mapping update mechanisms introduced as part
of the Upgrade API work, the NativeRealmMigrator is no longer needed or
used.  This commit removes the NativeRealmMigrator code and the
associated tests for it.

Original commit: elastic/x-pack-elasticsearch@5d2d7a582c
2017-07-19 12:19:48 -04:00
Tim Brooks 7ed5df4068 Make self-generated license type configurable (elastic/x-pack-elasticsearch#2000)
This is related to elastic/x-pack-elasticsearch#1778. This commit adds a setting
(xpack.license.self_generated.type) which allows the user to specify
what type of license will be self-generated on node startup. The allowed
types are basic or trial.

Original commit: elastic/x-pack-elasticsearch@0a16a59e10
2017-07-19 10:56:10 -05:00
Adrien Grand f15147b87b Remove assumption about how the timeout feature is implemented.
Original commit: elastic/x-pack-elasticsearch@7a4fa000c1
2017-07-19 17:29:49 +02:00
Tim Brooks a6bf79bd31 Remove the container work (elastic/x-pack-elasticsearch#2030)
This is related to elastic/x-pack-elasticsearch#1217. This commit removes the features that were
introduced in elastic/x-pack-elasticsearch#1832. This is because the containerized version of
x-pack can be have its password bootstrapped using the generalized
keystore mechanism introduced in elastic/x-pack-elasticsearch#1942.

Original commit: elastic/x-pack-elasticsearch@a9b6e870c1
2017-07-19 09:17:24 -05:00
Alexander Reelsen a64f9afa30 Tests: Ensure correct client is picked when security is enabled
Original commit: elastic/x-pack-elasticsearch@67d05553f8
2017-07-19 16:11:03 +02:00
markharwood ffe6966ccd Test fix - ensure forceMerged to single segment to avoid scoring variations in tests.
Removed test muting.

Original commit: elastic/x-pack-elasticsearch@3a584aadda
2017-07-19 13:27:07 +01:00
Alexander Reelsen 5e6c56bfc1 Watcher: Restore old WatcherStatsAction for BWC (elastic/x-pack-elasticsearch#2022)
To achieve backwards compatibility the easiest way is
to restore the old watcher stats, which are supposed to run
on the master node only.

The distributed watcher stats have been moved under the statsdist
package and the action name has been changed as well.

This way there is no need to have a serialization BWC layer,
we can just call different actions.

Note: With the current approach developers still need to change
their java applications if they try to receive watcher stats,
as by default we are now using the distributed stats in the
watcher client.

Original commit: elastic/x-pack-elasticsearch@49b3a45452
2017-07-19 13:34:14 +02:00
Simon Willnauer aeed4cb3e4 Followup refactoring for elastic/elasticsearch#25787
Original commit: elastic/x-pack-elasticsearch@ab0e5c45ef
2017-07-19 12:30:27 +02:00
David Roberts ac46b0b0a5 [ML] Include closing jobs in node capacity check (elastic/x-pack-elasticsearch#2034)
Closing jobs can still use some or all of the threads that communicate
with the C++ process - the number of threads used will decrease as the
close progresses, but at the beginning of the closure all are still in
use.  Therefore, to prevent the risk of EsRejectedExecution exceptions
for the autodetect communications threadpool, closing jobs need to be
considered when checking that enough threads exist to start a new
process.  An explicit check produces a much more understandable error
message than an EsRejectedExecution exception.

relates elastic/x-pack-elasticsearch#1364

Original commit: elastic/x-pack-elasticsearch@845bfe0188
2017-07-19 11:25:02 +01:00
David Roberts a41c33dd95 [ML] Allow jobs in the "closing" state to be killed (elastic/x-pack-elasticsearch#2026)
This is important for two reasons:

1. If a job hangs in the closing state for any reason there is now a
   way to get rid of it
2. The force delete endpoint (as used by the UI) killed open jobs before
   deleting them, but could not kill closing jobs, which created the
   possibility that if a job was deleted from the UI while in the closing
   state then the last few results could be indexed after the deletion
   completed

relates elastic/x-pack-elasticsearch#1796

Original commit: elastic/x-pack-elasticsearch@1471106e06
2017-07-19 08:54:24 +01:00
Tim Vernum 1bbc579cf3 [Security] [certgen] Option to generate PKCSelastic/x-pack-elasticsearch#12 (elastic/x-pack-elasticsearch#2013)
Add an option to the ssl certificate generation tool (certgen) that generates PKCSelastic/x-pack-elasticsearch#12 (.p12) files in addition to the certificate (.crt) and key (.key) files.
A PKCSelastic/x-pack-elasticsearch#12 store is a container format for storing multiple crypto objects in a single file, which means we can put the cert and key into the same file.

These format is particularly useful for .NET environments, where .NET Core requires a single into file for PKI authentication.

Also adds documentation for all the command-line options in certgen.

Original commit: elastic/x-pack-elasticsearch@d10f88f12d
2017-07-19 12:04:31 +10:00
Tim Sullivan ae62a67e61 [Monitoring] Add Cluster Alert for X-Pack License Expiration (elastic/x-pack-elasticsearch#1998)
* [Monitoring] Add Cluster Alert for X-Pack License Expiration

* work on scripts round 1

* updates per feedback

* spaces

* fix NPE error in transform

* condition to allow updating metadata in the alerts index in every interval

* custom subject message

* update name of indexing actions

* ensure ctx.metadata is updated even if alert is not resolved

* fix omission of absoluteTime

* skip info-level alerts for trial-type license

* move break above `fromNow` declaration

* fix test

Original commit: elastic/x-pack-elasticsearch@f13718f5b5
2017-07-18 15:39:13 -07:00
Andrew Cholakian ebc37feaeb [Logstash] new pipeline viewer schema (elastic/x-pack-elasticsearch#1845)
New metric types for logstash pipeline viewer.

Original commit: elastic/x-pack-elasticsearch@8e44b1fa5a
2017-07-18 11:28:01 -05:00
Alexander Reelsen 139513fdd3 Tests: Fix WatchBackwardsCompatibilityIT temporarily
the upgrade API is lacking some functionality in a special case,
where triggered_watches exists, but .watches does not. This
deletes the triggered watches index manually until we integrated
this properly in the upgrade API to fix the tests

Original commit: elastic/x-pack-elasticsearch@e9d1b0d35d
2017-07-18 18:00:55 +02:00
Ali Beyad 37cc602aef Adds upgrade API functionality for security (elastic/x-pack-elasticsearch#2012)
This commit adds the upgrade API functionality and script for security.
It also enables previously muted tests that would fail due to the lack
of security upgrade features in testing cluster restarts and old
security index backward compatibility.

Original commit: elastic/x-pack-elasticsearch@4abe9f1263
2017-07-18 11:44:28 -04:00
Jay Modi 8b608ef23b Restricted trust config delegates files to monitor to wrapped trust configuration (elastic/x-pack-elasticsearch#2017)
This change makes the restricted trust configuration delegate the list of files to monitor to the
trust configuration that it wraps so that all files that should be monitored for changes are
monitored for changes.

Relates elastic/x-pack-elasticsearch#1919

Original commit: elastic/x-pack-elasticsearch@227db92ac0
2017-07-18 08:52:04 -06:00
Luca Cavanna 7c58130eb2 Wipe security index using its concrete name (elastic/x-pack-elasticsearch#2011)
We were catching IndexNotFoundException, which was hiding the fact that delete index and update aliases APIs don't accept aliases anymore. Now that the exception changed this problem popped up. We now rather call get index providing .security as index name, then delete the concrete indices returned in the response.

Original commit: elastic/x-pack-elasticsearch@18f64f9a41
2017-07-18 15:41:32 +02:00
Alexander Reelsen 8200b18e9f Tests: Allow to run BWC in isolation
The current tests were only able to finish successfully, if the earlier
tests were run as well, you could not run the restart test in isolation.

This commit ensures an upgrade is executed if needed.

Original commit: elastic/x-pack-elasticsearch@616ebbd6eb
2017-07-18 15:29:07 +02:00
Alexander Reelsen 443cfb94be Tests: Ensure waiting time between stopping and starting watcher (elastic/x-pack-elasticsearch#2008)
Otherwise we might run into race conditions that prevent a useful
start up again.

Those tests can be massively improved (no need to run against the real
master node), once the watcher BWC compatible stats are in.

relates elastic/x-pack-elasticsearch#2004

Original commit: elastic/x-pack-elasticsearch@52ca77809c
2017-07-18 10:19:33 +02:00
Igor Motov b4031ee96f Upgrade API: add support for wait_for_completion in upgrade request (elastic/x-pack-elasticsearch#2019)
The wait_for_completion can be now specified with upgrade request to make it async

Original commit: elastic/x-pack-elasticsearch@b768a13ebd
2017-07-17 17:06:16 -04:00
Jay Modi 653b927628 LDAP calls that create a new connection use privilegedConnect (elastic/x-pack-elasticsearch#2018)
This change fixes some cases where calls to the LDAP library can result in a new connection being
created that were not wrapped in privileged connect calls. This would result in permission denied
errors when trying to make the connection.

Original commit: elastic/x-pack-elasticsearch@182c790dd4
2017-07-17 13:15:12 -06:00
Adrien Grand 44c9bba39c Remove `randomDynamicTemplates` usage.
It has been removed it core.

Original commit: elastic/x-pack-elasticsearch@ba9fd16ed6
2017-07-17 16:55:39 +02:00
Tim Brooks 1abc40c645 Rename elastic bootstrap password setting (elastic/x-pack-elasticsearch#2009)
This is related to elastic/x-pack-elasticsearch#1991.

Original commit: elastic/x-pack-elasticsearch@b265211e4a
2017-07-14 16:41:42 -05:00
jaymode 1a6e6411cf Test: use the same settings object to construct tribe settings
This changes the SecurityTribeIT tests to use the same settings object when creating the settings
for the tribe node. Previously two different objects were being created and we would read regular
settings from one and secure settings from another. This causes problems since randomization means
that there may be settings added on the first call that do not get added on the second call. One
example is the randomization of when to only use a keystore or to use both a keystore and a
truststore. On the first call, we would add settings for both a keystore and a truststore but on
the second call only the keystore settings were added. This lead to failures as we would not be
able to open a password protected truststore since the password was never added to the secure
settings.

relates elastic/x-pack-elasticsearch#2005

Original commit: elastic/x-pack-elasticsearch@bbdb3ec662
2017-07-14 14:12:32 -06:00
Drew Raines 49407c1069 [DOCS] Fix typo
Original commit: elastic/x-pack-elasticsearch@1aaaa88404
2017-07-14 14:31:53 -05:00
David Roberts 8365038584 [TEST] Mute failing test: SSLTrustRestrictionsTests testRestrictionsAreReloaded
See elastic/x-pack-elasticsearch#2007

Original commit: elastic/x-pack-elasticsearch@1deb4a7162
2017-07-14 16:30:47 +01:00
Jay Modi 6b4468ea5c Clear security caches on security index health changes (elastic/x-pack-elasticsearch#1957)
This change clears the caches in the native realm and the composite roles store when there is a
a change in the health of the security index that necessitates this. When the security index goes
to a red state, the caches are left in tact as this allows for management operations to be
performed for a limited amount of time. When the index transitions out of the red state or exists
when it didn't exist before, the caches will be cleared so that we remove any stale values.

relates elastic/x-pack-elasticsearch#1789

Original commit: elastic/x-pack-elasticsearch@914959ea6b
2017-07-14 09:28:28 -06:00
Tim Brooks 8ab167cccb Fix ReservedRealm test for failed authentication
Original commit: elastic/x-pack-elasticsearch@5759d9268c
2017-07-14 09:28:44 -05:00
Yannick Welsch dbbec0d37e Let primary own its replication group
Companion commit for elastic/x-pack-elasticsearch#25692

Original commit: elastic/x-pack-elasticsearch@ed93c56f07
2017-07-14 13:51:43 +02:00
David Roberts 81ec1a7ba5 [TEST] Mute failing test
See elastic/x-pack-elasticsearch#2003

Original commit: elastic/x-pack-elasticsearch@5f3611b1d3
2017-07-14 09:49:30 +01:00
Tim Vernum 1686add7ce The configured role-mapping file must be valid. (elastic/x-pack-elasticsearch#1940)
This adds a bootstrap-check that makes it an error to configure a role mapping file that doesn't exist or cannot be parsed.

We are still lenient on dynamic reload because
(a) killing a running node is quite drastic
(b) file writes aren't atomic, so we might be picking up a file that is half way through being written (etc).

If you rely on the default role mapping filename, then it doesn't need to exist (because you might be using the role mapping API instead) but if it does exist it has to parse successfully

Original commit: elastic/x-pack-elasticsearch@5424dea4c4
2017-07-14 15:04:26 +10:00
Tim Vernum d57e38fbed [LDAP] Support explicit "dn" attribute in group search (elastic/x-pack-elasticsearch#1995)
The default for group_search.attribute is to search by DN, but explicitly setting that to dn
wouldn't work because the DN is returned in a special value in the result, and not in the attributes list.

This change detects when user_attribute is set to dn and treats it the same way as the default value.

Original commit: elastic/x-pack-elasticsearch@1933410a0b
2017-07-14 13:12:28 +10:00
Tim Vernum 44a104cb26 [TESTS] Fix SSLTrustRestriction IntegrationTest
- Changes the reloading test to always trust the "trusted" cert so that the health-ping works
- Adds some more logging in case we get new failures

Original commit: elastic/x-pack-elasticsearch@993bf9c721
2017-07-14 13:10:29 +10:00