897 Commits

Author SHA1 Message Date
Nik Everett
acfc8d1bc5 Add hack to run internal test cluster if needed
Runs an internal test cluster if the test is not started with
an external rest test cluster.

Original commit: elastic/x-pack-elasticsearch@ce3d6f64e2
2017-06-29 12:07:39 -04:00
Nik Everett
f9da0b147f Cleanup after merge/core upgrade
This should get us compiling against core, at least as of
7f2bcf1f97c917bf7e5a5c0bf904f6092e576acb.

Original commit: elastic/x-pack-elasticsearch@3f27d83107
2017-06-29 09:35:02 -04:00
Nik Everett
0729c13f14 Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@66d102a6f8
2017-06-29 09:03:24 -04:00
Nik Everett
317cf0253e Move more tests to integration tests
Original commit: elastic/x-pack-elasticsearch@0a75293d97
2017-06-29 08:56:16 -04:00
Christoph Büscher
3ff5ee3f47 Adapting to merging GetField and SearchHitField to DocumentField (elastic/x-pack-elasticsearch#1860)
Follow up to changes in https://github.com/elastic/elasticsearch/pull/25361

Original commit: elastic/x-pack-elasticsearch@5b1ca009f6
2017-06-29 11:36:20 +02:00
Costin Leau
a9a2d14193 Backport changes from stand-alone repo:
use Joda ReadableTime interface
add support for GZIP
add Ansi colors in the CLI plus logo
remove net-client dependency from proto libs (avoids classpath leaks)

Original commit: elastic/x-pack-elasticsearch@0ce7c9690d
2017-06-29 00:22:32 +03:00
Simon Willnauer
2f10afa40e Use explicit settings for simple bool settings (elastic/x-pack-elasticsearch#1871)
We made the mistake to generate way to many settings in xpack which makes
finding out the right string and where it's defined super difficult. If
we use constants we can just use commandline tools to find where the settings
are defined. This also removes 1.x and 2.x BWC from the enabled settings which should
be removed in 6.x

Original commit: elastic/x-pack-elasticsearch@ec25e6c40c
2017-06-28 22:07:08 +02:00
Tim Brooks
f275a3f07b Support bootstrap password when in container (elastic/x-pack-elasticsearch#1832)
This is related to elastic/x-pack-elasticsearch#1217. This commit reads two environment variables on
startup: BOOTSTRAP_PWD and ELASTIC_CONTAINER. If BOOTSTRAP_PWD is
present, ELASTIC_CONTAINER must be set to true. Otherwise a new
bootstrap check will fail.

If ELASTIC_CONTAINER is set to true, the elastic user can be
authenticated with the BOOTSTRAP_PWD variable when its password
has not been explicitly set.

Original commit: elastic/x-pack-elasticsearch@78f53fd232
2017-06-28 12:48:49 -05:00
Chris Earle
231634251f [Monitoring] Enable read-only access for kibana_system user (elastic/x-pack-elasticsearch#1851)
Add read-only `.monitoring-*` to the `kibana_system` user.

Original commit: elastic/x-pack-elasticsearch@0991c062b8
2017-06-28 11:44:04 -04:00
Alexander Reelsen
edd5fa4ab4 Watcher: Allow unmet action conditions to reset action status (elastic/x-pack-elasticsearch#1859)
The logic of resetting acknowledgements is only executed, if the watch
wide condition is not met. However, if you dont specify a condition
(which makes it always true), but create a condition in your action
(this might make sense because it allows you to execute a transform and
then execute the condition), then after acking this action, it will
never get be unacked, because the watch wide condition is always met.

relates elastic/x-pack-elasticsearch#1857

Original commit: elastic/x-pack-elasticsearch@95aa402c27
2017-06-28 14:52:26 +02:00
Dimitris Athanasiou
1a076e2eb9 [ML] Improve naming of flush related methods (elastic/x-pack-elasticsearch#1872)
Original commit: elastic/x-pack-elasticsearch@eb3eb80b6e
2017-06-28 13:32:51 +01:00
Nik Everett
620404c095 Start moving tests to integration tests
These catch *lots* more issues.

Original commit: elastic/x-pack-elasticsearch@b0e157d7b4
2017-06-27 17:54:07 -04:00
Ali Beyad
a68fb27a23 Upgrade security index to use only one (the default) index type (elastic/x-pack-elasticsearch#1780)
The .security index used several different types to differentiate the
documents added to the index (users, reserved-users, roles, etc).  Since
types are deprecated in 6.x, this commit changes the .security index
access layer to only use a single type and have all documents in the
index be of that single type.  To differentiate documents that may have
the same id (e.g. the same user name and role name), the appropriate
type of the document is prepended to the id.  For example, a user named
"jdoe" will now have the document id "user-jdoe".  

This commit also ensures that any native realm security index operations
that lead to auto creation of the security index first go through the process
of creating the internal security index (.security-v6) and creating the alias
.security to point to the internal index. 

Lastly, anytime the security index is accessed without having been
upgraded, an exception is thrown notifying the user to use the
upgrade API to upgrade the security index.

Original commit: elastic/x-pack-elasticsearch@cc0a474aed
2017-06-27 17:53:58 -04:00
Igor Motov
d2bdd99308 Update upgrade API endpoints (elastic/x-pack-elasticsearch#1724)
Switch from /_xpack/_upgrade to /_xpack/migration/assistance and /_xpack/migration/upgrade

Original commit: elastic/x-pack-elasticsearch@b15217135c
2017-06-27 17:53:58 -04:00
Igor Motov
0de8cf47b1 Add REST point names
Relates to elastic/elasticsearch#24169

Original commit: elastic/x-pack-elasticsearch@d18a24a035
2017-06-27 17:53:58 -04:00
Igor Motov
3502a9901b Add watcher upgrade procedure (elastic/x-pack-elasticsearch#1603)
Relates to elastic/x-pack-elasticsearch#1214

Original commit: elastic/x-pack-elasticsearch@1017d60df4
2017-06-27 17:53:58 -04:00
Igor Motov
b564e6e102 Upgrade API should work with TRIAL license as well (elastic/x-pack-elasticsearch#1567)
Relates to elastic/x-pack-elasticsearch#1214

Original commit: elastic/x-pack-elasticsearch@2ad7e389fb
2017-06-27 17:53:58 -04:00
Igor Motov
568802bbcd Add Kibana upgrade procedure to Upgrade API (elastic/x-pack-elasticsearch#1498)
Adds upgrade infrastructure and support for Kibana index upgrades.

Original commit: elastic/x-pack-elasticsearch@91f343b00f
2017-06-27 17:53:58 -04:00
Igor Motov
ad6b86481e Make Upgrade API license aware (elastic/x-pack-elasticsearch#1401)
Upgrade API should only work with non-trial licenses

Relates to elastic/x-pack-elasticsearch#1214

Original commit: elastic/x-pack-elasticsearch@28fef0feeb
2017-06-27 17:53:58 -04:00
Igor Motov
bb034f42b8 Add Upgrade API Index Upgrade Info action (elastic/x-pack-elasticsearch#1264)
Adds a new Upgrade API with the first action, index upgrade info, that returns that list of indices that require upgrade in the current cluster before the cluster can be upgraded to the next major version.

Relates to elastic/x-pack-elasticsearch#1214

Original commit: elastic/x-pack-elasticsearch@761e7d2128
2017-06-27 17:53:58 -04:00
Ryan Ernst
84574c0367 Another fix for security tribe tests when ssl is disabled
Original commit: elastic/x-pack-elasticsearch@db2eca3522
2017-06-27 14:53:12 -07:00
Suyog Rao
e8472908c1 [Logstash] Change config management license to Gold (elastic/x-pack-elasticsearch#1843)
* [Logstash] Change management license to Gold

Previously the license type for LS config management was `BASIC`. In order to use the security features in Standard/Gold, we had to bump Logstash as well to Gold license.

relates elastic/x-pack-elasticsearch#1841

Original commit: elastic/x-pack-elasticsearch@29194b2417
2017-06-27 14:21:00 -07:00
Ryan Ernst
ad9797393a Fix tribe tests with secure ssl settings
Original commit: elastic/x-pack-elasticsearch@a06ef174b0
2017-06-27 14:01:21 -07:00
Tal Levy
8145b100f1 Introduce the deprecation API (elastic/x-pack-elasticsearch#1833)
Adds REST endpoint and Transport Action for retrieving breaking-changes deprecations that exist in current version. This PR is just the framework for such an API, future checks will be added to the appropriate branches.

Original commit: elastic/x-pack-elasticsearch@990e3468e9
2017-06-27 13:51:45 -07:00
Jason Tedor
b710f5906f Register primary context handoff as known handler
This commit registers the primary context handoff handler as known.

relates elastic/x-pack-elasticsearch#1856

Original commit: elastic/x-pack-elasticsearch@f91142a521
2017-06-27 15:18:44 -04:00
Ryan Ernst
9b3fb66394 Settings: Add secure versions of SSL passphrases (elastic/x-pack-elasticsearch#1852)
This commit adds new settings for the ssl keystore (not the ES keystore)
passphrase settings. New setting names are used, instead of trying to
support the existing names in both yml and the ES keystore, so that
there does not need to be complicated logic between the two. Note that
the old settings remain the only way to set the ssl passphrases for the
transport client, but the Settings object for transport clients are
created in memory by users, so they are already as "secure" as having a
loaded ES keystore. Also note that in the long term future (6.x
timeframe?) these settings should be deprecated and the keys/certs
themselves should be moved into the ES keystore, so there will be no
need for separate keystores/passphrases.

relates elastic/elasticsearch#22475

Original commit: elastic/x-pack-elasticsearch@be5275fa3d
2017-06-27 10:15:12 -07:00
Tal Levy
7c7bf475c1 reflect core change to method signature in sourceAsMap (elastic/x-pack-elasticsearch#1862)
x-pack change for https://github.com/elastic/elasticsearch/pull/25410

Original commit: elastic/x-pack-elasticsearch@908ddd3413
2017-06-27 08:59:29 -07:00
Alexander Reelsen
403cf8eba3 Watcher: Refactoring of TriggeredWatchStore (elastic/x-pack-elasticsearch#1848)
* The TriggeredWatchStore now only has one method to put triggered
  watches
* All code is async in TriggeredWatchStore, locking has been removed
* The dedicated WatchRecord.Fields interface has been removed
* TriggeredWatchTests integration test has been moved to a unit test

Original commit: elastic/x-pack-elasticsearch@bc4b5820fb
2017-06-27 17:47:00 +02:00
Simon Willnauer
9b0639d7f0 Remove usage of mapper.single_type from xpack tests (elastic/x-pack-elasticsearch#1853)
`index.mapper.single_type` will be removed in master. While there is still
one usage in the security template that we are working on, this change
will remove the remaining usage.

Original commit: elastic/x-pack-elasticsearch@6e7f63b9e0
2017-06-27 13:29:34 +02:00
Daniel Mitterdorfer
54345e6b8e Mute MlDistributedFailureIT#testFullClusterRestart()
Relates elastic/elasticsearch#25415

Original commit: elastic/x-pack-elasticsearch@01fa0e7601
2017-06-27 12:44:22 +02:00
Daniel Mitterdorfer
10a95c4268 Add AwaitsFix to KnownActionsTests#testAllTransportHandlersAreKnown
Relates elastic/x-pack-elasticsearch#1856

Original commit: elastic/x-pack-elasticsearch@ea16c49af0
2017-06-27 08:58:55 +02:00
Jason Tedor
c22494bcb7 Remove path.conf setting
This commit is a response to a change in core removing path.conf as a
valid setting.

Relates elastic/x-pack-elasticsearch#1844

Original commit: elastic/x-pack-elasticsearch@477a7eab71
2017-06-26 15:18:49 -04:00
Nik Everett
4d99a3c30e Fix startup
Original commit: elastic/x-pack-elasticsearch@20e5a15316
2017-06-26 14:24:02 -04:00
Alexander Reelsen
0912a85778 Tests: Add test for templatable URLs, that are URL encoded
This is just a workaround at the moment, but allows to use
mustache if you only provide the `url` part of a request,
instead of scheme, port, path, host, etc.

Original commit: elastic/x-pack-elasticsearch@3a4aa26665
2017-06-26 18:24:53 +02:00
Nik Everett
15626b870f Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@decda51b4f
2017-06-26 11:40:43 -04:00
Nik Everett
92e962cab5 Move tokens so they pass the license header test
This mirrors how painless works.

Original commit: elastic/x-pack-elasticsearch@f856714411
2017-06-26 11:39:55 -04:00
Nik Everett
933a51edef Checkstyle and license headers in plugin
Original commit: elastic/x-pack-elasticsearch@e7855b949f
2017-06-26 11:39:55 -04:00
Nik Everett
b504b0d2d9 More "normalization" to es-norms
Turned on checkstyle but turned off line length for sql because
now is not the time. That caught a few things like redundant
modifiers, out of order modifiers, and multiple top level classes
per file. Fixing them gave me a great little tour. So I'm going to
go and do the same in the plugin code now.

Original commit: elastic/x-pack-elasticsearch@8b52d8845d
2017-06-26 11:39:55 -04:00
Nik Everett
ea1be31d27 More nocommits and some checkstyle suppressions
Original commit: elastic/x-pack-elasticsearch@003d5e3aca
2017-06-26 11:39:55 -04:00
Nik Everett
ab2e9e0a84 Fix some dependency licenses
Original commit: elastic/x-pack-elasticsearch@4f25d4a573
2017-06-26 11:39:13 -04:00
Nik Everett
593e6572c1 Closer to compiling
Original commit: elastic/x-pack-elasticsearch@630bd7967d
2017-06-26 11:39:13 -04:00
Nik Everett
b00105f608 Switch to painless style regen task
Maybe not better but it is the way we've done it before so it is
easier to reason about.

Original commit: elastic/x-pack-elasticsearch@5c4f953f08
2017-06-26 11:39:12 -04:00
Nik Everett
6a8a9f33e7 Cleanup sql build
Make the projects more like the rest of Elasticsearch:
* Move integration tests from `itest` to `test`
* Make a build file per sql subproject
* Make sql work properly with elasticsearch-extra

Original commit: elastic/x-pack-elasticsearch@6fbb794f68
2017-06-26 11:37:08 -04:00
David Kyle
2ad9ce4840 [ML] Remove duplicated test classes (elastic/x-pack-elasticsearch#1850)
Original commit: elastic/x-pack-elasticsearch@cb5a1b544a
2017-06-26 16:20:59 +01:00
Dimitris Athanasiou
bd06a7b9b4 [ML] Do not write JSON records when metric value is not finite (elastic/x-pack-elasticsearch#1849)
relates elastic/x-pack-elasticsearch#1847

Original commit: elastic/x-pack-elasticsearch@26a69b840f
2017-06-26 14:45:00 +01:00
Alexander Reelsen
3db3cd0f0b Watcher: Reduce logging noise if no local watcher shards exist (elastic/x-pack-elasticsearch#1846)
Every cluster state update resulted in a log message, that watcher
pauses execution. This has been fixed to only log, if there was an
actual state switch from executing to pausing, but do nothing if
there are no local shards anyway.

This will reduce the logging noise in tests a lot.

Original commit: elastic/x-pack-elasticsearch@32ab86610c
2017-06-26 12:49:08 +02:00
David Kyle
232d59b855 [ML] Give kill a chance to stop autodetect before closing input (elastic/x-pack-elasticsearch#1824)
* Give kill a chance to kill the process before closing input

* Remove variable that can be refactored out

Original commit: elastic/x-pack-elasticsearch@42f7a3cece
2017-06-26 10:17:44 +01:00
Colin Goodheart-Smithe
27aa3094f6 Adds GraphExploreRequest as a remote index request (elastic/x-pack-elasticsearch#1836)
The graph API needs to be able to search in remote indices. Although it uses the Search API to perform the search and so doesn’t need to deal with remote indexes directly, the security feature needs to know it can be used with remote indexes so it knows to include remote indices in the list of indices accessible from the API for index level security

Original commit: elastic/x-pack-elasticsearch@e3cd84963e
2017-06-26 09:59:00 +01:00
Tim Vernum
a73be456ec [Security] Don't apply local restrictions on remote cluster search (elastic/x-pack-elasticsearch#1609)
This change removes all local security checks against remote cluster names.
Any user is allowed to attempt a cross-cluster search, and it is the responsibility of the remote cluster to authorise the search (or not).

This includes support for remote searches even if you have _no_ local search privileges.

Original commit: elastic/x-pack-elasticsearch@1620c3a8fa
2017-06-26 12:30:31 +10:00
Dimitris Athanasiou
701dc53c2a [ML] Refactor aggregation response handling to make more flexible (elastic/x-pack-elasticsearch#1795)
Currently, aggregated datafeeds construct JSON from the aggregation
response by traversing all nested aggregations. In order to
achieve this, multiple leaf aggregations are not supported. Also,
scenarios it makes it impossible to effectively use pipeline
aggregations as it will not ignore the intermediate bucket
aggregations.

This commit refactors AggregationToJsonProcessor in order to
support the above scenarios. This is achieved by only converting
the fields of interest, that is the job analysis fields.

Original commit: elastic/x-pack-elasticsearch@8b575956ca
2017-06-23 12:36:32 +01:00