Commit Graph

6755 Commits

Author SHA1 Message Date
Tal Levy 5a090c14c1 convert more request objects to writeable (elastic/x-pack-elasticsearch#2457)
* convert more to writeable

* migrate streamable tests to writeable tests

Original commit: elastic/x-pack-elasticsearch@56794e5760
2017-09-18 13:20:02 -07:00
Nik Everett bc03aa6c03 Line up sql serialization methods with core's (elastic/x-pack-elasticsearch#2538)
This renames that `write` and `read` methods in SQL to `writeTo` and
`readFrom` to line up with the names used in core. I don't have a
strong opinion whether or not any name is better than any other but
I figure there isn't a good reason for SQL to be different from the
rest of Elasticsearch.

Original commit: elastic/x-pack-elasticsearch@e5de9a4b81
2017-09-18 16:06:14 -04:00
Nik Everett 204e340397 Drop parameterized type from SQL's exception messages (elastic/x-pack-elasticsearch#2540)
The type parameter wasn't buying us anything.

Original commit: elastic/x-pack-elasticsearch@5005b26c09
2017-09-18 16:06:05 -04:00
Suyog Rao 3a9aad5ece [Logstash] Remove version field from config mgmt
relates elastic/x-pack-elasticsearch#2405

Original commit: elastic/x-pack-elasticsearch@5bfd1b7a6d
2017-09-18 12:35:21 -07:00
lcawley 3ff175b541 [DOCS] Remove old certgen examples
Original commit: elastic/x-pack-elasticsearch@f97d0ab00c
2017-09-18 12:22:45 -07:00
Lisa Cawley f71dc549a5 [DOCS] Update X-Pack installation re TLS changes (elastic/x-pack-elasticsearch#2523)
* [DOCS] Update xpack installation re TLS changes

* [DOCS] Clarify multi-node TLS requirement

Original commit: elastic/x-pack-elasticsearch@a961488fa1
2017-09-18 11:20:18 -07:00
Simon Willnauer 47214426e9 Use InputStreamStreamInputs validation to limit size of tokens (elastic/x-pack-elasticsearch#2537)
Relates to elastic/elasticsearch#26692
relates elastic/x-pack-elasticsearch#2493

Original commit: elastic/x-pack-elasticsearch@8e23868743
2017-09-18 19:25:08 +02:00
Nik Everett 759411571e Remove two NOCOMMITs
* TimeoutInfo - This is now tracked in the SQL tracker github issue
* AbstractProto - Convert to a TODO as we *can* handle it after
release. I've added it to the SQL tracker github issue in a special
section for low priority protocol stuff. Protocol stuff is special
because if we can make the change before release we don't have to
worry about backwards compatibility.

Original commit: elastic/x-pack-elasticsearch@dbef9db5f8
2017-09-18 12:54:40 -04:00
Nik Everett 52ee02da27 Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@c25c179ce6
2017-09-18 12:32:46 -04:00
Lisa Cawley 371953488b [DOCS] Create certgen command reference (elastic/x-pack-elasticsearch#2456)
* [DOCS] Create certgen command reference

* [DOCS] Added missing certgen parameters

* [DOCS] Finalized certgen command reference

Original commit: elastic/x-pack-elasticsearch@6d0b795eb4
2017-09-18 08:36:13 -07:00
Dimitris Athanasiou 9b25d0edf7 [ML] Correctly ignore extra aggregation buckets (elastic/x-pack-elasticsearch#2530)
The problem here is that the code was ignoring buckets
whose start time was before the start time of the extractor.
However, this is not a good enough condition. For example,
when there are no data in the bucket extra bucket that is
being queried, the first bucket will be the one containing
the start time.

This commit fixes the issue by changing the condition to
ignore buckets before the first bucket that includes the
start time of the extraction.

relates elastic/x-pack-elasticsearch#2519 

Original commit: elastic/x-pack-elasticsearch@15c7d2655f
2017-09-18 12:10:19 +01:00
Tim Vernum dea82a07a2 Infer KeyStore type from pathname (elastic/x-pack-elasticsearch#2514)
If the keystore type is not explicitly specified, infer it from the filename.
Treats .p12, .pfx and .pkcs12 as being PKCS12, all others as jks.

This will allow certgen to produce PKCSelastic/x-pack-elasticsearch#12 files by default and make it easy to use them as x-pack keystores

Original commit: elastic/x-pack-elasticsearch@fc361f0d87
2017-09-18 14:21:19 +10:00
Chris Earle 24c2c62ca2 [Monitoring] Ignore .marvel* indices (elastic/x-pack-elasticsearch#2520)
Beginning with 7.0, the cleaner service will no longer automatically cleanup .marvel indices regardless of their age.

Original commit: elastic/x-pack-elasticsearch@5b90e6f62a
2017-09-15 13:35:45 -07:00
jaymode 19de38665e Docs: remove incorrect name attribute from role snippet
relates elastic/x-pack-elasticsearch#2497

Original commit: elastic/x-pack-elasticsearch@ef15a1e36c
2017-09-15 12:51:37 -06:00
Tim Brooks b3914afd30 Reenable TribeWithSecurityIT tests (elastic/x-pack-elasticsearch#2511)
This is related to elastic/x-pack-elasticsearch#1996. These tests were disabled during the bootstrap
password work. They can now be reenabled. Additionally, I made the test
password used in tests consistent.

Original commit: elastic/x-pack-elasticsearch@5b490c8231
2017-09-15 12:50:54 -06:00
Simon Willnauer 96e01dce47 Only require TLS for standard/gold/platinum licenses elastic/x-pack-elasticsearch#2517
relates elastic/x-pack-elasticsearch#2463

Original commit: elastic/x-pack-elasticsearch@5213cf24f1
2017-09-15 20:21:15 +02:00
Chris Earle 9c9da2e1e4 [Monitoring] Remove Legacy Monitoring Indices (elastic/x-pack-elasticsearch#2513)
This changes Monitoring's Cleaner Service to remove any legacy Monitoring index that is appropriately old.

This includes any `.marvel-*` index and also the "data" indices used by both Marvel and 5.0 - 5.4 versions of X-Pack monitoring, as well as the legacy alerts index.

Original commit: elastic/x-pack-elasticsearch@8d99f5518b
2017-09-15 08:46:10 -07:00
jaymode 84dd719ab9 remove outdated comment
Original commit: elastic/x-pack-elasticsearch@06a51abb65
2017-09-15 09:03:36 -06:00
jaymode 344603e40f update text in TLSLicenseBootstrapCheck
Original commit: elastic/x-pack-elasticsearch@4ee6827566
2017-09-15 08:56:34 -06:00
Jay Modi 53d6d945f0 Update documentation to reflect the latest TLS changes and licensing (elastic/x-pack-elasticsearch#2508)
This commit updates to documentation and adds notes about TLS being required to install a
license.

Relates elastic/x-pack-elasticsearch#2463

Original commit: elastic/x-pack-elasticsearch@0d8bfb98ea
2017-09-15 08:44:03 -06:00
Simon Willnauer 4d20586b24 [TEST] add integration test that ensures we reject license upgrades if TLS is not enabled
Original commit: elastic/x-pack-elasticsearch@dfbadb5e5f
2017-09-15 14:47:28 +02:00
Simon Willnauer c3066d1a51 Merge branch 'master' into tls_6.0
Original commit: elastic/x-pack-elasticsearch@9ce33bc7c3
2017-09-15 09:51:16 +02:00
Simon Willnauer 023bdb72b2 Add common-analysis plugin to several xpack integ tests (elastic/x-pack-elasticsearch#2501)
Several tests miss the common-analysis plugin in the old-style integ
tests causing odd exceptions in the test logs. This adds the missing plugin reference.

relates elastic/x-pack-elasticsearch#2363

Original commit: elastic/x-pack-elasticsearch@e4e6735408
2017-09-15 09:25:36 +02:00
Lisa Cawley 895d28f462 [DOCS] Remove redundant users command info (elastic/x-pack-elasticsearch#2504)
Original commit: elastic/x-pack-elasticsearch@1c9fa91293
2017-09-14 15:47:21 -07:00
jaymode 8997792875 Test: use TLS for plugin integ tests
Original commit: elastic/x-pack-elasticsearch@99971d7256
2017-09-14 15:57:28 -06:00
lcawley c3b658df88 [DOCS] Fixed broken link to put watch API
Original commit: elastic/x-pack-elasticsearch@f36caaa371
2017-09-14 14:12:20 -07:00
Michael Basnight fa0b854fb6 Update rest-api-spec to use bad_request (elastic/x-pack-elasticsearch#2507)
ref #elastic/elasticsearch#26539

Original commit: elastic/x-pack-elasticsearch@8b79a0769a
2017-09-14 15:59:29 -05:00
Lisa Cawley df1e4e85a4 Format Watcher APIs (elastic/x-pack-elasticsearch#2382)
* [DOCS] Format Watcher APIs

* [DOCS] Removed master_timeout from Watcher APIs

* [DOCS] Added authority info to watcher APIs

Original commit: elastic/x-pack-elasticsearch@1e6de3b036
2017-09-14 13:01:47 -07:00
Andy Bristol 279c7e14fd [TEST] fix security template version check in rest tests (elastic/x-pack-elasticsearch#2506)
Since the template upgrade service was added, upgrades should
be performed by a node with the highest version in the cluster,
which may not be the master node.

Original commit: elastic/x-pack-elasticsearch@d66145de54
2017-09-14 12:16:20 -07:00
Jay Modi 57de66476c Disable TLS by default (elastic/x-pack-elasticsearch#2481)
This commit adds back the ability to disable TLS on the transport layer and also disables TLS by
default to restore the 5.x behavior. The auto generation of key/cert and bundled CA certificate
have also been removed.

Relates elastic/x-pack-elasticsearch#2463

Original commit: elastic/x-pack-elasticsearch@abc66ec67d
2017-09-14 12:18:54 -06:00
Simon Willnauer 1e14e14571 Prevent licenses to be upgraded to production unless TLS is configured (elastic/x-pack-elasticsearch#2502)
if a user tries to upgrade a license to a production license and has security
enabled we prevent the upgrade unless TLS is setup. This is a requirement now
if a cluster with security is running in prodcution.

Relates to elastic/x-pack-elasticsearch#2463

Original commit: elastic/x-pack-elasticsearch@d61ef3bcb1
2017-09-14 20:14:27 +02:00
Nik Everett 4994ff86f2 SQL: Enable tests for shared-proto
There used to not be tests in this project. There are now.

Original commit: elastic/x-pack-elasticsearch@81bd25479b
2017-09-14 12:06:47 -04:00
Nik Everett bac9afee7e Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@270ab71b19
2017-09-14 11:43:48 -04:00
Nik Everett 5d3f5cc4f8 Support scrolling in SQL's CLI (elastic/x-pack-elasticsearch#2494)
* Move CLI to TransportSqlAction
    * Moves REST endpoint from `/_cli` to `/_sql/cli`
    * Removes the special purpose CLI transport action instead
    implements the CLI entirely on the REST layer, delegating
    all SQL stuff to the same action that backs the `/_sql` REST
    API.
    * Reworks "embedded testing mode" to use a `FilterClient` to
    bounce capture the sql transport action and execute in embedded.
    * Switches CLI formatting from consuming the entire response
    to consuming just the first page of the response and returning
    a `cursor` that can be used to read the next page. That read is
    not yet implemented.
    * Switch CLI formatting from the consuming the `RowSetCursor` to
    consuming the `SqlResponse` object.
    * Adds tests for CLI formatting.
* Support next page in the cli
    * Rename cli's CommandRequest/CommandResponse to
    QueryInitRequest/QueryInitResponse to line up with jdbc
    * Implement QueryPageRequest/QueryPageResponse in cli
    * Use `byte[]` to represent the cursor in the cli. Those bytes
    mean something, but only to the server. The only reasonint that
    the client does about them is "if length == 0 then there isn't a
    next page."
    * Pull common code from jdbc's QueryInitRequest, QueryPageRequest,
    QueryInitResponse, and QueryPageResponse into the shared-proto
    project
        * By implication this switches jdbc's QueryPageRequest to using
     the same cursor implementation as the cli

Original commit: elastic/x-pack-elasticsearch@193586f1ee
2017-09-14 10:26:42 -04:00
Jay Modi 8d8baffe24 Add specific client and user for security index access (elastic/x-pack-elasticsearch#2492)
This change removes security index access from the xpack user by creating its own specific role
and adds a xpack security user that maintains the superuser role so that it can perform all
operations necessary for security.

Original commit: elastic/x-pack-elasticsearch@ad906bc913
2017-09-14 08:09:14 -06:00
David Roberts 104a3a323f [BUILD] Make AWS error message more informative
Original commit: elastic/x-pack-elasticsearch@42cca7ed82
2017-09-14 14:46:48 +01:00
Simon Willnauer 91b57ee63f Add bootstrap check that enforces TLS if a production license is in the local clusterstate (elastic/x-pack-elasticsearch#2499)
This change will enforce transport SSL to be enforced if security is enabled and the
license in the clusterstate is a production license. The cluster state is loaded from
local storage such that we don't need to join a cluster to make these checks. Yet, the cluster
might have already got a different license if the node got disconnected while the license got
downgraded and then TLS got disabled. This corner case requires manual intervention which
we consider ok given the simplicity of this change.

Relates to elastic/x-pack-elasticsearch#2463

Original commit: elastic/x-pack-elasticsearch@5765b7cd21
2017-09-14 13:52:53 +02:00
Hendrik Muhs 7d19264363 [ML-FC] Branch landing feature/ml (elastic/x-pack-elasticsearch#2500)
integrate forecasting feature branch into master

    - add endpoint xpack/ml/job/forecast to request forecasting on data of ml-jobs
       - current parameters: end time
    - persists forecast results into shared or own index
       - different runs are separated by a 'forecast id'

relates elastic/x-pack-elasticsearch#1838

Original commit: elastic/x-pack-elasticsearch@f9d701a6bc
2017-09-14 12:31:20 +02:00
Simon Willnauer 3b00251a96 Merge branch 'master' into tls_6.0
Original commit: elastic/x-pack-elasticsearch@4a36f0c2be
2017-09-14 07:43:19 +02:00
Jason Tedor 4f3e740ba8 Refactor bootstrap check results and error messages
This commit refactors the X-Pack bootstrap checks to respond to a change
in core Elasticsearch where the checks now return a single result
object.

Relates elastic/x-pack-elasticsearch#2495

Original commit: elastic/x-pack-elasticsearch@230b050529
2017-09-13 21:30:51 -04:00
lcawley 9ea36ef771 [DOCS] Added tip in users command
Original commit: elastic/x-pack-elasticsearch@3fb4e1819c
2017-09-13 17:21:15 -07:00
Igor Motov 6c0de3689f SQL: Provide hex version of toString for SQL's byte array (elastic/x-pack-elasticsearch#2469)
Original commit: elastic/x-pack-elasticsearch@8eadc9af99
2017-09-14 09:16:35 +09:00
Lisa Cawley 89d6c7e01e [DOCS] Create reference for users command (elastic/x-pack-elasticsearch#2480)
Original commit: elastic/x-pack-elasticsearch@d0afe8a20d
2017-09-13 17:16:06 -07:00
Aaron Bull Schaefer 447f224677 Add releaseTest option to CI script (elastic/x-pack-elasticsearch#2482)
This option runs a normal check but with `-Dsnapshot=false` (the flag
used to indicate a release build).

Related to https://github.com/elastic/infra/issues/2759 and
https://github.com/elastic/infra/issues/2739 from the ES side.

Original commit: elastic/x-pack-elasticsearch@e674e68905
2017-09-13 16:01:21 -07:00
Nik Everett 858f0b2dac Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@9945382d90
2017-09-13 16:45:27 -04:00
Nik Everett f15666f82e Fix links in deprecation checks (elastic/x-pack-elasticsearch#2490)
Some links must have moved since we wrote the tests and released
5.6.0.

relates elastic/x-pack-elasticsearch#2488

Original commit: elastic/x-pack-elasticsearch@ebceee7f3d
2017-09-13 16:32:40 -04:00
Simon Willnauer 01a921a8e3 Accept BootstrapContext in xpack (elastic/x-pack-elasticsearch#2486)
This is the xpack side of elastic/elasticsearch#26628

Original commit: elastic/x-pack-elasticsearch@f6c0599ee2
2017-09-13 22:14:29 +02:00
Jay Modi f30e5c3fee Register the legacy truststore password setting for the PKI realm (elastic/x-pack-elasticsearch#2487)
After the addition of the secure settings in 5.6, the truststore.password setting for the PKI realm
was no longer registered. This would cause new nodes to fail for customers that were upgrading and
had configured a PKI realm with a truststore. This change registers the setting and adds a test to
ensure a realm configuration with the old setting passes validation.

Relates elastic/support-dev-help#2505

Original commit: elastic/x-pack-elasticsearch@54da044a27
2017-09-13 13:11:54 -06:00
Simon Willnauer 0680e41f36 Prevent nodes from joining a non-TLS enabled cluster with a production license (elastic/x-pack-elasticsearch#2484)
This change prevents a node from joining a cluster with a production license (gold, platinum, standard) iff the cluster doesn't have TLS setup. This is mainly a BWC oriented change that prevents joining old 5.x clusters without a TLS setup.

Relates to elastic/x-pack-elasticsearch#2463

Original commit: elastic/x-pack-elasticsearch@21f5a58472
2017-09-13 20:40:35 +02:00
Dimitris Athanasiou 99ffbb1cd6 [ML] Add random offset to the maintenance task execution time (elastic/x-pack-elasticsearch#2483)
Currently the maintenance task is executed at 30 minutes past
midnight of each day. In the scenario where multiple clusters
are running on the same hardware infrastructure they all will
be running at the same time, competing for resources.

This commit changes this by adding a random offset to the
execution time which ranges from 0 to 119 minutes. The
minute granularity means that different offsets give at
least 1 minute for the maintenance task to end. Moreover,
the 2 hour window gives enough slots for different offsets
to occur and remains within what most people would think
as "middle of the night".

relates elastic/x-pack-elasticsearch#2273

Original commit: elastic/x-pack-elasticsearch@b538923aca
2017-09-13 14:53:44 +01:00