6126 Commits

Author SHA1 Message Date
Nik Everett
3759349db8 Remove many NOCOMMITs from cli protocol
* Move read from a static method to a ctor to mirror core.
* Make read and writes read and write the same data.
* Instead of the "header" integer use a byte for the response
type.
* For responses that do not make their request type obvious
then serialize the request type.
* Remove the request type member from requests and responses and
replace with an abstract method. These type members have caused
us trouble in core in the past.
* Remove the Message superclass as it didn't have anything in it.
* Pass client version to the request reader and response writer.
* Add round trip tests for the protocol.
* Force Requests and Responses to provide good `toString`, `equals`,
and `hashCode`.

Original commit: elastic/x-pack-elasticsearch@653ed8c27f
2017-07-10 11:36:08 -04:00
Nik Everett
0fe1e4bb48 Clean serialization for cli's CommandResponse
* Switch `data` member from Object to `String`
* Compress packages on server so easier to build `data` as `String`
* Move write of `data` member into `encode` method
* Move read of `data` member into ctor

Original commit: elastic/x-pack-elasticsearch@e3a52e7493
2017-07-08 07:24:37 -04:00
Nik Everett
9a08c93f38 Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@f789ea33ab
2017-07-07 17:01:52 -04:00
Nik Everett
65a0d44b2f Handle "current" being dropped from the list of released versions
Original commit: elastic/x-pack-elasticsearch@46668ce672
2017-07-07 16:01:21 -04:00
Nik Everett
1af9f56834 Remove IndexResponse.created
Core removed this in elastic/elasticsearch#25516.

Original commit: elastic/x-pack-elasticsearch@b1e7040fa2
2017-07-07 13:57:43 -04:00
Lisa Cawley
2e58164b96 [DOCS] Add ML limitation (elastic/x-pack-elasticsearch#1916)
Original commit: elastic/x-pack-elasticsearch@7730810f98
2017-07-07 10:17:55 -07:00
Lisa Cawley
2aa53f253f [DOCS] Update doc links for graph API (elastic/x-pack-elasticsearch#1884)
Original commit: elastic/x-pack-elasticsearch@9eebb6c9a6
2017-07-07 10:11:06 -07:00
Tim Sullivan
66ebdff447 [Monitoring] Email actions for Cluster Alerts (elastic/x-pack-elasticsearch#1879)
* [Monitoring] Email actions for Cluster Alerts

* fix quotations in email fields

* move email vars to transform, and rename for snake_case

* add state to email subject for cluster status alert

* remove types field in kibana_settings search

* simplify email action condition script

* uppercase the state for the email subject

* only append state to email subject if alert is new

* show state in email subject even when alert is resolved

Original commit: elastic/x-pack-elasticsearch@e6fdd8d620
2017-07-07 09:55:45 -07:00
Nik Everett
184d53f598 Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@ed7a977328
2017-07-07 10:35:09 -04:00
Nik Everett
f6fad00765 cli's gradle run works!
Including proper tty support!

Original commit: elastic/x-pack-elasticsearch@a780a99d1b
2017-07-07 10:32:23 -04:00
Simon Willnauer
c87d9278a6 Add validation for all transport.profile.* settings (elastic/x-pack-elasticsearch#1909)
Follow-up from elasticsearch/elastic#25508

Original commit: elastic/x-pack-elasticsearch@fe08e74ccc
2017-07-07 09:41:34 +02:00
Tim Vernum
c5012ac6e8 [DOC] Miscellaneous security doc updates (elastic/x-pack-elasticsearch#1908)
- Document refresh interval for role mapping files
- Fix obsolete shield reference in transport profile example 
- Clarify that AD & PKI don't support run_as
- Fix logstash conf examples
- Clarify interaction of SSL settings and PKI realm settings
- Document PKI DN format, and recommend use of pki_dn metadata
- Provide more details about action.auto_create_index during setup

Original commit: elastic/x-pack-elasticsearch@49ddb12a7e
2017-07-07 13:33:35 +10:00
Nik Everett
edcc87e30e gradle check -xforbiddenPatterns passes!
Hurray! Now we just have to keep it that way!

Original commit: elastic/x-pack-elasticsearch@98b8ede44e
2017-07-06 16:24:19 -04:00
Nik Everett
fc0bce0a3e Some more tests
Closer to `gradle check -xforbiddenPatterns` passing.

Original commit: elastic/x-pack-elasticsearch@fbce74a1f3
2017-07-06 15:58:12 -04:00
Jason Tedor
b636dcc366 Upgrade to Netty 4.1.13.Final
This commit updates the SHAs here as core upgraded the Netty dependency
from 4.1.11.Final to 4.1.13.Final.

Relates elastic/x-pack-elasticsearch#1936

Original commit: elastic/x-pack-elasticsearch@d1803fc331
2017-07-06 15:37:17 -04:00
Nik Everett
527541a66e More build fixes
* A unit test for cli
* Licenses for cli
* Remove licenses for protos (no more deps)
* `SHOW TABLES` returns results in order (makes testing easier)
* Clean up embedded jdbc server
* Wire up embedded cli server

Original commit: elastic/x-pack-elasticsearch@b98aaf446b
2017-07-06 14:37:09 -04:00
Lisa Cawley
0e322b525d [DOCS] Update doc links for security APIs (elastic/x-pack-elasticsearch#1883)
Original commit: elastic/x-pack-elasticsearch@660410008c
2017-07-06 11:06:03 -07:00
Tim Brooks
d95c365e64 Loosen setup mode restrictions for upgrade tests (elastic/x-pack-elasticsearch#1927)
This commit is related to elastic/x-pack-elasticsearch#1896. Currently setup mode means that the
password must be set post 6.0 for using x-pack. This interferes with
upgrade tests as setting the password fails without a properly
upgraded security index.

This commit loosens two aspects of the security.

1. The old default password will be accept in setup mode (requests
from localhost).
2. All request types can be submitted in setup mode.

Original commit: elastic/x-pack-elasticsearch@8a2a577038
2017-07-06 10:37:48 -05:00
Nik Everett
d25c23c3b4 Fix some forbiddens in cli
Original commit: elastic/x-pack-elasticsearch@3dc96c7279
2017-07-06 10:54:41 -04:00
Dimitris Athanasiou
4e03c657a3 [ML] Write model_memory_limit with units into the cluster state (elastic/x-pack-elasticsearch#1934)
This is step 2 of elastic/x-pack-elasticsearch#1604

This change stores `model_memory_limit` as a string with `mb` unit.
I considered using the `toString` method of `ByteSizeValue` but it
can lead to accuracy loss. Adding the fixed `mb` unit maintains
the accuracy, while making clear what unit the value is in.

Original commit: elastic/x-pack-elasticsearch@4dc48f0ce8
2017-07-06 15:40:19 +01:00
Nik Everett
bc2f0fe7ff Get jdbc tests passing
`gradle check -xforbiddenPatterns` now passes in jdbc.

This makes running the embedded HTTP server slightly more difficult,
you now have to add the following to your jvm arguments.
```
-ea -Dtests.rest.cluster=localhost:9200 -Dtests.embed.sql=true -Dtests.security.manager=false
```

Depending on your environment the embedded jdbc connection may give
spurious failures that look like:
```
org.elasticsearch.xpack.sql.jdbc.jdbc.JdbcException: RemoteTransportException[[node-0][127.0.0.1:9300][indices:data/read/search]]; nested: SearchPhaseExecutionException[]; nested: GeneralScriptException[Failed to compile inline script [( params.a0 > params.v0 ) && ( params.a1 > params.v1 )] using lang [painless]]; nested: CircuitBreakingException[[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead; this limit can be changed by the [script.max_compilations_per_minute] setting];
...
Caused by: Failed to execute phase [fetch],
..
Caused by: GeneralScriptException[Failed to compile inline script [( params.a0 > params.v0 ) && ( params.a1 > params.v1 )] using lang [painless]]; nested: CircuitBreakingException[[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead; this limit can be changed by the [script.max_compilations_per_minute] setting];
...
Caused by: CircuitBreakingException[[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead; this limit can be changed by the [script.max_compilations_per_minute] setting]
```

`gradle check` works around this by setting `script.max_compilations_per_minute`
to `1000`.

Another change is that we no longer support loading the test data by
uncommenting some code. Instead we load the test data into Elaticsearch
before the first test and we deleted it after the last test. This is
so that tests that required different test data can interoperate with
eachother. The spec tests all use the same test data but the metadata
tests do not.

Original commit: elastic/x-pack-elasticsearch@8b8f684ac1
2017-07-06 10:24:59 -04:00
David Roberts
984d2ca2ba [ML] Ignore unknown fields when parsing ML cluster state (elastic/x-pack-elasticsearch#1924)
ML has two types of custom cluster state:

1. jobs
2. datafeeds

These need to be parsed from JSON in two situations:

1. Create/update of the job/datafeed
2. Restoring cluster state on startup

Previously we used exactly the same parser in both situations, but this
severely limits our ability to add new features.  This is because the
parser was very strict.  This was good when accepting create/update
requests from users, but when restoring cluster state from disk it meant
that we could not add new fields, as that would prevent reloading in
mixed version clusters.

This commit introduces a second parser that tolerates unknown fields for
each object that is stored in cluster state.  Then we use this more
tolerant parser when parsing cluster state, but still use the strict
parser when parsing REST requests.

relates elastic/x-pack-elasticsearch#1732

Original commit: elastic/x-pack-elasticsearch@754e51d1ec
2017-07-06 13:36:11 +01:00
Dimitris Athanasiou
adc6fd5a0f [ML] Parse model_memory_limit both as number and string with units (elastic/x-pack-elasticsearch#1921)
This is the first step for elastic/x-pack-elasticsearch#1604.

Original commit: elastic/x-pack-elasticsearch@70010a216d
2017-07-06 12:29:12 +01:00
Lee Hinman
b66560fa85 [TEST] Use better REST endpoints for testing watch actions
This is the x-pack side of https://github.com/elastic/elasticsearch/pull/24437

It changes two things, for the disable tests, it uses a valid endpoint instead
of a previously invalid endpoint that happened to return a 400 because the
endpoint was bad, regardless of if watcher was disabled.

The other change is to create the watches index by putting a watch using the
correct API, rather than manually creating the index. This is because
`RestHijackOperationAction` hijacks operations like this and stops accessing the
endpoint in a regular manner.

Original commit: elastic/x-pack-elasticsearch@3be78d9aea
2017-07-05 10:55:25 -06:00
Igor Motov
c035adb568 Rename upgrade.* client methods to migration.* (elastic/x-pack-elasticsearch#1881)
This makes client names consistent with REST APIs and makes it simplifies client development.

Original commit: elastic/x-pack-elasticsearch@90913f485b
2017-07-05 09:47:58 -04:00
Dimitrios Athanasiou
2e0560528f [TEST] Fix MlBasicMultiNodeIT after changing flush response
Relates elastic/x-pack-elasticsearch#1914

Original commit: elastic/x-pack-elasticsearch@5175bf64d9
2017-07-05 13:30:25 +01:00
Nik Everett
18bc094dbc Remove LocalEsCluster
We aren't using it anymore.

Original commit: elastic/x-pack-elasticsearch@67f47c4f11
2017-07-05 08:21:35 -04:00
Nik Everett
9d83eccf27 Merge branch 'feature/sql' of github.com:elastic/x-pack-elasticsearch into feature/sql
Original commit: elastic/x-pack-elasticsearch@34d536c5e1
2017-07-05 08:15:11 -04:00
Nik Everett
b2484bebe8 Merge branch 'master' into feature/sql
This should get us OK until 0170e0e8d3fa210a539737c91251dbdd520bb458.

Original commit: elastic/x-pack-elasticsearch@e0bea08365
2017-07-05 08:14:24 -04:00
Dimitrios Athanasiou
3057f7f4b6 [TEST] Fix post data request in post_data.yml
Original commit: elastic/x-pack-elasticsearch@14e9083c02
2017-07-05 12:06:53 +01:00
Dimitris Athanasiou
15f9b1ed9c [ML] Impove mechanism for ignoring maintenance windows (elastic/x-pack-elasticsearch#1914)
Currently, the autodetect process has an `ignoreDowntime`
parameter which, when set to true, results to time being
skipped over to the end of the bucket of the first data
point received. After that, skipping time requires closing
and opening the job. With regard to datafeeds, this does not
work well with real-time requests which use the advance-time
API in order to ensure results are created for data gaps.

This commit improves this functionality by making it more
flexible and less ambiguous.

- flush API now supports skip_time parameter which
sends a control message to the autodetect process
telling it to skip time to a given value
- the flush API now also returns the last_finalized_bucket_end
time which allows clients to resume data searches correctly
- the datafeed start API issues a skip_time request when the
given start time is after the resume point. It then resumes
the search from the last_finalized_bucket_end time.

relates elastic/x-pack-elasticsearch#1913


Original commit: elastic/x-pack-elasticsearch@caa5fe8016
2017-07-05 11:33:42 +01:00
Lisa Cawley
595468a505 [DOCS] Update doc links for watcher APIs (elastic/x-pack-elasticsearch#1880)
Original commit: elastic/x-pack-elasticsearch@47521f1261
2017-07-04 11:16:47 -07:00
Lisa Cawley
04ff94a180 [DOCS] Update doc links for ML APIs (elastic/x-pack-elasticsearch#1882)
Original commit: elastic/x-pack-elasticsearch@cdc45f282d
2017-07-04 11:10:34 -07:00
Costin Leau
12e12f7a94 Improve exception reporting
Add the source spec in the error stacktrace

Original commit: elastic/x-pack-elasticsearch@bc7ed6ac60
2017-07-04 19:25:03 +03:00
Costin Leau
6f670c8d8e Testing fixes
- simplify handling of timezone in H2
- fix leaking threadpool in HttpServer
- update Csv tests
- keep the dates as long in internal Page

Original commit: elastic/x-pack-elasticsearch@43a804607f
2017-07-04 19:05:09 +03:00
Sophie Chang
0ff26e7566 [DOCS] Added limitation for time-based index patterns (elastic/x-pack-elasticsearch#1910)
* [DOCS] Added limitation for time-based index patterns

* [DOCS] Updated limitations

Original commit: elastic/x-pack-elasticsearch@9927105cc0
2017-07-04 08:40:42 -07:00
David Roberts
b93890b0df [ML] Add license downgrade warning (elastic/x-pack-elasticsearch#1912)
When applying a license that will cause ML to be disabled, a warning
to this effect is now included in the acknowledgement messages.

relates elastic/x-pack-elasticsearch#1888

Original commit: elastic/x-pack-elasticsearch@e453f20f58
2017-07-04 15:10:18 +01:00
Dimitrios Athanasiou
05a73a32bd [TEST] Correctly verify that aliases were deleted in MlJobIT
This fixes `testDeleteJobAfterMissingAliases` to not fail randomly.
The reason the test was failing is that at some point some aliases
are deleted and the cat-aliases API is called to verify they were
indeed deleted. This was checked by asserting an
index_not_found_exception was thrown by the cat-aliases request.
This was some times working as there were no other aliases. However,
that depends on whether other x-pack features had time to create their
infrastructure. For example, security creates an alias. When other
aliases had the time to be created, the cat-aliases request does not
fail and the test fails.

This commit simply changes the verification that the read/write
aliases were deleted by replacing the cat-aliases request with
two single get-alias requests.

Original commit: elastic/x-pack-elasticsearch@fe2c7b0cb4
2017-07-04 13:14:38 +01:00
Alexander Reelsen
fff72256a5 Watcher: Fix wrong logging in reporting attachment parser (elastic/x-pack-elasticsearch#1900)
The logging shows a wrong HTTP response status code from a previous
request. In addition the body now also gets logged, as debugging
is impossible otherwise.

Original commit: elastic/x-pack-elasticsearch@cc998cd587
2017-07-04 13:01:14 +02:00
Nik Everett
50a46d0ed2 Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@51fc29da6a
2017-07-03 18:04:09 -04:00
Costin Leau
83d73c9b3a More polishing
Original commit: elastic/x-pack-elasticsearch@043edf250b
2017-07-04 00:41:16 +03:00
Costin Leau
405b4a4166 Update test dataset name
Original commit: elastic/x-pack-elasticsearch@49951836c1
2017-07-04 00:00:11 +03:00
Nik Everett
56803bdd1b Port some of CLI's demos to tests
Original commit: elastic/x-pack-elasticsearch@5827c08268
2017-07-03 16:41:25 -04:00
Costin Leau
3eb1258b0d Add to CSV infra plus some refactoring of the JDBC one
Original commit: elastic/x-pack-elasticsearch@3e002c91c2
2017-07-03 21:01:29 +03:00
Clinton Gormley
fd518ea020 Include shared/attributes.asciidoc directly from docs master
Original commit: elastic/x-pack-elasticsearch@f3a0828c5d
2017-07-03 18:23:19 +02:00
Christoph Büscher
8d26996afd Remove QueryParseContext (elastic/x-pack-elasticsearch#1895)
This is the x-pack side of elastic/elasticsearch#25486.

Original commit: elastic/x-pack-elasticsearch@c90a3e096b
2017-07-03 17:31:18 +02:00
Simon Willnauer
20f6d66294 Adopt to network settings cleanup in elastic/elasticsearch#25489
Original commit: elastic/x-pack-elasticsearch@364bb260ee
2017-07-02 10:17:30 +02:00
Costin Leau
bc9c9a9151 Expanding the JDBC testing framework
Original commit: elastic/x-pack-elasticsearch@cc31f7b6a9
2017-07-01 17:20:48 +03:00
Costin Leau
16ea84defe Fix some checkstyles
Original commit: elastic/x-pack-elasticsearch@ff1c6a6e70
2017-07-01 17:19:46 +03:00
Chris Earle
02c0ad2aad [Monitoring] Reduce NodeStats Collection to required Data (elastic/x-pack-elasticsearch#1240)
This changes from collecting every index statistic to only what we actually want. This should help to reduce the performance impact of the lookup.

Original commit: elastic/x-pack-elasticsearch@80ae20f382
2017-06-30 19:42:44 -04:00