Commit Graph

6271 Commits

Author SHA1 Message Date
Lisa Cawley 2e58164b96 [DOCS] Add ML limitation (elastic/x-pack-elasticsearch#1916)
Original commit: elastic/x-pack-elasticsearch@7730810f98
2017-07-07 10:17:55 -07:00
Lisa Cawley 2aa53f253f [DOCS] Update doc links for graph API (elastic/x-pack-elasticsearch#1884)
Original commit: elastic/x-pack-elasticsearch@9eebb6c9a6
2017-07-07 10:11:06 -07:00
Tim Sullivan 66ebdff447 [Monitoring] Email actions for Cluster Alerts (elastic/x-pack-elasticsearch#1879)
* [Monitoring] Email actions for Cluster Alerts

* fix quotations in email fields

* move email vars to transform, and rename for snake_case

* add state to email subject for cluster status alert

* remove types field in kibana_settings search

* simplify email action condition script

* uppercase the state for the email subject

* only append state to email subject if alert is new

* show state in email subject even when alert is resolved

Original commit: elastic/x-pack-elasticsearch@e6fdd8d620
2017-07-07 09:55:45 -07:00
Nik Everett 184d53f598 Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@ed7a977328
2017-07-07 10:35:09 -04:00
Nik Everett f6fad00765 `cli`'s `gradle run` works!
Including proper tty support!

Original commit: elastic/x-pack-elasticsearch@a780a99d1b
2017-07-07 10:32:23 -04:00
Simon Willnauer c87d9278a6 Add validation for all `transport.profile.*` settings (elastic/x-pack-elasticsearch#1909)
Follow-up from elasticsearch/elastic#25508

Original commit: elastic/x-pack-elasticsearch@fe08e74ccc
2017-07-07 09:41:34 +02:00
Tim Vernum c5012ac6e8 [DOC] Miscellaneous security doc updates (elastic/x-pack-elasticsearch#1908)
- Document refresh interval for role mapping files
- Fix obsolete shield reference in transport profile example 
- Clarify that AD & PKI don't support run_as
- Fix logstash conf examples
- Clarify interaction of SSL settings and PKI realm settings
- Document PKI DN format, and recommend use of pki_dn metadata
- Provide more details about action.auto_create_index during setup

Original commit: elastic/x-pack-elasticsearch@49ddb12a7e
2017-07-07 13:33:35 +10:00
Nik Everett edcc87e30e `gradle check -xforbiddenPatterns` passes!
Hurray! Now we just have to keep it that way!

Original commit: elastic/x-pack-elasticsearch@98b8ede44e
2017-07-06 16:24:19 -04:00
Nik Everett fc0bce0a3e Some more tests
Closer to `gradle check -xforbiddenPatterns` passing.

Original commit: elastic/x-pack-elasticsearch@fbce74a1f3
2017-07-06 15:58:12 -04:00
Jason Tedor b636dcc366 Upgrade to Netty 4.1.13.Final
This commit updates the SHAs here as core upgraded the Netty dependency
from 4.1.11.Final to 4.1.13.Final.

Relates elastic/x-pack-elasticsearch#1936

Original commit: elastic/x-pack-elasticsearch@d1803fc331
2017-07-06 15:37:17 -04:00
Nik Everett 527541a66e More build fixes
* A unit test for cli
* Licenses for cli
* Remove licenses for protos (no more deps)
* `SHOW TABLES` returns results in order (makes testing easier)
* Clean up embedded jdbc server
* Wire up embedded cli server

Original commit: elastic/x-pack-elasticsearch@b98aaf446b
2017-07-06 14:37:09 -04:00
Lisa Cawley 0e322b525d [DOCS] Update doc links for security APIs (elastic/x-pack-elasticsearch#1883)
Original commit: elastic/x-pack-elasticsearch@660410008c
2017-07-06 11:06:03 -07:00
Tim Brooks d95c365e64 Loosen setup mode restrictions for upgrade tests (elastic/x-pack-elasticsearch#1927)
This commit is related to elastic/x-pack-elasticsearch#1896. Currently setup mode means that the
password must be set post 6.0 for using x-pack. This interferes with
upgrade tests as setting the password fails without a properly
upgraded security index.

This commit loosens two aspects of the security.

1. The old default password will be accept in setup mode (requests
from localhost).
2. All request types can be submitted in setup mode.

Original commit: elastic/x-pack-elasticsearch@8a2a577038
2017-07-06 10:37:48 -05:00
Nik Everett d25c23c3b4 Fix some forbiddens in cli
Original commit: elastic/x-pack-elasticsearch@3dc96c7279
2017-07-06 10:54:41 -04:00
Dimitris Athanasiou 4e03c657a3 [ML] Write model_memory_limit with units into the cluster state (elastic/x-pack-elasticsearch#1934)
This is step 2 of elastic/x-pack-elasticsearch#1604

This change stores `model_memory_limit` as a string with `mb` unit.
I considered using the `toString` method of `ByteSizeValue` but it
can lead to accuracy loss. Adding the fixed `mb` unit maintains
the accuracy, while making clear what unit the value is in.

Original commit: elastic/x-pack-elasticsearch@4dc48f0ce8
2017-07-06 15:40:19 +01:00
Nik Everett bc2f0fe7ff Get jdbc tests passing
`gradle check -xforbiddenPatterns` now passes in jdbc.

This makes running the embedded HTTP server slightly more difficult,
you now have to add the following to your jvm arguments.
```
-ea -Dtests.rest.cluster=localhost:9200 -Dtests.embed.sql=true -Dtests.security.manager=false
```

Depending on your environment the embedded jdbc connection may give
spurious failures that look like:
```
org.elasticsearch.xpack.sql.jdbc.jdbc.JdbcException: RemoteTransportException[[node-0][127.0.0.1:9300][indices:data/read/search]]; nested: SearchPhaseExecutionException[]; nested: GeneralScriptException[Failed to compile inline script [( params.a0 > params.v0 ) && ( params.a1 > params.v1 )] using lang [painless]]; nested: CircuitBreakingException[[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead; this limit can be changed by the [script.max_compilations_per_minute] setting];
...
Caused by: Failed to execute phase [fetch],
..
Caused by: GeneralScriptException[Failed to compile inline script [( params.a0 > params.v0 ) && ( params.a1 > params.v1 )] using lang [painless]]; nested: CircuitBreakingException[[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead; this limit can be changed by the [script.max_compilations_per_minute] setting];
...
Caused by: CircuitBreakingException[[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead; this limit can be changed by the [script.max_compilations_per_minute] setting]
```

`gradle check` works around this by setting `script.max_compilations_per_minute`
to `1000`.

Another change is that we no longer support loading the test data by
uncommenting some code. Instead we load the test data into Elaticsearch
before the first test and we deleted it after the last test. This is
so that tests that required different test data can interoperate with
eachother. The spec tests all use the same test data but the metadata
tests do not.

Original commit: elastic/x-pack-elasticsearch@8b8f684ac1
2017-07-06 10:24:59 -04:00
David Roberts 984d2ca2ba [ML] Ignore unknown fields when parsing ML cluster state (elastic/x-pack-elasticsearch#1924)
ML has two types of custom cluster state:

1. jobs
2. datafeeds

These need to be parsed from JSON in two situations:

1. Create/update of the job/datafeed
2. Restoring cluster state on startup

Previously we used exactly the same parser in both situations, but this
severely limits our ability to add new features.  This is because the
parser was very strict.  This was good when accepting create/update
requests from users, but when restoring cluster state from disk it meant
that we could not add new fields, as that would prevent reloading in
mixed version clusters.

This commit introduces a second parser that tolerates unknown fields for
each object that is stored in cluster state.  Then we use this more
tolerant parser when parsing cluster state, but still use the strict
parser when parsing REST requests.

relates elastic/x-pack-elasticsearch#1732

Original commit: elastic/x-pack-elasticsearch@754e51d1ec
2017-07-06 13:36:11 +01:00
Dimitris Athanasiou adc6fd5a0f [ML] Parse model_memory_limit both as number and string with units (elastic/x-pack-elasticsearch#1921)
This is the first step for elastic/x-pack-elasticsearch#1604.

Original commit: elastic/x-pack-elasticsearch@70010a216d
2017-07-06 12:29:12 +01:00
Lee Hinman b66560fa85 [TEST] Use better REST endpoints for testing watch actions
This is the x-pack side of https://github.com/elastic/elasticsearch/pull/24437

It changes two things, for the disable tests, it uses a valid endpoint instead
of a previously invalid endpoint that happened to return a 400 because the
endpoint was bad, regardless of if watcher was disabled.

The other change is to create the watches index by putting a watch using the
correct API, rather than manually creating the index. This is because
`RestHijackOperationAction` hijacks operations like this and stops accessing the
endpoint in a regular manner.

Original commit: elastic/x-pack-elasticsearch@3be78d9aea
2017-07-05 10:55:25 -06:00
Igor Motov c035adb568 Rename upgrade.* client methods to migration.* (elastic/x-pack-elasticsearch#1881)
This makes client names consistent with REST APIs and makes it simplifies client development.

Original commit: elastic/x-pack-elasticsearch@90913f485b
2017-07-05 09:47:58 -04:00
Dimitrios Athanasiou 2e0560528f [TEST] Fix MlBasicMultiNodeIT after changing flush response
Relates elastic/x-pack-elasticsearch#1914

Original commit: elastic/x-pack-elasticsearch@5175bf64d9
2017-07-05 13:30:25 +01:00
Nik Everett 18bc094dbc Remove LocalEsCluster
We aren't using it anymore.

Original commit: elastic/x-pack-elasticsearch@67f47c4f11
2017-07-05 08:21:35 -04:00
Nik Everett 9d83eccf27 Merge branch 'feature/sql' of github.com:elastic/x-pack-elasticsearch into feature/sql
Original commit: elastic/x-pack-elasticsearch@34d536c5e1
2017-07-05 08:15:11 -04:00
Nik Everett b2484bebe8 Merge branch 'master' into feature/sql
This should get us OK until 0170e0e8d3.

Original commit: elastic/x-pack-elasticsearch@e0bea08365
2017-07-05 08:14:24 -04:00
Dimitrios Athanasiou 3057f7f4b6 [TEST] Fix post data request in post_data.yml
Original commit: elastic/x-pack-elasticsearch@14e9083c02
2017-07-05 12:06:53 +01:00
Dimitris Athanasiou 15f9b1ed9c [ML] Impove mechanism for ignoring maintenance windows (elastic/x-pack-elasticsearch#1914)
Currently, the autodetect process has an `ignoreDowntime`
parameter which, when set to true, results to time being
skipped over to the end of the bucket of the first data
point received. After that, skipping time requires closing
and opening the job. With regard to datafeeds, this does not
work well with real-time requests which use the advance-time
API in order to ensure results are created for data gaps.

This commit improves this functionality by making it more
flexible and less ambiguous.

- flush API now supports skip_time parameter which
sends a control message to the autodetect process
telling it to skip time to a given value
- the flush API now also returns the last_finalized_bucket_end
time which allows clients to resume data searches correctly
- the datafeed start API issues a skip_time request when the
given start time is after the resume point. It then resumes
the search from the last_finalized_bucket_end time.

relates elastic/x-pack-elasticsearch#1913


Original commit: elastic/x-pack-elasticsearch@caa5fe8016
2017-07-05 11:33:42 +01:00
Lisa Cawley 595468a505 [DOCS] Update doc links for watcher APIs (elastic/x-pack-elasticsearch#1880)
Original commit: elastic/x-pack-elasticsearch@47521f1261
2017-07-04 11:16:47 -07:00
Lisa Cawley 04ff94a180 [DOCS] Update doc links for ML APIs (elastic/x-pack-elasticsearch#1882)
Original commit: elastic/x-pack-elasticsearch@cdc45f282d
2017-07-04 11:10:34 -07:00
Costin Leau 12e12f7a94 Improve exception reporting
Add the source spec in the error stacktrace

Original commit: elastic/x-pack-elasticsearch@bc7ed6ac60
2017-07-04 19:25:03 +03:00
Costin Leau 6f670c8d8e Testing fixes
- simplify handling of timezone in H2
- fix leaking threadpool in HttpServer
- update Csv tests
- keep the dates as long in internal Page

Original commit: elastic/x-pack-elasticsearch@43a804607f
2017-07-04 19:05:09 +03:00
Sophie Chang 0ff26e7566 [DOCS] Added limitation for time-based index patterns (elastic/x-pack-elasticsearch#1910)
* [DOCS] Added limitation for time-based index patterns

* [DOCS] Updated limitations

Original commit: elastic/x-pack-elasticsearch@9927105cc0
2017-07-04 08:40:42 -07:00
David Roberts b93890b0df [ML] Add license downgrade warning (elastic/x-pack-elasticsearch#1912)
When applying a license that will cause ML to be disabled, a warning
to this effect is now included in the acknowledgement messages.

relates elastic/x-pack-elasticsearch#1888

Original commit: elastic/x-pack-elasticsearch@e453f20f58
2017-07-04 15:10:18 +01:00
Dimitrios Athanasiou 05a73a32bd [TEST] Correctly verify that aliases were deleted in MlJobIT
This fixes `testDeleteJobAfterMissingAliases` to not fail randomly.
The reason the test was failing is that at some point some aliases
are deleted and the cat-aliases API is called to verify they were
indeed deleted. This was checked by asserting an
index_not_found_exception was thrown by the cat-aliases request.
This was some times working as there were no other aliases. However,
that depends on whether other x-pack features had time to create their
infrastructure. For example, security creates an alias. When other
aliases had the time to be created, the cat-aliases request does not
fail and the test fails.

This commit simply changes the verification that the read/write
aliases were deleted by replacing the cat-aliases request with
two single get-alias requests.

Original commit: elastic/x-pack-elasticsearch@fe2c7b0cb4
2017-07-04 13:14:38 +01:00
Alexander Reelsen fff72256a5 Watcher: Fix wrong logging in reporting attachment parser (elastic/x-pack-elasticsearch#1900)
The logging shows a wrong HTTP response status code from a previous
request. In addition the body now also gets logged, as debugging
is impossible otherwise.

Original commit: elastic/x-pack-elasticsearch@cc998cd587
2017-07-04 13:01:14 +02:00
Nik Everett 50a46d0ed2 Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@51fc29da6a
2017-07-03 18:04:09 -04:00
Costin Leau 83d73c9b3a More polishing
Original commit: elastic/x-pack-elasticsearch@043edf250b
2017-07-04 00:41:16 +03:00
Costin Leau 405b4a4166 Update test dataset name
Original commit: elastic/x-pack-elasticsearch@49951836c1
2017-07-04 00:00:11 +03:00
Nik Everett 56803bdd1b Port some of CLI's demos to tests
Original commit: elastic/x-pack-elasticsearch@5827c08268
2017-07-03 16:41:25 -04:00
Costin Leau 3eb1258b0d Add to CSV infra plus some refactoring of the JDBC one
Original commit: elastic/x-pack-elasticsearch@3e002c91c2
2017-07-03 21:01:29 +03:00
Clinton Gormley fd518ea020 Include shared/attributes.asciidoc directly from docs master
Original commit: elastic/x-pack-elasticsearch@f3a0828c5d
2017-07-03 18:23:19 +02:00
Christoph Büscher 8d26996afd Remove QueryParseContext (elastic/x-pack-elasticsearch#1895)
This is the x-pack side of elastic/elasticsearch#25486.

Original commit: elastic/x-pack-elasticsearch@c90a3e096b
2017-07-03 17:31:18 +02:00
Simon Willnauer 20f6d66294 Adopt to network settings cleanup in elastic/elasticsearch#25489
Original commit: elastic/x-pack-elasticsearch@364bb260ee
2017-07-02 10:17:30 +02:00
Costin Leau bc9c9a9151 Expanding the JDBC testing framework
Original commit: elastic/x-pack-elasticsearch@cc31f7b6a9
2017-07-01 17:20:48 +03:00
Costin Leau 16ea84defe Fix some checkstyles
Original commit: elastic/x-pack-elasticsearch@ff1c6a6e70
2017-07-01 17:19:46 +03:00
Chris Earle 02c0ad2aad [Monitoring] Reduce NodeStats Collection to required Data (elastic/x-pack-elasticsearch#1240)
This changes from collecting every index statistic to only what we actually want. This should help to reduce the performance impact of the lookup.

Original commit: elastic/x-pack-elasticsearch@80ae20f382
2017-06-30 19:42:44 -04:00
Nik Everett 4bb605d819 Fix licenses in server
Original commit: elastic/x-pack-elasticsearch@a3207814f8
2017-06-30 19:39:06 -04:00
Nik Everett 1a2cbea747 Checkstyle
Original commit: elastic/x-pack-elasticsearch@4d8e9b2a7a
2017-06-30 17:03:50 -04:00
Nik Everett 6de905c2e8 Enable more tests
And remove extra file

Original commit: elastic/x-pack-elasticsearch@aade36e030
2017-06-30 16:43:37 -04:00
Nik Everett 336980ec28 Add time zone support to the connection
Flows time zones through the `QueryInitRequest` and into the
`ExpressionBuilder` which attaches the time zones to date/time
expressions. Modifies the code that generates date aggs,
scripts, and extracts results to use the time zones.

Original commit: elastic/x-pack-elasticsearch@d6682580d1
2017-06-30 16:43:37 -04:00
Nik Everett 514b7290d7 Fix checkstyle
Original commit: elastic/x-pack-elasticsearch@f9b2d6ee47
2017-06-30 16:39:31 -04:00