Commit Graph

7247 Commits

Author SHA1 Message Date
Tanguy Leroux 3efd35cadf [Monitoring] Add missing mapping for interval_ms (elastic/x-pack-elasticsearch#3339)
# 2650 added the mapping for the interval_ms field in the Elasticsearch 
template but not for Kibana,Logstash and Beats templates.

Original commit: elastic/x-pack-elasticsearch@44fb501bb3
2017-12-19 09:14:10 +01:00
Nik Everett 4820bc757e SQL: Implement sorting and retrieving score (elastic/x-pack-elasticsearch#3340)
Accessing `_score` from SQL looks like:
```
--------------------------------------------------
POST /_sql
{
    "query": "SELECT SCORE(), * FROM library WHERE match(name, 'dune') ORDER BY SCORE() DESC"
}
--------------------------------------------------
```

This replaces elastic/x-pack-elasticsearch#3187

relates elastic/x-pack-elasticsearch#2887

Original commit: elastic/x-pack-elasticsearch@fe96348c22
2017-12-18 20:57:50 -05:00
Nik Everett dc69f92b49 SQL: Respond with nice error if there is a JOIN (elastic/x-pack-elasticsearch#3343)
`JOIN`s aren't supported right yet so we should send back a nice 400
level error to the user telling them that.

Also pulls out some common code in `RestSqlTestCase` that I've been
staring at for a while.

Relates to elastic/x-pack-elasticsearch#3176

Original commit: elastic/x-pack-elasticsearch@1c1bd1c355
2017-12-18 16:42:03 -05:00
Nik Everett 13428f4217 SQL: Do not compile against matrix aggs module
Since matrix aggs module isn't on the classpath at runtime, we shouldn't
compile with it on the classpath. Doing so makes it possible for us to
die with dignity when we can't load the class. Which happens right now.

Original commit: elastic/x-pack-elasticsearch@6d2ca5e367
2017-12-18 15:08:18 -05:00
Nik Everett 021e8dd111 SQL: Don't put aggs only queries into scroll context
SQL was adding scroll timeouts to aggregation only queries. At best this
is just confusing because we never scroll these queries. It *might*
leave behind a scroll context that we don't want. It is failing some new
validation that says that we have to have a `size` for every scroll
query.

Anyway, the simplest thing to do is to not put a scroll on aggregation
only queries.

Original commit: elastic/x-pack-elasticsearch@f6819a32b8
2017-12-18 13:57:58 -05:00
Nik Everett 4680e1e166 Merge branch 'master' into feature/sql_2
Original commit: elastic/x-pack-elasticsearch@2067b14cf8
2017-12-18 12:15:04 -05:00
Yannick Welsch 8ba08ccea9 [TEST] Create checkout directory on Windows before trimming path
The getShortPathName method can only be used on a directory that actually exists, otherwise it will fail with a cryptic message.

Original commit: elastic/x-pack-elasticsearch@44552dcfc8
2017-12-18 18:13:54 +01:00
Jason Tedor c92a216517 Revert "Fix elasticsearch-cli dependency"
This reverts commit elastic/x-pack-elasticsearch@68026168da.

Relates elastic/x-pack-elasticsearch#3349

Original commit: elastic/x-pack-elasticsearch@2c345ee5a4
2017-12-17 11:54:30 -05:00
Jason Tedor d97bfac8fc Fix elasticsearch-cli dependency
The API JAR POM picks up the wrong artifact name for the :core:cli
dependency, using the project name instead of the archive base
name. This commit fixes this issue by explicitly referring to the
artifact as a runtime dependency. With this change, the correct artifact
name is used in the API JAR POM.

Relates elastic/x-pack-elasticsearch#3336

Original commit: elastic/x-pack-elasticsearch@68026168da
2017-12-17 11:24:31 -05:00
Chris Earle 01f5318642 [Watcher] Use index.auto_expand_replicas: 0-1 (elastic/x-pack-elasticsearch#3284)
This changes the default behavior of .watch* indices to be green on one-node clusters, instead of constantly yellow.

Original commit: elastic/x-pack-elasticsearch@cdaee7cd72
2017-12-15 18:32:34 -05:00
Lisa Cawley 02129beec4 [DOCS] Reformatted machine learning overview (elastic/x-pack-elasticsearch#3346)
* [DOCS] Reformatted machine learning overview

* [DOCS] Added intro ML screenshot

Original commit: elastic/x-pack-elasticsearch@b6189000e0
2017-12-15 15:05:21 -08:00
Lee Hinman a8e9272994 SQL: Add remaining matrix aggregations (elastic/x-pack-elasticsearch#3330)
* Add remaining matrix aggregations

This adds the remaining [matrix aggregations](https://www.elastic.co/guide/en/elasticsearch/reference/6.1/search-aggregations-matrix-stats-aggregation.html).

These aggregations aren't currently implemented due to the inter-plugin
communication not being set up, so they return "innerkey/matrix stats not
handled (yet)".

For matrix aggs that share a name with an existing aggregation (like 'count'),
they have be prefixed with "matrix_", so, "matrix_count", "matrix_mean", and
"matrix_variance".

Relates to elastic/x-pack-elasticsearch#2885

* Return HTTP 400 for innerkey/matrix stats aggs

* Add integration test for unimplemented matrix aggs

Original commit: elastic/x-pack-elasticsearch@34459c59aa
2017-12-15 14:33:58 -07:00
Lisa Cawley cd245c8e86 [DOCS] Added xpack.ml.node_concurrent_job_allocations setting (elastic/x-pack-elasticsearch#3327)
* [DOCS] Added concurrent ML job setting

* [DOCS] Re-ordered ML settings

* [DOCS] Clarified concurrent job allocation setting

Original commit: elastic/x-pack-elasticsearch@cb2d501333
2017-12-15 11:19:11 -08:00
lcawley ab737982bb Merge remote-tracking branch 'upstream/master'
Original commit: elastic/x-pack-elasticsearch@5b94f9472a
2017-12-15 11:07:02 -08:00
lcawley d5e03f9bff [DOCS] Fixed troubleshooting titles
Original commit: elastic/x-pack-elasticsearch@4338580de6
2017-12-15 11:05:20 -08:00
Alexander Reelsen 5f8a0711f5 Watcher: Set index and type dynamically in index action (elastic/x-pack-elasticsearch#3264)
The index action allowed to set the id of a document dynamically,
however this was not allowed for the index or the type.

If a user wants to execute a search, modify the found documents and
index them back, then this would only work across a single index and a
single type. This change allows the watch writer to just take a search
result, read index and type out of that and configure this as part of
the index action.

On top of that the integration tests have been changed to become fast
running unit tests.

Original commit: elastic/x-pack-elasticsearch@640b085dd4
2017-12-15 16:59:29 +01:00
Lee Hinman 632c3e8238 SQL: Fix running `gradle run` from top-level directory (elastic/x-pack-elasticsearch#3329)
When running `gradle run` at the top level (at least with Gradle 4.4) it
attempts to run multiple instances of the server, causing the run to fail since
they can't both bind to 9200/9300.

This renames the tasks for the `qa` directories to be `runqa` and the task for
the `cli` directory to be `runcli`.

Original commit: elastic/x-pack-elasticsearch@734ab8e132
2017-12-15 08:42:18 -07:00
Alexander Reelsen f518501df4 Tests: Ensure that watcher is started in HipchatServiceTests
One of those tests requires watcher to be started, so a proper
assertBusy() block has been added to this tests.

relates elastic/x-pack-elasticsearch#3324

Original commit: elastic/x-pack-elasticsearch@324830316f
2017-12-15 16:05:43 +01:00
David Roberts b81c90d6fc [DOCS] Explain ML datafeed run-as integration/limitations (elastic/x-pack-elasticsearch#3311)
Docs for elastic/x-pack-elasticsearch#3254

Original commit: elastic/x-pack-elasticsearch@eec3c7ccce
2017-12-15 14:41:10 +00:00
Alexander Reelsen 758433a0fa Monitoring: Ensure all monitoring watches filter by timestamp (elastic/x-pack-elasticsearch#3238)
Only the Logstash and Kibana version mismatch watches contain a time
filter, the others are only sorting by timestamp. In combination with
searching in all `.monitoring-es-*` indices, this is IMO pretty resource
intensive, as we cannot exit early on any search request.

This commit adds time based filters to remaining three watches, using
the same range than the other two.

Original commit: elastic/x-pack-elasticsearch@3eb6bf0de2
2017-12-15 15:23:57 +01:00
Shaunak Kashyap 78f7c0e27a Fix license messaging for Logstash functionality (elastic/x-pack-elasticsearch#3268)
* Fix license messaging for Logstash functionality

With a Basic license, users are still able to perform CRUD operations on the `.logstash` index, therefore manage their Logstash pipelines. However, Logstash itself will not pick up any changes from this index and act on them. With an expired license Logstash functionality continues to operate as normal.

* Fixing messages after feedback

* Removing extraneous tabs at end of line

* Fixing typo

Original commit: elastic/x-pack-elasticsearch@bc069cf00f
2017-12-15 05:56:01 -08:00
Igor Motov 49a036cc5f SQL: Bring SQL Version in line with ES Version (elastic/x-pack-elasticsearch#3308)
It also makes it possible to use the Version class to parse the version that we get from Elasticsearch.

Original commit: elastic/x-pack-elasticsearch@73a3268b12
2017-12-14 22:01:19 -05:00
lcawley 4f1866db69 [DOCS] Updated titles of ML APIs
Original commit: elastic/x-pack-elasticsearch@3b3d856a89
2017-12-14 10:52:49 -08:00
Costin Leau f2792b8d93 Update cli to work on windows
Original commit: elastic/x-pack-elasticsearch@84f6ba3c1f
2017-12-14 18:53:21 +02:00
Nik Everett 28f1107dad SQL: Fix test for UnresolvedRelation
`UnresolvedRelation`'s equality test would sometimes not properly mutate
the object under test. This is because `ESTestCase#randomValueOtherThan`
will only run its provider one time if passed `null` for the "other
than" value.

Original commit: elastic/x-pack-elasticsearch@7b13e8dc98
2017-12-14 10:45:59 -05:00
Nik Everett 2268e592c6 Merge branch 'master' into feature/sql_2
Original commit: elastic/x-pack-elasticsearch@ded26a3834
2017-12-14 10:31:51 -05:00
Costin Leau 624a1530c0 SQL: Fix checkstyle / add missing hashCode (elastic/x-pack-elasticsearch#3323)
* Fix checkstyle / add missing hashCode
* Improve formatting

Original commit: elastic/x-pack-elasticsearch@a3ad2793bb
2017-12-14 17:17:26 +02:00
Nik Everett 59e6d34c29 SQL: Bundle the CLI into x-pack (elastic/x-pack-elasticsearch#3316)
This adds:

* The CLI jar itself into the `bin`. It is an executable jar.
* A shell and bat script to start the CLI. This isn't strictly required but folks will appreciate the consistency.
* Basic packaging tests for the CLI.

Relates to elastic/x-pack-elasticsearch#2979

Original commit: elastic/x-pack-elasticsearch@158f70a530
2017-12-14 09:57:03 -05:00
Nik Everett f5af60c7cf SQL: Fix error message on bad index (elastic/x-pack-elasticsearch#3312)
Fixes the error message that SQL produces when it sees unsupported
indexes. It was always returning all broken indexes as "unknown" even
though we have much better error messages. It was just throwing them
away.

I caught this originally when backporting to 6.x where we had a test
that we produced a useful error message when the user attempted to run
SQL on an index with more than one type. We couldn't run that in the
feature/sql branch because it is inside the full cluster restart tests
and the "old" version of Elasticsearch used in those tests in
feature/sql is 6.x which doesn't allow indexes with multiple types. When
I backported to 6.x the test failed because it hadn't been run before.

In addition to fixing that test and the problem, this adds another test
that will reveal the problem when run in the feature/sql and master
branch.

Original commit: elastic/x-pack-elasticsearch@c7b787baee
2017-12-14 09:56:36 -05:00
Colin Goodheart-Smithe 51b80aa998 Fixes test to support BytesSizeValue changes (elastic/x-pack-elasticsearch#3321)
Original commit: elastic/x-pack-elasticsearch@0bc8b0e847
2017-12-14 12:17:25 +00:00
Ioannis Kakavas d24921ea60 Check for existing x-pack directory when running the `users` CLI tool (elastic/x-pack-elasticsearch#3271)
Verify that the configuration directory `$ES_PATH_CONF/x-pack`
exists before attempting to run any of the `users` commands, and 
return a helpful error message to the user if it doesn't.

Original commit: elastic/x-pack-elasticsearch@6d663b6654
2017-12-14 13:45:53 +02:00
Chris Earle 876fc5612a [Monitoring] Use index.auto_expand_replicas: 0-1 (elastic/x-pack-elasticsearch#3282)
This changes the default behavior of .monitoring indices to be green on one-node clusters, instead of constantly yellow.

Note: This only impacts .monitoring* indices. Watcher indices currently still require a replica.

Original commit: elastic/x-pack-elasticsearch@6eb8a48a9f
2017-12-13 13:46:31 -05:00
Nik Everett e170021037 Revert "Revert "Fix version parsing in SQL""
This reverts commit elastic/x-pack-elasticsearch@a3ef6b43f9.

We'll merge this when we're good and ready.

Original commit: elastic/x-pack-elasticsearch@06d1e3311d
2017-12-13 10:19:54 -05:00
Costin Leau cdfe0f1f5a Revert "Revert "Merge branch 'feature/sql'""
This reverts commit elastic/x-pack-elasticsearch@cc79e19911.

We'll merge this when we're good and ready.

Original commit: elastic/x-pack-elasticsearch@b3ef4f2836
2017-12-13 10:19:31 -05:00
Costin Leau 2e60e831c0 Revert "Merge branch 'feature/sql'"
This reverts commit elastic/x-pack-elasticsearch@2b3f7697a5, reversing
changes made to elastic/x-pack-elasticsearch@b79f16673c.

We're backing out all of SQL.

Original commit: elastic/x-pack-elasticsearch@cc79e19911
2017-12-13 09:33:13 -05:00
Nik Everett 5a6cfafed2 Revert "Remove SQL ci script changes"
This reverts commit elastic/x-pack-elasticsearch@afb4347465.

We're backing out all of SQL.

Original commit: elastic/x-pack-elasticsearch@566729702f
2017-12-13 09:32:58 -05:00
Nik Everett d33c537711 Revert "Fix version parsing in SQL"
This reverts commit elastic/x-pack-elasticsearch@47020ed3aa.

We're backing out all of SQL.

Original commit: elastic/x-pack-elasticsearch@a3ef6b43f9
2017-12-13 09:32:44 -05:00
Jason Tedor 6ceee0be8f Fix version parsing in SQL
I have no idea why this logic was only looking at snapshot JARs (what is
suppose to happen in production?). This breaks the release tests though
which we really need to have stable. This commit fixes this by also
considering non-snapshot JARs.

Original commit: elastic/x-pack-elasticsearch@47020ed3aa
2017-12-12 22:15:59 -05:00
Lee Hinman ffeebc366d Add sql folder for upcoming migration (elastic/x-pack-elasticsearch#3307)
* Add sql folder for upcoming migration

This is the skeleton folder for SQL now that it's been merged.

Relates to elastic/x-pack-elasticsearch#3233

Original commit: elastic/x-pack-elasticsearch@c46548bd6c
2017-12-12 16:48:43 -07:00
Lee Hinman 86a04acb01 Rename folder x-pack-core -> core (elastic/x-pack-elasticsearch#3305)
* Rename folder x-pack-core -> core

The jar remains 'x-pack-core-*.jar'

* Put group in top-level build.gradle instead of plugin/core/build.gradle

Original commit: elastic/x-pack-elasticsearch@b23452fa55
2017-12-12 13:23:29 -07:00
Nik Everett c78328853f Remove SQL ci script changes
These were useful when SQL lived on its own branch but aren't now that
we've merged SQL to master.

Original commit: elastic/x-pack-elasticsearch@afb4347465
2017-12-12 14:49:46 -05:00
Nik Everett aef5ea6527 Merge branch 'feature/sql'
It is time.

Original commit: elastic/x-pack-elasticsearch@2b3f7697a5
2017-12-12 14:49:27 -05:00
Jay Modi d48ad6dbd6 Enhance security for alias and resize operations (elastic/x-pack-elasticsearch#3302)
This commit adds additional checks around resize operations and alias creation operations to
add an extra layer of security around these APIs.

Original commit: elastic/x-pack-elasticsearch@b79f16673c
2017-12-12 12:01:27 -07:00
Nik Everett eaa75cfdbb Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@b5fc2b484d
2017-12-12 11:29:23 -05:00
Nik Everett 2ca729afc2 Fix packaging tests after breaking up x-pack
The tests were failing after breaking up x-pack. This makes them pass
locally.

Original commit: elastic/x-pack-elasticsearch@af2a7866eb
2017-12-12 11:05:51 -05:00
Dimitris Athanasiou 92c40061c5 [ML] Skip index health checks for remote indices on datafeed node sel (elastic/x-pack-elasticsearch#3301)
Upon selecting a node to run a datafeed we normally check that the
data indices exists and their primaries are active. However, these
checks cannot be applied for CCS to a remote cluster. This commit
skips these checks for remote indices.

This removes the last obstacle for running CCS datafeeds.

Relates elastic/x-pack-elasticsearch#1071

Original commit: elastic/x-pack-elasticsearch@092f44feee
2017-12-12 15:12:33 +00:00
Tanguy Leroux 711254fd24 [Monitoring] Use the same Cluster State for all Collectors (elastic/x-pack-elasticsearch#3216)
This commit changes the Collectors so that they all use the 
same instance of ClusterState.

relates elastic/x-pack-elasticsearch#3156

Original commit: elastic/x-pack-elasticsearch@4f537b026c
2017-12-12 12:29:40 +01:00
David Kyle 0d46e9035c [ML] Get Filters should use executeAsyncWithOrigin (elastic/x-pack-elasticsearch#3295)
* Get Filters should use executeAsyncWithOrigin

Original commit: elastic/x-pack-elasticsearch@786c7bfd06
2017-12-12 11:15:54 +00:00
David Kyle 6113b86bdb [ML] Special events calendar CRUD endpoints (elastic/x-pack-elasticsearch#3267)
* Calendar CRUD endpoints

* Get calendars requires monitor permission

* Address review comments

* Add page params to get calendars

Original commit: elastic/x-pack-elasticsearch@badd1e6add
2017-12-12 09:21:44 +00:00
Nik Everett 6d56616d98 SQL: Ignore environment in CliFixture
If the currenct environment contains `JAVA_TOOLS` then the cli tests
will fail because they do not expect the notification about `JAVA_TOOLS`
that java prints to stdout. The clears the environment before starting
the CLI subprocess in an effort to remove the unexpected output.

Original commit: elastic/x-pack-elasticsearch@5729c1c09b
2017-12-11 16:42:52 -05:00