Commit Graph

7306 Commits

Author SHA1 Message Date
Nik Everett e989f465bb SQL: Switch is aggs only to OO (elastic/x-pack-elasticsearch#3432)
Switches the "is this query aggs only?" question from pattern matching
on the column tree to an OO-style "ask the columns if they can be aggs
only" method.

I expect this could have been less code if I was willing to define
`supportedByAggsOnlyQuery` as `true` at the top of the
`ProcessorDefinition` object tree and override it only on nodes
`ReferenceInput` and `ScoreProcessorDefinition` but this feels dirty to
me. I tend to think of a superclass as a list of questions that all the
subclasses have to answer rather than a list of behaviors to share.
Pulling the `return true` up saves a few lines of code but breaks my
ability to reason about subclasses.


Original commit: elastic/x-pack-elasticsearch@b1338543cb
2017-12-27 14:16:22 -05:00
Albert Zaharovits 3ecc433f43 AD authn: never clear passwords on Bind connections (elastic/x-pack-elasticsearch#3351)
It is unsafe to clear passwords of bind requests if the connection is live
and might be used latter (for eg for group searches). This is a temporary
fix that exposes passwords in memory.

Original commit: elastic/x-pack-elasticsearch@e2e1f1a358
2017-12-27 19:17:07 +02:00
Michael Basnight 220aa734ee Migrate actions to plugin core (elastic/x-pack-elasticsearch#3424)
This commit hacks up the xpack repo in such a way that it moves all
actions that the transport client uses to plugin core. It also moves
any classes that those actions use to plugin core, with a few
exceptions. I tried to split up any classes that pulled in server side
logic into the client, but that was not always 100% possible. Consider
this commit a guide, and since I do not know the codebase for each
plugin, consider it a best guess for what should be moved. A few other
things were merged in below.

This commit extracts the parts of Condition and AlwaysCondition that are
needed by classes that will be moved into the client package. The only
odd thing here is that since there are two parent classes (Condition and
AlwaysCondition) it was not possible to make AlwaysConditionInteral
extend from AlwaysCondition and still parse a ConditionInternal object,
which is what the ConditionFactory expects.

This commit removes the use of internal users in the User class, and
instead moves them to Authentication, where they are used.

[insert obligatory chuck norris karate image here]

ref elastic/x-pack-elasticsearch#2925

Original commit: elastic/x-pack-elasticsearch@42d0b72209
2017-12-27 11:31:16 -05:00
Hendrik Muhs f8d62d0ade [ML] wait for validation of forecast request and send error in case (elastic/x-pack-elasticsearch#3399)
change the forecast API to wait for the validation and return an error if validation fails.

fixes elastic/machine-learning-cpp#489

Original commit: elastic/x-pack-elasticsearch@5c0553b284
2017-12-27 11:25:26 +01:00
Jason Tedor 31839b2748 Increase Gradle heap space to 1536m
This commit increases the heap space for the Gradle processes to
1536m. Note that the default is 1024m.

Original commit: elastic/x-pack-elasticsearch@3b5cdc30b2
2017-12-26 19:22:07 -05:00
Igor Motov 3892da7a3d SQL: Suppress testShowTablesWorksAsAdmin() tests
These tests are relying on existence of an internal index that might or might not be present when we run this test.

Tracked by elastic/x-pack-elasticsearch#3423

Original commit: elastic/x-pack-elasticsearch@26a7795759
2017-12-26 12:53:21 -05:00
Costin Leau 3e7eca6f2e SQL: Improve generics usage inside Parser (elastic/x-pack-elasticsearch#3417)
Original commit: elastic/x-pack-elasticsearch@814e476b6f
2017-12-23 22:47:51 +02:00
Costin Leau 5a25213db9 Update security tests
Original commit: elastic/x-pack-elasticsearch@732a4c5850
2017-12-23 15:44:27 +02:00
Costin Leau b437341ddc Remove old class
Original commit: elastic/x-pack-elasticsearch@e075c3b972
2017-12-23 13:35:42 +02:00
Costin Leau 368099dcbd SQL: Add support for object/inner/dotted fields (elastic/x-pack-elasticsearch#3368)
* SQL: Add support for object/inner/dotted fields

Improve validation when selecting compound fields
Improve fallback to exact field for text fields
Consolidate various field attributes into one
Consolidate various field HitExtractors into one
Improve grammar for column declaration
Update response of embedded server
Improve index validation to allow use of internal indices
Add clean-up of scroll in case of server errors
By default thrown an exception on multi-valued fields

Original commit: elastic/x-pack-elasticsearch@45b73fe0dc
2017-12-23 13:19:54 +02:00
Igor Motov e18c7e92fa SQL: Switch CLI to REST protocol (elastic/x-pack-elasticsearch#3342)
Removes custom binary protocol used for CLI communication and switches CLI to REST.

relates elastic/x-pack-elasticsearch#3063

Original commit: elastic/x-pack-elasticsearch@b42bdcdc35
2017-12-22 20:42:13 -10:00
Nik Everett 2b11f8b3ba SQL: Preserve field order in response again
We accidentally stopped returning fields in the same order that the
user asked for them in but some docs tests caught it.

Original commit: elastic/x-pack-elasticsearch@586e3cf207
2017-12-22 17:16:15 -05:00
Costin Leau 1003cf3829 Fix formatting
Original commit: elastic/x-pack-elasticsearch@65f17636db
2017-12-22 21:09:43 +02:00
Costin Leau 2856050af9 SQL: Add javadoc and package description (elastic/x-pack-elasticsearch#3406)
SQL: Add project description as package javadoc

Original commit: elastic/x-pack-elasticsearch@106150d1e6
2017-12-22 20:29:36 +02:00
Nik Everett 5b30bc6c7a Merge branch 'master' into feature/sql_2
Original commit: elastic/x-pack-elasticsearch@5b23edc3ef
2017-12-22 09:49:25 -05:00
Lee Hinman 97eef004de SQL: Remove instanceof checks for field retrieval (elastic/x-pack-elasticsearch#3402)
* SQL: Remove instanceof checks for field retrieval

This removes the `instanceof` checks iterating through the columns determining
what fields need to be retrieved from the ES document. It adds an interface
`FieldExtraction` that collects the fields needed in a builder. The builder can
then build the fields necessary from a `SearchSourceBuilder`.

* Remove default implementation in favor of pushing down exception throwing

Original commit: elastic/x-pack-elasticsearch@11d3d69eb1
2017-12-21 17:11:31 -07:00
lcawley 648a4d9cd1 [DOCS] Fixed forecasting links
Original commit: elastic/x-pack-elasticsearch@42c326c3ce
2017-12-21 08:39:49 -08:00
Lisa Cawley b35f1909cc [DOCS] Add forecasting overview (elastic/x-pack-elasticsearch#3263)
* [DOCS] Restructure ML overview

* [DOCS] Added forecasting limitations

* [DOCS] Merged changes to ML overview

* [DOCS] Added forecasting screenshot

* [DOCS] Removed incorrect results info from forecast API

* [DOCS] Addressed feedback about forecasts

* [DOCS] Clarified default forecast duration

Original commit: elastic/x-pack-elasticsearch@1403f2cd2e
2017-12-21 08:14:52 -08:00
Chris Earle 01e3db3740 [Monitoring] Add Cluster Alert Blacklist (elastic/x-pack-elasticsearch#3326)
This adds a traditional blacklist that can be set [dynamically] at the exporter level to block specific cluster alerts (watches) from being created. It goes further and actually removes any watch that has been created already for the current cluster.

Original commit: elastic/x-pack-elasticsearch@1f67bb9501
2017-12-21 11:04:56 -05:00
Chris Earle 76cfdfcef7 [Logstash] Index Management should use auto_expand_replicas (elastic/x-pack-elasticsearch#3373)
This modifies the default template to use auto_expand_replicas so that a one node cluster can stay green, even while using the Logstash management feature.

Original commit: elastic/x-pack-elasticsearch@b1ddf3af6f
2017-12-21 10:58:12 -05:00
David Kyle b98706183e Fix Java 9 & 10 test failures comparing ZonedDateTimes
Original commit: elastic/x-pack-elasticsearch@2767bb3f16
2017-12-21 15:48:05 +00:00
David Kyle 0950a903f2 [ML] Get events for job and from all calendars (elastic/x-pack-elasticsearch#3401)
* Add get events for job and from all calendars

Original commit: elastic/x-pack-elasticsearch@8029ffb898
2017-12-21 14:05:54 +00:00
Alexander Reelsen e15f1c36b5 Docs: Fix cross project link
Original commit: elastic/x-pack-elasticsearch@83ec5bd47c
2017-12-21 11:42:57 +01:00
David Kyle 3f6478e798 Fix timing issue in test
Original commit: elastic/x-pack-elasticsearch@7fe221ef15
2017-12-21 09:19:14 +00:00
Alexander Reelsen 8c99a4dc78 Watcher: Add refresh parameter to index action (elastic/x-pack-elasticsearch#3350)
This allows to configure the refresh parameter to the index action,
which is either applied to the resulting index or bulk request.

Relates elastic/apm-dev#76

Original commit: elastic/x-pack-elasticsearch@aedb6adc66
2017-12-21 10:18:16 +01:00
Hendrik Muhs bf74c77fef [ML] allow forecast only on post 6.1 jobs (elastic/x-pack-elasticsearch#3362)
allow forecast only on post 6.1 jobs

discuss issue: elastic/machine-learning-cpp#494

relates elastic/x-pack-elasticsearch#3219

Original commit: elastic/x-pack-elasticsearch@c6884bc40f
2017-12-21 07:47:32 +01:00
lcawley 1cc73a0307 [DOCS] Fixed calendar API titles
Original commit: elastic/x-pack-elasticsearch@77fcbe7b37
2017-12-20 16:37:19 -08:00
Nik Everett a288dde22f SQL: Build CLI against compact profile again (elastic/x-pack-elasticsearch#3392)
Months ago SQL used `Desktop#open` to view the results of graphviz
description of plans. We've since removed this feature in favor of just
writing the file. So now we can build the CLI against the same java
profile that we build everything agianst.

Original commit: elastic/x-pack-elasticsearch@5535f1a6dd
2017-12-20 18:49:01 -05:00
Nik Everett d9776ef7b4 SQL: Ignore directory made by text editor (elastic/x-pack-elasticsearch#3391)
vscode-antlr4 does a fairly good job of syntax highlighting and
ctrl-click to jump to definition and the like. So I use it. But it
leaves behind this temporary directory that I really don't want to
commit.

Original commit: elastic/x-pack-elasticsearch@c8b802b26d
2017-12-20 18:48:54 -05:00
lcawley 5d31fd5000 Merge remote-tracking branch 'upstream/master'
Original commit: elastic/x-pack-elasticsearch@55a4bf7f90
2017-12-20 15:22:57 -08:00
lcawley 121bbd5689 [DOCS] Fixed typo in calendar API
Original commit: elastic/x-pack-elasticsearch@8b3989cd45
2017-12-20 15:22:38 -08:00
Nik Everett 1cf9d6e3f3 SQL: Document a few functions (elastic/x-pack-elasticsearch#3390)
* Starts to build the list of supported functions, adding links to
wikipedia when there is any doubt what the functions mean.
* Extracts an example of using the function from the test suite.
* Explicitly calls out how we round (half up) because there are
lots of ways to round.

Original commit: elastic/x-pack-elasticsearch@5fb64ba869
2017-12-20 17:42:29 -05:00
Chris Earle 2507a07ec9 [Watcher] Use auto_expand_replicas on triggered_watches index too (elastic/x-pack-elasticsearch#3371)
In the previous pass, only the ".watch*" templates were impacted. This also should use `auto_expand_replicas`.

Original commit: elastic/x-pack-elasticsearch@5d3a4f1f7b
2017-12-20 17:25:04 -05:00
Chris Earle 362ba75e6b [ML] Use auto_expand_replicas (elastic/x-pack-elasticsearch#3375)
This creates all ML templates using the `index.auto_expand_replicas` set as `"0-1"` so that a single node cluster (e.g., dev) can startup as green.

Original commit: elastic/x-pack-elasticsearch@7a243c5b06
2017-12-20 17:23:36 -05:00
Lee Hinman c26f039207 SQL: Add all QUERY() query options (elastic/x-pack-elasticsearch#3389)
This adds support for (almost) all of the options that the `query_string` query
supports.

Additionally, it reverts back to the default operator of `OR` rather than `AND`
for the `QUERY()` query.

Relates to elastic/x-pack-elasticsearch#3361

Original commit: elastic/x-pack-elasticsearch@da8b29b53c
2017-12-20 15:05:40 -07:00
lcawley 9c853cfe6f Merge remote-tracking branch 'upstream/master'
Original commit: elastic/x-pack-elasticsearch@c3d5b1ec9f
2017-12-20 13:53:54 -08:00
Lisa Cawley 15e738b184 [DOCS] Added ML calendar APIs (elastic/x-pack-elasticsearch#3328)
* [DOCS] Added ML calendar APIs

* [DOCS] Updated calendar job API titles

* [DOCS] Added more calendar APIs

Original commit: elastic/x-pack-elasticsearch@0910da09eb
2017-12-20 13:52:58 -08:00
Lisa Cawley f14ca25ffc [DOCS] Update screenshots for population job (elastic/x-pack-elasticsearch#3334)
* [DOCS] Update screenshots for population job

* [DOCS] Updated screenshots for population wizard

Original commit: elastic/x-pack-elasticsearch@21b7dc1734
2017-12-20 13:13:59 -08:00
Lee Hinman 5b47c67dec SQL: Add all MATCH() query options (elastic/x-pack-elasticsearch#3387)
* SQL: Add all MATCH() query options

This adds support for (almost) all of the options that the `match` query
supports.

Additionally, it reverts back to the default operator of `OR` rather than `AND`
for the `MATCH()` query.

Relates to elastic/x-pack-elasticsearch#3361

* Add getters required for Node usage

Original commit: elastic/x-pack-elasticsearch@144e2bec02
2017-12-20 12:35:32 -07:00
Nik Everett 65d22e7b3a SQL: Update docs (elastic/x-pack-elasticsearch#3386)
* Remove "overview". That is already covered in the index page.
* Write a basic getting started page.
* Fix the CLI startup instructions.

Original commit: elastic/x-pack-elasticsearch@dabe72b5cc
2017-12-20 13:58:32 -05:00
David Kyle e53ac4484c [ML] Calendar event actions (elastic/x-pack-elasticsearch#3365)
* Calendar event actions

* Add page params and date range tests

* Address review comments

* Support POSTing params in the body of a request


Original commit: elastic/x-pack-elasticsearch@22a7e17a8f
2017-12-20 17:39:44 +00:00
Nik Everett 42d1c62d03 SQL: Nicer failure message one EXISTS (elastic/x-pack-elasticsearch#3384)
We don't support the SQL `EXISTS` keyword. It looks like:
```
SELECT * FROM test WHERE EXISTS (SELECT * FROM foo WHERE test.id =
foo.id)
```

It is basically a `JOIN` that doesn't return columns. Since we don't
support `JOIN`, we don't support `EXISTS`.

This PR improves the error message from "unresolved blah blah blah" to
"EXISTS is not yet supported".

relates elastic/x-pack-elasticsearch#3176

Original commit: elastic/x-pack-elasticsearch@548d57c8c1
2017-12-20 12:04:46 -05:00
Nik Everett d2df25072a Merge branch 'master' into feature/sql_2
Original commit: elastic/x-pack-elasticsearch@f7d99dcbb7
2017-12-20 12:03:20 -05:00
Lee Hinman 38cee807b9 SQL: Parse all multi_match options from request (elastic/x-pack-elasticsearch#3361)
* Parse all multi_match options from request

Rather than hardcoding predicates that require more methods for each newly added
options, this encapsulates the state into a collection of appliers that can
accumulate what's needed in the QueryBuilder itself.

For example, the following:

```json
GET /_xpack/sql/translate
{
  "query": "SELECT foo,baz from i WHERE MATCH('baz', 'should clause', 'operator=AND;type=cross_fields')"
}
```

Then generates the following:

```json
{
  "size" : 1000,
  "query" : {
    "multi_match" : {
      "query" : "should clause",
      "fields" : [
        "baz^1.0"
      ],
      "type" : "cross_fields",
      "operator" : "AND",
      "slop" : 0,
      "prefix_length" : 0,
      "max_expansions" : 50,
      "zero_terms_query" : "NONE",
      "auto_generate_synonyms_phrase_query" : true,
      "fuzzy_transpositions" : true,
      "boost" : 1.0
    }
  },
  "_source" : {
    "includes" : [
      "baz"
    ],
    "excludes" : [ ]
  },
  "docvalue_fields" : [
    "foo"
  ]
}
```

And when an invalid field value is used:

```json
GET /_xpack/sql
{
  "query": "SELECT foo,baz from i WHERE MATCH('baz', 'should clause', 'operator=AND;minimum_should_match=potato')"
}
```

We get what ES would usually send back:

```json
{
  "error" : {
    "root_cause" : [
      {
        "type" : "query_shard_exception",
        "reason" : "failed to create query: {\n  \"multi_match\" : {\n    \"query\" : \"should clause\",\n    \"fields\" : [\n      \"baz^1.0\"\n    ],\n    \"type\" : \"best_fields\",\n    \"operator\" : \"AND\",\n    \"slop\" : 0,\n    \"prefix_length\" : 0,\n    \"max_expansions\" : 50,\n    \"minimum_should_match\" : \"potato\",\n    \"zero_terms_query\" : \"NONE\",\n    \"auto_generate_synonyms_phrase_query\" : true,\n    \"fuzzy_transpositions\" : true,\n    \"boost\" : 1.0\n  }\n}",
        "index_uuid" : "ef3MWf8FTUe2Qjz2FLbhoQ",
        "index" : "i"
      }
    ],
    "type" : "search_phase_execution_exception",
    "reason" : "all shards failed",
    "phase" : "query",
    "grouped" : true,
    "failed_shards" : [
      {
        "shard" : 0,
        "index" : "i",
        "node" : "VfG9zfk9TDWdWvEZu0a4Rw",
        "reason" : {
          "type" : "query_shard_exception",
          "reason" : "failed to create query: {\n  \"multi_match\" : {\n    \"query\" : \"should clause\",\n    \"fields\" : [\n      \"baz^1.0\"\n    ],\n    \"type\" : \"best_fields\",\n    \"operator\" : \"AND\",\n    \"slop\" : 0,\n    \"prefix_length\" : 0,\n    \"max_expansions\" : 50,\n    \"minimum_should_match\" : \"potato\",\n    \"zero_terms_query\" : \"NONE\",\n    \"auto_generate_synonyms_phrase_query\" : true,\n    \"fuzzy_transpositions\" : true,\n    \"boost\" : 1.0\n  }\n}",
          "index_uuid" : "ef3MWf8FTUe2Qjz2FLbhoQ",
          "index" : "i",
          "caused_by" : {
            "type" : "number_format_exception",
            "reason" : "For input string: \"potato\""
          }
        }
      }
    ]
  },
  "status" : 400
}
```

It even includes the validation that ES already does for things like `type`:

```json
GET /_xpack/sql
{
  "query": "SELECT foo,baz from i WHERE MATCH('baz', 'should clause', 'operator=AND;type=eggplant')"
}
```

```json
{
  "error" : {
    "root_cause" : [
      {
        "type" : "parse_exception",
        "reason" : "failed to parse [multi_match] query type [eggplant]. unknown type."
      }
    ],
    "type" : "parse_exception",
    "reason" : "failed to parse [multi_match] query type [eggplant]. unknown type."
  },
  "status" : 400
}
```

Resolves elastic/x-pack-elasticsearch#3257


Original commit: elastic/x-pack-elasticsearch@59f518af4a
2017-12-20 09:48:05 -07:00
Nik Everett 9765058e5c SQL: Remove whenClause from grammar (elastic/x-pack-elasticsearch#3378)
It wasn't used. Also remove tokens that we aren't using.

Related to elastic/x-pack-elasticsearch#3176

Original commit: elastic/x-pack-elasticsearch@3358b1e241
2017-12-20 11:06:54 -05:00
Boaz Leskes 5e51422f4d PkiAuthenticationTests & SslIntegrationTests should instead use NetworkAddress's format magic
Original commit: elastic/x-pack-elasticsearch@601e8e774d
2017-12-20 16:05:41 +01:00
Boaz Leskes 133d70bc6a PkiAuthenticationTests & SslIntegrationTests should properly handle ipv6 addresses
Original commit: elastic/x-pack-elasticsearch@b11d90e584
2017-12-20 15:53:22 +01:00
Dimitrios Athanasiou f977632a17 [DOCS] Change `detector_rules` to `rules` in ML docs
Original commit: elastic/x-pack-elasticsearch@49699286d3
2017-12-20 14:38:00 +00:00
Nik Everett 2c5cfcfae4 SQL: Proper errors on set qualifiers (elastic/x-pack-elasticsearch#3377)
`SELECT DISTINCT foo FROM bar` is not yet implemented and was returning
an "unplanned item" error which isn't useful to users so I replaced it
with `SELECT DISTINCT is not yet supported`.

`SELECT foo, COUNT(*) FROM bar GROUP BY ALL foo` is not supported.
Specifically the `ALL` part. It is fairly esoteric so see [1] for what
it does. Regardless of what it does it is not widely supported and even
Microsoft's SQL Server has deprecated it so we should never support it.
Probably.

[1]: https://technet.microsoft.com/en-us/library/ms175028(v=sql.90).aspx

Related to elastic/x-pack-elasticsearch#3176

Original commit: elastic/x-pack-elasticsearch@56e5ca3009
2017-12-20 09:25:00 -05:00
Boaz Leskes b89b1d5cc5 PkiAuthenticationTests & SslIntegrationTests shouldn't hard code local host
The test uses the bound address to determine how to speak to the node via http.
It currently takes the port but hard codes the host to `localhost`. This can lead
to mismatches where a port for ipv6 is used but localhost resolves to ipv4

relates elastic/x-pack-elasticsearch#3382

Original commit: elastic/x-pack-elasticsearch@e97363a521
2017-12-20 14:50:51 +01:00