Commit Graph

424 Commits

Author SHA1 Message Date
Costin Leau de5f924daa SQL: Adjust the precision and scale for drivers (#40467)
Fix #40357

(cherry picked from commit 1557d77b07bbeef99604857c1ca824e20771c283)
2019-03-28 10:08:09 +02:00
Marios Trivyzas 8e049c5f58
SQL: Fix getTime() methods in JDBC (#40484)
Previously, `getTime(colIdx/colLabel)` and
`getObject(colIdx/colLabel, java.sql.Time.class)` methods were computing
the time from a `ZonedDateTime` by applying day in millis modulo on the epoch millis
of the `ZonedDateTime` object. This is wrong as we need to keep the time
related fields at the timezone of the `ZonedDateTime` object and just
set the date info to the epoch date (01/01/1970).

Additionally fixes a testing issue as the original timezone id is converted
to an offset string when parsing the response from the server.
2019-03-27 17:14:08 +01:00
Andrei Stefan 99dca30197 SQL: MATCH and QUERY documentation; one list of functions (#40494)
* Document MATCH and QUERY function predicates.
* Polish the functions pages and add a list of functions to the main Functions & Operators page.

(cherry picked from commit 4cec0ae1b962ec7ea011a290aec72740386eb808)
2019-03-27 17:37:10 +02:00
Costin Leau 33737b6b21 SQL: Polish parsing of CAST expression (#40428)
(cherry picked from commit 9d291aa300bbb827eeae606e7d3e55eeef7cce00)
2019-03-27 12:20:58 +02:00
Costin Leau 7234d78747 SQL: Fix classpath discovery on Java 10+ (#40420)
(cherry picked from commit 2cef233cb34ee80d8ed9cd014cea76ea5096d206)
2019-03-26 08:16:37 +02:00
Costin Leau 61f49af497 SQL: Spec tests now use classpath discovery (#40388)
To avoid having to specify each spec by hand (which can miss specs to be
added), the test infrastructure now performs classpath discovery so that
each spec added, is automatically considered.

Relates #40358

(cherry picked from commit d0f60b4425c731509aa8ca765d55f563f866ef90)
2019-03-25 15:22:52 +02:00
Marios Trivyzas 143db10980
SQL: Fix issue timezone issues with JDBC getDate/getTime (#40360)
Previously, `getDate(int columnIdx)/getDate(String columnLabel)` and
were using legacy`java.util.Calendar` instead of the the `java.time.*`
classes to reset to the start of day. This resulted in different results 
for certain timestamps and timezones when calling
`getDate(col)` vs`getObject(col, java.sql.Date)`

Now only the methods (that must be implemented due to the JDBC spec)
`getDate(int columnIdx, Calendar cal)/getDate(String columnLabel, Calendar cal)`
are still using the `java.util.Calendar` for those conversion.

The same change was applied to
`getTime(int columnIdx)/getTime(String columnLabel)`
and
`getTimestamp(int columnIdx)/getTimestamp(String columnLabel)`

Fixes: #40289

(cherry picked from commit 44560671f18397e0c58e3647732880fcb73a5034)
2019-03-23 17:01:08 +01:00
Marios Trivyzas 17b8b54d5e
SQL: Fix metric aggs on date/time to not return double (#40377)
Previously metric aggregations on date fields would return a double
which caused errors when trying to apply scalar functions on top, e.g.:
```
SELECT YEAR(MAX(date)) FROM test
```

Fixes: #40376

(cherry-picked from commit 41d0a038467fbdbbf67fd9bfdf27623451cae63a)
2019-03-23 14:13:38 +01:00
Costin Leau 558adc0f28 SQL: Add missing handling of IP field in JDBC (#40384)
Fix #40358

(cherry picked from commit ee286fa4893817637c05d72b93b254b36efc0dae)
(cherry picked from commit d2296249499e31bd512390ac3d20bc38009612b3)
2019-03-23 12:58:10 +02:00
Andrei Stefan 150d1332cf SQL: Fix RLIKE bug and improve testing for RLIKE statement (#40354)
* Refactor RegexMatch to support both LIKE and RLIKE
* Add integration tests for RLIKE
* Polish the rest of tests

(cherry picked from commit 7562d6eeeb77c04794002649fe726f4b3a9a398b)
2019-03-23 06:37:53 +02:00
Costin Leau 87d3d16c5a SQL: JLine upgrade and polishing (#40321)
Upgrade JLine to 3.10.0
Switch to using JLine granular jars instead of the uber-one
Remove Jansi dependency (due to errors in closing streams)
Pin JNA dependency to our own artifact

Fix #40239

(cherry picked from commit 9afa65fa80111f3b68c13373c7b6db13c11dde31)
2019-03-22 23:55:51 +02:00
Costin Leau 496070fda6 SQL: CAST supports both SQL and ES types (#40365)
Extend CAST to support all data types notations (whether SQL or ES
specific)

Fix #40282

(cherry picked from commit eb2ee8a344da946920598839a5db76c8bb9bc3fe)
2019-03-22 23:55:51 +02:00
Costin Leau 980ee14f57 DOC: Expand section on ORDER BY aggs (#40332)
(cherry picked from commit 99d2f6fc9864ab972259ef5692129ab49e4a7ab8)
2019-03-22 10:04:52 +02:00
Andrei Stefan f9ab9afcc1 Extract the first value in an array when looking at the returned values (#40318)
(cherry picked from commit faf02e0f42a101985619abc0d30753851605e01d)
2019-03-22 06:43:37 +02:00
Andrei Stefan 35fe05308e SQL: rewrite ROUND and TRUNCATE functions with a different optional parameter handling method (#40242)
* Rewrite Round and Truncate functions to have a slightly different
approach to handling the optional parameter in the constructor. Until now
the optional parameter was considered 0 if the value was missing and the
constructor was filling in this value. The current solution is to have
the optional parameter as null right until the actual calculation is done.

(cherry picked from commit 3e314f8fa4cb322e67949e80857561ce51268726)
2019-03-22 06:43:37 +02:00
Costin Leau dd41ce0763 SQL: Preserve original source for cast/convert function (#40271)
Improve rule for pruning cast to preserve the original source
Fix #40239

(cherry picked from commit 7591cb1a1577320b3aec2ec557b0f881b6af744f)
2019-03-21 14:08:15 +02:00
Andrei Stefan 1a5ff05870 SQL: fix LIKE function equality by considering its pattern as well (#40260)
* Define a equals method for Like function so that the pattern used
is considered in the equality check. Whenever the functions are resolved
this check should be used.

(cherry picked from commit 4e5d5af58a140573b8ee19d57c7839db7b779e3b)
2019-03-21 11:44:57 +02:00
Andrei Stefan d485be631b Moving tests in locale-aware test file (#40254)
(cherry picked from commit 9beb31fd3c5a8323cb08cc524f1a2268e9c72c24)
2019-03-21 10:57:37 +02:00
Marios Trivyzas e1eb683c51 SQL: Fix issue with getting DATE type in JDBC (#40207)
Previously, calling getDate()/getTime()/getTimestamp() and getObject()
with the corresponding java.sql class on a column of SQL DATE type from
the JDBC result set would throw an Exception.
2019-03-21 01:48:06 +01:00
Marios Trivyzas f37f2b5d39
SQL: Fix issue with optimization on queries with ORDER BY/LIMIT (#40256)
Previously, when a trival plain `SELECT` or a trivial `SELECT` with
aggregations has also an `ORDER BY` or a `LIMIT` or both, then the
optimization to convert it to a `LocalRelation` was skipped resulting
in exception thrown. E.g.::
```
SELECT 'foo' FROM test LIMIT 10
```
or
```
SELECT 'foo' FROM test GROUP BY 1 ORDER BY 1
```

Fixes: #40211
2019-03-20 23:52:35 +01:00
Marios Trivyzas bc4c8e53c5 SQL: Fix issue with date columns returned always in UTC (#40163)
When selecting columns of ES type `date` (SQL's DATETIME) the
`FieldHitExtractor` was not using the timezone of the client session
but always resorted to UTC. The same behaviour (UTC only) was
encountered also for grouping keys (`CompositeKeyExtractor`) and
for First/Last functions on dates (`TopHitsAggExtractor`).

Fixes: #40152
2019-03-20 20:32:33 +01:00
Andrei Stefan 791814bb47 SQL: fix incorrect ordering of groupings (GROUP BY) based on orderings (ORDER BY) (#40087)
* Take into consideration aliases that can be used as aggregates
and in the ORDER BY element so that the groupings are re-ordered inside
the composite aggregation according to the ORDER BY ordering.

(cherry picked from commit 110c0b90b9cf2e9344ab3f412cfa8f8cd94ad71f)
2019-03-18 15:37:45 +02:00
Costin Leau 076a68007c SQL: Add multi_value_field_leniency inside FieldHitExtractor (#40113)
For cases where fields can have multi values, allow the behavior to be
customized through a dedicated configuration field.
By default this will be enabled on the drivers so that existing datasets
work instead of throwing an exception.
For regular SQL usage, the behavior is false so that the user is aware
of the underlying data.

Fix #39700

(cherry picked from commit 2b351571961f172fd59290ee079126bbd081ceaf)
2019-03-18 14:56:03 +02:00
Igor Motov a019af7690 SQL: Refactor Literals serialization method (#40058)
Since other classes besides intervals can be serialized as part of
the Cursor, the getNamedWritables method should be moved from Intervals
to a more generic class Literals.

Relates to #39973
2019-03-15 14:30:28 -04:00
Costin Leau 3960374a6f SQL: Introduce MAD (MedianAbsoluteDeviation) aggregation (#40048)
Add Median Absolute Deviation aggregation

Fix #39597

(cherry picked from commit 4f09613942a9249d06c74da64ad7e6f362e97f56)
2019-03-15 11:45:15 +02:00
Marios Trivyzas 4e9657f93f SQL: Fix bug with JDBC timezone setting and DATE type (#39978)
Previously, JDBC's REST call to the server was always sending UTC
instead of the timezone passed through connection string/properties.

Moreover the conversion to java.sql.Date was problematic as a
calculation on the epoch millis was used to set the time to 00:00:00.000
and the timezone info was lost. This caused the resulting java.sql.Date
object which is always using the JVM's timezone (no matter what timezone
setting is used in the connection string/properties) to be wrongly created.

Fixes: #39915
2019-03-14 13:41:53 +01:00
Andrei Stefan 4d1305b6df SQL: Extend the multi dot field notation extraction to lists of values (#39823)
(cherry picked from commit 300ae485dd08373727ca111a4d21276dd47d9a27)
2019-03-14 11:21:53 +02:00
Jim Ferenczi 7a7658707a
Upgrade to Lucene release 8.0.0 (#39998)
This commit upgrades to the GA release of Lucene 8

Closes #39640
2019-03-13 18:11:50 +01:00
Igor Motov 2f47e3d05a SQL: values in datetime script aggs should be treated as long (#39773)
When a query is translated into script terms agg where key has a date
type, it should generate a terms agg with value_type long instead of
date, otherwise the key gets formatted as a string, which confuses
hit extractor.

Fixes #37042
2019-03-11 17:41:12 -04:00
Costin Leau 92a87a45bf SQL: Wrap ZonedDateTime parameters inside scripts (#39911)
Painless allows ZonedDateTime objects to be passed natively to scripts
which creates problematic translate queries as the ZonedDateTime is
passed as a string instead.
Wrap this with a dedicated method to perform the conversion.

Fix #39877

(cherry picked from commit 4957cad5bda77257d10430ac102e93f5e062148a)
2019-03-11 17:44:03 +02:00
Costin Leau a079b9fd6d SQL: ConstantProcessor can now handle NamedWriteable (#39876)
Enhance ConstantProcessor to properly serialize complex objects
(Intervals) that have their own custom serialization/deserialization
mechanism

Fix #39875

(cherry picked from commit ed8a1f9340673e69a44ea7a89679cadb4762e43d)
2019-03-11 12:49:23 +02:00
Ryan Ernst 465343f12a
Bundle java in distributions (#38013)
* Bundle java in distributions

Setting up a jdk is currently a required external step when installing
elasticsearch. This is particularly problematic for the rpm/deb packages
as installing a jdk in the same package installation command does not
guarantee any order, so must be done in separate steps. Additionally,
JAVA_HOME must be set and often causes problems in selecting a correct
jdk when, for example, the system java is an older unsupported version.

This commit bundles platform specific openjdks into each distribution.
In addition to eliminating the issues above, it also presents future
possible improvements like using jlink to build jdk images only
containing modules that elasticsearch uses.

closes #31845
2019-03-08 11:04:18 -08:00
Marios Trivyzas c72a7998f5
SQL: Don't allow inexact fields for MIN/MAX (#39563)
MIN/MAX on strings are supported and are implemented with
TopAggs FIRST/LAST respectively, but they cannot operate on
`text` fields without underlying `keyword` fields => inexact.

Follows: #39427
2019-03-04 15:35:11 +01:00
Costin Leau e038ccef13 SQL: Fix merging of incompatible multi-fields (#39560)
Fix bug in IndexResolver that caused conflicts in multi-field types to
be ignored up (causing the query to fail later on due to mapping
conflicts).
The issue was caused by the multi-field which forced the parent creation
before checking its validity across mappings

Fix #39547

(cherry picked from commit 4e4fe289f90b9b5eae09072d54903701a3128696)
2019-03-02 10:30:02 +02:00
Costin Leau dfe81b260e SQL: Enable accurate hit tracking on demand (#39527)
Queries that require counting of all hits (COUNT(*) on implicit
group by), now enable accurate hit tracking.

Fix #37971

(cherry picked from commit 265b637cf6df08986a890b8b5daf012c2b0c1699)
2019-03-01 23:09:04 +02:00
Andrei Stefan 06d0e0efad Removed custom naming for DISTINCT COUNT (#39537)
(cherry picked from commit 9412a2ee01a60dd6449bbced1273ec0b37b65589)
2019-03-01 15:26:32 +02:00
Andrei Stefan ba44f28340 SQL: ignore UNSUPPORTED fields for JDBC and ODBC modes in 'SYS COLUMNS' (#39518)
* SYS COLUMNS will skip UNSUPPORTED field types in ODBC and JDBC, as well.
NESTED and OBJECT types were already skipped in ODBC mode, now they are
skipped in JDBC mode, as well.

(cherry picked from commit 9e0df64b2d36c9069dfa506570468f0522c86417)
2019-03-01 15:26:31 +02:00
Marios Trivyzas 9fb2f670dc SQL: Enhance checks for inexact fields (#39427)
For functions: move checks for `text` fields without underlying `keyword`
fields or with many of them (ambiguity) to the type resolution stage.

For Order By/Group By: move checks to the `Verifier` to catch early
before `QueryTranslator` or execution.

Closes: #38501
Fixes: #35203
2019-03-01 10:40:57 +01:00
Alan Woodward 71b8494181
Upgrade to lucene 8.0.0-snapshot-ff9509a8df (#39444)
Backport of #39350

Contains the following:

* LUCENE-8635: Move terms dictionary off-heap for non-primary-key fields in `MMapDirectory`
* LUCENE-8292: `TermsEnum` is fully abstract
* LUCENE-8679: Return WITHIN in `EdgeTree#relateTriangle` only when polygon and triangle share one edge
* LUCENE-8676: Nori tokenizer deals correctly with large buffers
* LUCENE-8697: `GraphTokenStreamFiniteStrings` better handles side paths with gaps
* LUCENE-8664: Add `equals` and `hashCode` to `TotalHits`
* LUCENE-8660: `TopDocsCollector` returns accurate hit counts if the total equals the threshold
* LUCENE-8654: `Polygon2D#relateTriangle` fix for when the polygon is inside the triangle
* LUCENE-8645: `Intervals#fixField` can merge intervals from different fields
* LUCENE-8585: Create jump-tables for DocValues at index time
2019-02-27 14:36:08 +00:00
Marios Trivyzas a2c07b5011
SQL: Use underlying exact field for LIKE/RLIKE (#39443)
Previously, if a text field had an underlying keyword field
the latter was not used instead of the text leading to wrong
results returned by queries filtering with LIKE/RLIKE.

Fixes: #39442
2019-02-27 14:46:54 +01:00
Andrei Stefan 542e2c55f6 SQL: change the default precision for CURRENT_TIMESTAMP function (#39391)
(cherry picked from commit dbb93310b083226c96e4bde3eef0079eb01cbca9)
2019-02-27 09:49:42 +02:00
Andrei Stefan 4deb69e9e4 SQL: introduce the columnar option for REST requests (#39287)
* Add "columnar" option for REST requests (but be lenient for non-"plain"
modes) for json, yaml, smile and cbor formats.
* Updated documentation

(cherry picked from commit 5b7e0de237fb514d14a61a347bc669d4b4adbe56)
2019-02-27 09:37:28 +02:00
Andrei Stefan d16edf0462 Randomize the timezone for equals and hashcode tests (#39353) 2019-02-27 07:39:06 +02:00
Tim Vernum 30687cbe7f
Switch internal security index to ".security-7" (#39422)
This changes the name of the internal security index to ".security-7",
but supports indices that were upgraded from earlier versions and use
the ".security-6" name.

In all cases, both ".security-6" and ".security-7" are considered to
be restricted index names regardless of which name is actually in use
on the cluster.

Backport of: #39337
2019-02-27 12:49:44 +11:00
Andrei Stefan c1018db404 SQL: enforce JDBC driver - ES server version parity (#38972)
(cherry picked from commit 822a21f29491f295b22dacd04b747781a69ffa61)
2019-02-20 11:29:02 +02:00
Andrei Stefan 92206c8567 Added "validate.properties" property to JDBC's list of allowed properties. (#39050)
This defaults to "true" (current behavior) and will throw an exception
if there is a property that cannot be recognized. If "false", it will
ignore anything unrecognizable.

(cherry picked from commit 38fbf9792bcf4fe66bb3f17589e5fe6d29748d07)
2019-02-20 11:29:01 +02:00
Andrei Stefan 7d78f4641b SQL: fall back to using the field name for column label (#38842)
(cherry picked from commit 0567bf24957be477e7649cff94872b0e7dc4d284)
2019-02-14 14:10:59 +02:00
Marios Trivyzas 032bcf99d6
SQL: Implement `::` cast operator (#38774)
`<expression>::<dataType>` is a simplified altenative syntax to
`CAST(<expression> AS <dataType> which exists in PostgreSQL and
provides an improved user experience and possibly more compact
SQL queries.

Fixes: #38717
2019-02-12 16:54:14 +02:00
Andrei Stefan b3695750bc Randomize the time zone properly for the current date test. (#38670)
(cherry picked from commit 29abbb8a590cdf4f9e0c0b447d6694bb7223648e)
2019-02-11 14:25:02 +02:00
Costin Leau 794ee4fb10 SQL: Prevent grouping over grouping functions (#38649)
Improve verifier to disallow grouping over grouping functions (e.g.
HISTOGRAM over HISTOGRAM).

Close #38308

(cherry picked from commit 4e9b1cfd4df38c652bba36b4b4b538ce7c714b6e)
2019-02-09 09:30:06 +02:00
Marios Trivyzas 871036bd21
SQL: Relax StackOverflow circuit breaker for constants (#38572)
Constant numbers (of any form: integers, decimals, negatives,
scientific) and strings shouldn't increase the depth counters
as they don't contribute to the increment of the stack depth.

Fixes: #38571
2019-02-09 09:18:21 +02:00
Marios Trivyzas af8a444caa
SQL: Replace joda with java time (#38437)
Replace remaining usages of joda classes with java time.

Fixes: #37703
2019-02-08 22:58:07 +02:00
Andrei Stefan 6359d988f0 Account for a possible rolled over file while reading the audit log file (#34909)
(cherry picked from commit 75cb6b38ed67dc9d32c9291b0c174ffa94e473bc)
2019-02-08 17:49:00 +02:00
Marios Trivyzas f96bd2ad71
SQL: Fix issue with IN not resolving to underlying keyword field (#38440)
- Add resolution to the exact keyword field (if exists) for text fields.
- Add proper verification and error message if underlying keyword
doesn'texist.
- Move check for field attribute in the comparison list to the
`resolveType()` method of `IN`.

Fixes: #38424
2019-02-06 16:25:06 +02:00
Costin Leau 1a02445ae1
SQL: Allow look-ahead resolution of aliases for WHERE clause (#38450)
Aliases defined in SELECT (Project or Aggregate) are now resolved in the
following WHERE clause. The Analyzer has been enhanced to identify this
rule and replace the field accordingly.

Close #29983
2019-02-06 12:08:32 +02:00
Marios Trivyzas 2c30501c74
SQL: Fix esType for DATETIME/DATE and INTERVALS (#38179)
Since introduction of data types that don't have a corresponding type
in ES the `esType` is error-prone when used for `unmappedType()` calls.
Moreover since the renaming of `DATE` to `DATETIME` and the introduction
of an actual date-only `DATE` the `esType` would return `datetime` which
is not a valid type for ES mapping.

Fixes: #38051
2019-02-05 23:12:52 +02:00
Julie Tibshirani 3ce7d2c9b6
Make sure to reject mappings with type _doc when include_type_name is false. (#38270)
`CreateIndexRequest#source(Map<String, Object>, ... )`, which is used when
deserializing index creation requests, accidentally accepts mappings that are
nested twice under the type key (as described in the bug report #38266).

This in turn causes us to be too lenient in parsing typeless mappings. In
particular, we accept the following index creation request, even though it
should not contain the type key `_doc`:

```
PUT index?include_type_name=false
{
  "mappings": {
    "_doc": {
      "properties": { ... }
    }
  }
}
```

There is a similar issue for both 'put templates' and 'put mappings' requests
as well.

This PR makes the minimal changes to detect and reject these typed mappings in
requests. It does not address #38266 generally, or attempt a larger refactor
around types in these server-side requests, as I think this should be done at a
later time.
2019-02-05 10:52:32 -08:00
Marios Trivyzas c9701be1e8
SQL: Implement CURRENT_DATE (#38175)
Since DATE data type is now available, this implements the
`CURRENT_DATE/CURRENT_DATE()/TODAY()` similar to `CURRENT_TIMESTAMP`.

Closes: #38160
2019-02-05 18:15:26 +02:00
Andrei Stefan cea81b199d
Change the milliseconds precision to 3 digits for intervals. (#38297) 2019-02-05 12:00:49 +02:00
Costin Leau 75f0750ff7
SQL: Remove exceptions from Analyzer (#38260)
Instead of throwing an exception, use an unresolved attribute to pass
the message to the Verifier.
Additionally improve the parser to save the extended source for the
Aggregate and OrderBy.

Close #38208
2019-02-03 22:32:16 +02:00
Costin Leau a088155f4d
SQL: Move metrics tracking inside PlanExecutor (#38259)
Move metrics in one place, from the transport layer inside the
PlanExecutor
Remove unused class

Close #38258
2019-02-03 22:31:35 +02:00
Andrei Stefan 6968f0925b
SQL: Generate relevant error message when grouping functions are not used in GROUP BY (#38017)
* Add checks for Grouping functions restriction to be placed inside GROUP BY
* Fixed bug where GROUP BY HISTOGRAM (not using alias) wasn't recognized
properly in the Verifier due to functions equality not working correctly.
2019-02-02 22:05:47 +02:00
Costin Leau 783c9ed372
SQL: Allow sorting of groups by aggregates (#38042)
Introduce client-side sorting of groups based on aggregate
functions. To allow this, the Analyzer has been extended to push down
to underlying Aggregate, aggregate function and the Querier has been
extended to identify the case and consume the results in order and sort
them based on the given columns.
The underlying QueryContainer has been slightly modified to allow a view
of the underlying values being extracted as the columns used for sorting
might not be requested by the user.

The PR also adds minor tweaks, mainly related to tree output.

Close #35118
2019-02-02 01:38:25 +02:00
Marios Trivyzas 4710a7472f
SQL: Implement FIRST/LAST aggregate functions (#37936)
FIRST and LAST can be used with one argument and work similarly to MIN
and MAX but they are implemented using a Top Hits aggregation and
therefore can also operate on keyword fields. When a second argument is
provided then they return the first/last value of the first arg when its
values are ordered ascending/descending (respectively) by the values of
the second argument. Currently because of the usage of a Top Hits
aggregation FIRST and LAST cannot be used in the HAVING clause of a
GROUP BY query to filter on the results of the aggregation.

Closes: #35639
2019-01-31 16:33:05 +02:00
Josh Soref 0154052335 spelling: java script -- not JavaScript (#37057) 2019-01-31 14:09:36 +02:00
Andrei Stefan 22d3290078
SQL: Added SSL configuration options tests (#37875)
* Added SSL configuration options tests
Removed the allow.self.signed option from the documentation since we allow
by default self signed certificates as well.

* Added more tests
2019-01-31 10:52:49 +02:00
Andrei Stefan 908c8def06
SQL: Skip the nested and object field types in case of an ODBC request (#37948) 2019-01-30 11:34:47 +02:00
Adrien Grand c8af0f4bfa
Use mappings to format doc-value fields by default. (#30831)
Doc-value fields now return a value that is based on the mappings rather than
the script implementation by default.

This deprecates the special `use_field_mapping` docvalue format which was added
in #29639 only to ease the transition to 7.x and it is not necessary anymore in
7.0.
2019-01-30 10:31:51 +01:00
Marios Trivyzas e9332331a3
SQL: Make error msg for validation of 2nd arg of PERCENTILE[_RANK] consistent (#37937)
Use `first` and `second` instead of `1st` and `2nd`.
2019-01-29 21:20:09 +02:00
Przemyslaw Gomulka 4f4113e964
Rename security audit.log to _audit.json (#37916)
in order to keep json logs consistent the security audit logs are renamed from .log to .json
relates #32850
2019-01-29 14:53:55 +01:00
Marios Trivyzas d1ff450edc
SQL: Fix casting from date to numeric type to use millis (#37869)
Previously casting from a DATE[TIME] type to a numeric (DOUBLE, LONG,
INT, etc. used seconds instead of the epoch millis.

Fixes: #37655
2019-01-25 23:29:10 +02:00
Martijn Laarman dfecb256cb
Exit batch files explictly using ERRORLEVEL (#29583)
* Exit batch files explictly using ERRORLEVEL

This makes sure the exit code is preserved when calling the batch
files from different contexts other than DOS

Fixes #29582

This also fixes specific error codes being masked by an explict

exit /b 1

causing the useful exitcodes from ExitCodes to be lost.

* fix line breaks for calling cli to match the bash scripts

* indent size of bash files is 2, make sure editorconfig does the same for bat files

* update indenting to match bash files

* update elasticsearch-keystore.bat indenting

* Update elasticsearch-node.bat to exit outside of endlocal
2019-01-25 16:44:33 +01:00
Christoph Büscher b4b4cd6ebd
Clean codebase from empty statements (#37822)
* Remove empty statements

There are a couple of instances of undocumented empty statements all across the
code base. While they are mostly harmless, they make the code hard to read and
are potentially error-prone. Removing most of these instances and marking blocks
that look empty by intention as such.

* Change test, slightly more verbose but less confusing
2019-01-25 14:23:02 +01:00
Jim Ferenczi 787acb14b9
Track total hits up to 10,000 by default (#37466)
This commit changes the default for the `track_total_hits` option of the search request
to `10,000`. This means that by default search requests will accurately track the total hit count
up to `10,000` documents, requests that match more than this value will set the `"total.relation"`
to `"gte"` (e.g. greater than or equals) and the `"total.value"` to `10,000` in the search response.
Scroll queries are not impacted, they will continue to count the total hits accurately.
The default is set back to `true` (accurate hit count) if `rest_total_hits_as_int` is set in the search request.
I choose `10,000` as the default because that's also the number we use to limit pagination. This means that users will be able to know how far they can jump (up to 10,000) even if the total number of hits is not accurate.

Closes #33028
2019-01-25 13:45:39 +01:00
Marios Trivyzas 74b6f308e9
SQL: Fix issue with complex expression as args of PERCENTILE/_RANK (#37102)
When the arguements of PERCENTILE and PERCENTILE_RANK can be folded,
the `ConstantFolding` rule kicks in and calls the `replaceChildren()`
method on `InnerAggregate` which is created from the aggregation rules
of the `Optimizerz. `InnerAggregate` in turn, cannot implement the method
as the logic of creating a new `InnerAggregate` instance from a list of
`Expression`s resides in the Optimizer. So, instead, `ConstantFolding`
should be applied before any of the aggregations related rules.

Fixes: #37099
2019-01-24 18:40:20 +02:00
Andrei Stefan 163a27b93c
SQL: Fix BasicFormatter NPE (#37804) 2019-01-24 15:40:51 +02:00
Marios Trivyzas 9357929309
SQL: Improve handling of invalid args for PERCENTILE/PERCENTILE_RANK (#37803)
Improve the Exception and the error message returned when 2nd argument
of PERCENTILE and PERCENTILE_RANK is not a constant.
2019-01-24 15:03:49 +02:00
Marios Trivyzas f707fa9e0a
SQL: Introduce SQL DATE data type (#37693)
* SQL: Introduce SQL DATE data type

Support ANSI SQL's DATE type by introducing a runtime-only
ES SQL date type.

Closes: #37340
2019-01-24 13:41:58 +02:00
Albert Zaharovits b6936e3c1e
Remove index audit output type (#37707)
This commit removes the Index Audit Output type, following its deprecation
in 6.7 by 8765a31d4e6770. It also adds the migration notice (settings notice).

In general, the problem with the index audit output is that event indexing
can be slower than the rate with which audit events are generated,
especially during the daily rollovers or the rolling cluster upgrades.
In this situation audit events will be lost which is a terrible failure situation
for an audit system.
Besides of the settings under the `xpack.security.audit.index` namespace, the
`xpack.security.audit.outputs` setting has also been deprecated and will be
removed in 7. Although explicitly configuring the logfile output does not touch
any deprecation bits, this setting is made redundant in 7 so this PR deprecates
it as well.

Relates #29881
2019-01-24 12:36:10 +02:00
Alexander Reelsen daa2ec8a60
Switch mapping/aggregations over to java time (#36363)
This commit moves the aggregation and mapping code from joda time to
java time. This includes field mappers, root object mappers, aggregations with date
histograms, query builders and a lot of changes within tests.

The cut-over to java time is a requirement so that we can support nanoseconds
properly in a future field mapper.

Relates #27330
2019-01-23 10:40:05 +01:00
Zachary Tong 2ba9e361ab
Add helper classes to determine if aggs have a value (#36020)
This adds a set of helper classes to determine if an agg "has a value". 
This is needed because InternalAggs represent "empty" in different 
manners according to convention. Some use `NaN`, `+/- Inf`, `0.0`, etc.

A user can pass the Internal agg type to one of these helper methods
and it will report if the agg contains a value or not, which allows the
user to differentiate "empty" from a real `NaN`.

These helpers are best-effort in some cases.  For example, several
pipeline aggs share a single return class but use different conventions
to mark "empty", so the helper uses the loosest definition that applies
to all the aggs that use the class.

Sums in particular are unreliable.  The InternalSum simply returns 0.0
if the agg is empty (which is correct, no values == sum of zero).  But this
also means the helper cannot differentiate from "empty" and `+1 + -1`.
2019-01-22 12:38:55 -05:00
Andrei Stefan 7507af29fa
SQL: Return Intervals in SQL format for CLI (#37602)
* Add separate CLI Mode
* Use the correct Mode for cursor close requests
* Renamed CliFormatter and have different formatting behavior for CLI and "text" format.
2019-01-22 14:55:28 +02:00
Adrien Grand e9fcb25a28
Upgrade to lucene-8.0.0-snapshot-83f9835. (#37668)
This snapshot uses a new file format for doc-values which is expected to make
advance/advanceExact perform faster on sparse fields:
https://issues.apache.org/jira/browse/LUCENE-8585
2019-01-22 11:44:29 +01:00
Andrei Stefan 90ae556d97
Define constants for REST requests endpoints in tests (#37610) 2019-01-22 10:01:51 +02:00
Igor Motov 54af8a4e7a
SQL: fix object extraction from sources (#37502)
Throws an exception if hit extractor tries to retrieve unsupported
object. For example, selecting "a" from `{"a": {"b": "c"}}` now throws
an exception instead of returning null.

Relates to #37364
2019-01-18 14:03:48 -05:00
Marios Trivyzas 1686c32ba9
SQL: Rename SQL type DATE to DATETIME (#37395)
* SQL: Rename SQL data type DATE to DATETIME

SQL data type DATE has only the date part (e.g.: 2019-01-14)
without any time information. Previously the SQL type DATE was
referring to the ES DATE which contains also the time part along
with TZ information. To conform with SQL data types the data type
`DATE` is renamed to `DATETIME`, since it includes also the time,
as a new runtime SQL `DATE` data type will be introduced down the road,
which only contains the date part and meets the SQL standard.

Closes: #36440

* Address comments
2019-01-17 10:17:58 +02:00
Marios Trivyzas ecf0de30ba
SQL: Lowercase the datatypes in validation error msgs (#37524)
To follow the ES convention display the datatypes in lowercase
in error messages thrown during validation if `IN` and conditional
functions.
2019-01-16 18:41:10 +02:00
Marios Trivyzas 2cf4b1863f
SQL: Lowercase es data type (mapping) returned from SQL Commands (#37531)
To follow the ES convention, convert the es data type, returned as
column `mapping` from SQL Commands, to lowercase.

Fixes: #37521
2019-01-16 18:08:33 +02:00
Costin Leau 19d2c29ca6 Remove SYS CATALOGS leftover
Relates #37506
2019-01-16 17:28:06 +02:00
Costin Leau 1f76b5fc31
SQL: Describe aliases as views (#37496)
When reporting metadata, several clients have issues with the 'ALIAS'
type. To improve compatibility and be consistent with the ANSI SQL
expectations and because they are similar, aliases targets are now
reported as views.

Close #37422
2019-01-16 17:26:00 +02:00
Andrei Stefan 659326fdd6
SQL: Add protocol tests and remove jdbc_type from drivers response (#37516) 2019-01-16 16:28:46 +02:00
Costin Leau 023bb2f1e4
SQL: Remove slightly used meta commands (#37506)
Remove SYS CATALOGS and SYS TABLE TYPES as they are a subset of SYS
TABLES (and thus somewhat redundant) and used only by JDBC.

Close #37409
2019-01-16 12:36:35 +02:00
Marios Trivyzas b594e81c86
SQL: Fix issue with field names containing "." (#37364)
Adjust FieldExtractor to handle fields which contain `.` in their
name, regardless where they fall in, in the document hierarchy. E.g.:

```
{
  "a.b": "Elastic Search"
}

{
  "a": {
    "b.c": "Elastic Search"
  }
}

{
  "a.b": {
    "c": {
      "d.e" : "Elastic Search"
    }
  }
}
```

Fixes: #37128
2019-01-15 09:41:41 +02:00
Julie Tibshirani 36a3b84fc9
Update the default for include_type_name to false. (#37285)
* Default include_type_name to false for get and put mappings.

* Default include_type_name to false for get field mappings.

* Add a constant for the default include_type_name value.

* Default include_type_name to false for get and put index templates.

* Default include_type_name to false for create index.

* Update create index calls in REST documentation to use include_type_name=true.

* Some minor clean-ups around the get index API.

* In REST tests, use include_type_name=true by default for index creation.

* Make sure to use 'expression == false'.

* Clarify the different IndexTemplateMetaData toXContent methods.

* Fix FullClusterRestartIT#testSnapshotRestore.

* Fix the ml_anomalies_default_mappings test.

* Fix GetFieldMappingsResponseTests and GetIndexTemplateResponseTests.

We make sure to specify include_type_name=true during xContent parsing,
so we continue to test the legacy typed responses. XContent generation
for the typeless responses is currently only covered by REST tests,
but we will be adding unit test coverage for these as we implement
each typeless API in the Java HLRC.

This commit also refactors GetMappingsResponse to follow the same appraoch
as the other mappings-related responses, where we read include_type_name
out of the xContent params, instead of creating a second toXContent method.
This gives better consistency in the response parsing code.

* Fix more REST tests.

* Improve some wording in the create index documentation.

* Add a note about types removal in the create index docs.

* Fix SmokeTestMonitoringWithSecurityIT#testHTTPExporterWithSSL.

* Make sure to mention include_type_name in the REST docs for affected APIs.

* Make sure to use 'expression == false' in FullClusterRestartIT.

* Mention include_type_name in the REST templates docs.
2019-01-14 13:08:01 -08:00
Jay Modi f3edbe2911
Security: remove SSL settings fallback (#36846)
This commit removes the fallback for SSL settings. While this may be
seen as a non user friendly change, the intention behind this change
is to simplify the reasoning needed to understand what is actually
being used for a given SSL configuration. Each configuration now needs
to be explicitly specified as there is no global configuration or
fallback to some other configuration.

Closes #29797
2019-01-14 14:06:22 -07:00
Costin Leau a4339ec7e9
SQL: Use declared source for error messages (#37161)
Improve error messages by returning the original SQL statement
declaration instead of trying to reproduce it as the casing and
whitespaces are not preserved accurately leading to small 
differences.

Close #37161
2019-01-13 01:40:22 +02:00
Marios Trivyzas 359222c55c
SQL: Make `FULL` non-reserved keyword in the grammar (#37377)
Since `full` can be common as a field name or part of a field name
(e.g.: `full.name` or `name.full`), it's nice if it's not a reserved
keyword of the grammar so a user can use it without resorting to quotes.

Fixes: #37376
2019-01-11 23:08:00 +02:00
Marios Trivyzas 85531f0285
SQL: [Tests] Fix and enable internalClusterTests (#37300)
SqlPlugin cannot have more than one public constructor, so for the testing
purposes the `getLicenseState()` should be overriden.

Fixes: #37191

Co-authored-by: Michael Basnight <mbasnight@gmail.com>
2019-01-11 22:43:17 +02:00
markharwood 434430506b
Type removal - added deprecation warnings to _bulk apis (#36549)
Added warnings checks to existing tests
Added “defaultTypeIfNull” to DocWriteRequest interface so that Bulk requests can override a null choice of document type with any global custom choice.
Related to #35190
2019-01-10 21:35:19 +00:00
Costin Leau 83f7423cd6
SQL: Fix bug regarding alias fields with dots (#37279)
Field of types aliases that have dots in name are returned without a
hierarchy by field_caps, as oppose to the mapping api or field with
concrete types, which in turn breaks IndexResolver.
This commit fixes this by creating the backing hierarchy similar to the
mapping api.

Close #37224
2019-01-10 22:18:53 +02:00