Commit Graph

5993 Commits

Author SHA1 Message Date
Adrien Grand 602d269059
Rename `datastream` to `data_stream`. (#60714)
The name of the feature having a space: "data stream", the key should
have an underscore.
2020-08-05 09:55:02 +02:00
Russ Cam e9c0bf1566 Remove body from indices.create_data_stream REST spec (#60705)
This commit removes the body property from the
indices.create_data_stream.json REST API spec
as the API does not support sending a body.

Update the description of the API to remove
that a data stream can be updated with the
API - data streams can only be created with
this API and attempting to update yields a
`resource_already_exists_exception`.

Closes #60704

(cherry picked from commit 2cab2e0ee094769852df31566dbe22b5df59d900)
2020-08-05 17:01:28 +10:00
Igor Motov 959690a64a
Refactor extendedBounds to use DoubleBounds (#60556) (#60681)
Refactors extendedBounds to use DoubleBounds instead
of 2 variables.

This is a follow up for #59175
2020-08-04 16:45:47 -04:00
Francisco Fernández Castaño b500b3d55a
Decrease restore rate limit value to enforce its usage on SearchableSnapshotsIntegTests#testMaxRestoreBytesPerSecIsUsed (#60650)
Fixes #59287. Backport of #59592
2020-08-04 17:44:47 +02:00
Alan Woodward b3ae5d26bd
Move mapper validation to the mappers themselves (#60072) (#60649)
Currently, validation of mappers (checking that cross-references are correct, limits on
field name lengths and object depths, multiple definitions, etc) is performed by the
MapperService. This means that any mapper-specific validation, for example that done
on the CompletionFieldMapper, needs to be called specifically from core server code,
and so we can't add validation to mappers that live in plugins.

This commit reworks the validation framework so that mapper-specific validation is
done on the Mapper itself. Mapper gets a new `validate(MappingLookup)`
method (already present on `MetadataFieldMapper` and now pulled up to the parent
interface), which is called from a new `DocumentMapper.validate()` method. All
the validation code currently living on `MapperService` moves either to individual
mapper implementations (FieldAliasMapper, CompletionFieldMapper) or into
`MappingLookup`, an altered `DocumentFieldMappers` which now knows about
object fields and can check for duplicate definitions, or into DocumentMapper
which handles soft limit checks.
2020-08-04 14:39:20 +01:00
Rene Groeschke bdd7347bbf
Merge test runner task into RestIntegTest (7.x backport) (#60600)
* Merge test runner task into RestIntegTest (#60261)
* Merge test runner task into RestIntegTest
* Reorganizing Standalone runner and RestIntegTest task
* Rework general test task configuration and extension
* Fix merge issues
* use former 7.x common test configuration
2020-08-04 14:46:32 +02:00
Adrien Grand 20ae1b75bd
Rename dataset to datastream (#60638)
Co-authored-by: ruflin <spam@ruflin.com>
2020-08-04 09:58:54 +02:00
Armin Braun 7ae9dc2092
Unify Stream Copy Buffer Usage (#56078) (#60608)
We have various ways of copying between two streams and handling thread-local
buffers throughout the codebase. This commit unifies a number of them and
removes buffer allocations in many spots.
2020-08-04 09:54:52 +02:00
Yang Wang 54aaadade7
API key name should always be required for creation (#59836) (#60636)
The name is now required when creating or granting API keys.
2020-08-04 13:28:47 +10:00
Tim Vernum c58e32bb27
Improve assertion failure when error is not empty (#60572)
This commit changes TokenAuthIntegTests so all occurrences of

    assertThat(x.size(), equalTo(0));

become

    assertThat(x, empty());

This means that the assertion failure message will include the
contents of the list (`x`) instead of just its size, which
facilitates easier failure diagnosis.

Relates: #56903
Backport of: #60496
2020-08-04 11:05:18 +10:00
Jake Landis bcb9d06bb6
[7.x] Cleanup xpack build.gradle (#60554) (#60603)
This commit does three things:
* Removes all Copyright/license headers for the build.gradle files under x-pack. (implicit Apache license)
* Removes evaluationDependsOn(xpackModule('core')) from build.gradle files under x-pack
* Removes a place holder test in favor of disabling the test task (in the async plugin)
2020-08-03 13:11:43 -05:00
Hendrik Muhs 1e01832b0c fix possible NPE introduced in #60591 2020-08-03 16:40:38 +02:00
Hendrik Muhs cd6492fc11 [Transform] fix regression of date histogram optimization (#60591)
fixes mix up of input and output field name for date histogram optimization.

minimal fix, more tests to be added with #60469

fixes #60590
2020-08-03 15:52:08 +02:00
Yannick Welsch b0d601fa63 Adjust searchable snapshot license (#60578)
No longer needs Platinum license for testing on staging.
2020-08-03 13:19:53 +02:00
Yannick Welsch 9e24a54382 Clean existing index folder when loading searchable snapshot (#60122)
Closing a regular index and mounting a snapshot-backed index into that existing index does not clean the existing index
folders of those preexisting shards.

This PR removes the existing Lucene / translog files once the searchable snapshot shard is starting up. Future PRs will
make reuse of the existing index files to populate the cache.
2020-08-03 13:19:11 +02:00
Yang Wang a76fc324d4
Fix get-license test failure by ensure cluster is ready (#60498) (#60569)
When a new cluster starts, the HTTP layer becomes ready to accept incoming 
requests while the basic license is still being populated in the background. 
When a get license request comes in before the license is ready, it can get 
404 error. This PR fixes it by either wrap the license check in assertBusy or 
ensure the license is ready before perform the check.

This is a backport for both #60498 and #60573
2020-08-03 19:40:03 +10:00
Tim Vernum 1a373b0c21
Only call listener once (SP template registration) (#60567)
This fixes a bug in the IdP's template registration that would
sometimes call the listener twice.

Resolves: #54285
Resolves: #54423

Backport of: #60497
2020-08-03 13:45:16 +10:00
Andrei Dan ac258f10d6
Data streams: throw ResourceAlreadyExists exception (#60518) (#60536)
For consistency reasons (and reducing the overload of IllegalArgumentException)
this changes the exception thrown when trying to create a data stream
that already exists.

(cherry picked from commit ac2184c4614bba0f3ee377da49aea0daed98bab4)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
2020-08-01 16:31:09 +01:00
Julie Tibshirani f1d4fd8c3e Correct name of IndexFieldData#loadGlobalDirect. (#60492)
It seems 'localGlobalDirect' was just a typo.
2020-07-31 10:53:21 -07:00
Rene Groeschke ed4b70190b
Replace immediate task creations by using task avoidance api (#60071) (#60504)
- Replace immediate task creations by using task avoidance api
- One step closer to #56610
- Still many tasks are created during configuration phase. Tackled in separate steps
2020-07-31 13:09:04 +02:00
Hendrik Muhs a721d6d19b [Transform] use correct version in BWC serialization test (#60500)
use correct version in BWC serialization test

fixes #60464
2020-07-31 11:23:05 +02:00
Julie Tibshirani 8ac81a3447 Remove IndexFieldData#clear since it is unused. (#60475)
This method was never called. It also seemed tricky that calling a method on
`IndexFieldData` could clear the contents of a shared cache.
2020-07-30 14:07:55 -07:00
Lisa Cawley 9425de8dd1
[DOCS] Clarify support for run as in OIDC realms (#60246) (#60485) 2020-07-30 13:39:41 -07:00
Mark Tozzi 970a0c8957
[7.x] Aggregation tests for Wildcard Field (#58507) (#60423) 2020-07-30 08:56:21 -04:00
Przemysław Witek 9e27f7474c
Make MlDailyMaintenanceService delete jobs that are in deleting state anyway (#60121) (#60439) 2020-07-30 09:53:11 +02:00
Hendrik Muhs aaed6b59d6
[7.x][Transform] add support for missing bucket (#59591) (#60390)
add support for "missing_bucket" in group_by

fixes #42941
fixes #55102
backport #59591
2020-07-30 08:26:51 +02:00
Bogdan Pintea 8c22adc447
SQL: Add option to provide the delimiter for the CSV format (#59907) (#60420)
* SQL: Add option to provide the delimiter for the CSV format (#59907)

* Add option to provide the delimiter to the CSV fmt

This adds the option to provide the desired character as the separator
for the CSV format (the default remains comma).
A set of characters are excluded though - like CR, LF, `"` - to avoid
slipping onto the CSV-dialects slope. The tab is also forbidden, the
user needs to choose the "tsv" format explicitely.

Update the doc to make it clear that the textual CSV, TSV and TXT
formats pass the cursor back to the user through the Cursor HTTP header.

(cherry picked from commit 3a8b00cc7480f7ada57fcea3cbac957facac08fc)

* Java8 fixes

- replace Set#of();
- URLDecoder#decode() requires a string (vs a charset) as 2nd arg.
2020-07-29 21:40:11 +02:00
Bogdan Pintea 30610d962a
Fix SYS COLUMNS schema in ODBC mode (#59513) (#60418)
* Fix SYS COLUMNS schema in ODBC mode (#59513)

* Fix SYS COLUMNS schema in ODBC mode

This fixes a regression when certain ODBC-specific columns that need to
be of the short type were returned as the integer type.

This also fixes the stubbing for the *-indices SYS COLUMN commands.

(cherry picked from commit 96d89dc9b1fd731e736ef804a16bd05496c1dea6)

* Java8 fix: avoid diamond notation in test.

Qualify anonymous class in test.
2020-07-29 21:19:32 +02:00
Bogdan Pintea 4c771485f6
SQL: fix NPE on ambiguous GROUP BY (#59370) (#60416)
* fix npe on ambiguous group by

* add tests for aggregates and group by, add quotes to error message

* add more cases for Group By ambiguity test

* change error messages for field ambiguity

* change collection aliases approach

* add locations of attributes for ambiguous grouping error

* Adress review comments

- remove Comparable implementations from Attribute and Location;
- add ad-hoc comparator for sorting locations in ambiguity message;
- remove added AttributeAlias class with Touple;
- add code comment to explain issue with Location overwriting.

* Fix c&p error in location ref generation comparator

Fix copy&paste error in dedicated comparator used for sorting ambiguity
location references.
Slightly increase its readability.

Co-authored-by: Nikita Verkhovin <verkhovin13@gmail.com>
(cherry picked from commit 9ba70a3483f0f4987229bec231cdc004f51b88a5)
2020-07-29 20:44:28 +02:00
Bogdan Pintea 79ef263fc2
Add test with alias reuse and grouping (#60396) (#60421)
Add test with alias reuse and grouping.

(cherry picked from commit 37ee819eb98fd10c1b16a61e4e1d446d0ee859de)
2020-07-29 20:43:04 +02:00
Mark Vieira 39fa1c4df0
Add compatibility testing for JDBC driver (#60409)
This commit adds compatibility testing of our JDBC driver against
different Elasticsearch versions. Although we are really testing the
forwards compatibility nature of the JDBC driver we model the testing
the same as we do existing BWC tests, that is, with the current branch
fetching the earlier versions of the artifact that is to be tested. In
this case, that's the JDBC driver itself.

Because the tests include the JDBC driver jar on it's classpath we had
to change the packaging of the driver jar in order to avoid jarhell and
other conflicting dependency issues when using an old JDBC driver with
later branches. For this we simply relocate all driver dependencies in
the shadow jar under a "shadowed" package. This allows the JDBC driver
to use the correct version of Elasticsearch libs classes, while the
tests themselves use their versions. Since this required a change to the
driver jar compatibility testing can only go back as far as that version
which at the time of this commit is 7.8.1.
2020-07-29 10:45:11 -07:00
David Roberts 2a0116f51b
[ML] Take more care that memory estimation uses unique named pipes (#60405)
Prior to this change ML memory estimation processes for a
given job would always use the same named pipe names.  This
would often cause one of the processes to fail.

This change avoids this risk by adding an incrementing counter
value into the named pipe names used for memory estimation
processes.

Backport of #60395
2020-07-29 17:29:55 +01:00
Armin Braun bfee7b91ff
Increase Timeouts in SLMBlockingIntegTests (#60356) (#60403)
The retention run goes through a number of steps and can randomly take more than 10s.
=> increased timeout to 30s like we did in other spots in this test

Also, noticed that we had a hard wait of 10s in this test, removed it and adjusted following
busy assert in a way that can deal with a missing snapshot (from when the assert runs before
the snapshot was put into the CS).

Closes #60336
2020-07-29 17:34:49 +02:00
Benjamin Trent 76359aaa53
[ML] always write prediction_[score|probability] for classification inference (#60335) (#60397)
In order to unify model inference and analytics results we
need to write the same fields.

prediction_probability and prediction_score are now written
for inference calls against classification models.
2020-07-29 10:58:14 -04:00
Nhat Nguyen 9d4a64e749
Allow CCR on nodes with legacy roles only (#60093)
CCR will stop functioning if the master node is on 7.8, but data nodes 
are before that version because the master node considers that all data
nodes do not have the remote cluster client role. This commit allows CCR
work on data nodes with legacy roles only.

Relates #54146
Relates #59375
2020-07-29 10:57:31 -04:00
Benjamin Trent a6da1fd73e
[ML] require alias when indexing to an alias that should be created (#60315) (#60394)
This sets up all indexing to one of our write aliases to require it actually be an alias.

This allows failures scenarios to be captured quickly, loudly, and then potentially recovered.
2020-07-29 10:52:36 -04:00
Jim Ferenczi 578749a5e8 Fix AsyncResultsServiceTests#testRetrieveFromMemoryWithExpiration (#60337)
This change ensures that the expiration time that is set in the test
is long enough to not be triggered by a slow execution.

Closes #60255
2020-07-29 09:47:47 +02:00
Hendrik Muhs 5eb04fb413 [Transform] fix performance regression introduced in #60196 (#60276)
re-work #60196, to not skip building change collectors as otherwise date histogram only
pivots would run slow

relates #60125
2020-07-29 09:44:03 +02:00
Armin Braun 753fd4f6bc
Cleanup and optimize More Serialization Spots (#59959) (#60331)
Same as #59626 for a few more spots.
2020-07-29 07:20:44 +02:00
Yang Wang 3a0e7f4294
Unmute kerberos tests for jdk 15 and mute for jdk 8u262 (#60279)
The JDK bug (https://bugs.openjdk.java.net/browse/JDK-8246193) is fixed since b26.
The tests can be unmuted since we are already using b33. However the same bug is now
affecting jdk 8u262, which is the base for current Zulu jdk 8.48. This PR mute the tests
for this specific jdk version.

Relates: #56507
2020-07-29 12:57:00 +10:00
Benjamin Trent 54c8936508
[ML] do not summerize importance for custom features (#60198) (#60333)
If a feature is created via a custom pre-processor,
we should return the importance for that feature.

This means we will not return the importance for the
original document field for custom processed features.

closes https://github.com/elastic/elasticsearch/issues/59330
2020-07-28 15:58:20 -04:00
Julie Tibshirani c7bfb5de41
Add search `fields` parameter to support high-level field retrieval. (#60258)
This feature adds a new `fields` parameter to the search request, which
consults both the document `_source` and the mappings to fetch fields in a
consistent way. The PR merges the `field-retrieval` feature branch.

Addresses #49028 and #55363.
2020-07-28 10:58:20 -07:00
James Rodewig 8edae3cd15
[DOCS] Update pre-existing data stream refs (#60289) (#60293) 2020-07-28 13:51:43 -04:00
Nhat Nguyen 416e51980c Relax ShardFollowTasksExecutor validation (#60054)
If a primary shard of a follower index is being relocated, then we
will fail to create a follow-task. This validation is too restricted.
We should ensure that all primaries of the follower index are active
instead.

Closes #59625
2020-07-28 13:46:49 -04:00
Nhat Nguyen 6ece629ec3 Set timeout of master requests on follower to unbounded (#60070)
Today, a follow task will fail if the master node of the follower
cluster is temporarily overloaded and unable to process master node
requests (such as update mapping, setting, or alias) from a follow-task
within the default timeout. This error is transient, and follow-tasks
should not abort. We can avoid this problem by setting the timeout of
master node requests on the follower cluster to unbounded.

Closes #56891
2020-07-28 13:46:49 -04:00
Zachary Tong 9f8ec3e3fb Mute SSLDriverTests#testCloseDuringHandshakePreJDK11
Tracking issue: https://github.com/elastic/elasticsearch/issues/59992
2020-07-28 13:20:53 -04:00
Zachary Tong 46f9c38c33 Mute tests while waiting on 58807
Bugurl: https://github.com/elastic/elasticsearch/issues/58807
2020-07-28 12:45:49 -04:00
markharwood e0286e9bd3
Search - remove allow-expensive-query checks from wildcard field. (#60273) (#60308)
Removing allow-expensive-query checks because we think this field type is fast enough.

Closes #60139
2020-07-28 17:12:33 +01:00
Dimitris Athanasiou ed7dcff7c4
[7.x][ML] Audit updates on data frame analytics jobs (#60126) (#60287)
Closes #59652

Backport of #60126
2020-07-28 16:33:35 +03:00
Dimitris Athanasiou 16ffcfb9f6
[7.x][ML] Ensure bulk requests are not over memory limit (#60219) (#60283)
Data frame analytics jobs that work with very large datasets
may produce bulk requests that are over the memory limit
for indexing. This commit adds a helper class that bundles
index requests in bulk requests that steer away from the
memory limit. We then use this class both from the results
joiner and the inference runner ensuring data frame analytics
jobs do not generate bulk requests that are too large.

Note the limit was implemented in #58885.

Backport of #60219
2020-07-28 16:04:03 +03:00