Commit Graph

45145 Commits

Author SHA1 Message Date
Benjamin Trent 22e688a709
Muting watcher tests for issue #35503 (#40527)
* Muting watcher tests for issue #35503

* blocking the two troublsome suites
2019-03-27 09:16:06 -05:00
James Rodewig cfe85c73bf Fix typo in rollup_index definition (#40520) 2019-03-27 10:04:49 -04:00
Bogdan Pintea 6182b464a3 SQL: ODBC: document extra connection string parameters (#40476)
* document ODBC's extra connection string parameters

Document the connection string parameters that are currently not
configurable from the GUI.

Add a note about SmartScreen possible warning on driver MSI
installation.

Add a note about the TLS certificate file not being supported as bundled
or password-protected.

* rephrasing and restructuring the list of params

- addressing PR review notes

* rephrasings and reference linking

- addressing PR review notes

(cherry picked from commit ca90190226c0a8c63f51c2603b27df299ce8d503)
2019-03-27 14:20:23 +01:00
Boaz Leskes 6efe748e2e Only recommend disabling allocation of replicas in a rolling upgrade (#40299)
The current documentation recommends disabling allocations of all shards. This prevents shards of
new indices from being allocated as well. That means that during a rolling upgrade, operations like
roll over and index shrinking will operate correctly. This, in turn, can cause issues for data ingestion
and ILM.
2019-03-27 14:18:17 +01:00
David Kyle 61845dd38b [ML] Fix serialisation of Start Data Frame request (#40483) 2019-03-27 12:56:34 +00:00
Przemyslaw Gomulka 65f01277ed
Parse composite patterns using ClassicFormat.parseObject backport(#40100) (#40501)
Java-time fails parsing composite patterns when first pattern matches only the prefix of the input. It expects pattern in longest to shortest order. Because of this constructing just one DateTimeFormatter with appendOptional is not sufficient. Parsers have to be iterated and if the parsing fails, the next one in order should be used. In order to not degrade performance parsing should not be throw exceptions on failure. Format.parseObject was used as it only returns null when parsing failed and allows to check if full input was read.
 
closes #39916
backport #40100
2019-03-27 13:51:44 +01:00
David Kyle c990b30019
[ML] Data Frame HLRC Get API (#40509) 2019-03-27 12:40:39 +00:00
Yannick Welsch b4b17e16e0 Remove TransportSingleItemBulkWriteAction as replication action (#40424)
The implementation of TransportIndexAction and TransportDeleteAction as
TransportReplicationAction existed for interoperability with older 5.x nodes, as these older nodes
coordinated single index / deletes as replication requests. This BWC layer is no longer needed in 7.x,
where these single actions are now mapped to bulk requests. Completely removing the deprecated
transport actions is not possible yet if we want to keep BWC with a 6.x transport client. The best
way here is to wait for the transport client to go away and then just remove the actions.
2019-03-27 13:16:58 +01:00
Dimitrios Liappis 856789f791
Wrap Dockerfile yum operations in retries (#40460)
While yum does retry retrieving files 10 times by default [1], slow
network fetches, governed by `minrate` cause immediate aborts without
getting retried.

Wrap yum commands in a 10 iteration retry loop.

[1] http://man7.org/linux/man-pages/man5/yum.conf.5.html

Backport of #40349
2019-03-27 14:14:47 +02:00
Benjamin Trent 12943c5d2c
[ML] Add data frame task state object and field (#40169) (#40490)
* [ML] Add data frame task state object and field

* A new state item is added so that the overall task state can be
accoutned for
* A new FAILED state and reason have been added as well so that failures
can be shown to the user for optional correction

* Addressing PR comments

* adjusting after master merge

* addressing pr comment

* Adjusting auditor usage with failure state

* Refactor, renamed state items to task_state and indexer_state

* Adding todo and removing redundant auditor call

* Address HLRC changes and PR comment

* adjusting hlrc IT test
2019-03-27 06:53:58 -05:00
Costin Leau 38dc005b8d SQL: Update JDBC class name in client screenshots
(cherry picked from commit 79a3b93deb858961ecbe4516ed9d8f209016666f)
2019-03-27 12:21:53 +02:00
Costin Leau 33737b6b21 SQL: Polish parsing of CAST expression (#40428)
(cherry picked from commit 9d291aa300bbb827eeae606e7d3e55eeef7cce00)
2019-03-27 12:20:58 +02:00
Jim Ferenczi fe05a4d511 Fix random failures in SearchResponseMerger#testMergeSearchHits (#40223)
This commit fixes the expectation in the test when the search hits are empty.

Closes #40214
2019-03-27 11:17:10 +01:00
Hendrik Muhs f4e56118c2 [ML] generate unique doc ids for data frame (#40382)
create and use unique, deterministic document ids based on the grouping values.

This is a pre-requisite for updating documents as well as preventing duplicates after a hard failure during indexing.
2019-03-27 08:27:05 +01:00
Alpar Torok 524e0273ae Testclusters: convert plugin repository-s3 (#40399)
* Add support for setting and keystore settings
* system properties and env var config
* use testclusters for repository-s3
* Some cleanup of the build.gradle file for plugin-s3
* add runner {} to rest integ test task
2019-03-27 08:40:16 +02:00
Ryan Ernst aa24669e95
Resolve JAVA_HOME at windows service install time (#39714)
On windows, JAVA_HOME is currently resolved when the windows service is started. However, this is contrary to what our documentation states. This commit moves resolution to service install. This has the side effect of making java existence checking optional in elasticsearch-env.bat, since the rest of the service commands do not require java.

closes #30720
2019-03-26 19:48:02 -07:00
Ryan Ernst f82f9423b7 Remove platform test command (#40280)
The platformTest gradle task was a packaging test meant to ensure unit
tests run on the various supported operating systems without relying on
CI to maintain a full matrix of platforms. Howevever, it never really
worked out as intended and is now additional code in our vagrant setup
to maintain. This commit removes the platformTest task.
2019-03-26 19:47:33 -07:00
lcawl 81510746d6 [DOCS] Adds anchors and x-pack icons for frozen indices 2019-03-26 15:53:18 -07:00
Jason Tedor cbdb20faba
Add more Java 12 distributions to testing (#40482)
This commit adds the Oracle JDK 12 and the Zulu JDK 12 distributions to
testing.
2019-03-26 18:48:43 -04:00
Julie Tibshirani 25954a8dd3 Stop clearing all watches in watcher integration tests. (#39724) 2019-03-26 13:14:33 -07:00
Ryan Ernst 562ebaefa6 Delete jdk extract dir before extracting (#40471)
This commit first deletes the directory we extract the bundled jdk into in order to make gradle's copy task happy.
2019-03-26 11:41:37 -07:00
Ryan Ernst e49ba6a9d9 Remove unused variable (#40468)
This was leftover from previous iterations of test clusters refactorings
2019-03-26 11:26:31 -07:00
alex101101 fb8ad0cf30 Add a soft limit to the field name length (#40309)
Adds an optional limit to the length of field names, throws an IllegalArgumentException if the limit is breached. 
Closes #33651
2019-03-26 17:58:32 +01:00
Daniel Mitterdorfer f2b5960f90 Add version 6.7.1 2019-03-26 17:38:14 +01:00
debadair 324a5400a4 [DOCS] Switched release state back to unreleased. Closes #40259 (#40470) 2019-03-26 09:20:41 -07:00
Yannick Welsch bf7b167bba Remove timeout task after completing cluster state publication (#40411)
Each cluster state publication schedules a cancellation task with the provided publication timeout
(30s by default). This scheduled cancellation keeps a reference to the publication, and therefore the
full cluster state that was published. In case of frequently updating a large cluster state, this results
in a large number of cancellation tasks keeping references to all previously published cluster states.
2019-03-26 17:13:57 +01:00
Jay Modi 9bd8600c2e
Use ephemeral ports for idp-fixture (#40333)
This change removes the use of hardcoded port values for the
idp-fixture in favor of the mapped ephemeral ports. This should prevent
failures due to port conflicts in CI.
2019-03-26 08:44:53 -06:00
Henning Andersen bf444b9f02 Store Pending Deletions Fix (#40345)
FilterDirectory.getPendingDeletions does not delegate, fixed
temporarily by overriding in StoreDirectory.

This in turn caused duplicate file name use after a trimUnsafeCommits
had been done, since a new IndexWriter would not consider the pending
deletes in IndexFileDeleter. This should only happen on windows (AFAIK).

Reenabled doing index updates for all tests using
IndexShardTests.indexOnReplicaWithGaps (which could fail due to above
when using mocked WindowsFS).

Added getPendingDeletions delegation to all elasticsearch
FilterDirectory subclasses that were not trivial test-only overrides to
minimize the risk of hitting this issue in another case.
2019-03-26 15:30:44 +01:00
Alpar Torok b5a8de9a7f Revert tmpdir (#40312)
* Revert "Configure TMP for test nodes on Windows (#39959)"
This reverts commit 97562a874fcb1f29fb05272ab860a0307e97d1aa.

* Configure a tmp dir without spaces
* Pass on TMP instead of changing it
2019-03-26 13:31:18 +02:00
Alpar Torok d50305bbee Upgrade to Gradle 5.3! (#40346)
Here are the highlights of this release:
 - Feature variants AKA "optional dependencies"
 - Type-safe accessors in Kotlin precompiled script plugins
 - Gradle Module Metadata 1.0

For more details see https://docs.gradle.org/5.3/release-notes.html
2019-03-26 13:23:40 +02:00
David Kyle 1354696db9
[ML] Data Frame HLRC Get Stats API (#40443) 2019-03-26 11:17:13 +00:00
Alpar Torok cd4eac5fa4 Testclusters links (#39831)
* Create hardlinks instead of sync to save on disk space
2019-03-26 13:10:13 +02:00
Alpar Torok 2b34923541 Add Gradle init script to configure artifactoy (#40412)
The script will be use in CI to configure our own Artifactory instance
so CI does not depend on external network resources nor do we generate
excessive load on these resources.
In case the repo is not accessible Gradle should fall back to using the
public ones.
2019-03-26 12:23:55 +02:00
Alan Woodward 12634850d6 IntervalQueryBuilderTests#testNonIndexedFields test fix (#40418)
This test checks that interval queries constructed against a field with no indexed
positions will throw exceptions. It uses a randomly-build IntervalsSourceProvider
against a fixed set of fields; however, the random source builder can occasionally
provide a source with a fixed field, meaning that even if the top-level query asks
for a set of intervals over a non-indexed field, the source will delegate to another
field, and no exception will be thrown.

This commit changes the test to always use a simple Match provider.

Fixes #40436
2019-03-26 08:33:42 +00:00
Armin Braun cafb83297c
Use Correct Enum in Wipe Snapshots Test Method (#40422) (#40438)
* Mistake was made in #39662
* The response deserialized here is `org.elasticsearch.action.admin.cluster.snapshots.get.GetSnapshotsResponse` which uses `org.elasticsearch.snapshots.SnapshotInfo` which uses `org.elasticsearch.snapshots.SnapshotState` and not the shard state
2019-03-26 09:04:15 +01:00
Martijn van Groningen 1d3ece1e96
Remove -Xlint exclusions in the percolator module. (#40372)
Relates to #40366
2019-03-26 07:55:02 +01:00
Costin Leau 7234d78747 SQL: Fix classpath discovery on Java 10+ (#40420)
(cherry picked from commit 2cef233cb34ee80d8ed9cd014cea76ea5096d206)
2019-03-26 08:16:37 +02:00
Ed Savage c20ea9a2dd [ML][TEST] Fix failing test testPersistJobOnGracefulShutdown_givenTimeAdvancedAfterNoNewData (#40363)
Ensure that there is at least a 1s delay between the time that state
is persisted by each of the two jobs in the test.

Model snapshot IDs use the current time in epoch seconds to
distinguish themselves, hence snapshots will be overwritten
by another if it occurs in the same 1s window.

Closes #40347
2019-03-25 17:55:10 +00:00
Lisa Cawley 6f03a6c546 [DOCS] Simplify ML upgrade step (#40006) 2019-03-25 10:26:29 -07:00
Nhat Nguyen 495dc11c9c Mute testPendingRefreshWithIntervalChange
Tracked at #39565
2019-03-25 11:47:08 -04:00
Armin Braun 3968d46a17
Remove Redundant Request Wrappers from RepositoryService (#40192) (#40404) 2019-03-25 16:36:02 +01:00
Costin Leau 61f49af497 SQL: Spec tests now use classpath discovery (#40388)
To avoid having to specify each spec by hand (which can miss specs to be
added), the test infrastructure now performs classpath discovery so that
each spec added, is automatically considered.

Relates #40358

(cherry picked from commit d0f60b4425c731509aa8ca765d55f563f866ef90)
2019-03-25 15:22:52 +02:00
James Rodewig e8a93b8915
Update Windows command to support prettified json licenses (#40407) 2019-03-25 09:00:03 -04:00
Benjamin Trent 7b4f964708
[ML] make source and dest objects in the transform config (#40337) (#40396)
* [ML] make source and dest objects in the transform config

* addressing PR comments

* Fixing compilation post merge

* adding comment for Arrays.hashCode

* addressing changes for moving dest to object

* fixing data_frame yml tests

* fixing API test
2019-03-25 07:16:41 -05:00
Hendrik Muhs 38afc9f27d refresh audit index before searching (#40401)
refresh the audit index before searching
2019-03-25 11:57:57 +01:00
Armin Braun dc5ff0fffc
Log Warning on Failed Blob Deletes in BlobStoreRepository (#40188) (#40340)
* Log Warning on Failed Blob Deletes in BlobStoreRepository
* We should not just debug log these spots, they all can and will lead to leaked files when snapshot deletion fails
2019-03-25 08:52:09 +01:00
Nhat Nguyen b9f96a8e1f
Expose external refreshes through the stats API (#38643)
Right now, the stats API only provides refresh metrics regarding
internal refreshes. This isn't very useful and somewhat misleading for
cluster administrators since the internal refreshes are not indicative
of documents being available for search.

In this PR I added a new metric for collecting external refreshes as
they occur and exposing them through the stats API. Now, calling an
endpoint for stats will yield external refresh metrics as well.

Relates #36712
2019-03-24 22:21:00 -04:00
Armin Braun 13d76239a0
Use Netty ByteBuf Bulk Operations for Faster Deserialization (#40158) (#40339)
* Use bulk methods to read numbers faster from byte buffers
2019-03-24 19:08:51 +01:00
Benjamin Trent a30bf27b2f
[ML] add auditor to data frame plugin (#40012) (#40394)
* [Data Frame] add auditor

* Adjusting Level, Auditor, and message to address pr comments

* Addressing PR comments
2019-03-23 18:56:44 -05:00
Benjamin Trent 2dd879abac
[ML] adds support for non-numeric mapped types (#40220) (#40380)
* [ML] adds support for non-numeric mapped types and mapping overrides

* correcting hlrc compilation issues after merge

* removing mapping_override option

* clearing up unnecessary changes
2019-03-23 14:04:14 -05:00